Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,8 @@ repos:
- id: ruff-check
args: [--fix]
- id: ruff-format

- repo: https://github.com/numpy/numpydoc
rev: v1.11.0
hooks:
- id: numpydoc-validation
94 changes: 94 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# AGENTS.md

Guidance for AI coding agents and human contributors working on `mxalign`.

## Project

`mxalign` aligns meteorological `xarray` datasets in space and time. Datasets carry
**traits** (`mlwp_space_trait`, `mlwp_time_trait`, `mlwp_uncertainty_trait` in `ds.attrs`).
The traits are defined by [mlwp-data-specs](https://github.com/mlwp-tools/mlwp-data-specs) and
attached by [mlwp-data-loaders](https://github.com/mlwp-tools/mlwp-data-loaders). Loading and
validating datasets happens in those packages. `mxalign` only reads the traits to decide how to
align.

Package layout (`src/mxalign/`):

| Package | Contents |
|---|---|
| `accessor/` | The `ds.mx` accessor (`MxAccessor`): trait predicates (`is_grid`, `is_forecast`, ...) and thin wrappers around alignment operations |
| `alignment/` | Time and space alignment (pairwise ops and the multi-dataset `align_time` / `align_space`), NaN broadcasting, and `interpolations/` (interpolator registry, xarray and Delaunay interpolators) |
| `pipeline/` | The YAML-driven pipeline: `cli` (the `mxalign` command), `runner`, `verification`, `transformations/`, and `config` (pydantic schema for the YAML config) with its `dates` / `save` helpers |
| `utils/` | Helpers shared by `accessor` and `alignment` (`projections`, `traits`) |

Public functions are re-exported from `mxalign/__init__.py`. Keep those re-exports working when
moving code.

## Development tools

- Python >= 3.12, managed with [uv](https://docs.astral.sh/uv/).
- Set up the environment: `uv sync --all-extras`
- Run the tests: `uv run pytest tests -v`
- Run the linters: `uvx pre-commit run --all-files`. This runs ruff check (with `--fix`), ruff
format, codespell, numpydoc-validation and the standard pre-commit hooks. CI runs the same hooks
(`.github/workflows/linting.yaml`), plus the tests (`.github/workflows/tests.yaml`).
- Add a dependency with `uv add <package>` (or `uv add --optional <extra> <package>`), and commit
both `pyproject.toml` and `uv.lock`.

## Coding conventions

- **Type annotations** on every argument (including `*args` / `**kwargs`) and every return value,
in `src/` and in `tests/`. Use modern syntax (`X | None`, `list[str]`) and concrete types
(`xr.Dataset`, `np.ndarray`, ...) rather than `Any`. Enforced by ruff's `ANN` rules.
- **Numpy-style docstrings** on every module, class, function and method, in `src/` and in
`tests/`. Each one has a one-line imperative summary ending in a period, then `Parameters`,
`Returns` (or `Yields`), and `Raises` where applicable. Class docstrings document the
`__init__` parameters, so `__init__` itself needs no docstring (it is excluded from
numpydoc's checks). pydantic models in `pipeline/config.py` document their fields under
`Attributes`. Enforced by ruff's `D` rules (numpy convention) and by `numpydoc lint`.
`Examples`, `See Also` and an extended summary are optional. Run `ruff` through pre-commit, which
pins the version: newer ruff releases enable more rules by default.

```python
def align_grid_point(
ds1: xr.Dataset, ds2: xr.Dataset, method: str = "xarray", **kwargs: object
) -> xr.Dataset:
"""Interpolate grid dataset ``ds1`` to the point locations of ``ds2``.

Parameters
----------
ds1 : xr.Dataset
Gridded source dataset.
ds2 : xr.Dataset
Point dataset whose locations are the interpolation targets.
method : str, default "xarray"
Name of a registered interpolator.
**kwargs : object
Passed to the interpolator's constructor.

Returns
-------
xr.Dataset
``ds1`` interpolated to the points of ``ds2``.
"""
```
- **Formatting** with `ruff format`. Don't hand-format against it.
- **Never mutate inputs**: alignment operations and accessor methods return new datasets (see
`MxAccessor`, which reads traits from `ds.attrs` on every call and never modifies `ds`).
- **Registries**: new interpolators and transformations are registered with
`register_interpolator` / `register_transformation` and imported in their package
`__init__.py`, so they register on `import mxalign`.
- **Config**: new pipeline config keys go into the pydantic models in
`pipeline/config.py`. Sections reject unknown keys unless they pass kwargs through to a function.

## Tests

- pytest, under `tests/`. Shared fixtures (`ds_fcst`, `ds_obs`, ...) are in `tests/conftest.py`.
- Test functions are annotated (`-> None`, fixture arguments typed) and have numpy-style docstrings
that state the behaviour being checked, the same as package code.
- Add or update tests together with any behaviour change.

## Changes and decisions

- Add an entry to `CHANGELOG.md` under `[Unreleased]` (Keep a Changelog format) with your GitHub
handle. Mark breaking changes with **Breaking:**.
- Record design decisions as ADRs in `doc/adr/` using `doc/adr/template.md`.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Changed
- All functions, methods and classes (package and tests) have type annotations and numpy-style docstrings, enforced in pre-commit/CI by ruff `ANN`/`D` rules and `numpydoc-validation`. `AGENTS.md` documents the coding conventions and development tools. @leifdenby
- **Breaking:** the pipeline config is validated against an explicit pydantic schema (`mxalign.pipeline.config.PipelineConfig`, loaded with `load_config`) instead of being read as a free-form dict. Unknown sections or keys, and missing required keys, raise a `ValidationError` when the `Runner` is created; `Runner.config` is now the validated model. Per-dataset `dates` entries override individual fields of the top-level `dates`. @leifdenby
- **Breaking:** file path patterns (`datasets.<name>.files` with a `dates:` section, and `alignment.save.path`) now use Python `str.format` syntax instead of earthkit-data patterns: `{reference_time:%Y%m%d%H}` instead of `{reference_time:strftime(%Y%m%d%H)}`, `{year:02d}` instead of `{year:int(%02d)}`. `{lead_time}` (integer hours) and `{valid_time}` are now supported in `files` patterns. `earthkit-data` is no longer a dependency. @michielv
- **Breaking:** temporal alignment always runs in the pipeline; `alignment.time` is optional and only accepts `lead_time` (`reference` | `intersection` | `union`, forecast reference only). Unknown keys such as `return_as` or `method` now raise an error. `align_time` takes `reference` and an explicit `lead_time` instead of `**kwargs`. @michielv
Expand Down
17 changes: 16 additions & 1 deletion examples/myloader.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,22 @@
"""Example custom loader for SYNOP observations, used by the example pipeline config."""

import xarray as xr


def load_dataset(file):
def load_dataset(file: str) -> xr.Dataset:
"""Load a SYNOP observation netCDF file and attach the mlwp traits.

Parameters
----------
file : str
Path to the netCDF file with a ``code`` station dimension.

Returns
-------
xr.Dataset
Observation point dataset with a ``point_index`` dimension, mlwp
traits in its attributes, and CF attributes on its coordinates.
"""
ds = xr.open_mfdataset([file], engine="h5netcdf")

ds = ds.rename_dims(code="point_index")
Expand Down
22 changes: 22 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,28 @@ build-backend = "hatchling.build"
[tool.pytest.ini_options]
testpaths = ["tests"]

[tool.ruff.lint]
# ANN: type annotations on all arguments and return values; D: docstrings
extend-select = ["ANN", "D"]

[tool.ruff.lint.pydocstyle]
convention = "numpy"

[tool.ruff.lint.per-file-ignores]
# Orphaned loader that imports removed modules; pending removal or port to mlwp-data-loaders
"src/mxalign/loaders/ifs_forecast.py" = ["ANN", "D"]

[tool.numpydoc_validation]
checks = [
"all", # report on all checks, except the below
"EX01", # no "Examples" section required
"SA01", # no "See Also" section required
"ES01", # no "Extended Summary" section required
]
# __init__ parameters are documented in the class docstring (numpydoc convention)
exclude = ['\.__init__$']
exclude_files = ['src/mxalign/loaders/ifs_forecast\.py']

[dependency-groups]
dev = [
"ipykernel>=7.2.0",
Expand Down
35 changes: 17 additions & 18 deletions src/mxalign/__init__.py
Original file line number Diff line number Diff line change
@@ -1,30 +1,29 @@
from .pipeline.transformations.transform import transform
from .pipeline.transformations.registry import (
available_transformations,
register_transformation,
)
"""Align and verify weather forecast and observation datasets with xarray."""

from . import accessor, alignment, pipeline
from .alignment.interpolations.interpolate import interpolate
from .alignment.interpolations.registry import (
available_interpolations,
register_interpolator,
)
from .alignment.time import align_time
from .alignment.space import align_space

from . import accessor
from . import alignment
from . import pipeline
from .alignment.time import align_time
from .pipeline.transformations.registry import (
available_transformations,
register_transformation,
)
from .pipeline.transformations.transform import transform

__all__ = [
"transform",
"available_transformations",
"register_transformation",
"interpolate",
"available_interpolations",
"register_interpolator",
"align_time",
"align_space",
"accessor",
"align_space",
"align_time",
"alignment",
"available_interpolations",
"available_transformations",
"interpolate",
"pipeline",
"register_interpolator",
"register_transformation",
"transform",
]
2 changes: 2 additions & 0 deletions src/mxalign/accessor/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
"""Dataset accessors provided by mxalign (registers ``ds.mx`` on import)."""

from . import mx

__all__ = ["mx"]
Loading
Loading