refactor: alignment

.gitignore

@@ -8,3 +8,6 @@ wheels/
 
 # Virtual environments
 .venv
+
+# Presentations
+presentation/

.pre-commit-config.yaml

@@ -24,3 +24,8 @@ repos:
       - id: ruff-check
         args: [--fix]
       - id: ruff-format
+
+  - repo: https://github.com/numpy/numpydoc
+    rev: v1.11.0
+    hooks:
+      - id: numpydoc-validation

AGENTS.md

@@ -0,0 +1,94 @@
+# AGENTS.md
+
+Guidance for AI coding agents and human contributors working on `mxalign`.
+
+## Project
+
+`mxalign` aligns meteorological `xarray` datasets in space and time. Datasets carry
+**traits** (`mlwp_space_trait`, `mlwp_time_trait`, `mlwp_uncertainty_trait` in `ds.attrs`).
+The traits are defined by [mlwp-data-specs](https://github.com/mlwp-tools/mlwp-data-specs) and
+attached by [mlwp-data-loaders](https://github.com/mlwp-tools/mlwp-data-loaders). Loading and
+validating datasets happens in those packages. `mxalign` only reads the traits to decide how to
+align.
+
+Package layout (`src/mxalign/`):
+
+| Package | Contents |
+|---|---|
+| `accessor/` | The `ds.mx` accessor (`MxAccessor`): trait predicates (`is_grid`, `is_forecast`, ...) and thin wrappers around alignment operations |
+| `alignment/` | Time and space alignment (pairwise ops and the multi-dataset `align_time` / `align_space`), NaN broadcasting, and `interpolations/` (interpolator registry, xarray and Delaunay interpolators) |
+| `pipeline/` | The YAML-driven pipeline: `cli` (the `mxalign` command), `runner`, `verification`, `transformations/`, and `config` (pydantic schema for the YAML config) with its `dates` / `save` helpers |
+| `utils/` | Helpers shared by `accessor` and `alignment` (`projections`, `traits`) |
+
+Public functions are re-exported from `mxalign/__init__.py`. Keep those re-exports working when
+moving code.
+
+## Development tools
+
+- Python >= 3.12, managed with [uv](https://docs.astral.sh/uv/).
+- Set up the environment: `uv sync --all-extras`
+- Run the tests: `uv run pytest tests -v`
+- Run the linters: `uvx pre-commit run --all-files`. This runs ruff check (with `--fix`), ruff
+  format, codespell, numpydoc-validation and the standard pre-commit hooks. CI runs the same hooks
+  (`.github/workflows/linting.yaml`), plus the tests (`.github/workflows/tests.yaml`).
+- Add a dependency with `uv add <package>` (or `uv add --optional <extra> <package>`), and commit
+  both `pyproject.toml` and `uv.lock`.
+
+## Coding conventions
+
+- **Type annotations** on every argument (including `*args` / `**kwargs`) and every return value,
+  in `src/` and in `tests/`. Use modern syntax (`X | None`, `list[str]`) and concrete types
+  (`xr.Dataset`, `np.ndarray`, ...) rather than `Any`. Enforced by ruff's `ANN` rules.
+- **Numpy-style docstrings** on every module, class, function and method, in `src/` and in
+  `tests/`. Each one has a one-line imperative summary ending in a period, then `Parameters`,
+  `Returns` (or `Yields`), and `Raises` where applicable. Class docstrings document the
+  `__init__` parameters, so `__init__` itself needs no docstring (it is excluded from
+  numpydoc's checks). pydantic models in `pipeline/config.py` document their fields under
+  `Attributes`. Enforced by ruff's `D` rules (numpy convention) and by `numpydoc lint`.
+  `Examples`, `See Also` and an extended summary are optional. Run `ruff` through pre-commit, which
+  pins the version: newer ruff releases enable more rules by default.
+
+  ```python
+  def align_grid_point(
+      ds1: xr.Dataset, ds2: xr.Dataset, method: str = "xarray", **kwargs: object
+  ) -> xr.Dataset:
+      """Interpolate grid dataset ``ds1`` to the point locations of ``ds2``.
+
+      Parameters
+      ----------
+      ds1 : xr.Dataset
+          Gridded source dataset.
+      ds2 : xr.Dataset
+          Point dataset whose locations are the interpolation targets.
+      method : str, default "xarray"
+          Name of a registered interpolator.
+      **kwargs : object
+          Passed to the interpolator's constructor.
+
+      Returns
+      -------
+      xr.Dataset
+          ``ds1`` interpolated to the points of ``ds2``.
+      """
+  ```
+- **Formatting** with `ruff format`. Don't hand-format against it.
+- **Never mutate inputs**: alignment operations and accessor methods return new datasets (see
+  `MxAccessor`, which reads traits from `ds.attrs` on every call and never modifies `ds`).
+- **Registries**: new interpolators and transformations are registered with
+  `register_interpolator` / `register_transformation` and imported in their package
+  `__init__.py`, so they register on `import mxalign`.
+- **Config**: new pipeline config keys go into the pydantic models in
+  `pipeline/config.py`. Sections reject unknown keys unless they pass kwargs through to a function.
+
+## Tests
+
+- pytest, under `tests/`. Shared fixtures (`ds_fcst`, `ds_obs`, ...) are in `tests/conftest.py`.
+- Test functions are annotated (`-> None`, fixture arguments typed) and have numpy-style docstrings
+  that state the behaviour being checked, the same as package code.
+- Add or update tests together with any behaviour change.
+
+## Changes and decisions
+
+- Add an entry to `CHANGELOG.md` under `[Unreleased]` (Keep a Changelog format) with your GitHub
+  handle. Mark breaking changes with **Breaking:**.
+- Record design decisions as ADRs in `doc/adr/` using `doc/adr/template.md`.

CHANGELOG.md

@@ -8,11 +8,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Changed
+- All functions, methods and classes (package and tests) have type annotations and numpy-style docstrings, enforced in pre-commit/CI by ruff `ANN`/`D` rules and `numpydoc-validation`. `AGENTS.md` documents the coding conventions and development tools. @leifdenby
+- **Breaking:** the pipeline config is validated against an explicit pydantic schema (`mxalign.pipeline.config.PipelineConfig`, loaded with `load_config`) instead of being read as a free-form dict. Unknown sections or keys, and missing required keys, raise a `ValidationError` when the `Runner` is created; `Runner.config` is now the validated model. Per-dataset `dates` entries override individual fields of the top-level `dates`. @leifdenby
+- **Breaking:** file path patterns (`datasets.<name>.files` with a `dates:` section, and `alignment.save.path`) now use Python `str.format` syntax instead of earthkit-data patterns: `{reference_time:%Y%m%d%H}` instead of `{reference_time:strftime(%Y%m%d%H)}`, `{year:02d}` instead of `{year:int(%02d)}`. `{lead_time}` (integer hours) and `{valid_time}` are now supported in `files` patterns. `earthkit-data` is no longer a dependency. @michielv
+- **Breaking:** temporal alignment always runs in the pipeline; `alignment.time` is optional and only accepts `lead_time` (`reference` | `intersection` | `union`, forecast reference only). Unknown keys such as `return_as` or `method` now raise an error. `align_time` takes `reference` and an explicit `lead_time` instead of `**kwargs`. @michielv
 - Remove dataset loading and dataset validation and outsource it to [mlwp-data-loaders](https://github.com/mlwp-tools/mlwp-data-loaders) and [mlwp-data-specs](https://github.com/mlwp-tools/mlwp-data-specs) @michielv
 - Store dataset properties as netCDF-safe individual attributes while keeping read compatibility with legacy `attrs["properties"]` dict/JSON data. [\#21](https://github.com/mlwp-tools/mxalign/pull/21) @observingClouds
 - Added CI test workflow with first unit tests. [\#21](https://github.com/mlwp-tools/mxalign/pull/21) @observingClouds
 - Added optional `ifs` dependency group with `cfgrib`, `eccodes`, and `eccodeslib`. [\#21](https://github.com/mlwp-tools/mxalign/pull/21) @observingClouds
 - Added CI action for package build and upload to pypi.org on releases. [\#28](https://github.com/mlwp-tools/mxalign/pull/28) @leifdenby
+- `transform()` and `interpolate()` return the same container type they are given: a one-item list now stays a list instead of being unwrapped to a bare dataset, and a dict with a single dataset no longer raises `KeyError`. @michielv
+- Delaunay interpolation weights are shared across datasets on the same grid, so aligning several models to the same points triangulates the grid once. When a multi-chunk `grid_index` is merged into a single chunk, the leading dimensions are re-chunked automatically so blocks stay near dask's `array.chunk-size`. @michielv
+
+### Fixed
+- `ds.mx.add_xy(crs=...)` and `ds.mx.unstack(crs=...)` no longer replace the dataset held by the cached `ds.mx` accessor, so later `ds.mx` calls work on `ds` itself instead of a copy with the CRS attached. @michielv
+- Interpolated point datasets no longer inherit the source grid's `crs` and `grid_mapping` attributes, so they can be saved with `to_netcdf`. @michielv
+- Saving aligned datasets with `alignment.save.datasets: merge` no longer fails with `UnboundLocalError`; the merged output fills `{name}` in the path with `merged`. @michielv
+- A pipeline config without a `transformations:` section no longer crashes. @michielv
+- `ds.mx.stack()` now stacks regular grids with `latitude`/`longitude` dimensions (it looked for `lat`/`lon`). @michielv
+- Delaunay interpolation could silently reuse the weights of a different grid with the same shape and corner coordinates, giving wrong values; weights are now cached by a hash of the full source and target coordinates. @michielv
 
 ## [0.1.0](https://github.com/mlwp-tools/mxalign/releases/tag/v0.1.0)
 

README.md

@@ -32,7 +32,7 @@ ds_fcst_aligned = mx.align_space(ds_fcst, reference=ds_obs, method="interpolatio
 
 # Align datasets temporally
 datasets = {"obs": ds_obs, "fcst": ds_fcst_aligned}
-aligned_datasets = mx.align_time(datasets, method="intersection")
+aligned_datasets = mx.align_time(datasets, reference="obs")
 ```
 
 For a more comprehensive interactive example, check out the [introductory notebook](./examples/introduction.ipynb).
@@ -43,7 +43,7 @@ For a more comprehensive interactive example, check out the [introductory notebo
 
 ### Configuration Contents
 
-The configuration file is divided into several main sections:
+The configuration file is divided into several main sections. It is validated on load against the schema in `mxalign.pipeline.config` (`PipelineConfig`): unknown sections or keys, and missing required ones such as `alignment.reference`, raise an error up front. Keys that are passed through to a function, such as loader, transformation, metric and save options, are accepted as-is.
 
 ```yaml
 datasets:
@@ -61,8 +61,9 @@ transformations:
 alignment:
   # Define reference dataset and alignment methods (space, time, NaN broadcasting)
   reference: obs_data
-  time:
-    method: intersection
+  # Time alignment always runs; options only apply to a forecast reference
+  # time:
+  #   lead_time: union  # reference (default) | intersection | union
 
 verification:
   # Specify the reference dataset and the metrics to calculate
@@ -92,7 +93,7 @@ mxalign slurm path/to/config.yaml --account your_account --queue your_queue --co
 You can also execute the entire configuration-driven pipeline directly from Python using the `Runner` class.
 
 ```python
-from mxalign.runner import Runner
+from mxalign.pipeline.runner import Runner
 
 # Initialize the runner with a YAML config file or a dictionary
 runner = Runner("path/to/config.yaml")

examples/advanced.ipynb

@@ -384,7 +384,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from mxalign.runner import Runner\n",
+    "from mxalign.pipeline.runner import Runner\n",
     "\n",
     "runner = Runner(\"./config.yaml\")"
    ]
@@ -455,7 +455,7 @@
     }
    ],
    "source": [
-    "runner.config()"
+    "runner.config"
    ]
   },
   {

examples/config.yaml

@@ -7,17 +7,17 @@ dates:
 
 datasets:
   la-1024-ea-00:
-    loader: anemoi-inference
-    files: /scratch/project_465000527/vandenbl/sg-la-comparison/inference-out/la-1024-01-ea-00/{reference_time:strftime(%Y%m%d%H)}.nc
+    loader: mlwp_data_loaders.loaders.anemoi.anemoi_inference
+    files: /scratch/project_465000527/vandenbl/sg-la-comparison/inference-out/la-1024-01-ea-00/{reference_time:%Y%m%d%H}.nc
     #grid_mapping: CERRA
     variables: ["2t","10u","10v","msl"]
   cerra:
-    loader: anemoi-datasets
+    loader: mlwp_data_loaders.loaders.anemoi.anemoi_datasets
     files: /scratch/project_465002133/datasets/cerra-rr-an-oper-0001-mars-5p5km-1984-2020-3h-v2-rmi.zarr
     #grid_mapping: CERRA
     variables: ["2t_2", "10u_10", "10v_10", "msl_0"]
   # synop:
-  #   loader: mxalign
+  #   loader: ./myloader.py
   #   files: /scratch/project_465002133/datasets/observations/cerra_synops_2020.nc
   #   variables: ["2t","10si","msl"]
 transformations:
@@ -42,15 +42,16 @@ transformations:
 alignment:
   broadcast_nans: False
   reference: cerra
-  time:
-    return_as: forecast
+  # Time alignment always runs; options only apply to a forecast reference
+  # time:
+  #   lead_time: union  # reference (default) | intersection | union
   space:
     interpolation:
       method: delaunay
   # save:
   #   method: to_netcdf
   #   datasets: all
-  #   path: "{name}_{year:int(%02d)}.nc"
+  #   path: "{name}_{year:02d}.nc"
 
 verification:
   reference: cerra