Repository navigation
Conversation
accessors/time.py and accessors/space.py held alignment operations rather than accessor code. Move them into align/time.py and align/space.py next to the multi-dataset align_time/align_space, and have the mx accessor import them from there. No behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r alignment Top-level layout becomes `accessor` (the ds.mx accessor) and `alignment` (time/space/nans alignment plus the interpolators it uses); the pipeline package follows separately. Public re-exports from mxalign/__init__.py are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The YAML-driven pipeline (cli, runner, verification, transformations and the config/dates/save helpers it alone uses) now lives in mxalign.pipeline. The console script points at mxalign.pipeline.cli:main. Public re-exports from mxalign/__init__.py are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the untyped Config dict wrapper with pydantic models (PipelineConfig and one model per section) loaded via load_config(). Unknown or missing keys now fail when the Runner is created instead of deep inside a run, and the Runner reads typed attributes rather than popping keys (which also stops align() mutating the config). Keys that are forwarded to loaders, transformations, metrics and save methods are kept as pass-through kwargs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…add AGENTS.md Every module, class, function and method in src/, tests/ and the example loader now has type annotations on all arguments and return values and a numpy-style docstring (Parameters/Returns/Raises). This is enforced in pre-commit (and therefore CI) by ruff's ANN and D rules (numpy convention) and the numpydoc-validation hook. AGENTS.md describes the package layout, development tools and coding conventions. No behaviour changes: apart from docstrings, annotations and import sorting, the only code edits are sorting mxalign.__all__ and dict literals in tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
is_deterministic(), is_ensemble() and is_quantile() read mlwp_uncertainty_trait on every call, like the space and time predicates (TODO_ENS U4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add ds_fcst_ens and ds_grid_ens fixtures (member coord with CF realization attrs) and lock in that pairwise and global time alignment and grid-to-point interpolation carry the member dim and the ensemble trait, and never match members between datasets (TODO_ENS U9). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Variables whose last dim wasn't grid_index were silently dropped, e.g. an ensemble with member after grid_index. Move grid_index last for the matmul and put point_index in its place in the output. Variables without grid_index are still skipped, now with a logged warning (TODO_ENS U6). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
broadcast_nans crashed whenever an ensemble met a dataset without member, which the pipeline does by default. The uncertainty dim (from the trait) is now reduced with any() before masks are compared and the mask is applied to all members, so the ensemble size stays constant; members are never matched between datasets. Other dim mismatches raise a ValueError. Decision recorded in ADR-003 (TODO_ENS U1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Time alignment and interpolation already carry a
memberdimension through. But every run with an ensemble then crashed inbroadcast_nans, which the pipeline runs by default. This PR makes the alignment step work for ensembles, on their own or mixed with deterministic forecasts and observations. Verification (chunking ofmemberinMetric, and which metrics apply to which uncertainty type) comes in a follow-up PR.Changes
ds.mx.is_deterministic(),is_ensemble()andis_quantile(). Like the space and time predicates, they read the trait on every call.broadcast_nansno longer crashes when an ensemble meets a dataset withoutmember. The uncertainty dimension, taken from the trait, is never matched between datasets. If any member is missing at a point, the ensemble counts as missing there, and the mask goes to all its members and to the other datasets. This keeps the ensemble size constant over the verification sample, and ensembles of different sizes work together. Any other difference in dimensions on a shared variable raises aValueError. Please review the decision in ADR-003 (status: proposed). Its main cost is that one missing member discards the whole ensemble at that point.grid_index, such as an ensemble withmemberaftergrid_index.point_indextakes the place ofgrid_index, and the other dimensions keep their order.ds_fcst_ensandds_grid_ensfixtures, whosemembercoordinate hasstandard_name="realization"andunits="1". Tests cover time alignment with ensembles (pairwise in all directions, and global), both interpolators withmemberfirst or last (numpy and dask), a newtests/test_nans.py, and a mixed deterministic + ensembleRunner.align.Breaking:
broadcast_nansnow needsmlwp_uncertainty_traiton every dataset. Datasets from mlwp-data-loaders always have it.Testing
Not in this PR
xskillscore.crps_ensemblestill fails becauseMetricsplitsmemberinto chunks, and every metric is still applied to every model.membercoordinate in the specs: Uncertainty trait:quantileis not a CF standard name, and the ensemblemembercoordinate is underspecified mlwp-data-specs#34 and Uncertainty trait: forbid uncertainty dims in deterministic, enforce member/quantile conventions mlwp-data-specs#35.🤖 Generated with Claude Code