Repository navigation
Add PangeoCMIP6 demo notebook - #288
ahmedmohiduet wants to merge 3 commits into
Conversation
Adds packages/third_party/pangeo_site_archive with PangeoCMIP6, an ArchiveIndex that reads CMIP6 variables from the Pangeo ARCO cloud archive, plus offline unit tests, CI install/coverage entries and a docs section. Closes ACCESS-Community-Hub#284. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Unqqtp91DWVC8ro9Y724iN
Returns the Pangeo CMIP6 catalogue as a pandas.DataFrame so users can discover available data. Includes a unit test.
Demonstrate catalogue discovery, lazy `retrieve`, global map, annual global mean and multiple variables with `PangeoCMIP6`, derived from `ArchiveIndex`. Link notebook from `Gallery.ipynb`. Refs ACCESS-Community-Hub#285
There was a problem hiding this comment.
🟡 Changes recommended
Invalid timestamps can return the full dataset, and ambiguous DCPP catalog entries may select an arbitrary store.
3 open findings
What changed in this PR
Adds a Pangeo CMIP6 cloud-data accessor and demonstration notebook to PyEarthTools.
Changes:
- Implements and tests the
PangeoCMIP6accessor. - Adds discovery, retrieval, visualization, and analysis examples.
- Integrates the package into documentation and CI.
| File | Description |
|---|---|
.github/workflows/python-app.yml |
Adds package coverage. |
requirements_cicd.txt |
Installs the new package in CI. |
docs/data.md |
Documents cloud CMIP6 access. |
docs/notebooks/Gallery.ipynb |
Adds the notebook to the gallery. |
docs/notebooks/data/Pangeo_CMIP6.ipynb |
Demonstrates the accessor workflow. |
packages/third_party/pangeo_site_archive/README.md |
Describes the package. |
packages/third_party/pangeo_site_archive/pyproject.toml |
Defines package metadata and dependencies. |
packages/third_party/pangeo_site_archive/src/site_archive_pangeo/__init__.py |
Exports and registers the accessor. |
packages/third_party/pangeo_site_archive/src/site_archive_pangeo/pangeo_cmip6.py |
Implements catalog lookup and Zarr loading. |
packages/third_party/pangeo_site_archive/tests/data/mini_catalog.csv |
Provides catalog test data. |
packages/third_party/pangeo_site_archive/tests/test_pangeo_cmip6.py |
Tests accessor behavior. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| Find the Zarr store of each variable. | ||
|
|
||
| Args: | ||
| querytime: Ignored, as a store holds all times. |
| raise ValueError( | ||
| f"Variable {variable!r} exists on several grids {grids}; pass grid_label to choose one." | ||
| ) | ||
| stores[variable] = str(rows.sort_values("version").iloc[-1]["zstore"]) |
| "| Applying data transforms | -- | [Applying Data Transforms](./data/Transforms.ipynb) | 18 Aug 2025 |\n", | ||
| "| Geospatial subsetting | -- | [Region Cutting](./data/RegionCutting.ipynb) | 18 Aug 2025 |\n", | ||
| "| Remote Zarr Loading | Demontsrates loading data in a remote zarr archive | [Remote Zarr Loading](./data/Remote_Zarr_Loading.ipynb) |2 June 2026 |\n" | ||
| "| Remote Zarr Loading | Demontsrates loading data in a remote zarr archive | [Remote Zarr Loading](./data/Remote_Zarr_Loading.ipynb) |2 June 2026 |\n", |
|
@ahmedmohiduet I've also now requested a copilot review for these PRs, to assist with the final review by Tennessee. As always, feel to use or disregard the suggestions by copilot based on your judgement. |
stevehadd
left a comment
There was a problem hiding this comment.
this looks good thanks @ahmedmohiduet . I made a suggestion how we could better demo a pyearthtools way of doing things (with transforms and operations) in the notebook, and shown what this would look like for the unit conversdaiton. It would be good to do the other changes in the notebook through a pipeline as well, to show off the pyearthtools pipeline concept.
There was a problem hiding this comment.
Thanks @ahmedmohiduet this looks really good. One change that would be good would be to do some of the operations in the notebooks using a pyearthtools pipeline. Like in python there is a "pythonic" way to do things, so there is a way of doing things we're trying to promote in pyearthtools,l especially in the tutorial/demo notebooks. So for example, doing the unit conversion could be done in a operation in a pipeline. Here is some example code I have create for this purpose. The transform class could just be in the notebook, or you could contribute it to the pipeline or data module. Here's something I put together:
class ConvertUnits(pyearthtools.data.Transform):
"""
This transform converts the unit in a dataset.
"""
def __init__(self, var_convert_dict, var_units):
self._var_convert_dict = var_convert_dict
self._var_units = var_units
if set(var_convert_dict.keys()) != set(var_units.keys()):
raise ValueError('The unit conversion function and unit names dictionaries must have the same keys.')
super().__init__(
)
self.record_initialisation()
def apply(self, sample):
def map_func(da, var_lut):
try:
ret_val = var_lut[da.name](da)
except KeyError:
ret_val = da
return ret_val
ret_sample = sample.map(map_func,var_lut=self._var_convert_dict)
for var_name, var_unit in self._var_units.items():
ret_sample[var_name].attrs['units'] = var_unit
return ret_sample
There was a problem hiding this comment.
You could then use either in a pipeline:
var_lut = {'tas': lambda arr1: arr1 - 273.15}
unit_lut = {'tas': 'c'}
cmip_pipe = pyearthtools.pipeline.Pipeline(
index,
ConvertUnits(var_convert_dict=var_lut, var_units=unit_lut),
)
cmip_pipe['2001-03']
or you could supply the transform to the index so you get the data object with the right units
var_lut = {'tas': lambda arr1: arr1 - 273.15}
unit_lut = {'tas': 'c'}
cmip_index = PangeoCMIP6(
['tas'],
source_id='CanESM5',
experiment_id='historical',
transforms=[ConvertUnits(var_convert_dict=var_lut, var_units=unit_lut)])
cmip_index['2001-03']



Closes #285 ( part of #276 )
Adds
docs/notebooks/data/Pangeo_CMIP6.ipynb, demo notebook forPangeoCMIP6accessor from #284 (parent #276).Covers:
PangeoCMIP6.catalog()CanESM5,historical)retrieveof one monthtasloadof full record, with annual global meangrid_labelAlso adds row in
Gallery.ipynb.Notes:
pet_nb_cpu. Outputs saved, no errors or warnings in them.cartopyfetches coastline data on first plot.!pip installcell fromRemote_Zarr_Loading.ipynbleft out, sincePyEarthTools-archive-Pangeois not confirmed on PyPI.xarrayreprs.