Skip to content

Add PangeoCMIP6 demo notebook - #288

Open
ahmedmohiduet wants to merge 3 commits into
ACCESS-Community-Hub:developfrom
ahmedmohiduet:285-demo-notebook
Open

ahmedmohiduet wants to merge 3 commits into
ACCESS-Community-Hub:developfrom
ahmedmohiduet:285-demo-notebook

Conversation

@ahmedmohiduet

Copy link
Copy Markdown

Closes #285 ( part of #276 )

Adds docs/notebooks/data/Pangeo_CMIP6.ipynb, demo notebook for PangeoCMIP6 accessor from #284 (parent #276).

Covers:

  • catalogue discovery with PangeoCMIP6.catalog()
  • accessor setup (CanESM5, historical)
  • lazy retrieve of one month
  • global map of tas
  • load of full record, with annual global mean
  • multiple variables and grid_label

Also adds row in Gallery.ipynb.

Notes:

  • Executed in full with kernel pet_nb_cpu. Outputs saved, no errors or warnings in them.
  • Needs network access for Google Cloud storage. cartopy fetches coastline data on first plot.
  • Colab badge and !pip install cell from Remote_Zarr_Loading.ipynb left out, since PyEarthTools-archive-Pangeo is not confirmed on PyPI.
  • File is about 320 KB, mostly xarray reprs.

ahmedmohiduet and others added 3 commits October 5, 2026 01:31
Adds packages/third_party/pangeo_site_archive with PangeoCMIP6, an ArchiveIndex that reads CMIP6 variables from the Pangeo ARCO cloud archive, plus offline unit tests, CI install/coverage entries and a docs section. Closes ACCESS-Community-Hub#284.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Unqqtp91DWVC8ro9Y724iN
Returns the Pangeo CMIP6 catalogue as a pandas.DataFrame so users can discover available data. Includes a unit test.
Demonstrate catalogue discovery, lazy `retrieve`, global map, annual global mean and multiple variables with `PangeoCMIP6`, derived from `ArchiveIndex`. Link notebook from `Gallery.ipynb`.

Refs ACCESS-Community-Hub#285
@stevehadd
stevehadd requested review from stevehadd and a balanced review from Copilot October 12, 2026 09:50

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Invalid timestamps can return the full dataset, and ambiguous DCPP catalog entries may select an arbitrary store.

3 open findings
What changed in this PR

Adds a Pangeo CMIP6 cloud-data accessor and demonstration notebook to PyEarthTools.

Changes:

  • Implements and tests the PangeoCMIP6 accessor.
  • Adds discovery, retrieval, visualization, and analysis examples.
  • Integrates the package into documentation and CI.
File Description
.github/​workflows/​python-app.yml Adds package coverage.
requirements_cicd.txt Installs the new package in CI.
docs/​data.md Documents cloud CMIP6 access.
docs/​notebooks/​Gallery.ipynb Adds the notebook to the gallery.
docs/​notebooks/​data/​Pangeo_CMIP6.ipynb Demonstrates the accessor workflow.
packages/​third_party/​pangeo_site_archive/​README.md Describes the package.
packages/​third_party/​pangeo_site_archive/​pyproject.toml Defines package metadata and dependencies.
packages/​third_party/​pangeo_site_archive/​src/​site_archive_pangeo/​__init__.py Exports and registers the accessor.
packages/​third_party/​pangeo_site_archive/​src/​site_archive_pangeo/​pangeo_cmip6.py Implements catalog lookup and Zarr loading.
packages/​third_party/​pangeo_site_archive/​tests/​data/​mini_catalog.csv Provides catalog test data.
packages/​third_party/​pangeo_site_archive/​tests/​test_pangeo_cmip6.py Tests accessor behavior.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Find the Zarr store of each variable.

Args:
querytime: Ignored, as a store holds all times.
raise ValueError(
f"Variable {variable!r} exists on several grids {grids}; pass grid_label to choose one."
)
stores[variable] = str(rows.sort_values("version").iloc[-1]["zstore"])
"| Applying data transforms | -- | [Applying Data Transforms](./data/Transforms.ipynb) | 18 Aug 2025 |\n",
"| Geospatial subsetting | -- | [Region Cutting](./data/RegionCutting.ipynb) | 18 Aug 2025 |\n",
"| Remote Zarr Loading | Demontsrates loading data in a remote zarr archive | [Remote Zarr Loading](./data/Remote_Zarr_Loading.ipynb) |2 June 2026 |\n"
"| Remote Zarr Loading | Demontsrates loading data in a remote zarr archive | [Remote Zarr Loading](./data/Remote_Zarr_Loading.ipynb) |2 June 2026 |\n",
@stevehadd

Copy link
Copy Markdown
Collaborator

@ahmedmohiduet I've also now requested a copilot review for these PRs, to assist with the final review by Tennessee. As always, feel to use or disregard the suggestions by copilot based on your judgement.

@stevehadd stevehadd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks good thanks @ahmedmohiduet . I made a suggestion how we could better demo a pyearthtools way of doing things (with transforms and operations) in the notebook, and shown what this would look like for the unit conversdaiton. It would be good to do the other changes in the notebook through a pipeline as well, to show off the pyearthtools pipeline concept.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ahmedmohiduet this looks really good. One change that would be good would be to do some of the operations in the notebooks using a pyearthtools pipeline. Like in python there is a "pythonic" way to do things, so there is a way of doing things we're trying to promote in pyearthtools,l especially in the tutorial/demo notebooks. So for example, doing the unit conversion could be done in a operation in a pipeline. Here is some example code I have create for this purpose. The transform class could just be in the notebook, or you could contribute it to the pipeline or data module. Here's something I put together:

class ConvertUnits(pyearthtools.data.Transform):
    """
    This transform converts the unit in a dataset.
    """
    def __init__(self, var_convert_dict, var_units):
        self._var_convert_dict = var_convert_dict
        self._var_units = var_units
        if set(var_convert_dict.keys()) != set(var_units.keys()):
            raise ValueError('The unit conversion function and unit names dictionaries must have the same keys.')
        
        super().__init__(
        )
        self.record_initialisation()

    def apply(self, sample):

        def map_func(da, var_lut):
            try:
                ret_val = var_lut[da.name](da)
            except KeyError:
                ret_val = da
            return ret_val
            
        ret_sample = sample.map(map_func,var_lut=self._var_convert_dict)

        for var_name, var_unit in self._var_units.items():
            ret_sample[var_name].attrs['units'] = var_unit
        
        return ret_sample
        

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could then use either in a pipeline:

var_lut = {'tas': lambda arr1: arr1 - 273.15}
unit_lut = {'tas': 'c'}
cmip_pipe = pyearthtools.pipeline.Pipeline(
    index, 
     ConvertUnits(var_convert_dict=var_lut, var_units=unit_lut),
)
cmip_pipe['2001-03']

or you could supply the transform to the index so you get the data object with the right units

var_lut = {'tas': lambda arr1: arr1 - 273.15}
unit_lut = {'tas': 'c'}
cmip_index = PangeoCMIP6(
    ['tas'], 
    source_id='CanESM5', 
    experiment_id='historical', 
    transforms=[ConvertUnits(var_convert_dict=var_lut, var_units=unit_lut)])
cmip_index['2001-03']

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Demo Notebook

3 participants