[anth-volk]

Fixes #734

Builds on #730 (issue #729), which is merged into main.

Summary

#730 checks requested years with supported_years(country_id, dataset_path), which needs the source file. So create_datasets could download before refusing a year, and ensure_datasets reused year files that already exist without any check. This PR records the certified dataset's supported years in the bundle manifest and checks requests for that dataset against the record, reading no dataset or year file.

The record

• Field: data_releases.<country>.certified_data_artifact.supported_years, a JSON list of integer years sorted ascending without duplicates. That is the shape supported_years returns and the execution receipt's supported_years uses.
• Why on certified_data_artifact: the years belong to one file under one model. That record already names the file (dataset, uri, sha256, build_id), and certification names the model. A top-level field would not say which of the release's datasets it covers. The UK release also routes local areas to microcosm_uk_2024_25_dense, and the US release routes them to populace_us_2024_acs_local.
• Values on the pinned bundle (no pin changes):

Country	Dataset and revision	Recorded years	How
US	populace_us_2024 @ populace-us-2024-spm-20260915 (sha256 6496cc43…)	2024–2035	python scripts/bundle.py record-supported-years --country us: policyengine-us 2.2.1, policyengine-core 3.33.3 and spm-calculator 1.0.0 in a uv run --isolated environment. The in-process supported_years on the locked environment gives the same list.
UK	microcosm_uk_2024_25 @ microcosm-uk-2024-25-national-20261025T091414Z-a6299504 (sha256 b96a8cd6…)	2024–2032	Recorded with python scripts/bundle.py record-supported-years --country uk on the certified file, downloaded with a token that can read policyengine/populace-uk-private, then python scripts/bundle.py generate --include-tros with the same token.

The US record starts at the data year because #730 now refuses US years before it: society-wide runs no longer accept 2022 or 2023 for populace_us_2024, which holds almost no data for them. An earlier version of this PR recorded 2022–2035.

How the record is produced and checked

• Certification (certify_data_release, so bundle.py certify-data and the UK candidate workflow's prepare) now writes the record. It makes the certified default dataset available as the runtime does, reusing a verified copy in ./data or downloading it and checking its sha256. It refuses a dataset without a certified sha256. It then runs dataset_years.supported_years_in_environment, which loads dataset_years.py by path in a throwaway uv run --isolated --no-project --no-config environment with exactly the certified model, the core (--core-version or the bundle pin) and, for the US, the bundle's spm-calculator. The environment's imported versions are checked against those pins. This mirrors the regional input-surface audit's isolated environment. It matters because the UK candidate workflow certifies from an environment that still holds the previous model. Certification now needs uv and PyPI access, and downloads the certified file.
• bundle.py record-supported-years --country <country> [--data-dir DIR] records the years for the current certificate under the bundle's own pins, without re-certifying. That is how the US value above was written.
• bundle.py check (run by the "Verify release inputs and bundle packages" job and by the certification workflow) refuses a malformed record via bundle_validation.validate_supported_years. A missing record passes, because certificates from before it fall back to the computed check. The runtime model CertifiedDataArtifact.supported_years (list[StrictInt]) applies the same check.
• TRACE sidecars: both *.trace.tro.jsonld files bind the manifest's bytes, so both were regenerated with bundle.py generate --include-tros --strict-tros after each record was written, last after the US years were re-recorded. Only the manifest digest and the composition fingerprint change. The UK one was regenerated from the certified UK release manifest bytes, which match the sidecar's pin (1ebdbcc3…), via POLICYENGINE_UK_DATA_MANIFEST_FILE. bundle.py check --published-spm --include-tros --strict-tros passes.
• Adding the record changes the US certificate digest (certificate_sha256(data_releases.us)).

How the year check now runs

• dataset_years.recorded_supported_years(country_id, dataset=None) returns the record when the request is the certified default: no dataset, its name, or its certified URI. The bundle must also serve that name with the certified sha256. Otherwise it returns None.
• create_datasets checks every request the record covers before any materialize_dataset call, so before any download. Requests it doesn't cover keep Refuse dataset years the country model does not project to #730's computed check after the source is local.
• ensure_datasets checks the covered requests before deciding to reuse or create year files, so an existing year file for an unsupported year of the certified dataset is refused. That includes a US year file an earlier release created. No source or year file is read.
• Not covered, as before: other datasets (the US ACS-local and UK dense files, overlays, a caller's own file) and certificates without the record. Their existing year files are still reused without a check.
• allow_unsupported_years=True turns off both checks.

Consumers of the manifest

• The Public API (policyengine_api/constants.py) reads the wheel's manifest.json as a plain dict with .get and takes only named keys. The v2 catalog reads attributes of .py's own CountryReleaseManifest, which ignores unknown fields. No JSON-schema or extra="forbid" validation of the manifest was found, so an unknown field is ignored.
• The simulation service (release_bundle.py, routing_publisher/bundles.py, stage12_bundle/parsing.py) reads named keys from the dict or from get_release_manifest. Its extra="forbid" models (Stage12CountryBundle and others) are built from those selected values, not from the raw manifest, so an unknown field is ignored. Its stage 12 bundle digest covers only the normalized values, so it does not change.

Tests

• New tests/test_recorded_supported_years.py:
  • the shipped manifest's record for every country, and that the US list starts in 2024, the data year;
  • malformed records refused by the model, by validate_supported_years and by bundle.py check;
  • certification writing the record with the exact packages;
  • the isolated-environment command, pins and refusals;
  • record_supported_years and its command.
• tests/test_dataset_years.py:
  • the recorded-years checks before download and on reuse, the fallback to the source, the opt-out and which requests the record covers;
  • Refuse dataset years the country model does not project to #730's real-dataset test now recomputes the years from the certified file and compares them with the record. It is skipped unless POLICYENGINE_UK_DEFAULT_DATASET or POLICYENGINE_US_DEFAULT_DATASET is set. It also refuses a file whose sha256 is not the certified one and an installed model other than the pinned one.
• Existing certification tests (tests/test_certify_data_release.py, tests/test_uk_qualification_pipeline.py) stand in for the new computation, which would download data and install packages.
• Run locally, by file or node ID only:
  • the updated US expectations against the earlier 2022–2035 record: 3 selected, 1 failed, 2 passed, and the US certified-dataset test failed;
  • tests/test_recorded_supported_years.py: 35 passed;
  • tests/test_dataset_years.py: 65 passed, 2 skipped;
  • test_the_recorded_years_equal_the_years_computed_on_the_certified_dataset on the certified files: [us] (sha256 6496cc43…) 1 passed, [uk] (sha256 b96a8cd6…) 1 passed;
  • bundle.py check --published-spm --include-tros --strict-tros passes;
  • ruff format --check . and ruff check --ignore UP038 . pass; mypy --python-version 3.13 src/policyengine: 212 errors.

🤖 Generated with Claude Code