Repository navigation
Conversation
policyengine.py 6.13.0 made create_datasets and ensure_datasets raise UnsupportedDatasetYearError for a year the source dataset doesn't support, which for the US means 2022 and 2023. The Public API still offers those years for society-wide runs, so _load_dataset now passes allow_unsupported_years=True until the Public API refuses them. The precompute calls to ensure_datasets stay strict. Fixes #1012 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1012
Summary
Moves the simulation executor from policyengine.py 6.12.0 to 6.14.0, and keeps the simulation service accepting US 2022 and 2023.
Since 6.13.0 (PolicyEngine/policyengine.py#730),
ensure_datasetsraisesUnsupportedDatasetYearErrorwhen it has to create a year file for a year the installed country model doesn't support the source dataset for. For the US, that includes 2022 and 2023, which come before the 2024 data year. The Public API still offers those years for society-wide runs, so without an opt-out those runs would fail partway through. 6.12.0'sensure_datasetshas noallow_unsupported_yearsargument, so the opt-out and the version change ship together.The opt-out is temporary. Once the Public API stops offering US 2022 and 2023 for society-wide runs and refuses them with a 400, this repository removes it.
This replaces #1011 (6.14.0), #1002 (6.13.0) and #990 (6.12.1) for the version change. They are left open.
Changes
projects/policyengine-simulation-executor/pyproject.toml:policyengine[models]==6.14.0independenciesand in themodal-simulation-imagegroup.projects/policyengine-simulation-executor/uv.lock: regenerated withuv lock(uv 0.13.6, the release in.uv-version). Only thepolicyengineentry changes. Both files are identical to chore(deps): update policyengine to 6.14.0 #1011's. The bundled versions stay the same: policyengine-core 3.33.3, policyengine-us 2.2.1, policyengine-uk 2.151.0, spm-calculator 1.0.0, and the same certified data releases (USpopulace-us-2024-spm-20260915, UKmicrocosm-uk-2024-25-national-20261025T091414Z-a6299504).simulation_runtime.py:_load_datasetpassesallow_unsupported_years=Truetoensure_datasets, with a comment saying it is temporary and when it is removed.tests/test_simulation_output_dataset_selection.py: a new test, and four existing tests that compare the loader's exact arguments now expect the new one.changelog.d/1012.changed.md.Every
ensure_datasetsandcreate_datasetscallsimulation_runtime._load_datasetprecompute_datasets.build_dataset_implPRECOMPUTE_YEARS: 2025, 2026, 2027precompute_datasetsper-state build (source year file)stage12_cache/precompute_datasets.build_dataset_implPRECOMPUTE_YEARS: 2025, 2026, 2027stage12_cache/precompute_datasetsper-state build (source year file)tests/native_spm_support.py_load_datasetis called from the main runtime (twice insimulation_runtime.py), the Stage 12 society-wide worker (stage12_runtime/simulation.py) and the baseline precompute (precompute_baselines._prepare_cohort_baseline). The baseline precompute therefore passes the flag too, but that changes nothing there: it loads onlyPRECOMPUTE_YEARS, from the year file the strict dataset precompute step built and uploaded first, andensure_datasetsloads an existing, current year file without the check.The repository makes no direct call to
create_datasets. Two tests replace it with a function that fails if called, to show thatensure_datasetsreuses a prepared state file.Other changes from policyengine.py 6.12.0 to 6.14.0
xandykeys from UK constituency and local-authority results and thexandycolumns from the packaged UK area lists. The executor reads none of them. Stop sending x and y on UK area results #919 and Remove x and y from the contract's UK area records #924 already removed the keys from this service's results and contract, and Accept UK area records with or without x and y #985 lets stored records that still carry them validate._uk_constituency_lookupreads onlycodeandnamefrom the constituency lookup CSV, and local-authority names come from the executor's own packaged files. The remainingxandymentions are contract, gateway and integration tests that check that stored records carrying those keys still validate.supported_yearsfunction. The executor doesn't call the candidate code, and the receipt's UK rule is unchanged.Environment variables and secrets
None added or read. The change passes a constant argument. The policyengine.py source changes from 6.12.0 to 6.14.0 add no environment variable reads and no dependencies.
Checks run locally
From
projects/policyengine-simulation-executor, withuv run --frozen --extra test pytest, each selected by node ID:tests/test_simulation_output_dataset_selection.py::test_load_dataset_lets_a_request_load_a_us_year_before_the_data_year(new;[2022]and[2023]): passed. Both fail withUnsupportedDatasetYearErrorwhen the opt-out is removed.tests/test_simulation_output_dataset_selection.py::test_load_dataset_delegates_default_selection_to_country_loader_with_receipt([us],[uk]): passed.tests/test_simulation_output_dataset_selection.py::test_load_dataset_passes_region_dataset_name_to_managed_loader: passed.tests/test_simulation_output_dataset_selection.py::test_load_dataset_maps_region_bundle_uri_to_managed_name([True],[False]): passed.tests/test_simulation_output_dataset_selection.py::test_load_dataset_reads_a_nondefault_dataset_from_the_folder_it_is_given: passed.Also passed: Ruff 0.15.22
format --checkandcheckon the changed Python files,python3 scripts/run_quality_guards.py, Pyright for the executor (python3 -m scripts.rules pyright projects/policyengine-simulation-executor),uv lock --check, and.github/scripts/check-changelog-fragment.sh. The rest of the test suite runs in CI.🤖 Generated with Claude Code