Calculate stored households through /v2/household-calculations on the v2 household functions by anth-volk · Pull Request #4092 · PolicyEngine/policyengine-api

anth-volk · 2026-10-27T20:57:37Z

Summary

The Public API now sends stored households to the Simulation Entrypoint's v2 household functions, records which catalog versions offer SPM settings, and checks both against its own /calculate in staging. Nothing calls the new routes yet: no frontend traffic reaches them.

POST /v2/household-calculations takes country_id, household_id, an optional policy_id, optional axes and add_missing_variables. It needs no account.
- It checks that the country's countries row allows household calculations (422 naming the country otherwise; a country with household calculations only is accepted), that the household and policy exist in that country, through the same services as /v2/households and /v2/policies (404 otherwise), and that the household has a default year (422 otherwise).
- It selects the country's running catalog version and sends the Simulation Entrypoint only IDs: the country, the household's and policy's IDs, the options (axes as given) and that version's PolicyEngine.py version. The Simulation Entrypoint reads the household, the policy's parameter values and the catalog version from these tables itself, through its own SELECT grant, converts the household, and calculates the household's default_year, which it sends the household function as the request's required year: the year of added variables, and of values and axes without a period.
- The answer is 202 with the calculation's ID and status. Callers cannot force a new calculation: an identical one that succeeded in the last 28 days is returned again.
GET /v2/household-calculations/{calculation_id} returns the calculation as it stands: once it succeeded, the calculated household with its warnings, the PolicyEngine.py version and, for US households, spm_config and spm_provenance; once it failed, its error and the household function's error detail.
Simulation service answers. The submission and each answer are checked against the generated simulation client's models, and kept in their own key order, because the household function lists warnings in the household's key order. Axes are checked against the service's limits (equal point counts within a group, at most 10,000 points) before anything is sent.
- 422, passed through with the simulation service's reason: the caller's data it refuses, which it checks before writing anything: a policy changing a parameter the catalog version lacks, a stored policy value it cannot take (such as an object), or a stored household it cannot take (such as a group no person belongs to, which /v2/households accepts). A request field it refuses gets 422 with each field.
- 503: its other refusals (a household, policy or catalog version it cannot find, or a household without a default year, which the Public API checks first, so such an answer means the two services disagree; or no v2 worker for the version), an unreachable service, an answer of another shape (including a 422 whose body is not text or a list of fields), a gateway token that cannot be fetched, or missing configuration.
SPM availability in the catalog. Catalog publication records, for each US catalog version, whether the Simulation Entrypoint's public GET /versions lists a certified SPM capability for its PolicyEngine.py version, with the settings schema and certified defaults; /v2/model-catalog serves it as spm. A version the service does not serve yet, or an unreadable answer, leaves it unrecorded for the next publication run. /versions itself is unchanged.
Route group and logs. /v2/household-calculations belongs to the household route group. Its migration request log records the Supabase read and the v2 workers on the Cloud Run Simulation Entrypoint, not the household group's v1 database and compute flags. The contract registry and the generated migration contract documents include both routes.
Staging checks. The parallel live staging job runs, against each tagged staging candidate: a submit, poll and result cycle; six calculations submitted at once, each accepted within 10 seconds and finished within 300 seconds; a household across 401 axis points, which must finish; and a parity set of three US and three UK households (with axes, reforms stored through /v2/policies, added variables, and a deprecated input that produces a warning), compared value by value with the Public API's /calculate and /calculate-full. The checks cannot see how many Simulation Entrypoint instances served them or how much memory the household functions used. The job's time limit rises from 20 to 30 minutes.

Route groups covered: household (new routes only). Everything else stays on its current path: /calculate, /calculate-full and the stored-household calculation routes still calculate in process. User-visible API changes: the two new routes, and spm in /v2/model-catalog.

The parity household that adds missing variables names its person you, because /calculate-full adds them for the first age period of a person named you (or else the current year) while the v2 path adds them for the household's default year; a test with the date fixed in 2027 checks both use the same year.

Testing

Run locally on the branch head (with the simulation client generated from PolicyEngine/policyengine-sim-api#957 at 3a2bbe02 installed over the locked release):

uv run pytest tests/unit tests/contract: 3717 passed;
against a disposable PostgreSQL 17 container: tests/integration/test_alembic_v2_lifecycle.py and tests/integration/test_v2_catalog_spm_availability.py, 14 passed;
ruff 0.15.12 (format --check and check), mypy, and scripts/run_quality_guards.py pass; scripts/export_migration_contracts.py regenerated the contract documents;
the two new live modules ran against a local server standing in for a staging candidate (stored households and policies, calculations, /calculate and /calculate-full, a 503 on each first submission): 9 passed; a lasting 500 fails at once and a lasting 503 fails after the outage limit. They have not run against staging: they run only in the staging deployment job.

Environment variables and operator steps

Operator step: a SELECT grant. The Simulation Entrypoint's runtime role simulation_runtime needs SELECT on tax_benefit_models, tax_benefit_model_versions, regions, datasets, parameters, parameter_values, policies and households, granted by the owner of these tables. Until then, calculations answer 503. It stays an operator step for now: once that role exists, a later change declares the grant in the v2 privileges manifest (policyengine_api/data/v2/privileges.py), so each deploy grants and checks it. This PR does not add it there, because the manifest's deploy-time reconciliation would fail in an environment where the role does not exist yet.

SIMULATION_ENTRYPOINT_URL
- Provider: GitHub environment secret, set in staging and production.
- Runtime: already in the job-level env: of both Cloud Run deploy jobs, and written to the revision by deploy_cloud_run_candidate.sh when it is set. The new routes always call this URL, whatever SIM_ENTRYPOINT selects (both environments select cloud_run_simulation_entrypoint). Without it, both routes answer 503.
- Catalog publication (new): passed in the publish step's env: in seed-v2-database.yml; that job declares environment: ${{ inputs.deployment_environment }} and is called with secrets: inherit. Publication stops before extraction when it is unset.
GATEWAY_AUTH_ISSUER, GATEWAY_AUTH_AUDIENCE, GATEWAY_AUTH_CLIENT_ID, GATEWAY_AUTH_CLIENT_SECRET_RESOURCE: organization secrets, unchanged. The deploy step passes them to the deploy script, which writes them with GATEWAY_AUTH_REQUIRED=1; the client secret is read from Secret Manager at runtime, as for every existing simulation call.
Live checks: API_BASE_URL and STAGING_API_TEST_PROBE_ID come from the deploy job's outputs in the live job's step env:, as before. STAGING_HOUSEHOLD_PARALLEL_COUNT, STAGING_HOUSEHOLD_SUBMIT_LIMIT_SECONDS, STAGING_HOUSEHOLD_FINISH_LIMIT_SECONDS, STAGING_HOUSEHOLD_CALCULATION_TIMEOUT_SECONDS and STAGING_HOUSEHOLD_TRANSIENT_LIMIT_SECONDS are optional limits with defaults; none is set.

Not verified: that the SELECT grant exists in staging or production; that each environment's SIMULATION_ENTRYPOINT_URL names that environment's Simulation Entrypoint (the existing release check covers this); and that the gateway's Auth0 client holds the permissions the Simulation Entrypoint will require for its household routes once it enforces them.

Waits for

Serve v2 metadata in parts loaded on demand, cacheable per catalog version #4042, the base, and the pull requests below it: Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023, Publish v2 catalog regions from checked-in reference files and the certified data release #4020, Rename the v2 table user_policies to user_policy_associations #4010 and Let saved reforms and households point at a new v2 policy or household #4005.
Add a v2 countries table that matches one country list in code, GET /v2/countries and a startup check #4012 (the countries table) and Check each release calls its own environment's Simulation Entrypoint #4003 (the release check), merged into this branch with plain merge commits. The countries revision is re-parented onto the catalog revisions in this branch only.
Calculate and store households through the v2 household functions policyengine-sim-api#957 (the household calculation routes), merged and deployed to staging first, with the SELECT grant above. The new staging checks call those routes, staging promotion waits on them, and production waits on staging promotion, so merging this earlier stops every release.
A release of policyengine-api-simulation-client that contains those routes (an operator step), then raising this PR's dependency (>=0.20261027.288, the latest published release, which lacks them) and the lock to it. Until then, the unit tests that import the client adapter fail in CI, and the routes would answer 500.
The v2 Alembic revision aeb5f3e8f662 (SPM availability) follows fb1a3e40d3d6 (countries). Migration pull requests in the v2 chain merge one at a time; another open revision on the same parent is moved onto the new head by whichever merges second.

Suppressions added

# noqa: PLC0415 twice in fastapi_routes/dependencies.py: the household calculation service and its simulation client are imported inside the default factory, so the generated client is loaded only on the first calculation request, as the other default factories do.
# type: ignore[import-untyped] for policyengine_observability in the simulation client module: the package has no type information, and the module is in mypy's file list.

CI: the Pull request workflow on 4c515e2 failed three jobs, as on every run since the branch was opened.

Type-check API v2 and Test: the locked policyengine-api-simulation-client 0.20261027.288 lacks the household calculation models, so mypy cannot find them and tests/unit/v2/test_household_calculation_simulation_client.py fails at import. This is the client release this PR waits for.
Cross-database integration tests: test_v2_catalog_installed.py::test_installed_policyengine_catalog_is_complete_and_bounded and the eight test_v2_catalog_metadata_parity.py tests fail with "the installed PolicyEngine.py does not publish catalog format 2's fields". Those tests come from Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023 in the base; the base PRs' runs skipped this job, and this PR changes neither the catalog extraction nor the installed PolicyEngine.py.

Every other job passed, including lint, the quality guards, both Alembic qualifications and the container builds.

🤖 Generated with Claude Code

The Public API listed its countries in several places: the v2 routes' `CountryId` type (`Literal["us", "uk"]`), an unused `SUPPORTED_COUNTRY_IDS` set in the query models, catalog extraction's `SUPPORTED_COUNTRIES` and its required packages and installed models, the set of countries the v1 routes copy into v2, and the household copy command's options. `policyengine_api/supported_countries.py` now holds the one list, `SupportedCountryId`, and `SUPPORTED_COUNTRY_IDS` derived from it; each of those places derives from it. The list lives in its own module, not in the query models, because the query models import the catalog package, which must import the list too. Tests: in a fresh interpreter, adding "zz" to the one list adds it to every derived list (this failed before the change: `CountryId` rejected "zz"); the catalog's per-country settings must cover exactly the listed countries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…reference Adds the v2 `countries` table: one row per country the Public API serves, with `household_enabled` and `society_wide_enabled` saying which calculations it supports (a check requires society-wide countries to be household capable). `users.primary_country` and the `country_id` of households, policies and the four user association tables reference it, with `ON DELETE RESTRICT`, instead of each checking `IN ('us', 'uk')`. The tables that pair v1 records with their v2 copies keep their checks. Revision `fb1a3e40d3d6` was autogenerated. Its one hand-written operation inserts the US and UK rows, both capable of both, between the table and the foreign keys, which otherwise fail on every database whose rows already name either country. This insert is an exception to the rule that revisions contain only generated operations, approved for this revision alone; the revision's comment and the Alembic skill say so. The generated downgrade restores the checks and drops the table with its rows. The privilege manifest gives the countries table no role access yet. Tests (failed first because the `countries` table and `Country` model did not exist): model structure and foreign keys; SQLite persistence (an unlisted country is refused, a listed country in use cannot be deleted, a household-only row lets records name that country); the revision's operations; and the Postgres lifecycle (upgrade keeps existing US and UK rows, unlisted countries are refused, downgrade and re-upgrade, a downgrade is refused while records name another country, and the previous release's table usage still works after the upgrade). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

`GET /v2/countries` lists every row of the countries table, ordered by ID: each country's `id`, `household_enabled` and `society_wide_enabled`. It takes no query parameters (any is refused with a 422), needs no user credential, and answers 503 when the v2 database is not configured or cannot be reached, and a secret-safe 500 for anything else, through the shared v2 error mapping. Requests are logged under the `metadata` route group. The route follows the v2 layout: a `countries` route package, and a service package with a read connector, a session container, a transformation and a read type. The Public API's runtime role gets SELECT on `countries` in the privilege manifest, which the deploy grants after the schema upgrade. The migration contract registry gains the `country_selection_v2` workflow, and the cross-database integration workflow runs the new integration test. Tests (failed first: the route and service tests could not import `policyengine_api.services.v2.countries`, the OpenAPI test did not find `/v2/countries`, the generated contract files were stale, and the integration workflow did not name the new test): route responses, the query refusal, 405 for other methods, 503 and 500 mapping and the OpenAPI document; the service against SQLite; and, against migrated disposable Postgres as the runtime role, the route returning the two rows the revision inserted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

`policyengine_api.asgi` now checks, while the application loads, that the v2 `countries` table holds one row for each country in `policyengine_api/supported_countries.py` and no other. A worker that finds a difference raises `CountryListMismatchError`, naming the listed countries without a row and the rows not listed, so it never serves a request; on Cloud Run the revision then fails its health checks. The check runs on Cloud Run (`K_SERVICE` is set) and wherever a v2 runtime database URL or its secret resource is configured. A local or test startup without v2 configuration opens no connection. A missing table, a missing privilege and unusable v2 configuration also stop startup. A database that cannot be reached is logged, by error type only, and startup continues: the v1 routes do not read the v2 database, and their copies into it already continue without it. The countries service gains `verify_countries_match_code_list`, which reads the row IDs through a new read connector and compares them in a validator. Tests (failed first: `policyengine_api.services.v2.countries.validators` did not exist): the comparison in every direction; the service against SQLite; the startup function's selection, each failure that stops startup, and the logged skip without the error's detail; importing the ASGI module with an unusable v2 runtime URL failing without printing the password; and, against migrated disposable Postgres as the runtime role, the check passing on the revision's rows and failing once an extra row exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Review found that the startup check sorted its failures by exception type in a way that stopped startup for a brief Secret Manager outage (reported as unusable v2 configuration) while only logging rejected credentials (a connection error). The check now stops startup only when the database answers and the answer disagrees with the country list: different countries, a missing table, or no privilege to read it. When it cannot get an answer, because the v2 configuration cannot be used, the secret cannot be resolved, or the database cannot be reached or queried, it logs why and startup continues; the readiness check already reports unusable v2 configuration as not ready and recovers once the secret resolves. The rule `api.startup.no-import-side-effects` now says what deployed startup does: it opens no network connections unless the process runs on Cloud Run or has a v2 runtime database configured. The country list and the startup check join the mypy file list and rule `api.quality.mypy-v2`. The country list's docstring and the Alembic skill now say that a deploy runs a country-adding revision before the new release starts, so the previous release refuses to start afterwards, and list the US-and-UK checks a new country must also widen. Tests (failed first: configuration errors and `InterfaceError` were raised, and importing the ASGI module with an unusable v2 URL exited with an error): configuration and secret failures are logged with their message and startup continues, connection and interface errors are logged by type only, and importing the ASGI module with an unusable v2 URL logs the skipped check without the password. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

In a full test run, the tests that run Alembic commands load Alembic's logging configuration, which disables every logger that already exists, including the startup check's. Its four log tests then captured nothing and failed, though they passed on their own. An autouse fixture now re-enables the check's logger for each of its tests. Tests (failed first: running the Alembic unit tests before the startup check tests failed all four log tests); with the fixture, both files pass together (46 passed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

PATCH /v2/user-policies/{id} now accepts policy_id. Supplied alone, it points the saved policy at that v2 policy and clears legacy_policy_id, which no longer describes the saved content. Supplied with legacy_policy_id, both are stored, and policy_id must be the v2 copy of that v1 policy when one exists, exactly as creation decides. The policy must exist in the row's country. The row keeps its ID, owner, name and creation time, so an edited reform keeps the ID that reports link to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

PATCH /v2/user-households/{id} now accepts household_id, with the same rules as a saved policy's policy_id: supplied alone, it points the saved household at that v2 household and clears legacy_household_id; supplied with legacy_household_id, both are stored, and household_id must be the v2 copy of that v1 household when one exists. The household must exist in the row's country, and the row keeps its ID, owner, name and creation time. Without this, editing a saved household changed its ID. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A household repoint can now fail with a reference conflict, and the household route tests sent only DELETE requests through the failure mapping. Send a repointing PATCH through it and pin each refusal's status: 400 for a country or reference conflict, 404 for a missing household or saved household, and 422 for a missing v1 household. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…mands The policy and user-association qualification commands counted saved policies through the UserPolicy model. Both run before every v2 upgrade, so once the model takes a new table name they would query a table the database does not have yet. The policy qualification counts only while its revision is pending, when the table is still named user_policies, so it names that table. The user-association qualification counts on every run, so it counts the table under whichever name the database has: user_policy_associations once renamed, otherwise user_policies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The saved-policy table now follows the naming of the other user-association tables. Revision fa7fa1035444 was autogenerated from the renamed model. Its one documented post-generation correction replaces the generated create-and-drop pair, which would have deleted every saved policy, with an in-place rename; each constraint named after the table is dropped under its old name and created under the generated name, and the generated changes to the two convention-named indexes and the three foreign keys that refer to the table are kept around the rename. The composite listing indexes keep their ix_user_policies_* names, as the other association tables keep their short index names. The privilege declarations and the head revision constant use the new name. Dual-write's saved-policy copy goes through the model and follows it. Unchanged: the route /v2/user-policies, the user_policy_id columns, legacy_user_policy_mappings and the v1 Cloud SQL table user_policies. Not included: a temporary view named user_policies for the release still running during the deploy. A view is not autogenerated, so it needs a human migration decision; a strict expected-failure test records the gap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- The revision test finds fa7fa1035444 by its ID instead of through the head constant, so a later revision no longer makes it read the wrong file, and it checks the revision's parent through the script directory. - The lifecycle test checks each privilege of the probe role on its own (has_table_privilege with a list is true when any one is held), and checks grants and delete cascades after the downgrade as well as after the upgrade. - The test for the missing user_policies view now stores a saved policy and upgrades to head before reading it under the old name, and only a missing relation counts as the expected failure, so an empty database or a failure while setting up no longer satisfies it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The merge table and the rule for Stage 10 mirror rows name the table by its current name. The rollback and schema compatibility sections no longer say head lets earlier releases run: until a view under the old name exists, a release from before the rename fails every statement on saved policies, from the schema upgrade until the new release takes traffic, and after a rollback until the rename is downgraded. They also say that downgrading the rename alone keeps every row, and that the Stage 12A compatibility test now maps the renamed table and no longer proves the parent release runs against head. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The reconciliation read only ordinary and partitioned tables when it checked which declared relations exist, so a view declared in the privilege manifest counted as missing and blocked every grant. It now reads views too. GRANT ... ON TABLE and has_table_privilege already work on views, and the granted-privilege query already read them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Revision fa7fa1035444 now creates a view named user_policies with every column of user_policy_associations after the rename, and drops it before the downgrade renames the table back. These two statements are the only hand-written SQL in the v2 chain: a maintainer approved them on 2026-10-22 as an exception to the generated-only rule, and a later revision drops the view. The revision comments and the Alembic skill say so, and the revision test allows exactly these two statements. A single-table view is automatically updatable, so the Public API release still serving during the deploy, or after a rollback to it, keeps reading and writing saved policies through it. The privilege manifest declares the view with its table's privileges, so the reconciliation grants the runtime role SELECT, INSERT, UPDATE and DELETE on it. Autogeneration does not reflect views, so alembic check and the head schema validation stay clean. Tests: the lifecycle test checks the view's columns after the upgrade and its absence after the downgrade, runs the parent revision's inserts, reads, updates and deletes through it, and reads a saved policy stored before the rename under the old name. A new integration test runs the saved-policy routes' services, the sign-in merge, report creation and the Stage 10 saved-policy copy as the runtime role with every statement naming user_policies. The Stage 12A document and the changelog describe the view. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…kflow The workflow runs an explicit list of integration modules, and test_v2_saved_policy_view.py was not in it. It now runs in the step for signed-in identity provisioning and user associations, and rule api.v2.runtime-privileges-tested-as-runtime-role names it, since it runs the saved-policy services as policyengine_v2_runtime. A unit test fails when an integration module that uses the managed v2 roles is missing from the workflow, or one that runs services as a managed role is missing from the rule's scope. No environment variable is added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The reconciliation read tables and views as one set of names, so a view satisfied a table entry of the same name and a table satisfied a temporary-view entry: before the rename, the user_policies table satisfied the user_policies view's entry. It now reads tables and views separately. A declared table must exist as a table and a TEMPORARY_VIEW_TABLES entry as a view; anything else is reported under "Declared tables missing" or "Declared views missing", nothing is granted, and verify reports every privilege declared on it as missing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Each file lists every region of one country with its code, label, type, parent, US state, the value that selects its households, and the v1 identifier it replaces. The UK file lists 1,016 regions: the United Kingdom, its four nations (Northern Ireland as country/northern_ireland, replacing country/ni), 650 constituencies and 361 local authorities by ONS code (constituency/E14001063 replaces constituency/Aldershot), each under its nation. The 361 include Ards and North Down (N09000011), which the Public API's own list now carries. The US file lists the United States, 51 states, 436 congressional districts with unchanged codes, and 333 places. The constituency and local-authority rows come from the Public API's own lookup files; the US states, districts and places, and every filter value, match the region registry of PolicyEngine.py 6.7.5. A loader validates each file: unique codes and v1 identifiers, codes that start with their type, and parents that lead to the national region. Nothing reads the files yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ta release Catalog extraction no longer reads PolicyEngine.py's region registry. It takes every region from the checked-in reference file of its country and gives it a dataset and filter from a scoping rule per country and region type: - US national, and UK national: the release's national dataset, no filter. - US states and congressional districts: the release's state and district dataset, a row filter on state_fips or congressional_district_geoid. - UK nations: the national dataset, a row filter on the stored region. - UK constituencies and local authorities: the national dataset, a weight_replacement filter whose field names the release's weight matrix (parliamentary_constituency_weights, local_authority_weights) and whose value is the region's ONS code. The matrix must come from the same release as the dataset, since weights attach to households by position. Extraction fails if the national region would not use the reviewed dataset, a dataset role names no certified dataset or a per-region template, a filtered region has no filter value or a malformed list, or a runnable region sits under one that is left out. Region types without a rule (US places) are left out and counted in the publication evidence. The installed 6.7.5 catalog now publishes 1,503 regions: 488 US and 1,015 UK. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… missing Before it publishes a catalog version that is not stored yet, the publication command checks each region against the certified artifact its filter reads, and stops with a clear error if: - a row filter's column (state_fips, congressional_district_geoid, region) is not a household column of the region's dataset; or - a weight matrix's row count differs from its row lookup, or the lookup has no row for a constituency's or local authority's ONS code. Compute finds a region's matrix row in the same lookup files (constituencies_2024.csv, local_authorities_2021.csv). The artifacts are read from Hugging Face at their pinned revisions with ranged requests, so only HDF5 metadata and the column-name arrays are downloaded: about a minute for the 9.8 GB US file. Pandas fixed and table formats, one frame per year, and policyengine-core files are understood. A read failure names only the artifact and the error type. Versions already stored are not read again, since publication only compares them. The seed workflow's publication step receives the organization secret PE_UK_PRIVATE_HF_READ_TOKEN as HUGGING_FACE_TOKEN, because the UK data repository is private. The release workflow passes it through its existing `secrets: inherit`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A catalog version is identified by its country and PolicyEngine.py version and never changes once published, but its regions now come from the Public API's reference files. Republishing a stored version compared every table, regions included, so a reference file change under an unchanged PolicyEngine.py version (6.7.5 today, whose stored version has the 826 regions of PolicyEngine.py's registry) made the seed job fail. For a stored version, publication now compares every table except regions exactly as before, keeps the stored region rows unchanged, and logs kept_published_catalog_regions with the country, the version and the counts of staged region rows the stored version lacks and of stored rows not staged. A version not yet stored still publishes, and is checked against, the reference-file regions. Region file changes therefore take effect in the next catalog version that is published. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The artifact test now checks that the UK release fixture's artifacts name the placeholder repository example-org/example-uk-data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Catalog publication's data reader imports both, which the project had only through PolicyEngine.py and PolicyEngine Core. They are now direct dependencies, constrained to keep the locked versions (h5py 3.16.0, huggingface-hub 1.12.2). uv.lock, refreshed with `uv lock` (uv 0.13.4, as the release workflow uses), changes only the project's dependency metadata. A test requires every third-party package the reader imports to be a direct dependency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The seed command configures no logging handler, so Python's last-resort handler printed only the warning's bare event name and dropped the structured attributes. The message now carries the country, the PolicyEngine.py version and both region-row counts; the attributes stay for structured handlers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Republishing a stored catalog version with different content failed with "persisted <country> catalog differs from PolicyEngine.py", even when the difference came from the Public API's own selections, such as a dataset. The error now names the country, the version and the first table that differs (model version, variables, parameter nodes, parameters, parameter values, datasets or regions), and still no row data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Verification reduced every read failure to the artifact's name and the error type, so a structure error such as "household group holds no data frame" reached the operator only as "CertifiedDataReadError". That error, which names only HDF5 entries and counts, now keeps its message; any other failure is still reduced to the name and type, since its text may hold a location. The error class moves to the catalog's errors module so verification can recognize it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The region data check reads stored catalog versions before publication checks the database revision, so on a database without the v2 schema the command failed with "catalog initialization failed unexpectedly" instead of the specific revision error. unpublished_countries now requires the expected Alembic revision first, as publication does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The UK credential document said the catalog publication step reads only UK artifacts, but the same authenticated file system also reads the public US dataset. The weight-matrix lookup comment and the catalog document now say that the certified data release lists no lookup file, so the check uses the Public API's own copies, and that on 2026-10-22 both copies were byte-identical to the files compute downloads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

When a stored catalog version kept its region rows, the publication evidence still counted the staged regions and reported the staged fallbacks and omissions, so the record kept before a release described regions that were never published. The evidence now counts the stored region rows for such a country, leaves its staged fallback and omitted summaries out, and lists it under kept_regions with the stored and staged region counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… set The parity set gains a US household under a stored reform that also adds missing variables (compared with /calculate-full) and a UK household under a stored reform, so the policy conversion and the added variables are checked against API v1 too. The memory check's description now says what it asks for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The generated client's models wrote the household with its groups before its people and each group's members after its values, in the request and in the answer. The household function lists warnings and calculated values in the household's key order, as API v1 does, so the order could differ from API v1's. The submission and the answers are now checked against the generated models but sent and returned as built and decoded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The default service factory built the simulation client before any check, so with the simulation service unconfigured a country without household calculations, or a household that does not exist, got 503 instead of 422 or 404. The client is now built only when a submission or read is sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

An axis without a period varied the country model's default period, the current year, rather than the year the household's values name. Such an axis is now sent with the household's default year as its period. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

/v2/household-calculations requests were logged with the route group "unknown"; they now belong to the household group. The contract registry records both routes as a typed v2 workflow, and the generated migration contract documents include them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Catalog publication now stops before extraction when SIMULATION_ENTRYPOINT_URL is unset, instead of after the catalog is written. When the simulation service's /versions answer cannot be read, publication logs a warning and leaves SPM availability unrecorded for the next run, so a slow or unreachable service no longer stops a release. The cache note and the stage 9 guide say that an instance keeps an earlier answer until it restarts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The simulation service answers 503 with Retry-After while its database or version manifest cannot be read. The live staging checks retried such answers when polling but not when submitting, so one short outage failed the job that staging promotion waits on. Submissions are now sent again, after the Retry-After wait, until the calculation's deadline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Requests to /v2/household-calculations were logged with the household route group's v1 database flags and its configured compute backend. They now log a Supabase read for a submission, no database read for a poll, no write, and the Cloud Run Simulation Entrypoint's v2 workers as the compute, whatever those flags select. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The household client held its HTTP client inside the generated AuthenticatedClient with an empty token but never called a generated route; it now keeps the HTTP client itself. Its description says the household's key order is the stored household's as the v2 database returns it. The SPM availability description and changelog entry say a version gets its record on the next publication run of its own PolicyEngine.py version, and that only an unreadable versions answer never fails publication. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…any order The live household checks sent a request again after any 5xx answer until the calculation's 600-second deadline, so a lasting failure kept each check waiting until the job's time limit and lost the failing answer. They now retry 502, 503 and 504 for at most 90 seconds in a row and fail at once, with the answer's body, on anything else. The parity set compares warnings regardless of order, since the v2 database orders stored keys itself, and one US household now has a deprecated input so that both calculations warn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

An invalid SIM_ENTRYPOINT or SIM_COMPUTE_HOUSEHOLD, which household calculations never read, reduced their migration log record to a flag error. The record is now built from what the route uses, checking only the route implementation. The guide and the live check helper say which requests are sent again after a 502, 503 or 504, and that failures the Public API answers with 503 take about 90 seconds to fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The helper that reads a simulation service answer took a generated model class as Any, with an ANN401 suppression; it now takes the model's from_dict reader, typed by the model it returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The simulation service now reads the stored household, the policy's parameter values and the catalog version from the Public API's tables itself. A household calculation's submission therefore carries only the country, the household's and policy's IDs, the options and the running catalog version's PolicyEngine.py version. The Public API still checks the country allows household calculations and that the household and policy exist in it, and fills an axis's missing period with the household's default year. The conversion of stored households and policies, which moved into the simulation service, is removed here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…eriod The household function now takes the run's year, the household's default year, from the simulation service and uses it for an axis without a period, so the Public API no longer fills that period itself. The stage 11 guide says the household's default year is also the year of the variables add_missing_variables adds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The simulation service now refuses the caller's data with 422 and its reason as text: a policy changing a parameter the catalog version lacks, or a stored policy value or household the household functions cannot take, such as a group no person belongs to. The Public API read only a list of refused fields, so such a refusal became 503. It now reads the service's refusal model and answers 422 with the reason, or with each refused field. The service's other refusals (400), for references the Public API has already checked, stay 503. The route's declared 422 answer, the adapter's description and the stage 11 guide say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s variables The staging parity household that adds missing variables named its person `adult`, so API v1's /calculate-full added them for the current calendar year while the v2 path adds them for the household's default year, and the two would differ from 2027. It now names its person `you`, whose age gives /calculate-full the household's year, and a test with the date fixed in 2027 checks both sides use the same year. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A test now sends an axis with an explicit period and one with a null period and checks both reach the simulation service unchanged, as they must now that the household function fills a missing period itself. The check that a household has a default year returns nothing, so it is named require_default_year. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The generated refusal model keeps a 422's detail as it came when it is neither text nor a list of field errors, such as null, an object, a number, a list of strings or an entry without a type, and building the reason from it raised, so the route answered 500. Such a refusal now means the simulation service's answer could not be read, and the route answers 503 as documented. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ault year The declared 503 answer now lists what is left for it (an unavailable dependency, an unreadable answer, a refused reference the Public API has already checked, or no v2 worker), and the stage 11 guide says a refused request field gets 422 with each field. The parity year test checks the default year the live check stores the household with, not a constant, and the axis-period tests are one parametrized test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

anth-volk and others added 30 commits October 22, 2026 16:59

Pin the UK catalog fixture's placeholder repository in the artifact test

5670b18

The artifact test now checks that the UK release fixture's artifacts name the placeholder repository example-org/example-uk-data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Name the weight matrix's dimension count instead of suppressing the lint

a9160d9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

anth-volk and others added 12 commits October 27, 2026 22:27

anth-volk temporarily deployed to staging October 27, 2026 20:57 — with GitHub Actions Inactive

anth-volk mentioned this pull request Oct 27, 2026

Calculate and store households through the v2 household functions PolicyEngine/policyengine-sim-api#957

Draft

anth-volk temporarily deployed to staging October 28, 2026 12:52 — with GitHub Actions Inactive

anth-volk temporarily deployed to staging October 28, 2026 13:10 — with GitHub Actions Inactive

anth-volk and others added 3 commits October 28, 2026 17:49

anth-volk temporarily deployed to staging October 28, 2026 13:53 — with GitHub Actions Inactive

anth-volk and others added 2 commits October 29, 2026 02:22

anth-volk temporarily deployed to staging October 28, 2026 22:25 — with GitHub Actions Inactive

This was referenced Oct 29, 2026

Give each live staging test its own pytest worker in the release run #4127

Merged

Serve household reports at /v2/reports #4132

Draft

anth-volk force-pushed the v2plan/1f-116-catalog-routes-on-demand branch 3 times, most recently from 7187955 to 0f1dae2 Compare November 2, 2026 17:52

Navigation Menu

Repository navigation

Calculate stored households through /v2/household-calculations on the v2 household functions - #4092

Calculate stored households through /v2/household-calculations on the v2 household functions#4092
anth-volk wants to merge 120 commits into
v2plan/1f-116-catalog-routes-on-demandfrom
v2plan/2b-204-household-calculations

anth-volk commented Oct 27, 2026 •

edited

Loading

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

anth-volk commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Environment variables and operator steps

Waits for

Suppressions added

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

anth-volk commented Oct 27, 2026 •

edited

Loading