Calculate stored households through /v2/household-calculations on the v2 household functions

.github/scripts/check_simulation_entrypoint_environment.sh

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+
+# Check that a deployment environment calls its own Simulation Entrypoint:
+# staging must call the staging service and production the production
+# service. The entrypoint names the Cloud Run revision that served a request
+# in its x-policyengine-simulation-revision response header, and a Cloud Run
+# revision name is its service name followed by a five-digit number and a
+# three-character suffix.
+#
+# The old gateway (SIM_ENTRYPOINT=old_gateway_direct) is one shared Modal app
+# that reports no environment, so it is not checked.
+#
+# Required env vars: DEPLOYMENT_ENVIRONMENT (staging or production),
+# SIM_ENTRYPOINT, and the URL variable that SIM_ENTRYPOINT selects.
+# Exit codes: 0 when the environments match, 1 when they do not or the
+# response names no revision, 2 for an unknown DEPLOYMENT_ENVIRONMENT, and
+# curl's own exit code when the request fails.
+
+set -euo pipefail
+
+source .github/scripts/simulation_entrypoint_env.sh
+
+deployment_environment="${DEPLOYMENT_ENVIRONMENT:-}"
+case "${deployment_environment}" in
+  staging)
+    expected_service="policyengine-simulation-entry-staging"
+    ;;
+  production)
+    expected_service="policyengine-simulation-entry"
+    ;;
+  *)
+    printf 'DEPLOYMENT_ENVIRONMENT must be staging or production, not %q\n' \
+      "${deployment_environment}" >&2
+    exit 2
+    ;;
+esac
+
+if [[ "${SIM_ENTRYPOINT:-}" == "old_gateway_direct" ]]; then
+  printf 'SIM_ENTRYPOINT=old_gateway_direct: the shared old gateway reports no environment, so there is nothing to check\n'
+  exit 0
+fi
+
+simulation_url="$(simulation_entrypoint_selected_url "${SIM_ENTRYPOINT:-}")"
+simulation_url="${simulation_url%/}"
+
+health_response="$(
+  curl --fail --silent --show-error --include \
+    --connect-timeout 10 --max-time 60 \
+    "${simulation_url}/health"
+)"
+
+# With --include, curl prints every response's headers; the last revision
+# header belongs to the final response.
+served_revision="$(
+  awk '
+    tolower($1) == "x-policyengine-simulation-revision:" {
+      gsub("\r", "", $2)
+      print $2
+    }
+  ' <<<"${health_response}" |
+    tail -n 1
+)"
+
+if [[ -z "${served_revision}" ]]; then
+  printf '%s/health did not name a Simulation Entrypoint revision; the %s deployment must call the stable URL of %s\n' \
+    "${simulation_url}" "${deployment_environment}" "${expected_service}" >&2
+  exit 1
+fi
+
+if ! [[ "${served_revision}" =~ ^(.+)-[0-9]{5}-[a-z0-9]{3}$ ]]; then
+  printf '%s, served by %s, is not a Cloud Run revision name\n' \
+    "${served_revision}" "${simulation_url}" >&2
+  exit 1
+fi
+served_service="${BASH_REMATCH[1]}"
+
+if [[ "${served_service}" != "${expected_service}" ]]; then
+  printf 'The %s deployment calls %s, which revision %s of %s serves; it must call %s\n' \
+    "${deployment_environment}" "${simulation_url}" "${served_revision}" \
+    "${served_service}" "${expected_service}" >&2
+  exit 1
+fi
+
+printf 'The %s deployment calls %s, served by revision %s of %s\n' \
+  "${deployment_environment}" "${simulation_url}" "${served_revision}" \
+  "${served_service}"

.github/scripts/run_live_staging_tests.sh

@@ -8,4 +8,6 @@ python -m pytest \
   tests/integration/test_live_budget_window_cache.py \
   tests/integration/test_live_calculate.py \
   tests/integration/test_live_economy.py \
+  tests/integration/test_live_v2_household_calculations.py \
+  tests/integration/test_live_v2_household_parity.py \
   -v

.github/workflows/push.yml

@@ -73,11 +73,14 @@ jobs:
       && (github.event.head_commit.message == 'Update PolicyEngine API')
     environment: staging
     env:
+      DEPLOYMENT_ENVIRONMENT: staging
       SIM_ENTRYPOINT: ${{ vars.SIM_ENTRYPOINT }}
       SIMULATION_ENTRYPOINT_URL: ${{ secrets.SIMULATION_ENTRYPOINT_URL }}
     steps:
       - name: Checkout repo
         uses: actions/checkout@v4
+      - name: Check the simulation service belongs to staging
+        run: bash .github/scripts/check_simulation_entrypoint_environment.sh
       - name: Install jq
         run: sudo apt-get install -y jq
       - name: Set up uv for installed bundle validation
@@ -339,7 +342,7 @@ jobs:
   parallel-live-simulation-tests-staging-cloud-run:
     name: Run live simulations against the staging candidate
     runs-on: ubuntu-latest
-    timeout-minutes: 20
+    timeout-minutes: 30
     needs: deploy-cloud-run-staging
     if: |
       (github.repository == 'PolicyEngine/policyengine-api')
@@ -438,11 +441,14 @@ jobs:
       && (github.event.head_commit.message == 'Update PolicyEngine API')
     environment: production
     env:
+      DEPLOYMENT_ENVIRONMENT: production
       SIM_ENTRYPOINT: ${{ vars.SIM_ENTRYPOINT }}
       SIMULATION_ENTRYPOINT_URL: ${{ secrets.SIMULATION_ENTRYPOINT_URL }}
     steps:
       - name: Checkout repo
         uses: actions/checkout@v4
+      - name: Check the simulation service belongs to production
+        run: bash .github/scripts/check_simulation_entrypoint_environment.sh
       - name: Install jq
         run: sudo apt-get install -y jq
       - name: Set up uv for installed bundle validation

.github/workflows/seed-v2-database.yml

@@ -66,3 +66,9 @@ jobs:
         run: uv run python scripts/publish_v2_metadata_catalog.py
         env:
           V2_DATA_WRITE_DATABASE_URL: ${{ secrets.V2_DATA_WRITE_DATABASE_URL }}
+          # Reads the certified UK data release, whose repository is private,
+          # to check each new catalog version's region filters.
+          HUGGING_FACE_TOKEN: ${{ secrets.PE_UK_PRIVATE_HF_READ_TOKEN }}
+          # Each version's SPM availability comes from this environment's
+          # simulation service's public versions answer.
+          SIMULATION_ENTRYPOINT_URL: ${{ secrets.SIMULATION_ENTRYPOINT_URL }}

.github/workflows/v2-integration-check.yml

@@ -73,16 +73,16 @@ jobs:
         run: uv run python scripts/reconcile_v2_privileges.py apply --create-missing-roles
       - name: Verify the managed v2 roles hold their declared privileges
         run: uv run python scripts/reconcile_v2_privileges.py verify
-      - name: Verify the installed PolicyEngine.py catalog interface
-        run: uv run coverage run --branch -m pytest -q tests/integration/test_v2_catalog_installed.py
+      - name: Verify the installed PolicyEngine.py catalog interface and its parity with v1 metadata
+        run: uv run coverage run --branch -m pytest -q tests/integration/test_v2_catalog_installed.py tests/integration/test_v2_catalog_metadata_parity.py
         env:
           RUN_V2_CATALOG_COMPATIBILITY: "1"
-      - name: Test v2 metadata publication and resource routes
-        run: uv run coverage run -a --branch -m pytest -q tests/integration/test_v2_catalog_publication.py tests/integration/test_v2_metadata_routes.py
+      - name: Test v2 metadata publication, resource routes and the countries list
+        run: uv run coverage run -a --branch -m pytest -q tests/integration/test_v2_catalog_publication.py tests/integration/test_v2_catalog_spm_availability.py tests/integration/test_v2_metadata_routes.py tests/integration/test_v2_country_routes.py
       - name: Test v2 policy persistence and immediate v1 mirroring
         run: uv run coverage run -a --branch -m pytest -q tests/integration/test_v2_policy_persistence.py tests/integration/test_v1_policy_dual_write.py tests/integration/test_mysql_policy_dual_write.py tests/integration/test_v2_user_policy_mirroring.py tests/integration/test_v1_user_policy_dual_write.py
       - name: Test signed-in identity provisioning and user associations
-        run: uv run coverage run -a --branch -m pytest -q tests/integration/test_v2_user_identity_persistence.py
+        run: uv run coverage run -a --branch -m pytest -q tests/integration/test_v2_user_identity_persistence.py tests/integration/test_v2_saved_policy_view.py
       - name: Test comparison-run persistence and privilege reconciliation as the managed roles
         run: uv run coverage run -a --branch -m pytest -q tests/integration/test_v2_stage12_comparison_persistence.py tests/integration/test_v2_privilege_reconciliation.py
       - name: Test v2 household persistence and immediate v1 copying

.gitignore

@@ -23,3 +23,8 @@ gha-creds-*.json
 ## vscode settings
 /.vscode
 *.db.init.lock
+
+# coverage.py data and reports written by `make test`
+.coverage
+.coverage.*
+coverage.xml

changelog.d/catalog-spm-availability.added.md

@@ -0,0 +1 @@
+Each published catalog version now records whether it offers SPM settings, with their schema and certified defaults, from the simulation service's public versions answer; the catalog publication step reads `SIMULATION_ENTRYPOINT_URL`, and a version published earlier gets it on the next publication run of its PolicyEngine.py version.

changelog.d/model-catalog-spm-availability.added.md

@@ -0,0 +1 @@
+`GET /v2/model-catalog` now returns `spm`: whether the catalog version offers SPM settings for a household, with their JSON schema and certified defaults when it does.

changelog.d/simulation-entrypoint-environment-check.added.md

@@ -0,0 +1 @@
+Each release now checks, in its staging and production model-version alignment jobs, that the environment's simulation URL is served by that environment's own Simulation Entrypoint service, and fails otherwise; staging must call the staging simulation service, and production the production one.

changelog.d/v2-countries-route.added.md

@@ -0,0 +1 @@
+Added `GET /v2/countries`, which lists every row of the countries table: each served country's ID and whether it supports household and society-wide calculations.

changelog.d/v2-countries-startup-check.added.md

@@ -0,0 +1 @@
+The Public API refuses to start when the v2 `countries` table and its country list in code name different countries, or when the table or the privilege to read it is missing. The check runs on Cloud Run and wherever a v2 runtime database is configured; when it cannot query the table, it logs why and startup continues.

changelog.d/v2-countries-table.added.md

@@ -0,0 +1 @@
+Added a v2 `countries` table that records, for each country the Public API serves, whether it supports household and society-wide calculations. Its revision lists the US and the UK, both with both. Users' primary country and the country of every v2 household, policy and user association now reference this table instead of a fixed US-and-UK check.

changelog.d/v2-household-calculations.added.md

@@ -0,0 +1 @@
+`POST /v2/household-calculations` sends a stored household, under a stored policy or current law, with optional axes, to the simulation service for calculation, and `GET /v2/household-calculations/{calculation_id}` returns its status and, once it succeeded, the calculated household with its warnings and, in the US, the SPM selection applied and its receipt. Neither needs an account; a country whose `countries` row does not allow household calculations is refused with 422.

changelog.d/v2plan-1f-115-catalog-metadata-fields.added.md

@@ -0,0 +1 @@
+Catalog format 2 stores the metadata the app's v3 workspace reads, taken only from PolicyEngine.py's public records: each model version's entities and basic inputs; each variable's input and hidden-input flags, module, the parameter path its adds or subtracts names, its enum values' labels, its unit and its definition period; each parameter's economy and household flags. Bracket schedules are stored as parameter nodes with their stored label. A generated revision adds the columns as nullable, so catalog versions published earlier keep them empty and are compared without them when republished. An opt-in parity test, run by the v2 integration check, compares every new field for the US and the UK with /{country}/metadata. Each country's default dataset now comes from the installed PolicyEngine.py release's manifest; the Public API no longer fixes its own (populace_us_2024 and microcosm_uk_2024_25), and publication stops if the release names no default or does not certify it.

changelog.d/v2plan-1f-116-catalog-routes-on-demand.added.md

@@ -0,0 +1,5 @@
+Every successful v2 metadata response carries a strong ETag (a hash of its body) and a public Cache-Control lifetime: a day when the request names its catalog with policyengine_version, five minutes for the running version, and never past the UTC midnight after the request for current parameter values (current=true). A request whose If-None-Match names the ETag gets 304 with no body. These responses and their 304s carry Vary: Origin, Accept-Encoding, each value once, and browser scripts can read the ETag.
+GET /v2/model-catalog returns only the small parts of one catalog version (model, model version, entities, basic inputs, regions, logical input datasets and economy choices, which are empty lists for a catalog without regions) and names the selected PolicyEngine.py version, so a client can request every other part for that exact version.
+GET /v2/model-catalog/variables returns every variable of a catalog version in one response, with the catalog format 2 fields (input and hidden-input flags, module, the parameter path an adds or subtracts list comes from, enum labels, unit and definition period), and the current-law values by date of each parameter such a list comes from. GET /v2/variables pages carry the same variable fields.
+GET /v2/model-catalog/parameters fetches up to 100 parameters of a catalog version by name in one request, each with its economy and household flags and its current-law values by date, plus the stored labels of their ancestor nodes for breadcrumbs and the requested names the version lacks. /v2/parameters pages and details carry the economy and household flags.
+Each API process keeps the serialized /v2/model-catalog and /v2/model-catalog/variables bodies and their ETags once built (up to eight route, country and version combinations), so only the first successful request per process reads them from the database; requests arriving during that read receive its outcome.

changelog.d/v2plan-1f-116-catalog-routes-on-demand.changed.md

@@ -0,0 +1 @@
+Every v2 metadata route reads its query through the shared query models, so unknown or repeated parameters, a missing or unsupported country_id and a malformed policyengine_version return 422 before any read (unsupported countries and malformed versions returned 400 before, and unknown parameters were ignored). /v2/parameters accepts repeated name keys and /v2/parameter-values repeated parameter_name keys to page only those parameters.

changelog.d/v2plan-1f-116-catalog-routes-on-demand.fixed.md

@@ -0,0 +1 @@
+v2 metadata reads run in autocommit sessions on the shared pool, and each process keeps the catalog versions it has selected. A request that makes one read for a version the process has already selected, such as a page of parameters or one parameter's values, now sends two statements to the database (the pool's liveness check and the read) instead of five; each further read in a request adds one statement.

changelog.d/v2plan-1f-121-repoint-saved-items.added.md

@@ -0,0 +1 @@
+PATCH /v2/user-policies/{id} accepts policy_id and PATCH /v2/user-households/{id} accepts household_id, so a saved policy or household can point at another v2 record in its country while keeping its ID, name and creation time. Supplied alone, the v2 reference clears the v1 reference (legacy_policy_id or legacy_household_id); supplied with it, both are stored, and the v2 reference must be the v2 copy of that v1 record when one exists. Unknown, other-country and conflicting records are refused as on creation, and saved policies maintained by the v1 saved-policy mirror still return 409.

changelog.d/v2plan-1f-122-rename-user-policies.changed.md

@@ -0,0 +1 @@
+The v2 table user_policies is renamed to user_policy_associations, matching the other user-association tables; its constraints and convention-named indexes take the new name. Saved policies, their v1 mappings and the reports that name them are kept. A temporary view named user_policies, with the runtime role's privileges, lets the release from before the rename keep reading and writing saved policies during the deploy and after a rollback; a later change removes it. The privilege reconciliation now finds, grants and verifies privileges on views. The route /v2/user-policies, the user_policy_id columns, legacy_user_policy_mappings and the v1 Cloud SQL table user_policies are unchanged.

changelog.d/v2plan-1f-123-region-reference-files.changed.md

@@ -0,0 +1 @@
+Catalog publication reads regions from the Public API's checked-in reference files, one per country, instead of PolicyEngine.py's region registry. Each region takes its dataset from the certified data release and a row filter from a scoping rule: state_fips or congressional_district_geoid for US states and districts, the UK region column for UK nations, and constituency_code_oa or la_code_oa on the local-area dataset the release routes UK constituencies and local authorities to. The UK now publishes 1,016 regions, with 650 constituencies and 361 local authorities (including Ards and North Down) by ONS code and Northern Ireland as country/northern_ireland; the national regions are "United States" and "United Kingdom". US places are listed in the reference file but left out of the catalog until they can run, and the publication evidence counts them. Before it publishes a catalog version that is not stored yet, the publication command reads the HDF5 metadata of the certified datasets the filters use, from Hugging Face at their pinned revisions, and stops if a filter's column is missing; the seed workflow's publication step receives the UK data read token as HUGGING_FACE_TOKEN for this. A catalog version that is already stored keeps the regions it was published with, so region file changes take effect in the next catalog version; republishing it logs the difference in region rows instead of failing, its evidence counts the stored regions and lists the kept version under kept_regions, and every other table must still match.