Repository navigation
Conversation
withheld_reasons tests every condition of reform withholding and gives one reason per condition and component or variable it holds for: suppressed_component_changed, small_component_left, small_balance_change, small_target_change and no_baseline. Each names the condition and the variable, and the published metric id of a partition component, never a count or an amount. reform_withheld keeps its verdict: any reason. Both reform figures keep the reasons under withheld_because, the long-format report gives them on each reforms and reform_targets row, the comparison carries them per side, and the comment lists them under the reform costings. A failing responsiveness detail in a withheld year names them too. Fixes #2654. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
measure() checks the target of every reform it costs, as on main, so a reason names the variable; a test pins the case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
privacy.py now describes reasons about groups the reform defines, and adds a deliberate exception: an own-target small_target_change beside the published responsive verdict bounds that small cell's total. withheld_reasons says that evaluating every condition can fail a reform in a year main withheld early. The comment lists at most 40 reason lines and points to the artifact for the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Test two small targets at once, a change of exactly CHANGE_GBP, a newly small double count and a target on benefit units, and add CHANGE_GBP to the Hypothesis deltas. - Qualify the numeric-text property: names such as bursary_fund_16_to_19 hold digits that count nothing; check the words around them. - Say in privacy.py that reasons across reform rows can bound a cell and that the reform list decides which rows exist, and pin that list. - Say that a no_baseline reason says nothing about the reform's records, and describe withheld_reasons as it behaves. - Name the comment's overflow count and the file and field that hold every reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The no-baseline test now changes one household's income tax and still records only no_baseline, and withheld_text's docstring says that names can hold digits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Add a case that weighted and unweighted benefit-unit counts tell apart: eight weighted of ten changed. - privacy.py: reasons across reform rows can pin an exact count when as many separate groups as a published count's records each change; in CI the head's standard_reforms.yaml decides which rows exist. - A variable that fails in any year fails the reform; the comment's overflow line names who gives the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A no_baseline note has no small cell behind it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Aggregate impact on certified data: passBase Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in Notes
Moves flagged for reviewNone. Spending by benefit and service
Caseloads
Standard reform costings (change in gov_balance)
The report suppresses a reform's costing and target change in these years, for the reasons below (
Standard reforms: change in each reform's target
Household net income by decileNo household's net income changes in any year. No change in any year: receipts by tax head; government balance, each component counted once; model totals; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles. Checks failing on both base and head (1)
Counted-once checkEach of 92 components was raised by £1 per unit in 2026-27, and every head of its totals was read household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A head that breaks a rule is measured again twice: with every other head held at its baseline value, and with every other head held at its value after the first £1 while the component rises again. It fails where either still shows the response. 0 failed; 0 were skipped.
Provenance
Dataset: Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
Fixes #2654
What and why
reform_withheld(policyengine_uk/utils/aggregate_impact/measure.py) withholds a standard reform's costing and target change in a year when they would come from a small cell, and the report recorded only that it did. The qualification gate in policyengine.py (reform_responsiveness) can't recompute the tool's verdict in such a year, so it returnsneeds_review, and a reviewer had to read the code to learn why.The report now records, for each withheld reform year, every condition that holds and the component or variable it holds for.
withheld_reasonstests every condition, where main stopped at the first, and gives one reason per condition and variable:conditionvariablemetric_idsuppressed_component_changedspending.state_pensionsmall_component_leftsmall_balance_changegov_balancein 1 to 9 householdsgov_balancenullsmall_target_changenullno_baselinenullnullmeasure()checks the target of every reform it costs, as main does, so asmall_target_changecan name another reform's target. Intest_a_small_change_in_another_reforms_target_names_that_target, the tool withholds a reform aimed at income tax because the reform moves Pension Credit, another reform's target, in three records.The reasons reach five places:
result.json, eachreforms.<id>andreform_targets.<id>figure carrieswithheld_because,{year: [reason, ...]}, for the years itssuppressedlists and no others.api.to_report), which qualification reads, eachreforms.<id>andreform_targets.<id>row carrieswithheld_because, a list of{"condition", "variable", "metric_id"}, empty for a published year. Other rows don't change.summary.jsonin the report artifact), a reform cell that either side withholds carrieswithheld_because: {"base": [...], "head": [...]}.summary.jsongives. The hand-built run below gives:responsive:<id>detail in a withheld year names the reasons, such as2026-27: income_tax did not move enough (the amount is withheld: the reform changes gov_balance in fewer than 10 households; the reform changes income_tax in fewer than 10 records). Main said "it comes from fewer than 10 records", which misstated the cause whenever the tool withheld a year to protect another cell.The tool withholds the same years and publishes the same figures as on main.
reform_withheldkeeps its signature and returns whether any reason holds, and every existing JSON key stays.TOOL_SCHEMAstayspolicyengine-uk-aggregate-impact/1. The tool states no rule for an added field, and the nearest stated rule (fiscal_partition.yaml: bumpschema_versionon any change to field meanings) doesn't apply, because no existing field changes meaning.metrics_versionchanges, as it does with any tool change, because it carries the tool's digest.Because
withheld_reasonsevaluates every condition, a variable that fails to compute under a reform now fails that reform (reform_runs:<id>) in a year where an earlier condition already withholds it. On main that error surfaced only in a year no earlier condition fired.Small cells
A reason holds three names and never a count, an amount or a share. Around the names, its words add no number but the 10 in "fewer than 10". The names come verbatim from the partition and the standard reforms, and some hold digits that count nothing, such as
bursary_fund_16_to_19. A reason about a group the reform defines says only that the group has fewer than 10, as a decile row's "fewer than 10 households change" already does. A published year records no reason.privacy.pystates three deliberate exceptions:suppressed_component_changednames the suppressed component the reform touches: a yes or no that can turn on that component's few records, as the counted-once verdicts the report already publishes for a component from fewer than 10 units do.small_target_changenaming the reform's own target, beside theresponsive:<id>verdict the report already publishes for that year, says whether fewer than 10 records together moved the target by at least the reform's minimum: a bound on that cell's total, never its value. Asmall_balance_changesays much the same whengov_balanceincludes the target and nothing offsets it there. The no-dominance counting rule already publishes more (one household's £10,000 beside ten pennies).standard_reforms.yaml, andtest_privacy_py_covers_the_stock_reform_rowspins its ten reforms, so a pull request that changes the list changes that test too.What qualification can read
reform_responsiveness(src/policyengine/provenance/uk_candidate.py, policyengine.py1e761047de22, its current main) keys rows by(metric_id, year)and readsvalueandsuppressedfrom eachreform_targets.<id>row._small_cell_problemschecks the small-cell fields and ignores fields it doesn't know, so the added field breaks neither. On thereform_targets.<id>row it already reads, the gate can now readwithheld_because.When a withheld year's reasons include neither
no_baselinenor asmall_target_changenaming the reform's own target, the reform changed that target in no weighted record or in 10 or more. The tool then withheld the year to protect another cell, and a reviewer can accept the tool's passing verdict without reading the code.no_baselinegives no such evidence: without a baseline record the tool tests no other condition, so a reform that moves its target in one heavily weighted record can passresponsive:<id>withno_baselineas its only reason.measure()keeps a baseline record for every year it costs, so only a direct call ofmeasure_reformswithout one recordsno_baseline.The gate's code doesn't change here, so it still returns
needs_reviewfor a withheld year until it reads the field. It measures with the tool at policyengine-uk1191ac5a4, the commit thatmetrics.tool.commitinsrc/policyengine/data/qualification/uk.jsonnames, so it seeswithheld_becauseonce that pin moves to a commit with this change.publishable_metrics_reportwrites the whole report,withheld_becauseincluded, to the public qualification ledger.Invariants
For every input on which every variable computes:
nulland its year insuppressed) exactly whenwithheld_becauseholds a non-empty list for it; a published year has no entry, and its long-format row has an empty list.reform_withheld.condition,variableandmetric_id, each a string ornull. Around the names it quotes, its text adds no number but the 10 in "fewer than 10", and no £.no_baselinereason stands alone.Test plan
ruff format --check,ruff checkandruff check --select F811,F821 policyengine_uk(ruff 0.17.2), andgit diff --check.changelog.d/2654.added.md.policyengine_uk/tests/aggregate_impact/test_aggregate_impact_withheld_reasons.pycollects 25 tests:no_baselinealone;double_counts.id; and Child Benefit on 40 benefit units in 20 households, which the tool counts with benefit-unit weights, in three cases, one of which tells weighted from unweighted counting;reform_withheld_main_reference.py, a copy of main's yes-or-no rule from da31e86. The second checks invariant 5.Every run below tested
d1882db34.origin/mainexport (2da212abc)ImportError: cannot import name 'component_metric_ids'KeyError: 'withheld_because'; the stock reform pin passes, because main ships the same listtest_aggregate_impact_measure.pytest_aggregate_impact_compare.pytest_aggregate_impact_reforms.pytest_aggregate_impact_partition.pytest_aggregate_impact_data_mode.pytest_aggregate_impact_dataset.pytest_aggregate_impact_childcare_provenance.pycode_health/test_aggregate_impact_workflow.pycode_health/test_cached_arrays_not_written_in_place.py, scan testsEach mutation below changes a scratch copy of
d1882db34, where the new file passes 25 of 25 unmutated, and the table counts the new tests that then fail.small_component_leftreason>=for>)Copies of the tool from
origin/main(2da212abc) and fromd1882db34measured the same model in separate processes, on aUKSingleYearDatasetof 40 hand-built synthetic households (no survey data). They costed Raise higher rate by 1pp, Raise VAT standard rate by 2pp and Raise the Universal Credit standard allowance by 10% in 2026-27 and 2027-28. Each side built all five simulations afresh, each from a fresh dataset: the baseline, a rebuilt baseline and one per reform. Both ran with the counted-once check off. Every published figure, suppressed year, exclusion and check verdict agreed, and only theresponsive:<id>details of the two withheld reforms differed.small_balance_change;small_target_changeonincome_taxsuppressed_component_changedonuniversal_credit;small_balance_change;small_target_changeonuniversal_credit#2719 (Budget 2026 measures) merged into main as
f2a6c3e6d. It changed none of the nine files this change touches, and git merged that main into this branch without a conflict. Draft #1963 also editsmeasure.py, in another function; at its head,69c919f60, git merges that file cleanly, and its conflicts lie in files this change doesn't touch.Reproducer
Reproducer: policyengine_uk/tests/aggregate_impact/test_aggregate_impact_withheld_reasons.py::test_two_conditions_together_are_both_recorded
axiom: n/a: CI tooling.
🤖 Generated with Claude Code