Repository navigation
Carry Schedule E line 32 by entity on the US PUF donor; retire the S-corp exact-zero universe - #1261
Merged
Merged
Conversation
The #982 income-rank predictor sends the top-ranked survey tax units, each standing for several hundred units, onto the PUF's top-ranked returns, which stand for one return or less (the synthetic Forbes-tail records weigh about 0.13). One draw put $141B of line-32 mass on a weight-644 unit in the #1251 rebuild. Weight-aware leaf draws (#481) cannot fix this: the slice such a donor belongs to is smaller than one recipient. QRF finalization now caps each raw amount draw at the positive mass per unit of slice weight of the donor's top slice of the recipient's own weight share, and floors losses from the bottom slice, on every amount output. The rule is bound into the primary PUF execution config (schema 6), and each bounded output writes a tail-bound diagnostics record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…corp exact-zero universe (#1258) The donor now carries the partnership and S corporation shares of line 32 as separate leaves from the TY2015 source rows: - partnership_income = E25940 + E25980 - E25920 - E25960 - E26110 - s_corp_income = E26170 + E26190 - E26160 - E26180 - E26100 Their sum equals passive + nonpassive exactly and E26270 within the booklet's per-field rounding; the passive subset is byte-identical. Aggregate-row buckets allocate each entity's total in proportion to the processed nonpassive amounts, as #1239 did for line 32. Retired: the stacked-spine universe-zero materializer and its whole-pool doctrine (spec, schema, semantics, receipts), the late ACS transfer's s_corp_income exclusion, the pool's primary_puf_exact_zero_universe provision, the donor's zero alias and the fiscal-refresh degenerate-input exclusion. s_corp_income is now a PUF-clone primary output that the late transfer carries to native rows; the passive reconciliation requires both line-32 components. SOI Table 1.4's partnership and S corporation legs (Chronicle measure ids) bind each entity's own leaf. Re-pinned with the repo's tools: generate_us_bundle_from_constants, repin_us_pool_engine_contracts (remaining-stage manifest, engine ABI lock), inventory and field-usage observations, the H3 fixture tool, and the counts that move with one more late-transfer target (71 -> 72 late, 119 -> 120 transfer, 135 -> 136 terminal, 44 -> 45 PUF producers). Coverage evidence regeneration follows in the next commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1258) tools/spec_engine_coverage.py --check: 42244/42244 configuration fields, 41/41 inventory checks. The field-usage pointer inventory digest, the engine-workflow multispine-pool tool's spec_sha256 and the loader golden vector follow the regenerated bundle. Adds a Hypothesis property test that each Table 1.4 entity's net-income and net-loss legs partition its signed leaf (income - loss == leaf; the return indicators partition nonzero units). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Snapped (sparse) outputs are bounded on their snap-grid donor values. A slice inside one grid value now caps at that value exactly; computing on unrounded values clipped every draw at an aged statutory maximum to the grid value below (1,163 qualified-tuition rows per seed in the sweep). - Every test that finalizes amount outputs passes a tail-bound diagnostics sink, now required because the recipient-share bound is always active. - Regenerated imputation.yaml (primary execution config schema 6), field usage counts and claim digest, inventory digests (seed protocol and map, producer registry, resource semantics, full checkpoint identity), the pointer inventory digest, coverage evidence, the pool-tool spec sha and the loader golden vector. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ests - Stacked publication and the pool loader required execution-config schema 5 while the builder wrote 6, so no current stacked pool could publish or load. All four sites now read PRIMARY_PUF_EXECUTION_CONFIG_SCHEMA_VERSION. - The comment and docstrings now call the bound an interim censor that limits a record to its slice (not a fix for concentration the slice has), name the root fixes (#958/#964, #481), and record the loosened links across columns, the rung dependence and the Social Security re-split. - Slices use positive-weight donors only; non-finite bounded draws are refused before any write; quantile records carry a rule discriminator. - Tests: zero weights on both sides; an LP optimality check replaces the tautological mass test; production-geometry differential test; tax-unit outputs, Social Security total, and per-leg calibration factors through finalization; the identity is pinned to a literal. - Re-pinned the seed protocol and map, coverage evidence, pool-tool spec sha and loader golden. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nish re-pins (#1258) Review round 1, finding 2: the primary pass accepted a passive output with only one of partnership_income and s_corp_income in the same pass, and the draw-coherence step summed whichever was present. That was harmless while the S-corp leaf was an exact zero; now an S-corporation-only unit would look line-32-free and lose its passive amount. Both now refuse a partial basis, matching the late transfer's reconciliation, with tests for each missing component. Pins that move with this commit or the previous ones: - seed protocol and seed map (puf_support's attested bytes), the loader golden vector and the US spec digest c74b09ea...; - spec-engine coverage evidence (42244/42244 fields, 41/41 checks); - counts that move with the 72nd late-transfer target in the shared spec-engine, head-to-head battery (136 scalar targets, 380 metric legs, 81 sign-separated), spec-bundle and completion-receipt fixtures; - the legacy pool-tool publication manifest golden (identity only; pool_h5 and agreement goldens unchanged). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- test_us_spec_bundle asserted execution-config schema 5; it now reads the shared constant. The operator-ordering doc and a pass-through fixture said 5 too. - Share-bound diagnostics records carry the recipient and donor weight totals, so rungs can be compared. - Tests: the non-finite check is shown to stop before the quantile cap's write; the LP oracle is scaled to a unit objective so solver tolerance cannot flake; a restated assertion is removed; the base-builder fake record carries the rule field. - Changelog: interim label; schema-5 stacked pools are no longer loadable. - Re-pinned the seed protocol and map, coverage evidence, pool-tool spec sha and loader golden. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 2, finding 4: both PUF entry points (in-process imputation and the chained-worker preparation) now run the signed-subset basis check up front, so a partial or misordered line-32 basis fails before any fit rather than at finalization after the QRF has run. The test now proves no fit runs. Re-derived: seed protocol and seed map (puf_support's attested bytes), the loader golden vector, the US spec digest 02c4227f..., and the spec-engine coverage evidence (42244/42244 fields, 41/41 checks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1258) Found by running the files CI stops in at its first failure: - engine-free: the late-callback occurrence counts (167 registered, 92 unique; s_corp_income is newly in the battery registry and has a primary and a transfer output), the sorted transfer-target digest, 45 PUF producers and 123 transfer-plus-deferred inputs. - engine-contract: the stacked-authority component digests (declared surface, late schedule, metric registry, post-PUF surface), the lineage dashboard counts (186 imputed variables, 135 amounts), the coverage-tool field and mode counts, and 230 compiled outputs. Review round 3: the pre-fit refusal test is renamed and now also proves the basis check runs ahead of donor validation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub's runner queue for the repository was stalled, so the engine-free and engine-us suites were run on Modal against this branch. They found: - For a snapped output the cap was lowered to the largest grid value within it, which could be zero: a carrier whose slice mass sits below the smallest positive donor value was erased. The cap is now lowered onto the positive grid only, so the bound moves no draw to zero; how many records carry a sparse output stays the donor-rate sparsification's job. - Two tests pinned the old field counts as literals. - Non-production stacked passes that ask for no diagnostics now hand finalization a discarded sink; production still requires one. - Four tests that isolate another mechanism on two- to twenty-unit frames (snapping control column, educator sparsification, farm signed-mass calibration, Social Security preservation) narrow the bound explicitly, through a new test-only passthrough on impute_us_puf_tax_detail_support. - Share-bound diagnostics records carry the recipient count, which is what differs between sample rungs. - Re-pinned the regenerated bundle and every digest derived from it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- test_support/spec_engine_inventory_coverage.py: the engine-contract inventory counts follow the S-corp transfer target (late targets 72, itemization targets 39, producer inputs 2747, compiled outputs 230). - Inherited from #1251, whose own CI is red on them at 45a5b98: - test_puf_qrf_chain.py locked the primary target order at 65; #1239's passive leaf made it 66 (the order digest was already current). - engine_workflow test_us_post_export_scoring.py pinned 18 smoke baseline keys; #1239's NIIT passive probe made it 19. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ew r4) A snapped output is bounded by the larger of its slice cap and its smallest positive donor value, so the bound never erases a carrier. The module comment promised the slice-mass bound without that exception. Comment only; re-pinned the seed protocol and map, coverage evidence, pool-tool spec sha and loader golden, which attest puf_support.py's bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_constant_derived_domain_counts_are_complete: primary-PUF targets are all PUF-clone scoped now (66, no whole-pool S-corp), the late DAG covers 72 targets, and the battery registry holds 136 targets (81 sign-separated). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The conflicts are all spec pins. Counts take ours + theirs - base. The repo's repin sequence regenerates the digests: the legacy pass, generate_us_bundle_from_constants, the compiler pass run twice, and spec_engine_coverage. Coverage reports 42,245 configuration fields and 41/41 inventory checks. puf_support.py is unchanged on main since this branch's base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The merge brings in us-puf-line32-passive-1239 at c50bda1, which carries main at 846b48a. The conflicts are spec pins plus one test comment, where #1251's note on the 66th primary target stays. Counts take ours + theirs - base. The repo's repin sequence regenerates the digests: the legacy pass, generate_us_bundle_from_constants, the compiler pass run twice, and spec_engine_coverage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1251 landed first, as d2183 ordered. Resolved count pins as ours + theirs - base and re-derived every digest with the repin sequence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1259 lands first, as d2183 ordered, and its head already holds main at f36a14a, so #1261's CI runs on the tree the land order produces. Resolved count pins as ours + theirs - base, kept #1259's bound asserts beside the retired-doctrine check, and re-derived every digest with the repin sequence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
marked this pull request as ready for review
October 31, 2026 13:47
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Claude merged |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #1258. Stacked on #1251. This PR targets
mainso CI runs, which means its diff also contains #1251's commits until #1251 merges. This PR's own commits start at85eb94b7b; to see only them, compare againstus-puf-line32-passive-1239.The US PUF donor now carries the partnership and S corporation shares of Schedule E line 32 as separate leaves:
partnership_income= E25940 + E25980 − E25920 − E25960 − E26110s_corp_income= E26170 + E26190 − E26160 − E26180 − E26100Before this PR, the whole line-32 total sat in
partnership_income, ands_corp_incomewas an exact-zero universe enforced in four places. This PR retires that machinery. It also binds SOI Table 1.4's partnership and S corporation legs to each entity's own leaf.Max ruled yes on d1489 and d2183 (see "Decision"). This PR lands last, after #1251 and #1259.
Why
From TY2021, SOI Table 1.4 publishes partnership and S corporation net income and net loss separately. For TY2022:
A frame whose S-corp leaf is zero by construction can bind only their sum. The raw TY2015 PUF carries every field the split needs.
The PUF fields
The roles come from the IRS SOI General Description Booklet for the 2014 Public Use Tax File (annotated Schedule E Part II). The same table appears in #1251.
Each entity's share is (g) + (j) − (f) − (h) − (i) of its own columns. Checked on the pinned TY2015 file (
puf_2015.csv, sha2560a7fd643…, all 207,692 regular records;out/entity_split_stats.txt):Other shares of regular returns:
The change
Donor (
puf_source_schedule_e.py,puf_support.py)schedule_e_partnership_s_corp_columnsreturns each entity's share. The line-32 rounding identity is now checked against their sum.source_year_puf_partnership_s_corp_incomereturnspartnership_income,s_corp_incomeandpassive_partnership_s_corp_income.s_corp_incometo zero, so a processed donor that requests any entity leaf without the supplement now fails with "cannot derive".On the real inputs (211,677 processed tax units, TY2015 dollars on the processed weights;
out/real_donor_vectors.json):6c76e077…).Retired: the S-corp exact-zero universe
Stacked spine:
_materialize_us_puf_s_corp_universe_zero,us_puf_s_corp_universe_zero_rule_identityandUS_PUF_S_CORP_UNIVERSE_ZERO_RULE_ID;whole_pool_output_universesdoctrine, in the primary execution config, the PUF-pass receipt and the late-resource validator;s_corp_income_universe_zeroreceipt.Native rows now stay null until the late transfer fills them, as
partnership_income's always have.Spec: the doctrine is gone from
imputation.yaml(regenerated),imputation.schema.jsonandimputation_semantics.py.Late ACS transfer:
s_corp_incomeis no longer excluded frompuf_tax_itemization. It joinspartnership_incomein batch 3, so the passive draw conditions on both entities. The post-transfer passive reconciliation now requires both line-32 components rather than reading a missing S-corp leg as zero.Late producer registry:
s_corp_incomeis a PUF-clone primary output like every other primary target.Pool: QBI reconciliation's
s_corp_incomeinput is provisioned by the pool input surface, likepartnership_income. Theprimary_puf_exact_zero_universeprovision is gone.Fiscal refresh: the reviewed degenerate-input exclusion for
s_corp_income("constant zero") is removed. The gate fails a stale exclusion, and the leaf is no longer constant.The execution-config payload keeps
schema_version5, and the registry keeps schema 17. Receipts embed the full content, and the exact doctrine-equality check refuses a pre-#1258 receipt; a test covers this. Bumping the versions would have cascaded into worker-identity authentication for no added safety.Table 1.4 entity legs (
fiscal_targets.py,variable_labels.py)The measure ids are Chronicle's (PolicyEngine/chronicle#180, as specified in chronicle#346):
partnership_net_income_{returns,amount}partnership_incomepartnership_incomepartnership_net_loss_{returns,amount}partnership_lossespartnership_incomes_corporation_net_income_{returns,amount}s_corp_incomes_corp_incomes_corporation_net_loss_{returns,amount}s_corp_lossess_corp_incomeThe materializer sums each person leaf to the tax unit before taking the part, which is return grain, as for the estate/trust legs.
out/compile_*.json):f5bd921d6b21(33,250 specs);national_statee1f9906bf0e0(6,102);scripts/compile_with_entity_legs.py). All eight compile ontonational_state, each binding its own leaf (out/compile_with_entity_legs.json).tax_unit_partnership_s_corp_income, which is consistent with the entity legs: line 32 is their sum.Re-pinned, every value re-derived by the repo's tools or from the observing code
tools/generate_us_bundle_from_constants.py:imputation.yaml,battery.yaml(s_corp sign-separated metric),catalogs.yaml(domainpuf_tax_itemization), and its reviewed counts (late 19/72, the itemization split [8,8,8,8,7], graph inputs 2747, compiled outputs 230).tools/repin_us_pool_engine_contracts.py: the remaining-stage manifest digest and the engine ABI lock.inventory_coverage.EXPECTED_HASHESand counts, frombuild_inventory_coverageobservations: authority, full checkpoint, graph nodes, late families, late resource semantics, late schedule, seed protocol and seed map.field_usagepins (battery, imputation family execution, imputation producer graph, totals) and the pointer-inventory digest.tools/spec_engine_coverage.py --checkpasses: 42,244/42,244 fields, 41/41 checks.tools/graph_us_post_transfer_fixture.py: the H3 fixture's pool digest line.02c4227f…(engine-workflow multispine tool test) and the loader golden vector.Invariants
These hold for every input. The Hypothesis property tests run 300–500 examples each.
test_entity_shares_partition_line_32_exactly).test_an_entity_without_columns_carries_nothing).puf_support's raw-field derivation equalspuf_source_schedule_e's columns for all three leaves (test_raw_array_path_agrees_with_the_source_year_columns).test_table_1_4_entity_legs_partition_each_signed_leaf).Do the tests catch wrong code? I seeded 13 mutants across the donor module,
puf_support, the transfer reconciliation, the target map and the materializer's sign rule. All 13 fail, each first on the test aimed at it, and the unmutated code passes (scripts/mutation_check.py,out/mutation.txt; the Hypothesis example database is cleared before each mutant). The mutants:s_corp_income;Modal evidence
I built a PUF-support base on Modal: run
entity-split-1258-base-d239b0f77b7a, rc 0, 8,644 s, about $7.81.us-puf-entity-split-1258-route-a(d239b0f). That is Carry Schedule E line 32 and its passive subset on the US PUF donor #1251's evidence commit d02fb2b (Route A's 4b57d15 plus US frame partnership/S-corp income is nonpassive-only: the donor builds it from E25980-E25960 + E26190-E26180, not line 32 (E26270) #1239), plus only this PR's donor-side files.line32-1239-base-d02fb2bca843. The two bases share every household, tax-unit and person id, so the only difference is the donor split.scripts/modal_compare_bases.py→out/compare_bases.json.Imputed against the donor. PUF-clone half (89.76M weight); donor scaled per unit weight (×89.76/161.18). Base initial weights:
person_columns_movedinout/compare_bases.json).w2_wages_from_qualified_business: −$54.3B (−30%)sstb_w2_wages_from_qualified_business: −$33.7B (−40%)Microsimulation. PolicyEngine-US 2.2.1, 2024, on both bases (
scripts/modal_microsim.py,scripts/microsim_totals.py→out/microsim_totals.json). Treatment minus control:These are base-level effects at fixed weights. A release re-solves its weights against the targets (HT2/CD A26270, and the Table 1.4 entity legs once Chronicle publishes them), so they are not release figures.
Under the Route A final weights, line 32 is $1,000.8B, against HT2 A26270 TY2022 of $1,033.3B. The entity split is partnership $439.8B and S corp $561.0B, against Table 1.4 TY2022 nets of $311.5B and $719.7B. The donor's TY2015 mix (37/63) carries a lower S-corp share than TY2022's (30/70); calibrating to the entity legs is what corrects that.
Review
Round 1 (Opus 5.5 lane, because the Codex lanes were full): APPROVE WITH NITS at aa47a3f. Response in
review-r1-response.md:Round 2 (Opus 5.5 lane): APPROVE WITH NITS at ba4dfa9. It confirmed:
Addressed in 48046eb:
Round 3 (Opus 5.5 lane): APPROVE WITH NITS at 48046eb. It confirmed the finding-4 fix in both entry points, that the re-pins agree with one another, and the corrected body. Its notes, and what was done:
_load_manifestalready refuses a production manifest whose target order differs from the current canonical order.out/pins-final.txtnow does.Round 4 (GPT-6.1 Sol): APPROVE at
487243ee9, covering the merges of Carry Schedule E line 32 and its passive subset on the US PUF donor #1251 and main.CI
Later commits fix count pins that CI and local sweeps found (f333029, e953167, 2126a5a). Two were inherited from #1251: its own CI is red on them at 45a5b98.
test_puf_qrf_chain.pylocks the primary target order at 66, not 65.test_us_post_export_scoring.pyhas 19 smoke baseline keys, not 18.Both PRs pass all 10 checks at their current heads.
At 2126a5a,
engine-free (3.13)passes the whole engine-free suite.Not in this PR
Table 1.4 vs HT2/CD. Once chronicle#180 lands, the Table 1.4 entity legs and HT2/CD's signed A26270 describe the same line 32. At one vintage, the signed net equals income less loss of the four legs exactly, so binding both nationally is an equality conflict. For TY2022, Table 1.4 implies $1,031.2B and HT2 says $1,033.3B. Which binds nationally is a choice for the HT2/Table 1.x control work (Rebase stale Historic Table 2 rows onto their concept-matched national controls #1247, Bind Historic Table 2 and district-file business, partnership and rental 'net income (less loss)' rows to signed model concepts #1230); recorded there.
Dollar basis. Both leaves keep the processed column's TY2015 dollars. The PUF donor's amounts stop at TY2021 levels, two columns at TY2015, and nothing ages them to the build year #1248 (PR Record the PUF donor uprating audit for #1248 #1223) owns aging the donor and can age each entity with its own series.
When the legs bind. Table 1.4 entity legs bind only once Chronicle publishes them (chronicle#180 / Informed-init L0 selection: warm-start reselection for pool changes (ACS/ASEC ingest forcing function) #346).
Legacy scoring. Pre-US PUF donor carries Schedule E line 32 as one number: s_corp_income is an exact-zero universe, so Table 1.4's partnership and S-corp legs cannot bind by entity #1258 attested artifacts need re-attestation for legacy scoring, as after Carry Schedule E line 32 and its passive subset on the US PUF donor #1251.
Decision
Merging changes what the next US build carries: S-corp income on every PUF-clone and native row, instead of zero. Max ruled yes on d1489 (10/24): land #1251, then this PR, with a fresh review that clears the top-tail draws, then a certified US build. Round 4 clears them only with #1259, and Max ruled yes on d2183 (10/31), so the order is #1251, #1259, #1261.
axiom: n/a: microsim data donor, imputation and calibration-target mapping only; no policy rule changes
🤖 Generated with Claude Code