[MaxGhenis]

WIP until independent review finishes. The measurements below ran at 96f03a8.

Problem

tools/pinned_tbs.py kept a pinned system only for the identical system object. Every simulation has its own system, so each simulation that reached a pinned formula rebuilt a 0.55 GB copy: New York's child credit (2018-2024 and from 2028), New York's EITC (2021), Alabama's federal tax deduction (2021).

Fix

• _get_pinned_tbs keys its cache on the source parameter tree and the pin name, not the system. Ordinary simulations share one tree, so they share one pinned tree.
• pin_spm_policy gives each simulation a cheap pinned system of its own: a private variable registry, private entities, a barrier and a provider snapshot around the shared pinned tree.
• The cache rebuilds when the source tree changes identity (a reform detaches a private copy) or when someone edits it in place (editing a parameter clears the root's at-instant cache, which holds a marker this code plants there).
• The cache holds parameter trees and weak references only, so it no longer pins a finished simulation or a base system.

This also fixes an existing staleness: a variable-only reform applied after a pin was built never reached that pin. The cache now compares the pinned system's variables with the base system's.

Measurements

Local, macOS, Python 3.12, one run at a time at 2 BLAS threads on a shared host, through PolicyEngineRunner from policyengine-taxsim 3.2.6. CPU time reads more reliably than wall time here. Main is 6572bce.

Input	Pin builds	CPU s	Peak memory (ru_maxrss), GB	Output
15,000 rows, 2024	12 → 1	228.9 → 123.3	5.58 → 4.24	byte-identical
1,500 rows heavy in pins: NY 2024, NY 2021, AL 2021, CA 2024	12 → 3	81.9 → 26.5	6.55 → 3.16	byte-identical
150 rows, NY 2024	3 → 1	32.0 → 14.5	3.34 → 2.19	byte-identical
150 rows, AL 2021	3 → 1	15.7 → 6.7	3.14 → 1.98	byte-identical

Three simulations of one New York 2024 household built three copies on main, taking 7.4 to 10.5 s each and peaking at 3.48 GB. With this change they build one, the second and third take 0.3 s, and the peak falls to 1.89 GB. ny_ctc stays $330.

Invariants

These hold for every sequence of reads, new simulations, in-place edits and reforms in a process:

1. Agreement with a fresh clone. A pinned system reads exactly what a fresh clone of the current source tree reads with the pin applied. test_pin_matches_its_source_after_any_sequence_of_changes runs this as a Hypothesis state model. test_a_shared_pin_gives_what_a_fresh_clone_gives checks the differential on real households, for all three pins.
2. One reader per pinned system. Two base systems never read through the same pinned system.
3. Isolation. A reform reaches only the simulation that applied it. Variables, entities and provider receipts stay private to each pinned system. Pinned systems share only the tree.
4. Bounded retention. The cache holds at most one pinned tree per pin, and nothing of a finished simulation (test_the_cache_keeps_nothing_of_a_finished_simulation).
5. Output identity. Unchanged outputs, shown by the table above.

Tests

test_pinned_tbs_sharing.py adds 12 tests. Locally they take 190 s and peak at 3.83 GB (macOS, Python 3.12). They join REST_CORE_HEAVY_TESTS, the core-heavy CI group; Linux CI timings will appear in that group's run.

Mutation check: 10 of 10 deliberate breakages fail these tests (no edit detection, ignoring a tree change, keying on the system, no variables check, reusing the last system, a strong reference, an inherited at-instant cache, a dropped provider snapshot, no private registry, no private entities).

Regression runs at 96f03a8: test_spm_simulation_isolation (28), test_spm_system (76), test_pinned_branch_credit_dependents (3), test_rest_python_groups (62), test_non_refundable_credit_downstream_consumers (1) and test_al_agi_dependent_invariants (3) all pass locally. I ran the 19 YAML files that reach a pin, each in its own process, under a 4 GB guard: 14 pass, and 5 hit the guard (ny_ctc, ny_ctc_post_2024_eligible, ny_ctc_pre_2024, ny_income_tax, ny_working_families_tax_credit). The first three stop at the same test on main, so that memory does not come from this change; I did not repeat the other two on main. test_spm_policy_family and test_federal_credit_limit_order also hit that guard. CI covers all of these.

Limits

• The cache keeps one slot per pin. A process that alternates between two source trees, such as a baseline and a reform, rebuilds on each switch, as it did before.
• A traced simulation has a private root, so it still builds its own tree, as it did before, and that build takes the slot.
• Core's YAML loader sets no bracket.parent, so an in-place edit to a scale bracket on a freshly loaded tree does not reach the root. The cache cannot see that edit, and ordinary reads at instants already cached go stale the same way. Cloned trees carry the parent, so a reform's scale edits reach it.

🤖 Generated with Claude Code