[MaxGhenis]

Fixes #732

Adds pe.us.flat_file_microsimulation and pe.us.prepare_flat_file, which run a researcher's own flat file through one state's rules or through the file's own states, plus a simulated instruments recipe at docs/simulated-instruments.md.

Why here

policyengine.py is the interface researchers should build on. The model fixes in PolicyEngine/policyengine-core#646 and PolicyEngine/policyengine-us#10574 cover direct use, but this package pins policyengine-core==3.33.3 and policyengine-us==2.2.1, and those fixes will not reach it soon. So the helper works at those pins by itself. No pin moves in this PR, and no bundled number changes.

What it does

prepare_flat_file checks a file and writes it in the one form the pinned loader reads correctly:

• Every column becomes variable__period, and every ID moves to each person's row. At the pinned core, a bare household_id column crashes the load with an IndexError.
• person_{group}_id numbers groups in file order, and {group}_id keeps the file's own IDs. The pinned loader sorts groups by ID but reads a one-row-per-group column in row order, so a file of one-person groups in any other order gave each group another's state and weight. File order makes the two agree for every entity at once. Keeping the file's IDs in {group}_id means results merge back on them.
• Text IDs, and IDs too large for the model's 32-bit storage, become integers in file order, with a log for the large ones. A missing person_id gets one.
• county_fips becomes five-digit text from 6037, 6037.0 or "6037".
• With state=, it sets state_fips in every year the file holds, and clears stored state labels, geography finer than a state, and what the model derives from that geography.
• Either way, it drops stored copies of what the model derives from the state, such as state_group, which otherwise hide the formula: a stale state_group kept Alaska on the contiguous poverty guideline.
• Without state=, it requires state_fips, and refuses a county, tract, block, congressional district or state label that contradicts it in any year the file holds, including a column carried from another year and the latest stored county that the model reads from any year.
• It logs a warning for a county code that names no county, since the model then uses the first county in the state.

It refuses, with a message that names the fix, a file that has no rows, no age, no group ID, a blank ID, a repeated person_id, group IDs numbered within households, or rows of one group that disagree on a group-level variable. The last two matter because they load without an error: the loader merges those people or takes the first person's value.

flat_file_microsimulation then builds a Microsimulation with a resolved SPM selection and stamps policyengine_bundle, recording flat_file as the runtime dataset, plus the year, the state and a SHA-256 digest of the prepared file.

Decisions

• SPM stays explicit. A file moved to a state has no county, so SPM thresholds need spm={"geography_kind": "national"}, and the recipe passes it. The helper does not change the bundle's default.
• state= clears every household's county, including households already in that state. Every state then runs the same population on the same terms, which is the point of the instrument. To keep real counties, leave state out.
• The helper drops columns that name no variable and logs them. A survey file carries many, and a double-underscore name would crash the pinned loader.

Invariants

• Preparing a file leaves the input frame unchanged.
• Moving a file twice lands where moving it once does, for every pair of states (Hypothesis).
• A prepared file prepares to itself (Hypothesis).
• A county code gives the same result as a number or as text (Hypothesis).
• Row order changes the order of results, not any household's membership (Hypothesis).
• Differential: a file with stale geography runs identically to one that never had it, after a move.

Tests

tests/test_us_flat_file.py: 80 passed at the pinned versions, including real Microsimulation runs from a CSV.

Real-data check

A 300-household Populace US sample with 822 people, read back from CSV, ran through the helper for CA, MS, MN, DC and NY, and through its own states. state_code was right in each, and national SPM thresholds matched across states. It logged five columns the pinned model does not know, as designed.

Differential at the pinned versions: the helper with state="MN" gives the same SNAP eligibility, Medicaid eligibility, income, state tax and net income as the same file loaded directly with geography dropped by hand. Means match, and values match once sorted, because group results follow file order.

axiom: n/a: microsim input handling, no policy change

🤖 Generated with Claude Code