Repository navigation
Conversation
scripts/household_set.py samples households from the dataset the installed policyengine.py bundle certifies, renames them mc_000 onward, computes every output on the installed policyengine-us with the reference conventions, and writes a board in the frozen layout with batches. On the frozen board's own households it reproduces all 1,935 scored 2.41.0 references exactly. The tool pilot takes --board: the engine and fix pins come from the board's last engine_upgrade revision, and a household set's arms are scored against the same model's no-tools run (--baseline-dir). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/batch_judge.py writes one arm's answers on a batch as a dashboard payload (each model a board model), so consensus-flags and the two-stage adversary run unchanged: two of three models agreeing off the reference flag a cell. The payload carries what the models saw: the v2-pilot household prompt (checked against each record's sent prompt hash), the v2-pilot definitions, and the condition they answered under. Derivations are pruned engine traces on the household set's own reference system. The adversary takes a payload's outputDefinitions and answerCondition; the default renders both stage headers byte for byte as before. Dry run on today's frozen-board no-tools answers (v2 wording): 23 of 1,935 scored cells flagged, 12 of them among the 2026-10-05 adversary's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-22: "in case youre running any claude api stuff please use the key on my clipboard". With JUDGE_API_KEY_SECRET naming an agent-secret entry, the judges bill that key and nothing else: it refuses a lane config, a lane token, AUDIT_ACCOUNT or the desktop opt-in alongside it; every call runs in a fresh empty config directory removed at exit; the key joins only the claude calls' allowlisted environment; and check-login requires a first-party api_key login from ANTHROPIC_API_KEY in that directory. Sidecars record the mode and the secret's name, never the key. The lane and desktop paths are unchanged. Why now: the desktop login's CLI session had expired, and the Subfleet lanes were closed, so the batch judges had no login to run on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The adversary prompt tells a judge to exclude policybench.org, policyengine.org and GitHub on every search. Codex's search tool takes no blocked_domains, so GPT-6.1 Sol did it with -site: operators, and the audit rejected those stage-1 outputs as searches naming PolicyEngine (rehearsal, 2026-10-22: 2 of 3). searched_for() drops a -site: exclusion of a blocked domain before the contamination check, in both runners' audits. A query that still names the engine or the benchmark (beside exclusions, under site:, -inurl:, or a look-alike domain) is still rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-23: "we can still test before that but i'd like v2 to be clean". household_set.py build takes --purpose v2 (the default) or test. A v2 set needs the installed policyengine.py bundle to certify its own dataset for the installed engine; anything else is refused unless --purpose test. A test set may name a published dataset (--dataset-repo/--dataset-file/ --dataset-revision); it records the release manifest's build engine and compatibility claim, purpose test and certified_pair false. Engines after policyengine-us 2.2.1 ignore Populace's filing_status_input and derive filing status from the household; the build records each sampled household whose stated status differs from the engine's (its scenario and reference carry the engine's). The sampler takes the dataset as an option; the default call is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-23, on what test batch 0 showed about the calculator ("yes go"):
- Names: a name written with spaces (as the prompt labels its facts) or a
benchmark output key (free_school_meals_eligible, head_medicaid_eligible)
is read as its engine variable, and the result's notes say how. Every model
had a call refused for a guessed name in all 10 households.
- Placement: an input set on another entity moves to the one unit that holds
the people it was set for (a group left out becomes one unit of everyone, as
the engine reads it); otherwise it stays an error.
- Overrides: a contribution set as the realized amount is read as its
_desired input (Sol and Opus put "traditional 401k contributions desired"
into traditional_401k_contributions in 6 households, moving tax), and an
input the engine computes and no stated fact sets is flagged as replacing
its formula (Sol set rent, not pre_subsidy_rent).
- Explanations: explain (up to 3 variables) returns the computation steps and
the parameters read, with values and cited sources. Opus overrode a correct
free-meals answer in Louisiana, which pays the reduced-price copay.
Tool version 1 (pe, web+pe) is byte for byte as before. The notices and tool
description now state the board's engine (a household set on 2.60.4 had been
told 2.41.0); the frozen board's text is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Max, 2026-10-23: "thinking we might want to do all these tests with a cheap model until we're super ready for v2 with the web and tool experiments". batch_judge.py payload takes --models (the frontier three by default). The payload's household prompt is the first tool-contract model's, checked against that model's records, and the payload records which models it holds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The calculator's six features (names, placement, overrides, explain, concise, flat) are each on or off, so a study can change one at a time. "pe2" keeps tool version 2 (the first four, its notice byte for byte), "pe3" offers all six, and any other set runs as "pe:" plus its features, e.g. "pe:names+flat". - concise: an explanation gives the value, its direct inputs and up to five parameters unless `detail` is "full". - flat: the household is described as a state, people with a role (head, spouse or dependent) and facts, and household facts; the tool builds the units the benchmark's own situations have and places each household fact on its entity. Tests: the v2 notice is reproduced from its features; each feature adds only its own behavior; flat builds the benchmark's units and refuses what it cannot place; and on the installed engine eight frozen-board households described flat compute every board output exactly as Scenario.to_pe_household does. Five mutants of the flat and arm logic are all caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The prompts label some facts in words rather than as the variable name with spaces: employment_income is "gross wages and salaries", hours_worked_last_week "usual weekly hours worked", has_esi "has employer-sponsored insurance" (policybench.prompts.INPUT_LABEL_OVERRIDES). In the 20-household test set nearly every household has at least one; hours appear in 17, employer coverage and premiums in 11 each. Tool version 2's names feature cannot read these, and its notice says the facts carry the variable names. A Flash-Lite smoke test of the flat format met three such errors and fell back to setting employer_sponsored_insurance_premiums to 0 and weekly_hours_worked, which the SNAP work rules do not read. The new labels feature reads a fact under its prompt label (case and punctuation aside, renames applied) and, as the reference situations do, also sets stated usual weekly hours as weekly_hours_worked_before_lsr (PE_INPUT_ALIASES). pe3 now offers all seven features. pe2 is unchanged and keeps its notice as it ran; every other feature set's notice says only what the tool reads. Tests: label reading, the alias (and leaving a set value alone), pe2 unchanged, the variable-name claim only in pe2's notice across every feature set, every prompt label reading back to its engine name, and a differential: twelve frozen-board households described flat with each fact as its prompt writes it compute every board output as the benchmark does. Three mutants (no alias, no renames, strict label matching) are caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The v2-pilot prompt relabels two facts itself ("home energy assistance
received" for spm_unit_energy_subsidy, "employer-paid health insurance
premiums ..." for employer_sponsored_insurance_premiums); the labels
feature now reads them, derived from V2_PILOT_CONVENTIONS so the two
stay in step. In the flat format a person given no name is named by
role (head, spouse, dependent_1, ...): Flash-Lite's first flat call in
two smoke households left the names out.
The label differential now writes each fact as the v2-pilot prompt
does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two of three Flash-Lite smoke households gave the head's rent as a household fact. pre_subsidy_rent is a person variable in policyengine-us, so the flat format refused it. Under the placement feature a person fact given for a household of one now goes to that person, as placement moves a person variable off a one-person unit; with two or more people it is still refused, now saying to give it under the person whose facts list it. The test registry's entities now match the engine's (pre_subsidy_rent is a person variable). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A differential against 5bbd571 (the commit the test batches ran) found version 2 requests carrying detail "full"; only execute_request reads it, through .get, so results were unchanged, but the request now matches exactly. 4,000 generated calls through versions 1 and 2, the notices, descriptions, schemas and 15 model profiles are identical to 5bbd571. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GPT-6.1 Sol's review (REQUEST_CHANGES, 7 findings) at 9c43f4f: 1. A model's arguments could turn on features its arm does not offer: the worker read "__tool_features__" (and, as at 5bbd571, "__tool_version__") from the arguments dict. A pool now sends features in a tuple frame no JSON can be, the worker validates them (names, concise beside explain), and keys like "__tool_version__" in a model's arguments are refused as unknown arguments. No run record ever carried them. 2. A resume could mix treatments. Each household record now keeps its tool features and calculator schema hash; a resume compares features, notice and schema, and refuses a non-legacy record without them. 3. Labels missed what the prompt renders. The label index is now built from the prompt's own renderer (_format_input_line, then v2_pilot_text) for every promptable, labeled and reference-stated input: fallback labels, the Marketplace plan line, "state", partnership or S-corp income. Facts the reference drops (PE_DROPPED_INPUTS) are ignored with a note. A label that names two of the engine's inputs is not read. 4. Flat refuses person keys other than name, role and facts. 5. Flat stores household facts under the name written, so normalization compares them once normalized; roles set the relationship inputs and a stated one that contradicts its role is refused. 6. The flat example uses employment_income; placement replaces the wrong-entity error sentence; "full" detail is the trace, capped. 7. Diagnostics read flat people by their default names. Evidence: 210 passed, 4 frozen-engine skips. A labels-only differential builds 30 frozen households from their prompts' facts and matches the reference situation on every output, and so do all 100 frozen and all 20 test-set households in a one-off run, with or without take-up inputs. pe and pe2 are identical to 5bbd571 over 4,000 calls and 2,000 pool messages. 11 mutants of the fixes are caught. An end-to-end Flash-Lite pe3 smoke run had 0 tool errors on 3 households. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- flat: a household fact repeated with a malformed value was dropped without a word. _same_value now reports the repeat's validation problems, and only a repeat validated as equal is dropped quietly. A malformed first value is reported once, by normal validation, and never as a conflict. - The fallback record for a household the harness failed on carried no treatment, so one failure blocked resuming any custom arm. The record is built by harness_error_record, which now includes the notice hash and profile_treatment. resume_treatment_problems checks only the records a resume keeps, since a rerunnable record is replaced under the current profile. pe, web+pe and pe2 still behave exactly as at 5bbd571: 4,000 calls and 2,000 pool messages match. 5 mutants of these fixes were all caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…core A rerun is not sure to replace a rerunnable record: a budget stop, lost workers or a narrower --scenarios leave it in place, and scoring read whatever was there. treatment_problems now checks every record in the folder, rerunnable ones included, and command_score refuses a folder that mixes treatments, rebuilding the notices on the board's engine. Runs written before 6e1578f (test batch 0's frontier calculator arms, b6d835a) were told policyengine-us 2.41.0 while computing on 2.60.4. A folder written wholly so is one treatment and scores with a note; a mix of the two texts is refused, and a resume into it stays refused, since its new records would state the board's engine. All 39 run folders score (628 records); the six frontier calculator folders take the note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) Reviewed head 6b00a6f (GPT-6.1 Sol r4 AGREE); full suite at that head: 218 passed, 4 skipped, ruff clean.
… ceiling scripts/confirmatory_run.py runs the frozen pre-registration's run procedure on the treatment harness (6b00a6f's tool_pilot_argon.py, unchanged) and writes the run records the confirmatory report checks: - two lanes per model, households alternating in draw order, each household's arms in its drawn order, then reruns in passes; - one ToolPool per model with no features of its own, each call framed with its arm's features; - admission against each model's cap and reserve, waiting while a pass's reruns are undecided; - the hard ceiling over every model's spend, through the harness's Spend; - the failure classes and actions, with operator pauses, harness and inconclusive stops, and replaced attempts saved before their reruns; - the 30-second host log and the guard's and power log's lines. Tests follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unit tests for the streams, arm strings and features, the classes and actions, the host evidence and the ceiling. Fake runs (a fake provider, pool and host log, on a fake board, with the windows scaled down) plant reruns, host trouble, an operator pause, a harness stop, the inconclusive stop, a refusal, the hard ceiling, a raising attempt, pool restarts and an interrupt. run_problems recomputes every rule of the run procedure from the files the runner wrote, and a property test checks it over random failures and spend. Two tests run the harness's own conversation under the runner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A model whose lanes never decide, because the ceiling was crossed before its first admission, wrote no admission log, and the confirmatory report reads a missing log as a lost one. The runner now creates each model's run and admission logs when it writes the model's manifest. Found by the differential test in the study's scripts repo, which runs this runner and checks its records with the report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cle fixes The wall clock steps back by up to about 0.1 ms between threads on the run's host (40 backward steps in 190 million readings over 12 threads, the largest 91 us). The run records order attempts, stops, admission decisions and the ceiling's crossing by their times, and the confirmatory report compares them exactly: an attempt's end and the next start in its lane can be microseconds apart, so a step back could make a valid run fail the report. `clock()` gives time.time(), never earlier than a time it gave before, and each of those times is taken under the lock that orders it. A test reads the wall clock up to 0.2 ms early everywhere; without `clock()` it fails every time (attempts overlap, a household runs before its admission). The test oracle had three faults, two of them exposed by the full suite: - A household a stop cut short was taken to finish at its last attempt. The runner counts it under way until the stop, as the report now does. A new test makes the case deterministic (it failed the old oracle with the flake's message, spend 0.02 vs 0.03). - The scaled host-log gap was 90 ms, and under a load of 12 the sampler stalled past it, so an untouched run got host reruns. The gap is 0.5 s in the tests, and the stall tests stall 0.8 s. - The oracle allows the harness's raw request times a second outside their attempt, as the report does. 121 tests pass, 8 runs of the full suite in a row at a load of 12 to 146. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… chains and anchors Answers runner-r1 (R1-R8) and the runner side of the pre-registration and report reviews: - A failed read of the guard's or the power log is tried five times; one that still fails is a runner error, which stops every model's lanes after settling and writing what had ended. A failed memory-pressure reading is no sample. - An attempt still running at an hour is abandoned: a harness-error record (AttemptDeadline) with its spend so far, a stop of cause deadline, and its later spend in late-cost events. An attempt that raises keeps the spend it counted. - Admission takes its time and checks the ceiling under the ceiling's lock. - A resume is accepted only if written after its pause began. - The signal handler only notes the signal; the main loop stops the lanes. - A sampler whose samples fail for five minutes, or whose thread ends, stops every model. - --workers must equal the design's. - Every stop names its cause. The error classes add the litellm subclasses the harness groups by isinstance, explain traces count as calls, and a Timeout on the escalated request after a truncated turn is class model. - Run, admission and host logs are hash chains; anchors.jsonl records every log's head every 15 minutes and at the end, and the design's anchor command (confirmatory_anchor.py) commits and pushes it. - In-flight records are fsynced before their rename. 189 tests pass (tests/test_confirmatory_run.py, tests/test_confirmatory_anchor.py). 19 mutants of the new rules each fail a named test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The docstring and EXCERPT_LEAD's comment said "the booking"; the code keeps the guard's and power log's lines from a day before the host log's start. No behaviour changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test sent its interrupt on a 0.15 s timer, and at load 49 the run had not started the household's first attempt by then, so no attempt was cut short and the test failed. The interrupt now arrives from inside that first attempt, which finishes only once the main loop has stopped the lanes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers the first part of review runner-r2: - The ledger can close. Ceiling.charge counts spend with the lock held and, once close() has run, counts nothing more and keeps the amount apart as unrecorded_usd. AttemptSpend now changes its totals under the ceiling's lock, so the attempt's totals and the ceiling's always agree. - The sampler's failure timer starts on every caught sampling exception (finding 3), so a reader that keeps raising stops the run after the silence limit instead of never. - Guard-log lines carry no offset, so guard_times reads each as both readings of a repeated or skipped local hour (New York repeats 01:00-02:00 on 2026-11-01). A pause opens at its earliest reading and closes at its latest; excerpts keep a line if either reading falls in the window. - ANCHOR_RETRY and ANCHOR_PUSH_TIMEOUT are defined for the anchor changes that follow; LATE_JOIN becomes one overall deadline. The suite has not run on this commit: the next commit runs it and finishes the round (stop_all, end_attempt, runner errors, startup, finish(), the resume race and the anchor push). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers review runner-r2 and finishes f210b6e: - Set and notify every model's stop state before event I/O. Retain ended attempts before pause/stop writes; recover their classes and final records best-effort before runner-error stops, and block new starts during recovery. - Finish against one abandoned-request join deadline. Close pools and host before the ledger, then record late costs, crossings and interrupts before the final anchor. Snapshot unresolved requests under the ledger lock and record their unknown future spend explicitly as a runner error. - Confirm persistent sampler exceptions stop the run and a good sample clears the failure timer. Validate a resume against the current pause under the condition that clears it. - Poll pool startup with supervision and process pending stops before every lane launch. Record startup/restart spans for the report's idle calculation; drain supervisors on runner errors before closing their host and ledger. - Kill the whole anchor command process group at ANCHOR_PUSH_TIMEOUT. Retry failed pushes using ANCHOR_RETRY and sync each outcome, anchor timestamp and push interval. Surface push-log I/O failures and retain final local coverage. - Exercise disk-error recovery, late cleanup spend, unresolved requests, persistent sampler errors, the resume race, startup signals and slow pools, failed pushes and real child/grandchild timeout termination. The report must subtract validated setup spans from lane idle time and compute remote exposure from successful pushed snapshots. Local cadence alone cannot support the draft's claimed fifteen-minute maximum; neither report nor draft is changed here. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers review runner-r3 findings 2, 4, 5 and 6, and orders push times: - Push a synced, atomically installed snapshot of the anchors file. Give each push its own path, including after a thread is abandoned, and keep the remote record named anchors.jsonl. - Try the final push up to three times per set of heads, a minute apart, writing a new anchor for each try without changing the covered logs. - Bound pipe draining after process-group termination and every push join. Record a stuck thread once as an error and reject its late outcome. - Ignore signals before the last cleanup check, then log and anchor a first signal that arrived in the gap. Keep first and second signal handling active throughout retries. - Take anchor cadence and push intervals from the runner's ordered clock. Exercise overlapping pushes, retry exhaustion, escaped descendants, abandoned threads, cleanup signals and a backward wall-clock step. The report's record fields and outcome names are unchanged. Push snapshots are copies, not records; the draft should distinguish final retry exhaustion from the ongoing run's retry cadence, and retries after newly logged signals. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers review runner-r4 finding 1 and tests the remaining clock and retry spacing requirements: - Hold an exclusive, non-blocking lock in the anchor clone's .git directory through the push. Refuse busy helpers and snapshots that do not extend the clone's anchors byte for byte; allow equality and never force-push. Leave a stale index.lock to fail normally. - Keep worker retirement and the next push, but say the retired command may still be running. A helper holding the lock makes later pushes fail as busy; final anchors left local are a deviation. - Exercise older helpers after newer pushes, shared-clone retirement, lock contention, interior edits, equality and stale git locks against a temporary clone and local bare remote. - Test anchor_due across a backward wall-clock step and measure each final retry from the preceding try's end, with a controlled clock. - Clarify that final retries wait a minute after a failed try ends. First and second signal handling stays active only until cleanup ignores signals before its last check; signals during re-anchoring after that check are ignored. State that write/fsync under _pushing can delay supervision and retirement beyond the thread join bound. The report's record fields and outcome names are unchanged. Retired-worker output now identifies the surviving-command risk; helper refusals are failed pushes. The clone's persistent lock file is auxiliary, outside run records. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers review runner-r5 finding 1 and fills the retry and conversation coverage gaps: - Pass the clone's flock descriptor to every Git command. A surviving Git writer, hook or remote helper retains the lock after helper-only death; later helpers still refuse busy clones and non-extending snapshots. Disable fsmonitor startup, whose daemon closes inherited descriptors. Keep ordinary, non-force pushes and leave Git's own index.lock alone. - Kill the real helper after Git has entered pre-commit, pre-push and local pre-receive hooks. Check busy refusals, inherited hook descriptors and unchanged clone file, index, HEAD and remote, then newer pushes and every remote commit's byte-prefix extension. Test that configured fsmonitor is never started. - Pin the registered 900-second anchor interval and 60-second retry wait independently of production constants, citing draft line 218. - Compare fake OpenAI Responses and Gemini JSON conversations against the unchanged treatment harness, including complete requests and calculator inputs. Exercise both through the runner's lanes. Use the existing Gemini 3.1 JSON profile; both confirmatory Geminis retain their tool contracts. - Describe surviving Git lock holders and advisory/custom-command limits. The report's record fields and outcome names are unchanged. Busy refusal output now identifies a helper or Git process as the holder. The existing clone lock remains auxiliary, outside run records. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers review runner-r6's two P3 items: - Narrow the fsmonitor claim to disabling normal index-triggered startup. A configured hook can explicitly start the descriptor-closing daemon; configured background clone writers must retain the locked descriptor without unlocking it. The flock is local, while an external remote's safety rests on the byte-prefix check and ordinary non-force pushes. - Describe credential-cache and other descriptor holders keeping later helpers busy for their lifetimes, including the default 900-second cache. Every try while the lock is retained, including all three final tries, fails as busy and is reported as a failed push. - Block real Git clean filters during add and status, kill only the helper, and check later busy refusals leave file, raw index, staged contents, HEAD and remote history unchanged. Release the writer and verify newer snapshots push with prefix-preserving history. - Double-fork a hook descendant into a new session. Let its hook and helper finish, check the detached holder still prevents later pushes, and verify pushes succeed after it exits. Check each new regression against a temporary helper with inheritance removed only from its target command. Production behavior, report fields and outcomes are unchanged. This adds no dependencies and leaves the report implementation and draft untouched. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Catch recursion failures while reading saved finals, archived attempts, baseline exclusions and the CLI design so deeply nested JSON becomes a refusal naming the file. Include the baseline path in its validation errors, and require the CLI design to be a JSON object before stage initialization. Valid stage decisions stay the same. Saved records and baseline exclusions must refuse decoder recursion failures with a path-bearing StateError. The CLI must refuse unreadable, undecodable and non-object designs with their path and exit 1 before initializing a stage. Invalid UTF-8 must remain a decoding refusal. Synthetic regressions exercise every reader in a subprocess with a timeout and require the path before provider calls; final and archive cases also check preflight. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Check board inputs as strict UTF-8 and required JSON objects before delegating to the pinned harness. Attribute metadata engine checks, CSV parsing, exclusion validation and scenario reconstruction failures to their source files. Preserve absent optional exclusions and return the harness's loaded board for usable inputs. Saved finals and archived attempts must refuse numeric cost overflow with a path-bearing StateError at startup and preflight. Nested fixtures must control the child's recursion limit, derive a deeper payload and require the decoder's RecursionError as the refusal's cause. Allocation exhaustion continues to propagate. Provider-free regressions cover each board decoder, both cost boundaries and valid board results. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Give the CLI option test a real small board so it exercises the runner's input boundary and unchanged harness loader. Reference agreement failures must name both the reference CSV and exclusions JSON because either input can make that validation unusable. Preserve the checkpoint branch and its existing commits. The delivery branch retains two commits while including this fixture update and the paired reference paths. The CLI dry run must parse all model options and name its owed work before provider calls. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hash the bytes used by every board input check and guard the unchanged harness loader. Exclusions existence failures must name the exclusions file. Delegated I/O failures must name their filename or the board folder. Other delegated failures must name files whose checked bytes changed, or the board folder when no change is established, while preserving the original cause. Saved finals and archives must contain an object totals with a present numeric cost_usd. Booleans and other nonnumbers are refused. Costs must convert to finite nonnegative floats. Startup and calculator preflight use this reader; calculator attempt_cost and valid spend arithmetic stay unchanged. Provider-free regressions record 65 failures and 49 passes against 6928b06 before these fixes; final verification follows. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Check every saved final and archive against the pinned none-arm treatment before providers or calculator admission. Model, arm, prompt version, wording, notice, explicit null calculator fields, board outputs and provider turns must match. A completed reply or nonnull answer requires its contract, sent prompt and rebuilt hashes. Empty failed attempts may omit that evidence; any evidence present must match. Cheap finals must be in the draw and retain a predictions object. Preflight loads the checked board lazily to apply these same rules. Final link numbers must be consecutive integers excluding booleans. Archived prefixes retain the report equality rule. Known link fields must match while extra keys and an absent status matching null remain permitted, and earlier prefixes are preserved. Archive classification failures must name the archive. Valid numeric spend, rerun eligibility, interrupted-archive recovery, locking and worker execution keep their existing rules. Keep the minimal state-machine fixtures explicit about their injected treatment validator. Add real-board fixtures with harness-built evidence, conditional failure controls and exact archive/spend resume checks. Record 164 agreement failures and 32 controls against the original reader before these fixes. The final selection passes 315 cases, including legacy assertions updated for the new refusal reasons. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s accept 0300257 checks each saved no-tools record against the pinned none arm, and a cheap model's final must hold a predictions object. complete_preflight_design still wrote bare finals, so every preflight refused each model's first household and five tests never reached their planted fault. The fixture now builds each final through the harness with the records tests' full_record and writes the board files the no-tools loader checks. The harness and ceiling faults use the form a failed call saves: no answer and no prompt evidence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record provider-free regressions for denied history enumeration, saved-evidence metadata failures, hash and decoder buffer disagreement, and board engine failures through calculator preflight and main. Each unusable file or folder must produce a refusal naming its path before providers. Baseline and eligible-pool hashes must bind the bytes decoded, and main must print all collected problems and return 1. The original production code fails all 38 regression cases. Two controls establish that runner startup, its stage check, and the separate report accept object-form variables with the same saved spend. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Enumerate saved histories and calculator evidence explicitly. Missing optional folders retain empty inventories; covered listing and stat failures must name their paths. Preserve sorted JSON histories, saved spend and attempt decisions. Guard exclusions metadata, model discovery, archive overwrite checks, summaries, calculator records and host logs, and the preregistration hash read. Read baseline exclusions and the eligible pool once each. Their hashes must bind the exact buffer decoded with strict UTF-8. Guard both calculator engine reads with the metadata sidecar path. Preflight retains its collected problems, and main prints every problem and returns 1. Valid board engine checks and provider execution retain their existing paths. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explicitly list each candidate calculator stage before checking its literal households child. A covered failure listing that parent must become a preflight problem naming the stage folder. Preserve the same household path, record filter and early refusal once saved calculator records are found. The final regression run exposed this missing parent listing after the first implementation: 39 cases passed and the denied-stage-listing case still failed. Keep the original failing regression and verify the corrected inventory and preflight paths. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three test modules added for this round take descriptive names: test_confirmatory_unreadable_evidence.py holds the folder, listing and stat refusals; test_confirmatory_input_reads.py holds the single-buffer hash and decode checks for the baseline exclusions and the eligible pool, and the saved-variables control shared with the report; test_confirmatory_board_engine.py holds the board-engine refusals through preflight and main. The control's imported report module drops the round number from its name too. No test or production code changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… per record Runner review r16 found two gaps. A no-tools model folder, a calculator arm or a calculator stage that a listing returned was dropped when its stat failed with ENOENT: a dangling link (an optional model's folder on an unmounted volume) or an entry removed after the listing. The report refuses such an entry. listed_is_dir now refuses every stat failure of a listed entry, at startup, in stage_problems and in the preflight. A missing root, households folder or attempts folder, which the code names itself, still reads as empty. history decoded each saved record with one read and hashed it with another, and Stage.archive hashed the runner's final again before the move. read_record now returns the digest of the bytes it decoded; an archive's link carries that digest, and the history keeps each final's digest (the bytes read at startup, or the bytes write_json wrote). Before the move, archive refuses a final whose bytes no longer hash to it. history now lists a model's finals before its attempts folder, as the report does, so a households folder that cannot be listed or searched is the folder both programs name (report review r15 finding 4). New tests: a dangling or vanished model entry (startup and preflight), a dangling or vanished calculator arm or stage (preflight), a file at or a stat failure on an optional model's households or attempts folder (startup and preflight), a second read of an archive or a final that differs from the hashed bytes, an archive link equal to the digest its final was read from, and finals changed after the runner read or wrote them. The board engine test now requires the preflight guard's own line and every saved-treatment line beside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In suites r10, three of 0ae6df5's new cases failed in all three runs: - test_archive_refuses_a_final_changed_after_the_runner_wrote_it: Stage.archive created the attempts folder before it checked the target and the final's digest, so a refused move left an empty attempts folder in a history that had none. Both refusals now come first, and the folder exists only when the move proceeds. - test_saved_record_hash_and_decoder_share_one_buffer[archive] and [final]: the test hashed the files itself after installing the read counter, so its own read counted as a second runner read. It now hashes them before counting. The runner read each record once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… files
Review runner-r17 (sol61, REQUEST_CHANGES) found two gaps between the
runner and the report and three smaller items. This commit fixes each.
- The calculator preflight now refuses what the report refuses.
stage_problems refuses any model's archive that no final lists, and
any final whose action is rerun or pause-rerun. That covers optional
models, and cheap-model archives outside the draw. Startup recovery
still resumes both states.
- Saved records now go through the report's descriptor guard.
read_regular opens with O_NONBLOCK and refuses a descriptor that is
not a regular file, then reads through that descriptor. Before this, a
FIFO with no writer could hang startup while it held the invocation
lock. The archive move's digest checks use the same reader.
- The archive move hashes the moved bytes again after the rename. If a
write landed between the first hash and the rename, it moves the bytes
back and refuses. The docstring states the limit that remains: the
runner cannot see a write to the same inode during either hash or
after the second. Every later reader hashes each archive against its
link.
- New tests:
- a FIFO or device in place of a final or an archive, at startup and
preflight, bounded by a writer that releases a blocked open after
5 s;
- unfinished optional and outside-draw histories;
- a write between the digest check and the rename;
- a target collision that keeps both files;
- a digest refusal that leaves no attempts folder, and the missing
folder after a target-stat refusal;
- the changed-written-final case across every cheap model, which pins
the other models' continuation.
- The optional no-reply control now expects the preflight to refuse its
final, which owes a rerun.
- The read-counting helper also counts read_regular. It patches that
reader only where it exists, so the same cases run against c2ca2dc.
Against c2ca2dc, 14 of the new cases fail: 8 special-file, 1 move-back
and 5 unfinished-history. The collision, folder and continuation cases
pin existing behaviour.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI runs `uv run ruff check .` and `ruff format --check .` with the locked ruff 0.15.2. CI has not run on this draft. Run locally on c2ca2dc's tree, both fail on files this PR adds: four files needed formatting, and four lines ran past 88 characters. This commit formats those four files and rewraps the one string the formatter could not split. Every changed file parses to the same AST as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A mutant check of 274e1c7 found one fault that the tests missed. Its digest read in Stage.archive used the blocking sha256_file. The new case replaces a prepared final with a FIFO after startup. The archive move must refuse it at once by name, within the 5 s bound, and leave no attempts folder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he abandon suites-r12's first runner pass at 0193811 failed test_an_attempt_past_its_deadline_is_abandoned_and_stops_the_model with 0.02 == 0.01, and the next two passes held 730 passed. The test gave each turn 0.2 s against a 0.3 s deadline, so its second turn spent at 0.4 s, and it expected the waiting thread to abandon the attempt inside that 0.1 s. A wake 0.1 s late put the second turn's cost in the attempt's record, which is the spend so far that the runner's rule asks for. The runner was right and the test's expectation assumed a wake time. A plugin that delays the abandon by 0.25 s reproduces the same assertion at the same line. The first turn now spends at once and the second waits for the abandon itself, through a new `hold` key in FakeProvider. The deadline rises to 1 s and the other model's attempts to 0.2 s, which leaves 0.8 s of room against that model reaching its own deadline. The assertions are unchanged. The test passed 25 of 25 runs alone and 25 of 25 with the 0.25 s delay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eline rules Answers review runner-r18. stage_problems dropped the cost history returned and never compared it with spent_before, and the preflight never read the baseline. Calculator spending could therefore start from an understated ceiling, or on a baseline the report refuses, and the report would then withhold the results. stage_problems now sums every model's validated cost and refuses a spent_before below that sum by more than MONEY_TOLERANCE, the report's value. It refuses an unusable value by the report's money rule: finite and nonnegative, with numeric text and booleans counting as numbers there too. check_baseline is the startup check moved unchanged, so startup and the preflight share one rule. Tests: a full preflight against spend in a final, an archive and an optional model; each unusable spent_before; and each baseline fault. The second hash after a rename gets its own FIFO case, because the earlier case attacks the first hash. Two execution cases fail the second hash and the move back, then check that the other models finish, the lock is released and startup recovers the archive. The deadline test now records the join's timeout: its hold on the second turn hides when the abandon came, so a half deadline passed it. The ceiling test waits for the crossing where it slept 0.3 s against an unbounded competitor. The archive docstring now limits later digest checks to surviving links: after an interrupt, startup rebuilds the last archive's digest from the bytes then present. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers review runner-r19 and item 3 of confirmatory-report-r18. The preflight added each cost to a running total, and the report grouped its total by model, so the two programs rounded differently. A spent_before at the tolerance passed one and failed the other, and $2 of saved spend could round away beside a $1e16 cost. history now lists each saved cost, and exact_spend takes their correctly rounded sum, the same in any order and grouping. The report takes the same sum. An intermediate overflow means a total past every float, which the comparison refuses. Startup's spend uses the same sum. Tests: every order of two sets of costs at the tolerance, costs a running total rounds away, and a total past every float; a spent_before given as numeric text or a boolean, which counts as its number in both programs; and a design without spent_before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review runner-r20 found that restoring a running total at startup alone would pass every test. A new case saves costs whose running total stays under the cap and whose exact sum reaches it, and asserts that no owed attempt starts and that the invocation lock comes free. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s release Review runner-r21 found two gaps in the startup test. Rounding each model's costs first escaped it, so a second test saves costs whose subtotals fall one float under the cap and whose exact sum reaches it. The test's lock assertion read bookkeeping, so both tests now take the lock from a second open. `exact_spend`'s docstring limits the claim to a flat list of costs, in any order, and to histories that pass validation. Production code changes only in that docstring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review runner-r22 found that startup's overflow rule had no test. Replacing `exact_spend(costs)` with `math.fsum(costs)` at startup alone would have raised a raw OverflowError and passed every test, because the existing overflow cases reach the preflight or refuse one invalid cost. A new test saves two valid 1.5e308 finals and requires that startup takes infinity for the total, stops every owed attempt and frees the lock. The helper takes the expected total as an argument. Production code doesn't change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
CI's test job failed 22 confirmatory tests on Linux at 3846666; they had passed on the Mac that ran every earlier suite. Fourteen reached a preflight check that compares the installed policyengine-us with the board's engine, and the lockfile installs 2.41.0 where the fixture board names 2.60.4. Eight reached host_power, which calls macOS's pmset. World already fakes the board's engine, so it now fakes the installed version and the power reading too. Its tests no longer read either from the host. A test that wants a different reading still sets it after World, as the lid-closed test does. The eight stubs in the unreadable-evidence tests and the ones in pool_design and engine_fault repeat World's and go. Production code does not change. A pytest plugin that hides pmset and reports 2.41.0 reproduced CI's 22 failures on this Mac at 3846666, by name. With this commit all 772 runner tests pass under that plugin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
World now reports the installed engine and the power reading, so no test could see the checks that read them, and no earlier test had held those checks with the real values either. An installed engine other than the board's must now refuse in the preflight twice, in its own check and through the no-tools treatment check, and again in main's final check, which must keep the preflight's problems. Each test requires that no provider is called. host_power gets its first test, on the four mixes of power source and lid state, with the text pmset and ioreg print on this Mac. Production code does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
30 of 33 tasks
Contributor
Author
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI runs only on pull requests into
main, and #244's base isv2-microcosm-batches, so its code has had no lint or test job. This draft runs them on the same head (3846666), thetests/directory included. It closes when those jobs end, and nothing merges from it.🤖 Generated with Claude Code