[MaxGhenis]

Stacked on #240 (v2-microcosm-batches). Rounds 6 and 7 agreed on the calculator stage at 9b30a77 and 4002f8e. Commit 047c868 adds the no-tools stage's script. Rounds 8 to 10 asked for changes, made in 08d7b18 and 5a9735b, then in c60086b, 248e777 and 4ce7f8e, and then in 2f795ea, 6597adb and 9acc920, which round 11 reviewed. Round 11 asked for four P3 changes (watcher-startup and descriptor-order tests, three wordings), made in a1114fe (tests only) and 89dd946 (docstrings and comments only), which round 12 reviewed. Round 12 asked for one wording, in the pre-registration draft. The report's own round-11 review found that a record nested deeply enough to make Python's JSON decoder raise RecursionError escaped its reader; this runner's reader had the same gap, closed in 4dafb83, which round 13 reviewed. Round 13 asked for three changes: the board files that the harness's loader decoded for the script escaped its refusals or lost their names, a saved cost too large for a float escaped with no path, and the pre-registration draft needed to say that the named refusals exclude running out of memory. Commits 040d206 and 6928b06 made the first two, the draft made the third, and round 14 reviewed them. Round 14 asked for four changes: the harness's reread of the board, after the script's own checks, could still escape with a raw I/O error or an unnamed recursion error; the script accepted saved costs that the report refuses (true, "0.25", null, [] and a missing cost); the differential test's nesting case decoded on the pinned Python; and this description overstated how many new cases use a real board. Commit 8b6c8f7 guards the reread and adopts the report's cost rule. Commit 0300257 applies the report's other checks of a saved no-tools record, which a comparison of the two programs turned up. The new checks refused the bare no-tools finals that five of the calculator runner's preflight tests wrote, so those tests stopped at each model's first household before their planted fault; commit 4fb80d9 builds the finals through the harness instead. The study's scripts repo fixes the differential test, and this description now counts the cases correctly. Round 15 reviewed them. Round 15 asked for four changes: a history folder the script could not list looked empty, so an unreadable optional model's history passed the preflight and added no spend; the script hashed the baseline exclusions file in one read and decoded a second; the calculator runner's preflight and main read the board's engine with no guard, so a malformed sidecar escaped and the preflight lost the named problem the no-tools check had recorded; and this description and the draft claimed more than some tests and guards show. Commit 5e334d9 adds the regressions, d31c1d7 lists saved evidence directly, reads the baseline exclusions file and the eligible pool once and decodes the bytes it hashes, and guards both engine reads, and cccc381 adds a listing the preflight still skipped; 4d93b40 names the new test modules for what they check. Round 16 reviewed them. Round 16 asked for changes: an entry that a listing returned, and whose metadata read then found nothing (a link to nothing, or an entry removed after the listing), still looked absent, so an optional model's folder on an unmounted volume passed with none of its spend; the script decoded each saved record in one read and hashed it in another; some faults would pass every test; and this description and the draft claimed more than the tests cover. Commit 0ae6df5 makes the code changes and adds the tests, and c2ca2dc fixes the three of its cases that failed in the suites: the archive move created the attempts folder before its refusals, and one test counted its own read as the script's. Round 17 reviewed both. Round 17 asked for changes: unfinished optional histories passed the calculator runner's preflight; a saved record that was no regular file passed or blocked; a final could change between its hash and its rename; two archive refusals had no test; and this description and the draft overstated what a refused move leaves behind. Commit 274e1c7 makes the code changes and adds the tests, d3673c3 formats four files with the ruff that the repository's lock pins, 0193811 adds a case for the descriptor guard before the move, and a8226dd makes the deadline test's second turn wait for the abandon, because a loaded host's late wake failed that test once. Round 18 reviewed all four and asked for changes: the calculator runner's preflight compared no saved no-tools spend with spent_before, and it never read the baseline, so either could start a stage whose results the report would withhold; several faults passed the tests; and this description and the draft overstated later digest checks. Commit 4441364 makes the code changes and adds the tests. Round 19 reviewed it and asked for changes. The preflight and the report grouped and rounded saved spend differently, so a spent_before at the tolerance could pass one and fail the other. This description's invariant also claimed more than the differential covers. Commit ceae93f sums every saved cost with math.fsum, as the report's c714369 does, and adds the tests. Round 20 requested narrower wording. It also found that a running total restored only at startup would pass every test. Commit 3c80377 tests that startup stops at the correctly rounded cap. Its production diff changes only a docstring. Round 21 reviewed it and asked for narrower wording here and one count. It also found two test gaps: a startup that rounded each model's costs first would pass, and the test read the lock's bookkeeping. Commit 58ba64b tests the sum across models and the lock's release. Its production diff changes only a docstring. Round 22 reviewed it and asked for narrower wording here. It also found that startup's overflow rule had no test. Commit 3846666 adds that test and changes no production code. Round 23 agreed at that head, with one wording fix in the draft. CI's test job then failed 22 runner tests on Linux, where the lockfile installs policyengine-us 2.41.0 and pmset does not exist. Fourteen reached the check that the installed engine matches the board's, and eight reached host_power. Commit 2ee8acf makes the tests' World fixture report policyengine-us 2.60.4 and AC power with the lid open. Commit 0ab2b1e adds 6 tests of the checks that read those facts. Neither changes production code, and the job passes at 0ab2b1e. Round 24 agreed at that head, with four wording fixes. The run waits for the operator to freeze the pre-registration this PR implements and for Max's go.

scripts/confirmatory_run.py runs the v2 calculator-interface study's confirmatory calculator stage as its draft pre-registration specifies, and writes the run records the confirmatory report checks before it scores anything. It runs the treatment harness unchanged: tool_pilot_argon.py and policybench here are 6b00a6f's, and the runner refuses to start if they differ.

Nothing here runs a model. The confirmatory run waits on the frozen pre-registration and Max's go.

What it does

• Lanes. Two per model. Households alternate between them in draw order. A lane runs a household's arms in the order drawn for it (stream order/<model>), then its reruns in passes, and only then decides its next household. The six models run at once.
• One engine pool per model. The pool starts once, with no features of its own. Each call carries its arm's features in the harness's call frame. In development, starting a pool per arm cost 2.8 to 3.6 minutes each time.
• Admission. Each model has a cap and a reserve per household under way. After one refusal, every later decision is a refusal. A decision waits while a pass's reruns remain undecided, so the runner always knows whether a household has finished.
• The hard ceiling. Overall cap plus $25, counted over every model through the harness's Spend. When spend crosses it, no attempt starts, conversations under way end budget_stop, and every model's log gets a ceiling event. Admission checks the ceiling and takes its time under the ceiling's lock, so it decides no household after the crossing.
• The attempt deadline. The runner abandons a conversation still running an hour after it started (the longest development attempt took 714 s). Its attempt gets a harness-error record, AttemptDeadline, with the spend so far, and the model's lanes stop with cause deadline. The conversation ends budget_stop at its next turn; its spend after the abandon, which can come after the hour, still counts toward the ceiling and goes to the run log as late-cost events. An attempt whose conversation raises keeps the spend it counted.
• Failures. Each attempt gets a class from its record and the host evidence, in the report's precedence, and an action from its class. The class lists hold every litellm subclass the harness groups by isinstance, explain traces count as calls, and a Timeout on the escalated request after a truncated turn is class model. An operator failure pauses the model's lanes until confirmatory_run.py resume, accepted only if written after the pause began. A harness failure stops them, and so does one inconclusive error that ends runs in more than five households. The runner saves replaced attempts, and fsyncs in-flight records, before their reruns.
• Stops name their cause: harness, inconclusive, deadline, pool, runner-error, interrupt or sampler (ceiling events have cause ceiling). An exception in the runner itself stops every model's lanes after settling and writing what had ended. The signal handler only notes the signal, and the main loop stops the lanes.
• Host log. A sample every 30 seconds, with the monotonic clock beside the wall clock, then the guard's and the power log's lines for the booking. A failed memory-pressure reading is no sample. The runner tries a read of the guard's or the power log five times before it counts as a runner error. A pass's classes wait until its host evidence is complete, which takes at most about 90 seconds. A sampler whose samples fail for five minutes, or whose thread ends, stops every model.
• Chains and anchors. Every line of the run, admission and host logs carries prev, the sha256 of the line before. Every 15 minutes and at the end, anchors.jsonl gets every log's line count and head and every manifest's sha256, and the design's anchor_command runs on its own snapshot of the file (so no push sends a line half written) in the background. scripts/confirmatory_anchor.py is that command: it commits the snapshot to a clone and pushes it without forcing, so a remote outside the run holds each anchor's time. It holds an exclusive lock in the clone and passes the locked descriptor to every Git command it runs, so a helper, hook or remote helper that outlives a push makes later pushes fail as busy, never writing beside them. It refuses a snapshot that does not extend the clone's file byte for byte. The runner kills a push still running after 5 minutes with its process group, and it pushes the final heads up to three times, a minute apart. Each push's anchor, times, return code and outcome go to anchor-pushes.jsonl.
• The run's end. The runner waits up to 60 seconds for abandoned conversations, closes the pools and the host log, then the spend ledger. The runner logs late costs, a crossing of the ceiling and any signal by then before the final anchor, and it logs a request still running when the ledger closes as late-unresolved with the spend it had counted. After its last check the runner ignores signals, checks once more, and anchors again if a first interrupt arrived in between.
• Setup spans. The runner logs each pool start and restart as a span, which the report's idle check sets aside.
• Workers. --workers must equal the design's engine workers per model.
• Logs from the start. Each model's run and admission logs exist from its manifest on, so a model that never decides (the ceiling crossed first) still has both.
• A clock that never goes back. The host's wall clock steps back by up to about 0.1 ms, even within a thread. The study's clock probe saw 29 backward steps in 177 million readings by twelve threads, the largest 99 µs, at a load of about 150. The run records order attempts, stops, admission decisions and the ceiling's crossing by their times, and an attempt's end and the next start can be microseconds apart. So clock() returns time.time(), never earlier than a time it returned before, and the runner takes each such time under the lock that orders it. The harness's request times stay raw. The report allows them 0.1 s of slack, widened by the largest backward step the host log shows.

The no-tools stage

scripts/confirmatory_no_tools.py (047c868) runs the stage before the calculator stage: the three cheap models answer every drawn household without tools, and their answers only flag references for audit. It calls the harness's run_household directly with each model's none profile and a tool function that raises, so the harness's own resume, which folds earlier attempts into their status and totals, is never used.

• Threads. Each model runs in its own thread, one household at a time in draw order.
• Classes and actions. The calculator runner's classify and action, unchanged, with no host evidence. A provider or none attempt reruns at once; an operator attempt pauses its model until the operator runs the script again; a harness attempt stops its model; a ceiling attempt stops every model. At most two reruns.
• Attempts. Before a rerun, the script moves the final record byte for byte to households/attempts/<household>-<n>.json, and the new final lists every replaced attempt (number, file, sha256, status, class, reason). Since 0ae6df5 the script decodes and hashes each saved record from one read, an archive's link carries the digest of the bytes it decoded or wrote, and before the move it hashes the final again; a final whose bytes differ stops that model with a StateError naming the file. The target-exists and digest checks precede the attempts folder's creation, so either refusal leaves the history on disk unchanged (c2ca2dc). After the rename the script hashes the moved bytes again; when a write landed between the first hash and the rename, it moves them back and stops that model the same way (274e1c7). It cannot see a write to the same file during either hash or after the second. Later readers verify surviving archive links and refuse mismatches. If no later saved record holds the last archive's link, startup rebuilds it from the bytes then present. The script resumes from these files alone, refuses prior_attempts, numbering gaps, links that disagree with the files and archives no rule reruns, and resumes an archive left without a final as the next attempt. It lists the stage's runs folder and each model's final and attempts folders itself (d31c1d7), a model's finals before its attempts folder, as the report lists them (0ae6df5). One of these folders that does not exist, a link to nothing included, holds no records. A folder it cannot list or whose metadata it cannot read, a final or attempts path that is no folder, and a model entry the runs folder's listing returned whose metadata it cannot read, a link to nothing or an entry removed since the listing included (0ae6df5), each give a StateError naming the path, before any provider call, so a failed listing or metadata read cannot hide saved records or their spend. It opens each saved record without blocking, as the report does, refuses a FIFO, a device or another special file at once, naming it, and still follows a link to a regular file (274e1c7). Before any provider call it refuses, with a StateError naming the file, a saved final or archive that it cannot read or decode: each must be a JSON object in strict UTF-8, and the script refuses one nested too deeply to decode the same way, and one whose totals is no object, or whose totals.cost_usd is missing, null, a boolean, a string, a list or another non-number, or is negative, non-finite or too large to convert to a float: the report's own rule (8b6c8f7). It also checks each saved record against the pinned none arm by the report's treatment rule (0300257): model and arm, prompt version, wording and notice hashes, null calculator fields, the board's outputs and each turn's served model, and, for a record with a completed reply or an answer, its contract, sent prompt and prompt hashes, which it rebuilds through the harness. A cheap model's final must name a drawn household and hold a predictions object, and a final's attempt links must number 1, 2 and so on, as integers. The calculator runner's preflight reports each of these as a problem naming the file.
• Spend. Every saved record's cost, final and replaced, of every model folder, counts at startup, through the correctly rounded sum of those costs (ceae93f). All threads share the $5 cap, and the script checks it before each attempt and, through the harness, before each turn. A saved ceiling attempt stops later invocations. An attempt that raises keeps the spend it counted. Concurrent requests and requests within a turn, among them a request's retries and the harness's unchecked resend at a larger output budget after a reply cut short for length, can take the total past $5.
• Baseline exclusions. The design names baseline_exclusions ({path, sha256}): the board's exclusions as they stood before the stage's first call. Before any provider call, the script refuses to start unless that file hashes as declared and has the form the report reads: an object whose exclusions list holds one object per cell, with nonempty scenario_id and variable strings and no cell twice ({"exclusions": []} when empty). The script reads the file once, checks the hash of those bytes and decodes the same bytes (d31c1d7), all before it takes the invocation lock or loads a board, and it refuses a baseline file that nests too deeply, is a JSON array or scalar, or holds invalid UTF-8, naming its path. A design the CLI cannot read or decode, or one that is not a JSON object, gives the usual cannot-start message naming the design's path, with exit 1. Before the harness's unchanged loader reads the board, the script checks each board file itself: the metadata sidecar, reference_outputs.csv, reference_exclusions.json and scenarios.csv with each scenario's JSON. A file it cannot read, a CSV that is not strict UTF-8 or that pandas cannot parse, JSON that is not an object in strict UTF-8 or that nests too deeply, and a sidecar whose engine differs from the design's give the same cannot-start message naming that file, with exit 1. A board with no exclusions file has none, as in the harness, and the same refusal covers the check of whether that file exists. The harness's loader then reads the board again (8b6c8f7): an I/O failure there names the file the error names, or the board folder when it names none, and a covered non-I/O failure (a ValueError, TypeError, AttributeError, KeyError, RecursionError, OverflowError or SystemExit) names each board file whose bytes changed after its check, or the board folder when none changed; any other exception can end the script without naming a file. None of these refusals covers running out of memory: an allocation failure while reading, hashing or decoding these files can end the script with an error that need not name the file, always before any provider call, and the script releases the invocation lock first.
• One invocation at a time. An exclusive, non-blocking flock on .confirmatory-no-tools.lock in the stage's runs folder, held for the whole invocation, dry runs included. A second invocation refuses.
• Interrupts. For the run's duration the main thread handles SIGINT and SIGTERM and restores the previous handlers afterwards. The first signal's handler takes no lock and raises nothing beyond allocation failure: it records the signal and writes a byte to a nonblocking pipe. A watcher thread reads it and sets the interrupt event under a short admission lock, which archive moves and attempt admissions also take for the interrupt check and the move or the admission alone. A move or attempt admitted before the watcher sets the interrupt goes ahead; the script refuses every later one. Provider calls run outside the lock. Attempts already admitted finish and save, and the script prints each model's state and the spend. A signal received before the script restores the handlers makes the exit 1, even when every model finished. The interrupt is separate from the budget stop. A second signal takes no lock: it writes to stderr without blocking and exits with 128 plus the signal number. The stage loses every unsaved attempt then under way, with its spend; saved finals, archives and the spend they record survive.
• Failures. An exception escaping execute, a thread that fails to start included, sets the interrupt under the admission lock and waits for every started thread before the script restores the handlers and releases the invocation lock. The script tracks each thread's start and finish with its own events, resumes a wait cut short by another exception, and gives a thread whose start() raised 10 seconds to begin; one that begins later returns before touching a file. It tracks the watcher thread the same way. It restores the handlers before it closes the watcher's pipe. The original exception propagates. An exception caught in a model's thread marks that model error and does not set the interrupt; the other models go on, and the exit is nonzero.
• Exit. 0 only when every selected model has a standing final for every drawn household. --dry-run lists the standing households and the next attempt owed, with no provider call.

The calculator runner changes in its preflight (5a9735b) and, since d31c1d7, in its listings of saved evidence and in main. It calls stage_problems, which calls no provider. It applies the script's own history and treatment checks, which classify every saved final (since 248e777, an optional model's too) and, since 0300257, refuse a cheap model's final outside the draw. It loads the board, with the script's checks, when the first saved record needs it (0300257), and it refuses to start while any no-tools history fails them or any cheap model lacks a standing final for a drawn household. Since 274e1c7 it also refuses, as the report does, any model's final that the script would rerun, at once or after the operator's pause, and any archived attempts that no final lists. Startup accepts these histories for recovery; execution resumes only the selected models' drawn households. Since 4441364, like startup, the preflight checks the baseline exclusions file, and it refuses a spent_before that cannot convert to a finite, nonnegative number or that lies more than the report's $0.000001 under the spend the saved records hold, every model's included. For histories that pass validation, the preflight and the report each sum every saved cost with math.fsum. Both map an OverflowError to infinity and refuse that total (ceae93f). It also refuses no-tools and calculator runs roots that are the same folder or where one contains the other. It reads the eligible pool once, checks the hash of those bytes and decodes the same bytes; a pool it cannot read or decode is a problem naming the pool's path. A board engine it cannot read is a problem naming reference_outputs.csv.meta.json, and main guards its own engine check the same way, prints every problem under "The run cannot start" and exits 1 before it builds a run or calls a provider. A saved final, run log or host log that the preflight or the summary cannot list or examine gives a problem or refusal naming the path. An entry that its listing of a model's arms or an arm's stages returned, and whose metadata it then cannot read, a link to nothing included, is a problem naming the entry (0ae6df5).

Invariants

The tests' oracle (run_problems in tests/test_confirmatory_run.py) checks these on every fake run, and the property test checks them on random mixes of planted failures:

• Every attempt has one run-log line, and every record and replaced-attempt file hashes as its line logs it.
• Each logged class and action equals the one recomputed from the record and the host evidence.
• Each lane runs its households in draw order. Each household runs its drawn arm order, then its reruns in passes. No two attempts on a lane overlap. No arm has more than two reruns.
• Each admission decision equals the rule applied to the finished households' spend and the households under way. A household a stop cut short is under way until the stop. After one refusal, every decision refuses.
• clock() never returns a time earlier than one it has returned, including under a wall clock that steps back (test_the_clock_never_goes_back and test_a_wall_clock_that_steps_back_keeps_the_run_procedure). The runner writes an attempt's line when its household ends, and a ceiling event carries the crossing's time, so timestamps can appear out of order in the run log. Every check that orders events reads their times, never their line order.
• Every record's request times lie within its attempt, to within 1 s.
• No attempt starts during a pause, after a stop, or after the ceiling's crossing.
• Conservation: the ceiling's total equals the spend before the stage plus every record's cost plus every late-cost event.
• Each log is an unbroken chain from its seed; each anchor matches the logs' prefixes at its line counts; the last anchor matches every log whole, and its push returned 0.
• Every stop names a cause. A stop of that cause follows each harness, deadline and inconclusive trigger within 5 s, no attempt starts after a harness or deadline trigger, and every stop of those causes has its trigger.

For histories that pass validation, the preflight and the report sum the flat list of saved costs with math.fsum, so their total holds for any order of those costs. test_the_exact_sum_ignores_order and the preflight's permutation test check that total. The differential requires both programs to reach one spend decision for generated costs in three cheap finals, an archive and an optional final. It also checks the listed boundaries, declaration forms and baseline files.

Tests

• tests/test_confirmatory_no_tools.py (205 tests) and tests/test_confirmatory_no_tools_records.py (200), fake household implementations, plus dedicated conversations through the real harness: draw order and one thread per model; a rerun links the unchanged archived bytes; a third provider failure stands with two archives; operator pause and resume; harness and budget stops that persist; a crash after the archive move; spend carried across invocations and optional models; refusals of foreign, gapped and corrupted histories; an attempt that raises keeps its spend; the cap agrees with the report's. Commit 08d7b18 adds a real os.kill(os.getpid(), signal.SIGINT) from inside a fake provider call (the attempt saves, no later household starts, the exit is 1, the script restores the handlers), a second Stage refused while the first holds the lock, both baseline refusals, archive identity and rerun eligibility, a harness stop at attempt 3, and a conversation through the real run_household that ends at its next turn after another model's spend crosses the cap. c60086b to 4ce7f8e add real signals between the interrupt check and an archive or an attempt (both refused); a second thread that fails to start while the first holds a fake provider (an independent flock probe shows the invocation lock held until the first finishes); a second signal while a worker holds the output lock (exit 128 plus SIGTERM within the deadline); optional and off-draw finals that the script cannot classify, refused at startup and by the preflight; and baseline files that are a hashed bare list, invalid JSON or missing a field. Commit 2f795ea replaces the handler's lock with the watcher and adds tests that run each signal path in a bounded subprocess: a real SIGINT inside the cleanup's own Event condition (the worker saves, the handlers stay installed and the invocation lock held until it does); exceptions injected once and twice into the cleanup's waits; a start() that raises before or after the native thread exists, within and past the start grace; a signal while a model's thread holds its admission (with the lock removed from the worker or from the watcher, all four fail); a second signal while the watcher waits for admission or while a worker holds the output lock; and a signal after every model finished (exit 1). The tests that slept 50 ms now wait for the interrupt event, and the forced-exit deadline is 30 s. Commit 9acc920 fixes a race in one new probe: its control thread could exit the process before the main thread's handler did (at 6597adb it failed that way in two of three runs: once in run 2, twice in run 3). The two forced-exit probes then pass five more runs in a row. a1114fe adds six cases in bounded subprocesses: the script must restore the handlers before either of the watcher's pipe descriptors closes, and a real SIGINT after a new file takes a descriptor's number must write nothing to its new owner (with the teardown order reversed, both fail); a watcher whose start() raises before or after the native thread exists, or whose target begins after the start grace (the original error propagates, the script releases the handlers and descriptors, and a late target returns without reading a closed descriptor); and a first SIGINT after every model finished (exit 1, every final saved, no interrupt set and no announcement). All six pass on the script at 9acc920, which needed no change. Commit 89dd946 qualifies the docstrings and comments on the handler and the cleanup's waits; its executable code is identical to 9acc920's. Commit 4dafb83 adds 12 cases: a saved final, an archive, a baseline file and a design that each nest too deeply, hold a JSON array or hold an invalid UTF-8 byte inside a string, and the script must refuse each, naming its path. Each case runs in a bounded subprocess. At 89dd946 all four nested cases escaped as RecursionError, and the baseline and design refusals did not name their files (8 failed, 4 passed with the new tests on the old script). Commits 040d206 and 6928b06 add 16 cases, twelve on a small real board and four on saved costs: the script must refuse a metadata sidecar, an exclusions file and a scenario's JSON that each nest too deeply, hold a JSON array or hold an invalid UTF-8 byte, and a reference CSV with an invalid byte, naming the file each time (10 cases); it must refuse an oversized saved cost in a final and in an archive, naming the file, at startup and in the preflight (4); and a valid board, with an empty exclusions file and with none, must load through the harness exactly as the harness loads it (2). At 4dafb83 the 14 fault cases fail and the two valid boards pass. The CLI test now runs on a real board. Commits 8b6c8f7 and 0300257 add 310 cases. In the first file, the harness's reread fails with an I/O error, or meets a sidecar or a scenario's JSON rewritten after its check to JSON nested too deeply, and the exclusions file's existence check fails; each must stop the script naming the file (or the board folder, when the error names no file), with no provider called. Saved costs of true, "0.25", null, [] and a missing cost, in a final and in an archive, the script must refuse each, naming the file, at startup and in the preflight. The new second file builds saved records with the harness on a real board and changes one treatment field, attempt link or prediction at a time: a change the report refuses, the script must refuse at startup and in the preflight, and a change the report accepts, the script must accept. At 6928b06, 229 of the new cases fail and 81 controls pass. Commits 5e334d9 to 4d93b40 add 40 cases in three new files. tests/test_confirmatory_unreadable_evidence.py makes listings of saved evidence fail (the no-tools runs folder, a model's final and attempts folders, and the calculator runner's model, arm, stage and households folders) and metadata reads fail (those folders, a listed model entry, the run and host logs, an archive's target and the board's exclusions file), at the script's startup, in the calculator runner's summary and through the full preflight, and requires a refusal or problem naming the folder or file, with no provider called. tests/test_confirmatory_input_reads.py replaces the baseline exclusions file's and the eligible pool's bytes between the hash and a second read, with invalid JSON and with an invalid UTF-8 byte, and requires the refusal; it also saves a record whose variables is an object, which the report accepts, as a final and as an archive, and requires the script to accept both. tests/test_confirmatory_board_engine.py runs the real board_engine against a malformed and an unreadable sidecar through the preflight and through main, and requires the sidecar's name, no provider call and, for main, exit 1 with "The run cannot start". At 4fb80d9, 38 of the new cases fail and the two variables controls pass. Commit 0ae6df5 adds 21 cases to these files. In the first: a model entry that is a link to nothing or that vanishes after the listing, at startup and in the preflight; a calculator arm or stage that does the same, in the preflight; and a file at, or a failed metadata read on, an optional model's final or attempts folder, at startup and in the preflight. Each requires a refusal or problem naming the entry, with no provider called. In the second: an archive and a final that the script must read once, hashing the bytes it decoded; an archive link whose digest matches the bytes the script read for its final, a control; and a final changed after the script read it, and after it wrote it, which must stop the model before the move. tests/test_confirmatory_board_engine.py now requires the preflight guard's own line, once, and each cheap model's saved-treatment line beside it. Run against the script at 4d93b40, 12 of the new and changed cases from 0ae6df5 and c2ca2dc fail and 13 pass. The tests in 274e1c7 and 0193811 require the preflight to refuse, and startup to resume, an unfinished optional history and archived attempts outside the draw. They refuse a FIFO and a link to a device as a final and as an archive within 5 seconds, at startup and in the preflight. They replace a final with a FIFO before its move, and they put an archive at the target's name and require both files to stay. A refused move must leave no attempts folder, and the script must move back a final changed just before its rename. Run against the script at c2ca2dc, 15 of those cases fail and 7 pass, and a test catches each of 10 mutants of the fixes. The tests in 4441364 put saved spend in a final, an archive and an optional model's final on both sides of the tolerance. They refuse a spent_before that is no number, infinite, null, nonnumeric text, negative or too large to convert, and they plant each baseline fault at the preflight. They replace an archive target with a FIFO after its rename. When a second read or a move back fails during execution, the other models must finish, the lock must come free and startup must recover the archive. The deadline test now records the join's timeout, and the ceiling test waits for the crossing. With nt.MONEY_TOLERANCE written as 1e-6 in the probe's test copy, 15 of the 24 selected cases failed at a8226dd, and a test caught each of 21 mutants. The tests in ceae93f permute saved costs at the tolerance, preserve costs a running total loses and refuse an overflowing total. They also check numeric-text and boolean declarations and refuse a design without spent_before. Against the script at 4441364, 7 of the 17 cases selected failed, and a test caught each of 24 mutants. The test in 3c80377 saves costs whose running total stays under the stage cap and whose exact sum reaches it, and requires that no owed attempt starts and that the lock comes free. A test caught each of 25 mutants. The new test in 58ba64b saves costs whose subtotals per model fall one float under the cap and whose exact sum reaches it, and both tests take the invocation lock from a second open. A test caught each of 27 mutants. The new test in 3846666 saves two valid finals of 1.5e308 and requires that startup takes infinity for the total, starts no attempt and frees the lock. A test caught each of 28 mutants. Commit 0ab2b1e adds 6 tests: the preflight and main must refuse an installed engine other than the board's, and host_power must read the four mixes of power source and lid state. A test catches each of 33 mutants. With the existing files and the three newer ones, 778 pass at 0ab2b1e, one worker, in each of three runs in a row, with the report at 02d25d0 (772 at 3846666, 771 at 58ba64b, 770 at 3c80377, 769 at ceae93f, 752 at 4441364, 730 at a8226dd, 714 at c2ca2dc, 693 at 4d93b40, 653 at 4fb80d9, 343 at 6928b06, 327 at 4dafb83, 315 at 89dd946, 309 at 9acc920, 297 at 4ce7f8e). The two forced-exit probes pass five more runs. For this branch, CI requires a pull request into main, so temporary draft CI run for #244 (closes when the run ends) #248 carries this head to it. Its test job runs tests/ on Linux with the lockfile's engine, except the tests marked slow, and none of this PR's tests carries that mark. At 0ab2b1e the job reports 6,675 passed, 92 skipped and 47 deselected (Actions run 41179707195). CI has no report checkout, so the acceptance controls that need CONFIRMATORY_REPORT skip there and run in the three runs above. This PR changes only scripts/confirmatory_*.py, tests/test_confirmatory_*.py and tests/fake_anchor.py. Commit 060b287 makes two host-log tests (test_the_host_log_samples_and_closes and test_pressure_near_an_attempt_is_host_trouble_once_settled) wait for the samples they need instead of sleeping a fixed time; on a loaded host the 0.02 s sampler had missed both windows.
• tests/test_confirmatory_run.py and tests/test_confirmatory_anchor.py: 248 pass at 4002f8e, one worker. The anchor tests run the helper against a real temporary clone and a local bare remote. They kill the helper inside Git's pre-commit, pre-push and pre-receive hooks, inside a clean filter during git add and during git status, and with a detached descendant holding the lock. Each checks that later helpers fail as busy and leave the clone's file, index, HEAD and the remote unchanged, and the clean-filter and descendant tests each fail when a fault removes inheritance from only the command they target. Tests compare fake OpenAI Responses and Gemini JSON conversations with the unchanged treatment harness's, complete requests and calculator inputs included.
• Earlier, at f5099db: 189 passed (load about 90). f5099db makes the interrupt test independent of timing: it now interrupts from inside a provider call and waits for the lanes to stop, and it passed 5 of 5 at load 91. Each of 19 mutants of the rules added in fdbd411 fails a named test; the one that first survived (a resume kept until a pause) is now caught by a resume with a misdated time. Until fdbd411, 121 tests passed in 8 runs of the full suite in a row at loads from 12 to 146. They cover unit tests (streams, arms, classes, host evidence, the ceiling), fake runs with planted failures, the harness's own conversation under the runner, and the property test. Stacked PRs into v2-microcosm-batches get no CI, since ci.yml runs on pull requests to main.
  • Three oracle faults, fixed in 808ccc8. First, the oracle took a household that a stop cut short to have finished at its last attempt. The runner counts it as under way until the stop, and a new test makes that case deterministic. Second, under load the sampler stalled past the 90 ms scaled host-log gap, so an untouched run got host reruns. The gap is now 0.5 s in the tests. Third, the oracle now allows the harness's raw request times 1 s of slack.
  • The clock. A test reads the wall clock up to 0.2 ms early everywhere. Without clock() it fails every time: attempts overlap, and a household runs before its admission.
• Differential test, in the study's scripts repo (test_confirmatory_differential.py). It runs this runner's Run on a copy of the development board. The harness's run_household is real, and so are this runner's host log and lanes. Only three things are fakes: the model (replying in each API's shape: chat tool, chat JSON, Responses), the engine pool and the host's readers. The confirmatory report's analyse then checks every record and log. At the report's 7cf22e2 against 4002f8e, 27 tests pass, and a 60-example run of its property test passed. At the report's ef297fa against 047c868, 34 pass, among them the no-tools script's own output checked by the report, and a 60-example run of the property test passed. At the report's 0b8e275 against 5a9735b, 39 pass, among them a harness stop and a ceiling stop on the last household, each of which the report refuses, and a 60-example run of the property test passed. At the report's 35005db against 89dd946, 64 pass, among them 15 baseline files that this runner and the report must both accept or both refuse, and a 60-example run of the property test passed. At the report's 31a2964 against 6928b06, 66 pass, among them a baseline file with an invalid byte inside it, which both programs must refuse; its baseline file meant to fail decoding by its depth decoded on the pinned Python (round 14). At the report's 563ef03 against 4fb80d9, 76 passed, among them a baseline file nested 20,000 deep, which both programs must refuse with the decoder's recursion error as the cause, and saved costs of true, "0.25", null, [] and a missing cost, in a final and in an archive, which both must refuse before any provider call, and a 60-example run of the property test passed. At the report's 8df8531 against 4d93b40, 78 passed, among them a no-tools final folder and an attempts folder that neither program can list, which both must refuse before any provider call, each naming the folder, and a 60-example run of the property test passed. At the report's 61e5eef against c2ca2dc, 79 pass, among them a model entry that is a link to nothing, which both programs must refuse before any provider call, naming the entry, and the unlistable final folder, which each must name as itself, and a 60-example run of the property test passed. At the report's 84a9f63 against a8226dd, 84 pass, among them a final and an archive that are FIFOs, which both programs must refuse without blocking, and an optional model's history left by a pause, an interrupt or a crash, which both must refuse, and a 60-example run of the property test passed. At the report's 58e4533 against 4441364, 90 passed, among them baselines and saved spend on which the preflight and the report must give one answer, in listed cases and in a property over saved costs, and a 60-example run of test_any_runner_run_passes_the_report passed. At the report's b52164b against ceae93f, 125 passed, among them saved finals, archives and optional finals at five floats around the spend tolerance. A 60-example run of test_any_runner_run_passes_the_report and of test_any_saved_spend_gets_one_answer_from_both_programs passed. At the report's 1476850 against 3c80377, 128 passed. Both programs must refuse a saved total that overflows and accept a declaration of the largest float when the total rounds to it. A 60-example run of both properties passed. At the report's a816d91 against 58ba64b, 128 passed, and a 60-example run of both properties passed. At the report's 02d25d0 against 3846666, 128 passed, and a 60-example run of both properties passed. At the report's 02d25d0 against 0ab2b1e, 128 pass, and a 60-example run of both properties passed. (At eda54e1 against f5099db, 22 passed, among them the same jittered clock.)
  • It found one defect here: the missing admission log, fixed in ec31e83.
  • It found four rules the report applied more strictly than the procedure. The report fixes all four:
    • an arm left empty by a stop;
    • a rerun cut off by a stop;
    • a household cut short by a stop, which counts as under way until the stop;
    • a truncated, retried request at the ceiling.
  • Six planted runner faults each fail the report, so the test can fail:
    • a skipped provider rerun;
    • admission past the cap;
    • a runner blind to memory pressure;
    • a runner that ignores a harness failure;
    • the ceiling's crossing logged an hour late;
    • a lane that stops deciding.

Plan

• Runner
• Tests
• Differential test (in the study's scripts repo): run this runner on fakes and check its records pass the confirmatory report
• Independent review, round 1 (Opus 5.5): REQUEST_CHANGES, nine findings, all answered in fdbd411
• Differential test updated for fdbd411's API (in the study's scripts repo)
• Round 2 (GPT-6.1 Sol): REQUEST_CHANGES, six findings, answered in f210b6e and 36d553a (a ledger that closes, the sampler's failure timer, ambiguous local hours in the guard log, stop states before event I/O, the run's end, setup spans, push retries)
• Round 3 (Opus 5.5): REQUEST_CHANGES, six findings; the runner's four answered in 00d267a (push snapshots, three final tries, bounded joins, signals at the end, push times on the runner's clock)
• Round 4 (GPT-6.1 Sol): REQUEST_CHANGES; its runner finding, a retired push's command that could still write, answered in 019f842 (the clone lock, the byte-prefix check, no force)
• Round 5 (GPT-6.1 Sol): REQUEST_CHANGES, one finding, Git children that outlived the lock, answered in 9b30a77 (the descriptor goes to every Git command)
• Round 6 (GPT-6.1 Sol): AGREE at 9b30a77, two minor items answered in 4002f8e (docstring wording and three more regressions)
• Round 7 (GPT-6.1 Sol): AGREE at 4002f8e
• The no-tools stage's script (047c868), for the pre-registration review's finding that the stage had no checked rerun procedure
• Round 8 (GPT-6.1 Sol): REQUEST_CHANGES, four findings; the runner's answered in 08d7b18 and 5a9735b (a coordinated interrupt, the invocation lock, the baseline exclusions, the calculator preflight's check on the no-tools stage, more tests), the report's in its own repo
• Round 9 (GPT-6.1 Sol): REQUEST_CHANGES, six findings (an archive or attempt admitted after a signal, a failure releasing the lock under a live thread, the preflight skipping optional and off-draw finals, a second signal blocking on output, the baseline's form, the ceiling wording); answered in c60086b, 248e777 and 4ce7f8e
• Round 10 (GPT-6.1 Sol): REQUEST_CHANGES, four findings (a signal inside the cleanup's own lock could deadlock, a cut-short wait could release the invocation lock under a live thread, admission tests that passed without the lock, two wordings); answered in 2f795ea, 6597adb and 9acc920
• Round 11 (GPT-6.1 Sol): REQUEST_CHANGES at 9acc920, four P3 findings (no regression for the handlers' restoration before the pipe closes or for a watcher that fails or starts late; three wordings: a first signal after the watcher shuts down, unguarded cleanup instructions, the 6597adb failure count); answered in a1114fe and 89dd946
• Round 12 (GPT-6.1 Sol): REQUEST_CHANGES at 89dd946, one P3 wording in the pre-registration draft (the handlers raise nothing beyond allocation failure) and a count in this description (the whole-directory run reached 63% with seven unnamed failures); the report's round 11 found a nested-record escape that this runner's reader shared, answered in 4dafb83
• Round 13 (GPT-6.1 Sol): REQUEST_CHANGES at 4dafb83: board files decoded by the harness's loader escaped or lost their names (P2), an oversized saved cost escaped with no path (P2), and the pre-registration draft needed an allocation-failure limit on its named refusals (P3); answered in 040d206 and 6928b06 and in the draft
• Round 14 (GPT-6.1 Sol): REQUEST_CHANGES at 6928b06: the harness's reread of the board escaped (P2), the script accepted saved costs the report refuses (P2), the differential test's nesting case decoded (P3), and this description overstated the real-board cases (P3); answered in 8b6c8f7 and 0300257, in the study's scripts repo and here
• Round 15 (GPT-6.1 Sol): REQUEST_CHANGES at 4fb80d9: unlistable history folders looked empty (P2), the baseline's hash did not bind the bytes decoded (P2), the preflight lost the named board-engine problem (P2), and claims in this description and the draft exceeded the evidence (P3); answered in 5e334d9 to 4d93b40 and here
• Round 16 (Opus 5.5, in session, while Fleet ops held the Subfleet lanes): REQUEST_CHANGES at 4d93b40: a listed entry whose metadata read found nothing still looked absent (P2), the script decoded and hashed each saved record from separate reads (P3), faults that would pass every test (P3), and claims in this description and the draft that exceeded the tests (P3); answered in 0ae6df5 and here
• Round 17 (GPT-6.1 Sol): REQUEST_CHANGES at c2ca2dc: unfinished optional histories passed the preflight (P2), saved records that are no regular file disagreed with the report or could hang startup (P2), a final could change during its move (P3), two archive refusals had no test (P3), and this description and the draft overstated a refused move (P3); answered in 274e1c7 to a8226dd and here
• Round 18 (GPT-6.1 Sol): REQUEST_CHANGES at a8226dd: the preflight compared no saved spend with spent_before (P2) and never read the baseline (P2), several faults passed the tests (P3), and this description and the draft overstated later digest checks (P3); answered in 4441364 and here
• Round 19 (GPT-6.1 Sol): REQUEST_CHANGES at 4441364: the preflight and the report summed saved spend differently (P2), and this description's invariant exceeded the tests (P3); answered in ceae93f and here
• Round 20 (GPT-6.1 Sol): REQUEST_CHANGES at ceae93f: this description's total and overflow claims needed narrowing and its new sentences a voice pass (P3), and a running total restored at startup alone would have passed (a nonblocking gap); answered in 3c80377 and here
• Round 21 (GPT-6.1 Sol): REQUEST_CHANGES at 3c80377: this description's total claim needed its limit to histories that pass validation, two sentences a split and one count a fix (P3), and a sum per model and the lock's release went untested (nonblocking gaps); answered in 58ba64b and here
• Round 22 (GPT-6.1 Sol): REQUEST_CHANGES at 58ba64b: one sentence here needed a split and seven phrases an actor or a provenance fix (P3), and startup's overflow rule had no test (nonblocking gap); answered in 3846666 and here
• Round 23 (GPT-6.1 Sol): AGREE at 3846666, with one wording fix in the draft
• CI: the test job failed 22 host-dependent tests at 3846666; answered in 2ee8acf and 0ab2b1e; at 0ab2b1e, 6,675 passed, 92 skipped and 47 deselected (Actions run 41179707195)
• Round 24 (GPT-6.1 Sol): AGREE at 0ab2b1e, with four wording fixes, made here and in the draft
• Merge
• The pre-registration frozen (rounds 20, 21, 23 and 24 agreed; the draft still needs the certified build, the engine version and the booking's dates)
• The run, on Max's go

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 31, 2026 5:51am UTC