Repository navigation
Add GPT-6.1 Terra and move the references to policyengine-us 2.60.0 (release, 51 models) - #247
Merged
Merged
Conversation
Argon's finish/freeze driver plus Muse Wonder's reference install, built on a Subfleet lane (run 20261026-141122-pb-terra-driver; 164 tests passed, ruff clean there). Review r1 (GPT-6.1 Sol) requested changes; the fixes follow. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
validate_verdicts checked the judge's requested and reported model, runner and timestamp only when a new model was among a case's wrong models. A case re-opened because the installed build moved its reference, with Terra right and only incumbents wrong, passed with any judge and no timestamp. Every verdict that is not carried over from the seed (and any verdict naming a new model) must now show JUDGE_MODEL requested and reported, the runner the judge step runs (JUDGE_RUNNER, scripts/run_audit_claude.sh) and an ISO 8601 judged_at_utc in UTC. The judge step runs JUDGE_RUNNER, so the two cannot drift apart. Tests cover a moved reference with Terra right and an incumbent wrong under a wrong judge, a missing or non-UTC timestamp and a wrong runner, in the driver and at the freeze, plus a property test of the identity gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The note finish_terra wrote into docs/terra/judge_provenance.json was release 20261014's: one setup-token login, effort xhigh, judges run "two at a time and then one at a time", and a retry history. A Terra run on another listed account or at AUDIT_EFFORT=high would have published those claims. provenance_note now builds the note from the entries alone: how many new verdicts there are, and tallies of the judge model each sidecar reports, the effort and template each records and whether each judge ran isolated, plus what the verifier checks of an isolated entry. Logins stay in the group counts. verify_judge_provenance refuses a record whose note is not the one its entries give, so export and the freeze reject a hand-edited or inherited note. Ports Argon's judge provenance and transcript tests (record keys, tallies, withheld addresses, token login, every transcript defect, the differential check against the GPT-6.1 Sol driver's port) against this driver, and adds tests that runs on each listed account at efforts high, xhigh and max get a note stating only what their sidecars record, a round-trip property for the note's tallies, and refusals of an inherited or edited note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
freeze_terra read stage.json without its early and partial flags and
checked only the export receipt's partial field, so an audited stage marked
early, its receipt's stage hash refreshed, passed the freeze's own gates.
The driver's resume check read the flags by truthiness, so a missing flag
raised KeyError and a 0, null, [] or {} flag passed.
require_release_stage requires both flags to be the boolean False. The
driver's resumed steps and the freeze's verify_upgrade (before anything else
it reads) call it. Tests mutate each flag to True, missing and several
non-boolean values, at every resumed step and at the freeze, and a stage.json
that is not an object; the freeze fixtures now carry the two flags a real
prepare writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each later step refuses a stage that overlaps a read-only input, holds a symlink, has changed staged inputs, or lacks the installed build or its exclusion record. The seed section exercises the real prepare_cases on a synthetic three-household board: a seed the committed digest does not record, an edited or removed carried verdict, prompt partitions moved between lists, a forged manifest the predictions contradict, bind-seed disagreement before and after prepare, and a moved reference without an installed build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A synthetic Terra run on every reference key of release 20261014's snapshot (read from git) folds onto two synthetic incumbents. Each preparation gate gets a failing case: household completeness, unknown or missing outputs, synthetic run states, treatment fingerprints, provider identity, call and usage disclosures, registration and pricing, a stale by_model folder, a short fold and a snapshot off its pins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The judge loop runs against a scripted runner on a prepared seeded stage with the real collect_audit: a verdict the identity, template or model gate refuses, or a hedged one, is set aside and re-judged; three empty or refused passes stop the step; an edited carried verdict or a missing seed binding stops it before any judge runs. The triage port covers missing, invalid and hedged verdicts, flags and unresolved rows, the case count, exclusion agreement, record edits and drops, hand-written restatements, date conventions and wording amendments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A stage with the build installed and bound, Terra's pinned run and an empty provenance record exports a receipt binding the build copy, the run and the prepare-time hashes. Each refusal leaves no payload, data.json or receipt: a staged or committed reference off the build's pins, a run off its input pins, prepare-time hashes that no longer hold, Terra rows that differ from the run, a provenance record the stage does not give, a payload the dashboard schema refuses (also with the real validator), a roster other than the incumbents plus Terra, and scope drift. Early and partial exports never get a receipt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The exact-build pins refuse any changed byte first, and those tests stay. These tests re-pin REFERENCE_BUILD, the sidecar's CSV and actions hashes and the audit's publication record to the mutated bytes, so each of 47 mutations is refused by its own gate: the sidecar revision, the CSV against its changed list, the actions, the traces, the explanations and the exclusion record. An unchanged build re-pinned the same way passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add section H to tests/test_freeze_terra.py: one synthetic stage, built by the real driver steps (triage, write_judge_provenance, export) on the pinned reference build, that freeze_terra.main accepts both as a dry run and as a real freeze. Every gate in main runs unpatched. The stand-ins are the data sources and the freezer's internals, each marked MOCK in the section's header: Terra's run and the incumbent's answers, the base payload, the two exporters' boards, the incumbent baseline rows, the seed digest, the input pins as committed at HEAD, and freezer.main with its serving and registry lookups. The driver's, freeze's and freezer's paths point at a temporary checkout. From that accepted stage, each test makes one change and asserts the freeze refuses it before writing anything: receipt identity and binding, the release spec and date, rehearsal flags, the provenance record and note, payload edits, the export scope check, the strict dashboard validator, roster and counts, verdicts, seed and prompt partitions, a forged wrong-model claim, prepare-time hashes, Terra's rows, input pins, annotation and adjudication record edits, the installed build's narratives, drops, exclusions and scoring, held references, incumbent prediction drift, incomplete run state, treatment off the registry, an answer before the response window, and each defect in what the freezer writes, which verify_frozen must catch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tighten the former unhardened-verdict acceptance test to require refusal. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ation Supply hardened authentication and transcript evidence before each identity mutation, preserving the existing refusal assertions. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reject unlisted verdict directories before export or freeze and pass validated case IDs to the shared tally. Its historical default remains unchanged. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep prediction discovery and writes out of audit consumers, including when a shadow appears after the entry guard. Preserve shared audit defaults for historical releases. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run the complete CLI export path in an independent stage copy and promote its diagnostics, annotations and payload only after every gate passes, with the receipt last. Exercise unresolved triage and downstream refusals against existing stage bytes. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe checks that run after writes begin and distinguish invalidated repository bindings from pointer-only failures and identical repeated freezes. Preserve the freeze write order. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/terra/input_pins.json records the sha256 of the supervised run's predictions.csv and run_state.json, written by finish_terra.py --step pin-inputs from terra-runs/terra/run. prepare copies those files into the stage, and export and the freeze refuse a staged copy whose bytes differ from these pins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release installs reference_audit/2026-10-22-engine-upgrade/reference-build/, which was computed on policyengine-us 2.60.0. pyproject.toml and uv.lock now pin that version (uv lock --upgrade-package policyengine-us; no other package moved), so the install step and the freeze check the build against the engine the repository declares. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add schema enforcement to the fake CLI for StructuredOutput retries within one session. Cover case model bounds and IDs, a single-model case, and the exact schema digest in the sidecar. Keep malformed-answer injection available for the existing extraction and coverage tests. The five regressions fail against the runner at 0805c28. The fixture limits Git discovery above its temporary tree so workspace-local test directories retain the nested-repository isolation checks. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Build the CLI schema in classify_one from the shared audit schema and the case manifest. Require the manifest model count and constrain model IDs to that case so schema refusals can be corrected within the same judge session. Record the exact per-case schema argument's SHA-256 in verdict.meta.json and describe the constraints in the runner header. Keep the shared schema and post-extraction validator authoritative for duplicate IDs and missing manifest cases. Update the isolation expectation and extractor fixture arguments for the per-case schema. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The setup-token opt-in's token (sha256 6a6daf56361b) was declared by its opt-in date in releases 20261013b and 20261014. This release's launcher reads it from agent-secret's pb-judge and declares it that way, so provenance generation found no group for the declared account. Map the second declaration to the same isolated setup-token group, and test that both declarations tally as one login. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generated by finish_terra.py --step provenance; note unedited. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
CI note at 9a3bd31: the one failing test, 🤖 Generated with Claude Code |
Export, the freeze and the publication check ran on 2677b11 (logs in ~/reviews/policybench-gpt-6-1-terra/freeze-run/). This commit holds what they wrote and the release package around it: - the frozen snapshot, annotations, app data and pins for the 51-model board; - the release note, paper, benchmark card, docs/paper.md and sensitivity prose, rendered and re-pinned (scripts/freeze_snapshot.py --rendered-only); two paper sentences on the chunked accommodation now open with words ("Of the 51 models, 10 ..."); - docs/terra/design.md's lead completion record; - scripts/date_terra_judge_verdicts.py, which writes docs/terra/judge_verdicts_20261027.json (the stage verdict of each of the 42 restated decisions) and fills release 20261014's 2026-10-14 triage wave with its commit bf845cb (#221); - finish_terra.py load_build accepts a newer publication-check engine only when it moves no scored output; freeze_snapshot.py knows the 2026-10-22 upgrade's builder; paper_results counts the upgrade summary in words; - test pins moved to the new board, and new tests for the dating script (including a differential test against the bf845cb evidence and a Hypothesis test of judge_disagreements). The release venv was synced to uv.lock before rendering (uv sync --extra docs --extra dev --inexact): it held policyengine-us 2.41.0 and lacked the render extra. uv.lock and pyproject.toml are unchanged. On this tree the full Python suite passed (6,791 passed, 72 skipped) and so did the app's (183 passed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The independent review of the Terra release (r1) asked for four of the changes here. The paper reports the current references' engine and its publication check. It drops the September move's upload time, its 2.17.0 publication check and the October 12 move's timing sentence; those records stay in their audit folders. The policyengine.py compatibility sentence names the current engine. The disclosure tests rebuild the current engine's two sentences from the 2026-10-22 timing record and refuse the earlier moves' time fields in the paper. The PDF test reads a sentence across a page break, where pdftotext puts the page number between two words. The paper's account of the derivation narratives now states what the builds enforce: the writer gets each change's recorded basis, an upgrade build refuses a narrative that omits the new amount, and hand corrections replace narratives found wrong. No check reads every mechanism a narrative states. Contrastive "X, not Y" and "rather than" constructions are rewritten in the paper, the benchmark card, the Claude thinking sensitivity note, the cost comment in config.py and the app's methodology, failure-mode and exclusion text. The card's title is in sentence case. The cost-basis test pins the new override sentence in all four places that state it. The Muse Wonder freeze test pins its October 12 builder by path. Appending Terra's builder had moved the old alias to October 22. The test that checks the mock setup now reads the frozen revision from release commit bc30579 in git, so it no longer skips when the local build is absent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record the frozen manifest core while retaining the export hashes. Finalization requires a completed release stage and strict receipt, accepts only publication timing and rendered paper changes, re-gates changed timing through the installed build, and verifies the manifest core, served pins and app snapshot. Preserve base data and historical versions, bind removed files to null, accept exact finalized outputs on repeated freezes, and replace the receipt atomically after every check passes. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Exercise freeze, publication checks, rendering, re-pinning and repeated finalization while preserving export bindings. Require allowed repository names, an unchanged manifest core, current rendered pins, a valid publication build gate and matching app metadata. Cover strict stage and receipt checks, immutable base data, historical versions, exact finalized hashes, removed-file absence and atomic receipt replacement with unchanged bytes on refusal. Check random subsets of bound inputs through the real gates. Cache only successful reads of immutable base blobs by repository, full commit and path so repeated stage checks reduce host load while mutable inputs and custom readers stay live. Co-Authored-By: GPT-6.1 Sol (Subfleet) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the --finalize command after the rendered-only re-pin and describe what it validates and records in the receipt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CLI export runs triage and export in a disposable copy of the stage and then promotes a fixed list of generated files. Triage also applies the listed reasoning amendments to the adjudication record in that copy, and the receipt binds the amended bytes, but the list left the record out. The stage kept the unamended record, so the freeze refused the staged evidence. Export now promotes every file the receipt binds whose bytes differ from the stage's, before the receipt. A new test rewrites the record during triage and requires the stage to hold every byte the receipt names; it fails without the change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… tag The release's restatement test required every decision's reasoning to equal release 20261014's. Export now promotes the adjudication record that triage amends, so the five reasoning amendments the wording review listed reach the record. The test now applies this release's listed reasoning amendments (the items us_wording-amendments.json appends after release 20261014's) to release 20261014's reasoning, requires each to change its decision's text, and still requires every other field to match. A comment in the Muse Wonder freeze tests spelled that release's tag, which their rule test refuses; it now names the release without the tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release ran again on 2026-10-29 from 4984698 (logs in ~/reviews/policybench-gpt-6-1-terra/freeze-run-r2/): export, the freeze dry run and the freeze, then the publication check, the dated judge verdicts, the render, the re-pin and --finalize. This commit holds what they wrote: - the annotations triage changed, among them 1,522 wording amendments in 117 cases (us_wording-amendments.json); - the frozen snapshot's run data and manifest, the app data, and the rendered paper (PDF sha256 84bdb41e, 17 web files) with its paperSnapshot.json pins; - the publication check on policyengine-us 2.76.0, PyPI's newest at 08:27 UTC: publication_check_2_76_0.csv and sweep_timing.json. It gives the same value as 2.60.0 for all 1,935 scored outputs and moves one excluded output by $10.98. The release note, the benchmark card and docs/terra/design.md now name 2.76.0 where they named 2.74.2; - docs/terra/design.md's completion record for this run, its suite attempts included. On this tree with d2f5b5f's two test repairs, the full Python suite passed (6,838 passed, 71 skipped; 12:16 to 13:39 UTC) and so did the app's (183 passed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release review r2 found that scenario_122's scored Minnesota refundable
credits reference of $0 rests on mn_renters_credit_qualifying_crp, a
Certificate of Rent Paid input no prompt states and policyengine-us defaults
to false. reference_audit/2026-10-29-mn-renters-crp traces the household on
2.60.0 and 2.41.0 and sweeps every Minnesota output: the CRP moves this
output alone. The spec records ruling "crp": the output stays scored in this
release, whose reviewed 2026-10-22 build fixes the exclusion record, and
Max's d2180 decides it for the next release.
The release note now dates its publication check ({checkedOn}, the day the
release read PyPI), and its test recomputes the date from sweep_timing.json
and requires the checked engine to be PyPI's newest at that read. The
benchmark card drops the superseded 2.17.0 PyPI check, as round 1 did for
the paper.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
25d779d recorded scenario_122's CRP ruling as a key in docs/terra/spec.json. install-references binds the spec's hash in stage.json (references_installed and exclusions_installed), so the freeze dry run refused the stage: "stage.json's references_installed is not the installed build's record". The ruling changes no exclusion, adjudication or triage decision, and no driver code reads the spec's rulings, so the spec goes back to the bytes the install bound (b2679418) and docs/terra/design.md records the ruling beside the r24 cells'. The audit's README now points there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Release review r2 noted that docs/benchmark_card.md kept the 2.17.0 PyPI check of 2026-09-29, a release chronology that round 1 moved out of the paper and into the audit records. 25d779d removed the sentence, but test_card_states_the_engines_behind_scored_and_excluded_references still required it, and the round-3 Python suite failed that test alone (1 failed, 6,837 passed, 71 skipped; freeze-run-r3/suite-python.log). The test now requires the card to leave that check out and drops its 14:58 UTC read time from the clock times the card may show. It still requires the audit record's timing file to name 2.17.0 as the newest release at its PyPI read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release ran a third time on 2026-10-29 (logs in ~/reviews/policybench-gpt-6-1-terra/freeze-run-r3/): export at 25d779d, then, after 3654e5d restored the bound spec, the freeze dry run and the freeze, the publication check, the dated judge verdicts, the render, the re-pin and --finalize. This commit holds what they wrote: - the annotations export changed, with the stage's 1,573 wording amendments: round 2's 1,522 and 51 more in seven cases, each answering a finding of release review r2 (us_wording-amendments.json); - the frozen snapshot's run data and manifest (payload 5de82780), the app data, and the rendered paper with its paperSnapshot.json pins; - the publication check on policyengine-us 2.78.0, PyPI's newest at its 16:52 UTC read: publication_check_2_78_0.csv (replacing publication_check_2_76_0.csv) and sweep_timing.json. It gives the same value as 2.60.0 for all 1,935 scored outputs and moves one excluded output by $10.98. The release note's checkedOn and checkedEngine, the benchmark card and docs/terra/design.md name 2.78.0 where they named 2.76.0; - docs/terra/design.md's round-3 completion record, the first dry run's refusal and round 2's temp leftovers included. The full Python suite on this tree failed one stale test (6,837 passed, 71 skipped; 17:07 to 19:19 UTC), which 065202d repairs; with 065202d tests/test_disclosures.py passed (38) and so did the app's suite (183). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… read
Release review r3 (AGREE with five P3 items) found two hand-written
sentences that claimed more than their records:
- The release note said "On {checkedOn} the newest policyengine-us
release was {checkedEngine}", but four releases took turns as the
newest on 2026-10-29. It now says "When PolicyBench checked PyPI on
{checkedOn}, the newest policyengine-us release was {checkedEngine}",
which the timing record's 16:52 UTC read supports. The facts and their
test binding stay as they were.
- docs/terra/design.md said 2.78.0 was "still the newest at 17:02 UTC",
a PyPI read no file records. The clause is gone.
The review's other three items are wording amendments in the stage,
which the round-4 export applies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release ran a fourth time on 2026-10-30 (logs in ~/reviews/policybench-gpt-6-1-terra/freeze-run-r4/): export at 162c166, the freeze dry run and the freeze, the dated judge verdicts, the render, the re-pin and --finalize. Round 3's publication check on policyengine-us 2.78.0 stands, since round 4 leaves the reference build and its engine as they were. This commit holds what the run wrote: - the annotations export changed, with the stage's 1,578 wording amendments: round 3's 1,573 and five more in three cases, each answering a finding of release review r3 (scenario_122's case note and claude-fable-5's row, scenario_064's case note and reasoning, and scenario_120's reasoning); - the frozen snapshot's run data and manifest (payload 67cb0d14), the app data, and the rendered paper with its paperSnapshot.json pins; - docs/terra/design.md's round-4 completion record. The first Python suite stopped on the run's disk guard at 02:40 UTC: the launch lacked PYTEST_ADDOPTS=-o tmp_path_retention_policy=failed, so the basetemp kept every passing test's folder. The lead relaunched the run with that setting, and the suite then passed from 02:43 to 03:47 UTC (6,838 passed, 71 skipped), as did the app's suite (183). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The public board gains GPT-6.1 Terra on 2026-10-31, on Max's go (d2200). The note takes that date and says PolicyBench added the model to its 2026-10-27 snapshot. The tag, the payload and the paper keep their frozen bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
changed the title
MaxGhenis
marked this pull request as ready for review
October 31, 2026 11:43
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Max approved publication on 2026-10-31 (d2200). The data release
dashboard-data-20261027went up at 11:41 UTC. The site reads its board from the pointer in this PR, so it keeps release 20261014 until this PR merges.This release adds OpenAI's GPT-6.1 Terra to
dashboard-data-20261014, taking the board to 51 models, and moves the references from policyengine-us 2.41.0 to 2.60.0. The model card landed in #245 and the reference build in #242.Results
GPT-6.1 Terra answered every output for all 100 households, at $1.2532 in all and a median 22.4 seconds per household.
Four scored references move: scenario_031's SNAP, and
free_school_meals_eligiblefor scenarios 028, 117 and 123. The exact rates of 47 of the 50 earlier models fall by 0.03 to 0.07 points, three rise by 0.02, and Claude Sonnet 5.5 drops from #4 to #6. Every model keeps 1,935 scored outputs of 1,984; the release excludes nothing new and restores nothing.What the release does
docs/terra/input_pins.jsonpins the run's predictions and state.scripts/finish_terra.pyprepares the run, installs the reference build, re-judges the cases the run or the moved references re-open, triages, and exports a hash-bound stage.scripts/freeze_terra.pyrebuilds the payload from that stage's bound inputs and validates its release inputs before writing repository artifacts. It checks the frozen outputs afterward. The shared freezer gains a keyword-only parameter whose default keeps earlier releases' behaviour.federal_income_tax_before_refundable_credits, by $10.98. PyPI's newest reached 2.88.6 by publication, and the release did not run the check again.docs/terra/judge_provenance.json). The release restates 42 committed decisions with their new verdicts (docs/terra/judge_verdicts_20261027.json), and triage flagged no reference.annotations/us_full_run_20260612_policyengine_4_16_1_populace/us_wording-amendments.json). Each amendment changes text, and none moves a class, score or reference.tests/test_notes.pyrecomputes the facts of release note2026-10-31-gpt-6-1-terra-joins-the-board. The note dates the public board's change to 2026-10-31 and names the 2026-10-27 snapshot. The release rescores the sensitivity files on the 51-model board, updates the benchmark card and re-renders the paper (PDF sha256 501fb903).Invariants (tested)
test_release_20261027_note,test_release_20261027_note_claims).Checks
dashboard-data.json67cb0d14,predictions.csv.gzb2fb90d9.Review
GPT-6.1 Sol's last rounds agreed at 8d01221 (driver) and f52886b (frozen release). Two P3 wording items ride the next release. Its delta review of fb5b1d2, which changes only the note's date and first sentence, its tests and the design record, agreed after corrections to this description.
After merge
dashboard-data-20261027Latest, then check the board, the note, the model page and the paper PDF on policybench.org.🤖 Generated with Claude Code