diff --git a/examples/request-routing/README.md b/examples/request-routing/README.md
index 3203b1174..5d6f6eaa9 100644
--- a/examples/request-routing/README.md
+++ b/examples/request-routing/README.md
@@ -1,4 +1,7 @@
-# Route requests without a generation call on every message
+# Route requests to handlers
+
+This example chooses a handler for a request. To train a new decision function
+from reviewed examples, use [Train a router](../train-a-router/README.md).
For a single native classification call with your own handler labels, see
the [direct routing example](./direct). The recorded cascade below uses a
diff --git a/examples/request-routing/cost/README.md b/examples/request-routing/cost/README.md
index d5e3c574e..c35383897 100644
--- a/examples/request-routing/cost/README.md
+++ b/examples/request-routing/cost/README.md
@@ -1,4 +1,9 @@
-# Send each request to a cheap or a strong LLM
+# Historical cost-routing study
+
+This is a separately installable study of cheap/strong LLM routing, with its own
+locked environment and archived answer costs. It is independent of the parent
+handler example and the new [Train a router](../../train-a-router/README.md)
+learning example. The figures below describe the dated recorded study.
## What this shows
diff --git a/examples/request-routing/score.py b/examples/request-routing/score.py
index c73f1ace0..787580ba4 100755
--- a/examples/request-routing/score.py
+++ b/examples/request-routing/score.py
@@ -329,7 +329,10 @@ def recorded_inputs(directory):
require(
type(recorded["conf"]) in (int, float)
and math.isfinite(recorded["conf"])
- and math.isclose(confidence, recorded["conf"], rel_tol=1e-6, abs_tol=1e-7),
+ # Float32 BLAS reductions differ across CPUs (e.g. Apple Accelerate
+ # versus OpenBLAS). Observed max absolute drift on the pinned 3,000
+ # cases is 3.8e-6. Route order and the threshold branch stay exact.
+ and math.isclose(confidence, recorded["conf"], rel_tol=1e-5, abs_tol=1e-6),
"recorded_front_confidence",
)
require((confidence >= THRESHOLD) == (recorded["conf"] >= THRESHOLD), "front_threshold")
diff --git a/examples/train-a-router/.gitignore b/examples/train-a-router/.gitignore
new file mode 100644
index 000000000..46dd3e1d9
--- /dev/null
+++ b/examples/train-a-router/.gitignore
@@ -0,0 +1,7 @@
+.venv/
+__pycache__/
+.ruff_cache/
+.pytest_cache/
+evidence/
+run-output/
+training-input*/
diff --git a/examples/train-a-router/QUICKSTART.md b/examples/train-a-router/QUICKSTART.md
new file mode 100644
index 000000000..fb98fb593
--- /dev/null
+++ b/examples/train-a-router/QUICKSTART.md
@@ -0,0 +1,78 @@
+# Train a router: quickstart
+
+Choose **Qwen3.5-4B** when it should answer correctly and **Qwen3.8-27B-FP8**
+when the extra capability is worth paying for. Measure savings against always
+using 27B and the **quality gap**: how many percentage points of answer accuracy
+you lose. [Full walkthrough and metrics](README.md).
+
+The GSM8K study uses **500 / 2,000 training questions**, **1,000 calibration
+questions** to choose the routing cutoff, and **3,000 test questions**.
+The primary embedding result is **20.9% savings / 0.47-point quality gap**.
+The declared secondary setting gives **33.8% / 0.83 points**: **$6.30 instead
+of $9.51 per 10,000 questions**, with about 83 fewer correct answers. Its gap
+interval is 0.33–1.30 points; results on other workloads can differ.
+
+| Step | What changes | Savings / quality gap |
+| --- | --- | --- |
+| [1. Always strong](README.md#baseline) | Send everything to 27B. | **0% / 0 points** |
+| [2. Laya](README.md#laya) | Use a fixed decision model. | **21.6% / 1.50 points** |
+| [3. LoRA](README.md#lora) | Adapt Laya on the training examples. | **16.4% / 0.80 points** |
+| [4. Embeddings](README.md#embedding-router) | Train two small classifiers on fixed Qwen embeddings. | **20.9% / 0.47 points** |
+| [5. Teacher labels](README.md#teacher-labels) | Compare teacher and reference labels on 1,539 matched examples. | **19.7% / 0.33 points** |
+
+Solver costs are estimates at the recorded Cloud prices. Embedding savings
+include charges shared across each collection batch; Laya savings exclude
+router hosting. Cutoffs target a one-point calibration gap; test gaps can differ.
+
+**Requirements:** Git, [uv](https://docs.astral.sh/uv/getting-started/installation/),
+and an **NVIDIA CUDA machine for LoRA**, locally or rented. The reference uses
+a Modal H100. CPU works for the other steps with the supplied recordings.
+
+```sh
+# From the repository root, install dependencies, fetch evidence, and replay stages 1–5.
+cd examples/train-a-router
+uv sync --frozen --group outcomes
+uv run --frozen --group outcomes python router.py fetch --study evidence
+uv run --frozen --group outcomes python router.py replay --study evidence --output run-output/replay
+```
+
+The default download is about 159 MB. Read `run-output/replay/comparison.md`;
+no API key or model calls are needed.
+
+```sh
+# Prepare a separate folder for your own training runs.
+uv run --frozen --group outcomes python router.py prepare --study evidence --output run-output/my-router
+```
+
+```sh
+# Stage 3: on your CUDA machine, adapt Laya and evaluate its saved routing rule.
+uv sync --frozen --group outcomes --group training
+uv run --frozen --group outcomes --group training python router.py fit-native --run run-output/my-router
+uv run --frozen --group outcomes --group training python router.py evaluate --run run-output/my-router --method native --output run-output/lora.json
+```
+
+Commands default to `--study evidence --population expanded` (2,000 examples).
+Use `--population original` on fit and evaluate for 500 examples.
+
+```sh
+# Stages 4–5: train from saved embeddings and teacher references, then evaluate both.
+uv run --frozen --group outcomes python router.py fit-embedding --run run-output/my-router
+uv run --frozen --group outcomes python router.py fit-teacher --run run-output/my-router
+for method in embedding teacher; do
+ uv run --frozen --group outcomes python router.py evaluate --run run-output/my-router --method "$method" --output "run-output/$method.json"
+done
+```
+
+Create a [SIE Console](https://console.superlinked.com) account and API key to
+use your available free credits on a new routed answer:
+
+```sh
+# Connect to SIE Cloud and answer a new question using your trained embedding router.
+export SIE_BASE_URL=https://api.superlinked.com
+export SIE_API_KEY='paste-your-key'
+uv run --frozen --group outcomes python router.py answer --run run-output/my-router --question 'A rectangle has perimeter 30 and length 9. What is its area?' --target 1 --output run-output/new-answer
+```
+
+Use a fresh output folder for each run. [Live Laya and timing commands](docs/serving.md)
+include its additional inference dependencies. The [self-hosted setup](self-host/README.md)
+uses the same SDK against your own GPU server.
diff --git a/examples/train-a-router/README.md b/examples/train-a-router/README.md
new file mode 100644
index 000000000..2ed7ed933
--- /dev/null
+++ b/examples/train-a-router/README.md
@@ -0,0 +1,426 @@
+# Train a router that saves money
+
+Use a smaller answer model when it is likely to succeed, and pay for the stronger
+model when its extra capability is useful. This example measures **dollars
+saved and correct answers lost**, compared with always using the stronger model.
+
+On **3,000 separate test questions**, the embedding router trained on 2,000
+examples saves **20.9%** with a **0.47-point quality gap** at the primary
+setting. A second setting, also chosen before testing, saves **33.8%** with
+a **0.83-point gap**. That reduces the estimated bill from **$9.51 to $6.30
+per 10,000 questions**, with about **83 fewer correct answers**.
+The two settings trade money against answer quality; neither guarantees
+the same result on a different workload.
+
+Follow these steps, or use the [one-page quickstart](QUICKSTART.md):
+
+1. **Establish the always-strong baseline** [savings **0%**; quality gap **0 points**].
+2. **Use the unchanged Laya decision model** [savings **21.6% before hosting**; quality gap **1.50 points**].
+3. **Adapt Laya with LoRA** [savings **16.4% before hosting**; quality gap **0.80 points**].
+4. **Train small classifiers on Qwen embeddings** [savings **20.9%**; quality gap **0.47 points**].
+5. **Train from a teacher's answers when answer keys are unavailable** [savings **19.7%**; quality gap **0.33 points**].
+
+All code is in this folder. The example's [Hugging Face download](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/71e0d5109e46f8b9c698806e7d3e47b2aa0da87d/evaluations/2026-10-19/train-a-router-arithmetic) contains the
+questions, answer keys, recorded predictions, embeddings, and optional model
+weights. You can reproduce the comparison without paying for those model calls
+again. LoRA is a full training stage and requires an NVIDIA CUDA machine.
+
+## Dataset and the measures of success
+
+We use arithmetic word problems from [GSM8K](https://github.com/openai/grade-school-math).
+For every question, we record answers from **Qwen3.5-4B** and
+**Qwen3.8-27B-FP8**, then check their final answers against the dataset's answer
+key. A router learns when the 27B model is worth its extra cost.
+
+| Group | Questions | Purpose |
+| --- | ---: | --- |
+| Small training set | 500 | Start with fewer labeled examples. |
+| Expanded training set | 2,000 | Repeat the same recipe with more examples; includes the 500-question set. |
+| Calibration | 1,000 | Choose the score cutoff for sending a question to 27B. |
+| Test | 3,000 | Measure the saved models and cutoffs on separate questions. |
+
+The routers see the question text, not the answer key or either model's answer.
+Training labels record whether each answer model was correct. Both models can
+succeed, both can fail, and sometimes only the smaller one succeeds.
+
+We group duplicates and similar question templates before splitting the data.
+No detected family crosses training, calibration, and test. The 2,000 training
+questions had already been used during development; calibration and test use
+previously unused families. These study groups differ from GSM8K's official
+split: training contains its 1,319 official test questions and 681 official
+training questions; our calibration and test questions come from the remaining
+official training split. These are custom routing results, not standard GSM8K
+benchmark scores. The checks do not establish what the foundation models saw during
+pretraining. See the [measurement method](docs/methodology.md) for the complete
+split, grading, and review rules.
+
+- **Answer accuracy:** the percentage of correct final answers. Equivalent
+ numeric forms, such as `1/2` and `0.5`, count as equal.
+- **Quality gap:** always-strong accuracy minus routed accuracy, in percentage
+ points. Dropping from 96% to 95% is a **1-point gap**: 100 fewer correct answers
+ per 10,000 questions. Gains when only the smaller model succeeds count too.
+- **Savings:** the percentage reduction in cost versus always using 27B. A
+ drop from $100 to $70 is **30% savings**.
+- **Cost per 10,000 questions:** the chosen answer-model charges plus known
+ router charges. The report also keeps costs per 1,000 for other traffic levels.
+
+The main comparison chooses cutoffs for a **1-point gap on calibration**, then
+measures the actual test gap. That setting is not a guarantee on new questions.
+The report includes 95% uncertainty intervals and all four declared targets:
+0, 1, 2, and 5 points.
+
+**Primary setting: a 1-point calibration target, evaluated on 3,000 test questions.**
+
+| Method | Answer accuracy | Quality gap, points (95% interval) | Cost / 10,000 | Savings |
+| --- | ---: | ---: | ---: | ---: |
+| Always Qwen 27B | 95.90% | 0.00 (0.00–0.00) | $9.51 | 0.0% |
+| Always Qwen 4B | 91.03% | 4.87 (3.97–5.93) | $2.29 | 75.9% |
+| Laya, unchanged | 94.40% | 1.50 (0.97–2.07) | $7.46 + hosting | 21.6% |
+| Laya + LoRA, 2,000 examples | 95.10% | 0.80 (0.40–1.23) | $7.95 + hosting | 16.4% |
+| Embeddings + classifiers, 2,000 examples | 95.43% | 0.47 (0.07–0.87) | $7.52 | 20.9% |
+| Embeddings + teacher labels, 1,539 examples | 95.57% | 0.33 (0.03–0.67) | $7.63 | 19.7% |
+| Same 1,539 questions, reference labels | 95.30% | 0.60 (0.20–1.03) | $7.41 | 22.0% |
+
+Answer costs use recorded physical token counts and a saved SIE Cloud price
+table; they are Cloud-price estimates. Embedding savings include observed
+embedding credits, shared across the questions in each collection batch:
+about **$0.059 per 10,000 questions**. The one-question timing calls used one
+credit each, equivalent to **$0.10 per 10,000** at that rate; individual calls
+can therefore cost slightly more than the recorded batches. Laya's hosting cost
+is unpriced, so its savings are **before router hosting**. Training and teacher
+labeling are separate one-time expenses.
+
+At the primary setting, LoRA reduces the unchanged decision model's
+quality gap from **1.50 to 0.80 points**, while saving less on answer calls.
+The embedding router then reaches **0.47 points** and saves **20.9%**,
+including its embedding bill. These are the recorded point estimates;
+the intervals show the uncertainty in each quality gap.
+
+**Secondary setting: a 2-point calibration target.** The same saved models
+use a more permissive cutoff, chosen before opening the test answers.
+
+| Method | Savings | Measured test gap, points (95% interval) |
+| --- | ---: | ---: |
+| Laya, unchanged | 26.9% before hosting | 1.77 (1.17–2.43) |
+| Laya + LoRA, 2,000 examples | 29.4% before hosting | 1.53 (0.97–2.17) |
+| Embeddings + classifiers, 2,000 examples | 33.8% | 0.83 (0.33–1.30) |
+| Embeddings + teacher labels, 1,539 examples | 35.2% | 1.03 (0.50–1.53) |
+| Same 1,539 questions, reference labels | 36.7% | 1.13 (0.60–1.67) |
+
+For the 2,000-example embedding router, secondary savings are **33.8%**
+(95% interval **32.0–35.6%**). Its **0.83-point** test gap has an interval
+of **0.33–1.30 points**; this does not establish a guaranteed gap below
+one point. The primary setting remains the main comparison. Both are
+available as saved rules: `--target 1` and `--target 2`.
+
+The full comparison plots retain both training sizes and the controls:
+[primary setting](https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/71e0d5109e46f8b9c698806e7d3e47b2aa0da87d/evaluations/2026-10-19/train-a-router-arithmetic/arithmetic/figures/cost-and-quality-1pp.png)
+and [secondary setting](https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/71e0d5109e46f8b9c698806e7d3e47b2aa0da87d/evaluations/2026-10-19/train-a-router-arithmetic/arithmetic/figures/cost-and-quality-2pp.png).
+
+The headings below show the primary setting, using 2,000 training
+examples except for the matched 1,539-example teacher comparison.
+
+## Setup
+
+Install Git and [uv](https://docs.astral.sh/uv/getting-started/installation/).
+For the complete walkthrough, use **an NVIDIA CUDA machine**, locally or rented.
+The reference LoRA training ran on a Modal H100; a
+[Modal GPU shell](https://modal.com/docs/cli/latest/shell) is one way to rent a
+machine. Run the training commands inside that machine's checkout. The remaining training and
+replay steps work on CPU with the saved embeddings. SIE Cloud supplies the
+embedding and answer models for new requests; the CUDA machine trains Laya.
+
+From the root of a clone of this repository:
+
+```sh
+# Enter the example and install its locked replay and classifier dependencies.
+cd examples/train-a-router
+uv sync --frozen --group outcomes
+```
+
+```sh
+# Download the verified data, recorded results, and embeddings for this example.
+uv run --frozen --group outcomes python router.py fetch --study evidence
+```
+
+`evidence/` contains the downloaded reference material. `run-output/` contains
+your results. The default download is about **159 MB**. Use a fresh output
+directory when repeating a command.
+
+
+
+## 1. Establish the always-strong baseline [savings 0%; quality gap 0 points]
+
+**CPU.** Recompute the comparison from saved answers and routing decisions.
+This makes no model calls and needs no API key.
+
+```sh
+# Recompute costs, answer accuracy, and uncertainty intervals for the saved decisions.
+uv run --frozen --group outcomes python router.py replay \
+ --study evidence --output run-output/replay
+```
+
+Open `run-output/replay/comparison.md`. Always using 27B answers **95.90%** correctly at an estimated **$9.51 per
+10,000 questions**. Always using 4B costs **$2.29**, but accuracy falls to
+**91.03%**: a **4.87-point gap**, or about 487 fewer correct answers per
+10,000. The router aims to keep much of the saving while recovering
+most of those answers.
+
+Every router chooses between the same two recorded answers for each question.
+That isolates the effect of its routing choices from variation in newly
+generated answers. Replaying the published results is a reproduction;
+further experiments need new test questions.
+
+
+
+## 2. Use Laya [savings **21.6% before hosting**; quality gap **1.50 points**]
+
+**CPU for replay; CPU or NVIDIA GPU for new decisions.** Laya is a decision
+model: it scores possible choices instead of generating an answer. Here it
+predicts whether each of the two answer models will be correct.
+
+Start with **Laya, unchanged; costs from 2,000 examples** in the report. Its roughly 421 million
+model weights stay fixed. Training records supply expected answer costs, and
+calibration chooses the cutoff. The baseline uses the same cost estimates as
+the corresponding LoRA run, so their comparison isolates the effect of
+adapting the weights.
+
+The unchanged decision model saves **21.6% before hosting**, but its
+**1.50-point** test gap exceeds the one-point calibration target. It
+provides a concrete baseline for learning the decisions from our own
+answer outcomes.
+
+
+
+## 3. Adapt Laya with LoRA [savings **16.4% before hosting**; quality gap **0.80 points**]
+
+**NVIDIA CUDA GPU for training; CPU for replay.** LoRA, or low-rank adaptation,
+trains small added weight updates while keeping most of Laya fixed. We also
+train its final decision layers. One shared model learns both correctness
+questions.
+
+```sh
+# Create your own training folder while preserving the downloaded reference results.
+uv run --frozen --group outcomes python router.py prepare \
+ --study evidence --output run-output/my-router
+```
+
+```sh
+# On your CUDA machine, install the locked LoRA training dependencies.
+uv sync --frozen --group outcomes --group training
+```
+
+The first training run also downloads the public Laya checkpoint. No SIE API
+key is needed for this stage.
+
+```sh
+# Train and evaluate LoRA with 500 examples, then repeat the same recipe with 2,000.
+for population in original expanded; do
+ uv run --frozen --group outcomes --group training python router.py fit-native \
+ --study evidence --run run-output/my-router --population "$population"
+ uv run --frozen --group outcomes --group training python router.py evaluate \
+ --study evidence --run run-output/my-router --population "$population" \
+ --method native --output "run-output/laya-$population.json"
+done
+```
+
+One reserved portion of the training data, keeping whole question families
+together, chooses how many passes, or **epochs**, to make through the data.
+Selection checks zero through eight epochs, then starts
+again from the original checkpoint and trains on all training rows for the
+chosen duration. Calibration and test do not select that duration.
+
+With 500 examples, LoRA saves **14.9% before hosting** with a **1.17-point
+gap**. With 2,000 it saves **16.4%** with a **0.80-point gap**. At this
+primary setting, adaptation preserves more correct answers than
+unchanged Laya but spends more on 27B. At the secondary setting, the
+2,000-example LoRA reaches **29.4% savings / 1.53-point gap**, compared
+with unchanged Laya's **26.9% / 1.77 points**. The full report retains
+both training sizes and every declared cutoff target.
+
+The reference runs selected one epoch for both training sizes. Selecting,
+fitting, and exporting took **2.4 minutes for 500 examples** and **8.1 minutes
+for 2,000**, excluding environment startup. The allocated H100 lifetime, including
+startup and input scoring, was **18.4 minutes**: about **$1.48** at the recorded
+hardware rate, an estimate rather than an invoice.
+
+
+
+## 4. Train the embedding router [savings **20.9%**; quality gap **0.47 points**]
+
+**CPU with saved embeddings.** Qwen3-Embedding-4B converts each question into
+2,560 numbers representing its text. Two small probability models, called
+**logistic classifiers**, learn from those numbers: one predicts correctness
+for Qwen 4B, the other for Qwen 27B. The encoder stays fixed.
+
+```sh
+# Fit and evaluate the two classifier training sizes using the saved Qwen embeddings.
+for population in original expanded; do
+ uv run --frozen --group outcomes python router.py fit-embedding \
+ --study evidence --run run-output/my-router --population "$population"
+ uv run --frozen --group outcomes python router.py evaluate \
+ --study evidence --run run-output/my-router --population "$population" \
+ --method embedding --output "run-output/embedding-$population.json"
+done
+```
+
+The routing score is the predicted improvement in correctness divided by the
+expected extra cost of using 27B. Expected costs come from training records;
+the router never sees a new question's future answer length or correctness.
+
+With 500 examples, this router saves **18.2%** with a **0.70-point gap**.
+With 2,000 it reaches **20.9% / 0.47 points**: lower measured cost and
+higher answer accuracy than the 2,000-example LoRA run. The corresponding
+secondary setting reaches **33.8% / 0.83 points**.
+
+More data does not automatically increase savings at every setting:
+the 500-example secondary rule saves **43.9%** but loses **1.73 points**.
+Inspect both measures together; a higher saving alone can simply mean
+sending more questions to the cheaper model.
+
+New requests need one embedding call and two small CPU calculations. Adding
+training data does not increase that work. It changes which questions go to
+27B, which can increase or decrease the answer bill. A saved rule that always
+chooses one model skips the embedding call. Classifier fitting took **6 seconds
+for 500 examples** and **11 seconds for 2,000** on the reference CPU worker.
+
+
+
+## 5. Train from teacher labels [savings **19.7%**; quality gap **0.33 points**]
+
+**CPU with recorded teacher answers.** When answer keys are unavailable, ask a
+strong teacher to solve the training questions independently. Compare both
+answer models' outputs with the teacher's reference, then train the same two
+classifiers. The teacher sees neither the benchmark answer nor the two outputs.
+
+The recorded teacher is **GPT-6 Astra**, which supplied 1,539 independent numeric
+references. Both teacher-label and
+reference-label classifiers use exactly those same 1,539 questions, with the
+same settings and training groups. The remaining questions were unavailable;
+they were not removed based on correctness.
+
+Teacher references agree with the answer key on **98.25%** of these training
+questions. The derived correctness labels agree on **98.70%** of cheap-model
+answers and **98.51%** of strong-model answers. These training-label checks
+are separate from the held-out routing comparison. Incorrect teacher answers
+remain in the teacher labels.
+
+```sh
+# Refit the teacher-label classifiers and a reference-label control on the same questions.
+uv run --frozen --group outcomes python router.py fit-teacher \
+ --study evidence --run run-output/my-router
+uv run --frozen --group outcomes python router.py evaluate \
+ --study evidence --run run-output/my-router --method teacher \
+ --output run-output/teacher-comparison.json
+```
+
+The teacher-label router saves **19.7%** with a **0.33-point gap** at
+the primary setting. The same-question reference-label control saves
+**22.0%** with a **0.60-point gap**. At the secondary setting, the pair
+reaches **35.2% / 1.03 points** and **36.7% / 1.13 points**, respectively.
+Teacher labels support a useful router here; these differing cutoffs
+do not establish that teacher labels are better than reference labels.
+
+These commands reuse the recorded references and incur no labeling charges.
+The numeric teacher run used **$6.40** of known API usage; an earlier formatting
+run used **$8.75**. Unknown usage is retained separately, so this is not a
+complete invoice. Calibration and test still use the benchmark answer key.
+Without any existing labels, keep a small independently reviewed sample to
+check quality; agreement with the training teacher alone is insufficient.
+
+## Use your router with SIE Cloud or your own server
+
+Create an account and API key in the [SIE Console](https://console.superlinked.com).
+Use the free credits shown in your account for new model calls. Replaying
+and training from the downloaded records uses no SIE credits.
+
+```sh
+# Connect the SDK to SIE Cloud using the API key from your console account.
+export SIE_BASE_URL=https://api.superlinked.com
+export SIE_API_KEY='paste-your-key'
+```
+
+```sh
+# Select an answer model for a new question and save its answer and observed API charges.
+uv run --frozen --group outcomes python router.py answer \
+ --study evidence --run run-output/my-router --method embedding \
+ --question 'A rectangle has perimeter 30 and length 9. What is its area?' \
+ --target 1 --output run-output/cloud-answer
+```
+
+The result records the chosen model, answer, and API charges. `--target 1`
+selects the saved calibration rule; it does not guarantee a correct answer.
+Use `--method teacher` for the classifiers trained from teacher labels. For
+your adapted decision model, include its inference dependencies:
+
+```sh
+# Load your adapted Laya router on CPU and use SIE Cloud for the chosen answer model.
+uv run --frozen --group outcomes --group training python router.py answer \
+ --study evidence --run run-output/my-router --method lora --device cpu \
+ --question 'A rectangle has perimeter 30 and length 9. What is its area?' \
+ --target 1 --output run-output/lora-cloud-answer
+```
+
+Use `--device cuda` for NVIDIA inference. Omit `--run` to use recorded routers.
+Recorded Laya inference needs the optional weights download; your own trained
+checkpoint already contains its weights. The optional recorded checkpoint
+bundle is about **10.1 GB**:
+
+```sh
+# Fetch the optional decision-model checkpoints for using recorded Laya routers.
+uv run --frozen --group outcomes python router.py fetch --study evidence --weights
+```
+
+The [self-hosted setup](self-host/README.md) supplies a three-model SIE deployment
+for Linux and NVIDIA GPUs. Its validation hardware uses three H100 80 GB GPUs,
+one per model. That full inference setup is larger than the single GPU needed
+for LoRA training. After starting your own endpoint:
+
+```sh
+# Use the same router against the supplied trusted local SIE deployment.
+SIE_BASE_URL=http://127.0.0.1:8080 SIE_API_KEY='' \
+uv run --frozen --group outcomes python router.py answer \
+ --study evidence --run run-output/my-router --method embedding \
+ --question 'A rectangle has perimeter 30 and length 9. What is its area?' \
+ --target 1 --self-hosted --output run-output/self-hosted-answer
+```
+
+Use your deployment's API key if it requires authentication. The flag accepts
+responses without Cloud billing receipts and preserves model checks. It does
+not start the server. Model components and the routed SDK request passed
+Modal GPU validation; the complete Docker Compose stack has not been run as
+one deployment.
+
+## Serving cost, speed, and model size
+
+| Router | Work per routed request | What you train |
+| --- | --- | --- |
+| Laya / Laya + LoRA | Two correctness decisions through one shared 421-million-weight model | Small weight updates and its decision layers |
+| Qwen embeddings + classifiers | One 4-billion-weight encoder call, then two small CPU calculations | Two sets of 2,560 weights and a bias |
+
+The embedding encoder is roughly ten times larger than Laya. Architecture,
+pretraining, and model size all differ; this comparison does not isolate which
+causes a performance difference. Laya is smaller to host. Using the Cloud
+embedding endpoint avoids hosting the encoder yourself.
+
+We timed the same 20 training questions sequentially after one warmup:
+
+| Router and setup | Median | 95th percentile |
+| --- | ---: | ---: |
+| LoRA on a resident Modal H100 | 15.23 ms | 16.00 ms |
+| The same LoRA on a Modal CPU worker | 436.77 ms | 550.16 ms |
+| SIE Cloud embedding + classifiers, including network time | 107.67 ms | 128.58 ms |
+
+The classifier calculation alone took **0.36 ms median**. Native timings
+exclude model loading and HTTP transport. These are different serving setups,
+not equal-hardware or concurrent-throughput comparisons. The one-question CLI
+loads Laya each time; its startup is slower than resident-model inference.
+
+Laya's answer-model savings are the maximum amount you can spend on hosting
+the router before breaking even. A GPU kept idle between requests can consume
+those savings. The embedding API has a per-use charge, already included in
+the embedding-router table; the small classifier's CPU work remains unpriced.
+
+Use the [serving guide](docs/serving.md) to repeat the timing checks and account
+for router hosting. The [methodology](docs/methodology.md) explains the frozen
+decisions, teacher comparison, cost estimates, and uncertainty intervals.
diff --git a/examples/train-a-router/arithmetic_answers.py b/examples/train-a-router/arithmetic_answers.py
new file mode 100644
index 000000000..211151623
--- /dev/null
+++ b/examples/train-a-router/arithmetic_answers.py
@@ -0,0 +1,70 @@
+"""Arithmetic grading with an explicit clock-hour answer contract."""
+
+from __future__ import annotations
+
+import re
+
+import teacher_format
+from solver_answers import extract_final_answer
+
+
+def clock_question(question):
+ request = re.split(r"(?<=[.?!])\s+", question.strip())[-1].casefold()
+ if re.search(
+ r"\b(?:how\s+long|how\s+many\s+(?:hours|minutes|seconds)|time\s+(?:taken|spent|needed|elapsed))\b", request
+ ):
+ return False
+ return bool(
+ re.search(
+ r"\bwhat(?:\s+is|\s+would\s+be)?\s+(?:the\s+)?"
+ r"(?:(?:earliest|latest|starting|finishing|arrival|departure)\s+)?time\b",
+ request,
+ )
+ )
+
+
+def clock_value(value):
+ text = value.strip().strip("$")
+ text = re.sub(r"\\(?:text|mathrm|textrm)\{([^{}]*)\}", r"\1", text)
+ text = text.replace("{:}", ":")
+ text = re.sub(r"\\[,!; ]|~", " ", text)
+ text = re.sub(r"^\s*at\s+", "", text, flags=re.IGNORECASE).strip()
+ text = re.sub(r"\s*o['’]clock\s*$", "", text, flags=re.IGNORECASE)
+ match = re.fullmatch(r"(\d{1,2})(?::([0-5]\d))?\s*([ap]\.?m\.?)?\.?", text, re.IGNORECASE)
+ if match is None:
+ return None
+ hour, minute = int(match[1]), int(match[2] or 0)
+ period = match[3].replace(".", "").lower() if match[3] else None
+ if hour > 23 or (period is not None and not 1 <= hour <= 12):
+ return None
+ return {"hour": hour, "minute": minute, "period": period, "explicit_minutes": match[2] is not None}
+
+
+def grade_reference(question, reference, response):
+ final = extract_final_answer(response)
+ expected = clock_value(reference) if clock_question(question) else None
+ actual = clock_value(final) if expected is not None and final else None
+ if expected is None or actual is None:
+ return teacher_format.grade_reference(question, reference, response)
+ if expected["period"] is None and expected["hour"] <= 12:
+ correct = expected["hour"] % 12 == actual["hour"] % 12 and expected["minute"] == actual["minute"]
+ contract = "The numeric source reference gives a 12-hour clock value without an AM/PM distinction."
+ else:
+
+ def minutes(value):
+ hour = value["hour"]
+ if value["period"]:
+ hour = hour % 12 + (12 if value["period"] == "pm" else 0)
+ return 60 * hour + value["minute"]
+
+ correct = minutes(expected) == minutes(actual)
+ contract = "The source reference specifies a period or a 24-hour clock value."
+ return {
+ "correct": bool(correct),
+ "status": "graded",
+ "final_answer": final,
+ "clock_reference": expected,
+ "clock_prediction": actual,
+ "clock_contract": contract,
+ "original_reference": reference,
+ }
diff --git a/examples/train-a-router/arithmetic_calibration_review.py b/examples/train-a-router/arithmetic_calibration_review.py
new file mode 100644
index 000000000..8be07165f
--- /dev/null
+++ b/examples/train-a-router/arithmetic_calibration_review.py
@@ -0,0 +1,145 @@
+"""Apply an explicit, hash-bound calibration answer-format review."""
+
+from __future__ import annotations
+
+import hashlib
+import shutil
+import tempfile
+from concurrent.futures import ProcessPoolExecutor
+from pathlib import Path
+
+import arithmetic_study as study
+from data import digest, file_hash, write
+
+
+def text_hash(value):
+ return hashlib.sha256(value.encode()).hexdigest()
+
+
+def apply_decision(row, flag, decision, reference):
+ """Only an unresolved comparison with the exact reviewed bytes can change."""
+ arm = flag["arm"]
+ current = row[f"{arm}_grade"]
+ study.require(
+ flag["status"] in study.UNRESOLVED
+ and current["status"] == flag["status"]
+ and decision["id"] == flag["id"] == row["id"]
+ and decision["arm"] == arm
+ and decision["record_sha256"] == flag["record_sha256"] == row[f"{arm}_record_sha256"]
+ and decision["question_sha256"] == text_hash(flag["question"])
+ and decision["reference_sha256"] == text_hash(reference) == text_hash(flag["reference"])
+ and decision["final_answer_sha256"] == text_hash(flag["final_answer"])
+ and current["final_answer"] == flag["final_answer"]
+ and type(decision["correct"]) is bool
+ and bool(decision["reason"]),
+ "Calibration adjudication does not match the exact unresolved comparison",
+ )
+ row[f"{arm}_correct"] = decision["correct"]
+ row[f"{arm}_grade"] = {
+ "correct": decision["correct"],
+ "status": "graded",
+ "final_answer": flag["final_answer"],
+ "base_grade": current,
+ "adjudication_sha256": digest(decision),
+ "method": "Explicit calibration answer-format review before cutoff selection",
+ }
+
+
+def verified_scan(source, run, collection, prices, manifest, workers):
+ # The original function performs every source, acquisition and record check,
+ # and writes a complete scan in a fresh directory, preserving the first scan.
+ with tempfile.TemporaryDirectory(prefix="arithmetic-calibration-validation-") as temporary:
+ scratch = Path(temporary)
+ shutil.copyfile(run / "plan.json", scratch / "plan.json")
+ try:
+ study.grade(source, scratch, collection, prices, "calibration", workers=workers)
+ except ValueError as error:
+ if (
+ str(error)
+ != f"{len(manifest['decisions'])} unresolved comparisons; retain every row and review the complete list"
+ ):
+ raise
+ else:
+ raise ValueError("The reviewed unresolved comparison no longer exists")
+ fresh_scan = scratch / "unresolved-calibration.json"
+ study.require(file_hash(fresh_scan) == manifest["original_scan_sha256"], "Recomputed review scan differs")
+ diagnostic_path = run / "unresolved-calibration.json"
+ if not diagnostic_path.exists():
+ with diagnostic_path.open("xb") as stream:
+ stream.write(fresh_scan.read_bytes())
+ diagnostic_path = run / "unresolved-calibration.json"
+ study.require(
+ file_hash(diagnostic_path) == manifest["original_scan_sha256"],
+ "The complete calibration-format scan differs from the reviewed scan",
+ )
+ return diagnostic_path
+
+
+def grade(source, run, collection, prices, manifest_path, *, workers=8):
+ source, run, collection, prices, manifest_path = map(Path, (source, run, collection, prices, manifest_path))
+ study.unopened(run)
+ if (run / "grades-calibration.json").exists():
+ raise FileExistsError(run / "grades-calibration.json")
+ manifest = study.read(manifest_path)
+ study.require(
+ manifest["format"] == "sie-arithmetic-calibration-adjudication-v1"
+ and manifest["split"] == "calibration"
+ and manifest["base_source_sha256"] == study.code_hashes()
+ and manifest["decisions"],
+ "Review must bind the unchanged base grader and calibration-only scope",
+ )
+ diagnostic_path = verified_scan(source, run, collection, prices, manifest, workers)
+ decisions = {(item["id"], item["arm"]): item for item in manifest["decisions"]}
+ study.require(len(decisions) == len(manifest["decisions"]), "Duplicate review decisions")
+ plan, inputs, metadata = study.checked_plan(source, run)
+ gold = study.read(source / "gold-for-scoring-only.json")
+ rates, public_rows = study.benefit_costs.rates_from_snapshot(study.read(prices))
+ acquisition = study.read(collection / "collection.json")
+ jobs = []
+ for question, meta in zip(inputs["calibration"], metadata["calibration"], strict=True):
+ records = {arm: study.read(collection / f"{question['id']}-{arm}.json") for arm in study.outcomes.ARMS}
+ hashes = {arm: file_hash(collection / f"{question['id']}-{arm}.json") for arm in study.outcomes.ARMS}
+ jobs.append((question, meta, gold[question["id"]], records, hashes, rates))
+ with ProcessPoolExecutor(max_workers=workers) as pool:
+ computed = list(pool.map(study._grade_pair, jobs, chunksize=8))
+ base_rows_sha256 = digest([row for row, _ in computed])
+ applied = []
+ for row, flags in computed:
+ for flag in flags:
+ key = flag["id"], flag["arm"]
+ study.require(key in decisions, "An unresolved comparison has no explicit adjudication")
+ apply_decision(row, flag, decisions[key], gold[flag["id"]])
+ applied.append(key)
+ study.require(set(applied) == set(decisions) and len(applied) == len(decisions), "Review coverage differs")
+ result = {
+ "format": "sie-arithmetic-adaptation-grades-v1",
+ "split": "calibration",
+ "study_plan_sha256": file_hash(run / "plan.json"),
+ "source_plan_sha256": plan["source_plan_sha256"],
+ "collection_plan_sha256": file_hash(collection / "collection-plan.json"),
+ "collection_summary_sha256": file_hash(collection / "collection.json"),
+ "acquisition_receipt_mapping_sha256": digest(acquisition["receipt_sha256"]),
+ "acquisition_verification": (
+ "The unchanged base grade command verifies every selected record and the complete acquisition mapping."
+ ),
+ "gold_sha256": file_hash(source / "gold-for-scoring-only.json"),
+ "prices_sha256": file_hash(prices),
+ "public_price_rows": public_rows,
+ "grading_source_sha256": {
+ name: study.code_hashes()[name]
+ for name in ("arithmetic_answers.py", "teacher_format.py", "solver_answers.py")
+ },
+ "test_access_sha256": None,
+ "cost_scope": "Every physical call priced at pinned uncached Cloud rates; not observed Cloud bills or Modal spend.",
+ "grading_extension": {
+ "source_sha256": file_hash(Path(__file__)),
+ "manifest_sha256": file_hash(manifest_path),
+ "original_scan_sha256": file_hash(diagnostic_path),
+ "base_rows_sha256": base_rows_sha256,
+ "applied": len(applied),
+ "scope": "One exact calibration format review before cutoff selection; no row exclusion.",
+ },
+ "rows": [row for row, _ in computed],
+ }
+ write(run / "grades-calibration.json", result)
+ return result
diff --git a/examples/train-a-router/arithmetic_controls.py b/examples/train-a-router/arithmetic_controls.py
new file mode 100644
index 000000000..9c3e2e461
--- /dev/null
+++ b/examples/train-a-router/arithmetic_controls.py
@@ -0,0 +1,242 @@
+"""Separate cost-estimate controls for unchanged Laya weights."""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+import numpy as np
+
+import arithmetic_study as study
+from data import digest, file_hash, write
+
+FORMAT = "sie-arithmetic-cost-controls-frozen-v1"
+PROTOCOL = {
+ "purpose": "Isolate weight adaptation from changing expected solver costs.",
+ "weights": "Reuse the unchanged reference Laya probabilities; no training or new model calls.",
+ "costs": "Use the same arithmetic500/2000 conditional cost means as each corresponding LoRA model.",
+ "selection": "Apply the unchanged calibration cutoff rule at targets1/0/2/5.",
+ "scope": "Separate pre-test controls; all original core models, predictions and cutoffs remain immutable.",
+}
+
+
+def names():
+ return {f"base_arithmetic_{size}" for size in (500, 2000)}
+
+
+def _bindings(run):
+ run = Path(run)
+ return {
+ "core_frozen_sha256": file_hash(run / "frozen.json"),
+ "study_plan_sha256": file_hash(run / "plan.json"),
+ "calibration_grades_sha256": file_hash(run / "grades-calibration.json"),
+ "native_scores_sha256": file_hash(run / "native-scores.json"),
+ "native_trained_sha256": {str(size): file_hash(run / f"native-{size}/trained.json") for size in (500, 2000)},
+ "source_sha256": file_hash(Path(__file__)),
+ }
+
+
+def scores(probabilities, statistics, count):
+ values = np.asarray(probabilities, dtype=float)
+ study.require(
+ values.shape == (count, 2) and np.isfinite(values).all() and ((values >= 0) & (values <= 1)).all(),
+ "Need finite, unchanged cheap/strong correctness probabilities",
+ )
+ return study.outcomes.normalized_scores(values, statistics)[0].tolist()
+
+
+def freeze(source, run, controls):
+ run, controls = Path(run), Path(controls)
+ if controls.exists():
+ raise FileExistsError(controls)
+ study.unopened(run)
+ _, inputs, _ = study.checked_plan(source, run)
+ core = study.read(run / "frozen.json")
+ bindings = _bindings(run)
+ study.require(
+ core["study_plan_sha256"] == bindings["study_plan_sha256"]
+ and core["calibration_grades_sha256"] == bindings["calibration_grades_sha256"]
+ and core["model_artifacts_sha256"]["native-scores.json"] == bindings["native_scores_sha256"]
+ and core["source_code_sha256"] == study.code_hashes(),
+ "The existing core freeze or executed sources changed",
+ )
+ native = study.read(run / "native-scores.json")["models"]["reference_base"]
+ calibration = study.read(run / "grades-calibration.json")["rows"]
+ ids = {split: [row["id"] for row in inputs[split]] for split in ("calibration", "test")}
+ study.require(
+ [row["id"] for row in calibration] == ids["calibration"]
+ and all(native["splits"][split]["ids"] == ordered == core["ids"][split] for split, ordered in ids.items()),
+ "Cached probabilities or calibration rows differ from the frozen question order",
+ )
+ methods = {}
+ for size in (500, 2000):
+ trained = study.checked_native(source, run, size)
+ statistics = trained["conditional_cost_statistics"]
+ values = {
+ split: scores(native["splits"][split]["probabilities"], statistics, len(ordered))
+ for split, ordered in ids.items()
+ }
+ methods[f"base_arithmetic_{size}"] = {
+ "design": "native",
+ "training_rows": 0,
+ "cost_training_rows": size,
+ "weights_changed": False,
+ "cost_model_source": f"arithmetic {size} training outcomes",
+ "conditional_cost_statistics": statistics,
+ "scores": values,
+ "policies": {
+ str(target): study.solver_report.select_cutoff(
+ calibration, values["calibration"], target, router_credits=None
+ )
+ for target in study.TARGETS
+ },
+ }
+ result = {
+ "format": FORMAT,
+ "protocol": PROTOCOL,
+ **bindings,
+ "ids": ids,
+ "base_trained_sha256": native["trained_sha256"],
+ "base_probabilities_sha256": {split: digest(native["splits"][split]["probabilities"]) for split in ids},
+ "methods": methods,
+ "fresh_test_answers_opened": False,
+ }
+ write(controls / "frozen.json", result)
+ return result
+
+
+def checked_frozen(source, run, controls):
+ run, controls = Path(run), Path(controls)
+ _, inputs, _ = study.checked_plan(source, run)
+ result = study.read(controls / "frozen.json")
+ study.require(
+ result["format"] == FORMAT
+ and result["protocol"] == PROTOCOL
+ and all(result[key] == value for key, value in _bindings(run).items())
+ and set(result["methods"]) == names(),
+ "Cost-control protocol, source or original study bindings changed",
+ )
+ native = study.read(run / "native-scores.json")["models"]["reference_base"]
+ calibration = study.read(run / "grades-calibration.json")["rows"]
+ study.require(result["base_trained_sha256"] == native["trained_sha256"], "Base checkpoint binding changed")
+ for size in (500, 2000):
+ item = result["methods"][f"base_arithmetic_{size}"]
+ statistics = study.checked_native(source, run, size)["conditional_cost_statistics"]
+ study.require(
+ item["training_rows"] == 0
+ and item["cost_training_rows"] == size
+ and item["weights_changed"] is False
+ and item["conditional_cost_statistics"] == statistics
+ and set(item["policies"]) == {str(target) for target in study.TARGETS},
+ "Unchanged-weight control or matched cost means changed",
+ )
+ for split in ("calibration", "test"):
+ ordered = [row["id"] for row in inputs[split]]
+ probabilities = native["splits"][split]["probabilities"]
+ expected = scores(probabilities, statistics, len(ordered))
+ actual = np.asarray(item["scores"][split], dtype=float)
+ study.require(
+ result["ids"][split] == ordered == native["splits"][split]["ids"]
+ and digest(probabilities) == result["base_probabilities_sha256"][split]
+ and actual.shape == (len(ordered),)
+ and np.isfinite(actual).all()
+ and np.allclose(actual, expected, rtol=1e-12, atol=1e-15),
+ "Cost-control input probabilities or score formula changed",
+ )
+ for target in study.TARGETS:
+ expected = study.solver_report.select_cutoff(
+ calibration, item["scores"]["calibration"], target, router_credits=None
+ )
+ actual = item["policies"][str(target)]
+ study.require(
+ actual["mode"] == expected["mode"] and actual["cutoff"] == expected["cutoff"],
+ "Cost-control cutoff differs from the declared calibration selection",
+ )
+ return result
+
+
+def authorize_test(source, run, controls):
+ run, controls = Path(run), Path(controls)
+ for path in (run / "grades-test.json", controls / "test-access.json", controls / "report.json"):
+ if path.exists():
+ raise FileExistsError(path)
+ frozen = checked_frozen(source, run, controls)
+ access = study.read(run / "test-access.json")
+ study.require(
+ access["frozen_sha256"] == frozen["core_frozen_sha256"],
+ "Core access must bind the unchanged pre-test model freeze",
+ )
+ result = {
+ "format": "sie-arithmetic-cost-controls-test-access-v1",
+ "frozen_sha256": file_hash(controls / "frozen.json"),
+ "core_test_access_sha256": file_hash(run / "test-access.json"),
+ "core_frozen_sha256": frozen["core_frozen_sha256"],
+ "statement": "Both cost-only controls were frozen before reading fresh test answers.",
+ }
+ write(controls / "test-access.json", result)
+ return result
+
+
+def report(source, run, controls):
+ run, controls = Path(run), Path(controls)
+ if (controls / "report.json").exists():
+ raise FileExistsError(controls / "report.json")
+ frozen = checked_frozen(source, run, controls)
+ access = study.read(controls / "test-access.json")
+ grades = study.read(run / "grades-test.json")
+ study.require(
+ access["frozen_sha256"] == file_hash(controls / "frozen.json")
+ and access["core_test_access_sha256"] == file_hash(run / "test-access.json")
+ and grades["test_access_sha256"] == access["core_test_access_sha256"]
+ and [row["id"] for row in grades["rows"]] == frozen["ids"]["test"],
+ "Cost-control access or shared held-out grades changed",
+ )
+ methods = {}
+ for name, item in frozen["methods"].items():
+ policies = {}
+ values = np.asarray(item["scores"]["test"])
+ for target, policy in item["policies"].items():
+ decisions = (values >= policy["cutoff"]).tolist() if policy["mode"] == "threshold" else policy["mode"]
+ policies[target] = {
+ "mode": policy["mode"],
+ "cutoff": policy["cutoff"],
+ "metrics": study.solver_report.summarize(
+ grades["rows"], decisions, router_credits=None if policy["mode"] == "threshold" else 0
+ ),
+ }
+ methods[name] = {
+ "label": f"Unchanged Laya + arithmetic {item['cost_training_rows']} cost estimates",
+ **{key: item[key] for key in ("design", "training_rows", "cost_training_rows", "weights_changed")},
+ "cost_model_source": item["cost_model_source"],
+ "show_by_default": False,
+ "policies": policies,
+ }
+ result = {
+ "format": "sie-arithmetic-cost-controls-report-v1",
+ "n": len(grades["rows"]),
+ "ids": frozen["ids"]["test"],
+ "primary_target_pp": 1,
+ "frozen_sha256": file_hash(controls / "frozen.json"),
+ "core_frozen_sha256": frozen["core_frozen_sha256"],
+ "test_grades_sha256": file_hash(run / "grades-test.json"),
+ "methods": methods,
+ "cost_scope": "Gross solver-token savings; native hosting is unpriced. Each cost estimate matches its LoRA training population.",
+ }
+ write(controls / "report.json", result)
+ return result
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("command", choices=("freeze", "authorize-test", "report"))
+ for name in ("source", "run", "controls"):
+ parser.add_argument("--" + name, type=Path, required=True)
+ args = parser.parse_args()
+ result = {"freeze": freeze, "authorize-test": authorize_test, "report": report}[args.command](
+ args.source, args.run, args.controls
+ )
+ print(result["format"])
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/arithmetic_data.py b/examples/train-a-router/arithmetic_data.py
new file mode 100644
index 000000000..50a39e5c5
--- /dev/null
+++ b/examples/train-a-router/arithmetic_data.py
@@ -0,0 +1,333 @@
+"""Prepare a new arithmetic study with exposed training and fresh grouped splits."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+from collections import Counter
+from pathlib import Path
+
+import numpy as np
+
+import solver
+from data import SOURCE_HASHES, digest, duplicate_components, file_hash, render_request, write
+from generalization_data import HASHES, REVISION, SOURCE_URL, final_answer
+
+COUNTS = {"train": 2000, "train_small": 500, "calibration": 1000, "test": 3000}
+NEAR_DUPLICATES = {
+ "character_ngram": 4,
+ "plain_cosine": 0.80,
+ "numeric_masked_cosine": 0.85,
+ "plain_review_cosine": 0.80,
+ "numeric_masked_review_cosine": 0.85,
+}
+FORMAT = "sie-arithmetic-adaptation-source-v1"
+
+
+def read(path):
+ return json.loads(Path(path).read_text())
+
+
+def source_rows(source, math_source):
+ import pyarrow.parquet as pq
+
+ source, math_source = Path(source), Path(math_source)
+ rows = []
+ for name, expected in SOURCE_HASHES.items():
+ if file_hash(math_source / name) != expected:
+ raise ValueError("Pinned competition source changed")
+ rows.extend(
+ {
+ "id": f"e2h-amc-{name.split('.')[0]}-{index}",
+ "problem": row["problem"],
+ "dataset": "competition",
+ }
+ for index, row in enumerate(pq.read_table(math_source / name).to_pylist())
+ )
+ for name, expected in HASHES.items():
+ if file_hash(source / name) != expected:
+ raise ValueError("Pinned arithmetic source changed")
+ rows.extend(
+ {
+ "id": f"gsm8k-{name.split('.')[0]}-{index}",
+ "problem": row["question"],
+ "gold": final_answer(row["answer"]),
+ "dataset": "arithmetic",
+ "original_split": name.split(".")[0],
+ }
+ for index, row in enumerate(map(json.loads, (source / name).read_text().splitlines()))
+ )
+ return rows
+
+
+def _character_matrix(texts, masked):
+ from scipy.sparse import csr_matrix
+
+ vocabulary, indices, indptr, values, document_counts = {}, [], [0], [], Counter()
+ for text in texts:
+ normalized = " ".join(re.findall(r"[a-z]+|\d+", text.casefold()))
+ if masked:
+ normalized = re.sub(r"\d+", "#", normalized)
+ normalized = f" {normalized} "
+ grams = Counter(normalized[index : index + 4] for index in range(max(0, len(normalized) - 3)))
+ for gram, count in grams.items():
+ column = vocabulary.setdefault(gram, len(vocabulary))
+ indices.append(column)
+ values.append(1 + np.log(count))
+ document_counts[column] += 1
+ indptr.append(len(indices))
+ matrix = csr_matrix((values, indices, indptr), shape=(len(texts), len(vocabulary)), dtype=np.float64)
+ idf = np.log((1 + len(texts)) / (1 + np.array([document_counts[i] for i in range(len(vocabulary))]))) + 1
+ matrix = matrix.multiply(idf).tocsr()
+ norms = np.sqrt(np.asarray(matrix.multiply(matrix).sum(axis=1)).ravel())
+ if np.any(norms == 0):
+ raise ValueError("Every question must contain character features")
+ return matrix.multiply(1 / norms[:, None]).tocsr()
+
+
+def grouped_sources(rows, *, progress=None):
+ """Union lexical/template families using only the question text."""
+ parent = list(range(len(rows)))
+
+ def find(index):
+ while parent[index] != index:
+ parent[index] = parent[parent[index]]
+ index = parent[index]
+ return index
+
+ def union(left, right):
+ left, right = find(left), find(right)
+ parent[max(left, right)] = min(left, right)
+
+ for component in duplicate_components(rows):
+ for index in component[1:]:
+ union(component[0], index)
+ pairs = {}
+ for masked, threshold, review_threshold in (
+ (False, NEAR_DUPLICATES["plain_cosine"], NEAR_DUPLICATES["plain_review_cosine"]),
+ (True, NEAR_DUPLICATES["numeric_masked_cosine"], NEAR_DUPLICATES["numeric_masked_review_cosine"]),
+ ):
+ matrix = _character_matrix([row["problem"] for row in rows], masked)
+ for start in range(0, len(rows), 128):
+ similarities = (matrix[start : start + 128] @ matrix.T).toarray()
+ for offset, current in enumerate(similarities):
+ left = start + offset
+ for right in np.flatnonzero(current[:left] >= review_threshold):
+ if rows[left]["dataset"] == rows[int(right)]["dataset"] == "competition":
+ continue
+ pair = tuple(sorted((left, int(right))))
+ entry = pairs.setdefault(pair, {"plain_cosine": None, "numeric_masked_cosine": None})
+ entry["numeric_masked_cosine" if masked else "plain_cosine"] = float(current[right])
+ if current[right] >= threshold:
+ union(*pair)
+ if progress is not None and start % 1024 == 0:
+ progress(f"Source-only {'masked' if masked else 'plain'} similarity: {start}/{len(rows)}")
+ components = {}
+ for index in range(len(rows)):
+ components.setdefault(find(index), []).append(index)
+ review = [
+ {
+ "ids": [rows[left]["id"], rows[right]["id"]],
+ **scores,
+ "same_family": find(left) == find(right),
+ }
+ for (left, right), scores in sorted(pairs.items())
+ ]
+ return list(components.values()), review
+
+
+def assign(rows, components, exposed, counts=None):
+ counts = COUNTS if counts is None else counts
+ known = {row["id"]: row for row in rows}
+ exposed_ids = [row["id"] for row in exposed]
+ if len(exposed_ids) != counts["train"] or len(set(exposed_ids)) != len(exposed_ids):
+ raise ValueError("Exposed training pool count or identities differ")
+ for row in exposed:
+ if row["id"] not in known or row["request"] != render_request("GSM8K", known[row["id"]]["problem"]):
+ raise ValueError("Exposed requests must match the pinned source exactly")
+ exposed_set, families, candidates = set(exposed_ids), {}, []
+ excluded, grouped = [], []
+ for component in components:
+ members = [rows[index] for index in component]
+ identities = sorted(row["id"] for row in members)
+ family = digest(["arithmetic-family-v1", identities])
+ for identity in identities:
+ families[identity] = family
+ arithmetic = [row for row in members if row["dataset"] == "arithmetic"]
+ if not arithmetic:
+ continue
+ touches_exposed = bool(exposed_set.intersection(identities))
+ if len(arithmetic) != len(members):
+ if touches_exposed:
+ raise ValueError("An exposed arithmetic training family overlaps competition source")
+ excluded.append({"ids": identities, "reason": "overlaps_competition_source"})
+ continue
+ if len(members) > 1:
+ grouped.append({"ids": identities, "family": family, "touches_exposed": touches_exposed})
+ if touches_exposed:
+ excluded.extend(
+ {"ids": [row["id"]], "reason": "same_family_as_exposed_training"}
+ for row in arithmetic
+ if row["id"] not in exposed_set
+ )
+ continue
+ if any(row["original_split"] != "train" for row in arithmetic):
+ raise ValueError("Every unexposed arithmetic family must come from source train")
+ representative = min(arithmetic, key=lambda row: digest(["arithmetic-representative-v1", row["id"]]))
+ candidates.append(representative)
+ candidates.sort(key=lambda row: digest(["arithmetic-fresh-split-v1", families[row["id"]]]))
+ required = counts["calibration"] + counts["test"]
+ if len(candidates) < required:
+ raise ValueError("Not enough unused disjoint source families for declared splits")
+ selected = {
+ "train": [known[identity] for identity in exposed_ids],
+ "calibration": candidates[: counts["calibration"]],
+ "test": candidates[counts["calibration"] : required],
+ }
+ training_families = {}
+ for row in selected["train"]:
+ training_families.setdefault(families[row["id"]], []).append(row["id"])
+ reachable = {0: ()}
+ for family in sorted(training_families, key=lambda key: digest(["arithmetic-small-v1", key])):
+ size = len(training_families[family])
+ for total, chosen in list(reachable.items()):
+ if total + size <= counts["train_small"] and total + size not in reachable:
+ reachable[total + size] = chosen + (family,)
+ if counts["train_small"] in reachable:
+ break
+ if counts["train_small"] not in reachable:
+ raise ValueError("Whole families cannot satisfy the declared small training count")
+ small = [identity for family in reachable[counts["train_small"]] for identity in training_families[family]]
+ split_families = {split: {families[row["id"]] for row in population} for split, population in selected.items()}
+ if any(
+ split_families[left] & split_families[right]
+ for left, right in (("train", "calibration"), ("train", "test"), ("calibration", "test"))
+ ):
+ raise ValueError("Source families cross study roles")
+ return (
+ selected,
+ families,
+ small,
+ {
+ "available_unused_families": len(candidates),
+ "unused_after_selection": len(candidates) - required,
+ "grouped_arithmetic_families": grouped,
+ "excluded": excluded,
+ },
+ )
+
+
+def prepare(source, math_source, exposed_directory, output, *, progress=None):
+ output, exposed_directory = Path(output), Path(exposed_directory)
+ if output.exists():
+ raise FileExistsError(output)
+ exposed_plan = read(exposed_directory / "source-plan.json")
+ exposed = read(exposed_directory / "inputs.json")
+ if (
+ exposed_plan.get("source_revision") != REVISION
+ or exposed_plan.get("source_hashes") != HASHES
+ or exposed_plan["files_sha256"]["inputs.json"] != file_hash(exposed_directory / "inputs.json")
+ ):
+ raise ValueError("Exposed development source differs")
+ rows = source_rows(source, math_source)
+ components, pairs = grouped_sources(rows, progress=progress)
+ selected, families, small, audit = assign(rows, components, exposed)
+ inputs = {
+ split: [{"id": row["id"], "request": render_request("GSM8K", row["problem"])} for row in population]
+ for split, population in selected.items()
+ }
+ metadata = {
+ split: [
+ {"id": row["id"], "family": families[row["id"]], "original_split": row["original_split"]}
+ for row in population
+ ]
+ for split, population in selected.items()
+ }
+ audit |= {
+ "question_only": True,
+ "solver_answers_or_scores_read": False,
+ "arithmetic_source_counts": dict(
+ Counter(row["original_split"] for row in rows if row["dataset"] == "arithmetic")
+ ),
+ "grouping": NEAR_DUPLICATES,
+ "initial_grouping": "Pinned data.duplicate_components canonical and five-token Jaccard rules",
+ "limitation": "Lexical/template grouping does not prove absence of all semantic relatives or model pretraining.",
+ }
+ write(output / "inputs.json", inputs)
+ write(output / "rows.json", metadata)
+ write(output / "train-small-ids.json", small)
+ write(
+ output / "gold-for-scoring-only.json",
+ {row["id"]: row["gold"] for population in selected.values() for row in population},
+ )
+ write(output / "source-audit.json", audit)
+ write(output / "near-duplicate-candidates.json", pairs)
+ plan = {
+ "format": FORMAT,
+ "counts": COUNTS,
+ "source_revision": REVISION,
+ "source_url": SOURCE_URL,
+ "source_hashes": HASHES,
+ "competition_source_hashes": SOURCE_HASHES,
+ "exposed_source_plan_sha256": file_hash(exposed_directory / "source-plan.json"),
+ "exposed_inputs_sha256": file_hash(exposed_directory / "inputs.json"),
+ "exposed_rows_use": "All prior 2000 arithmetic evaluation rows are now exposed training/development only.",
+ "primary_target_pp": 1,
+ "secondary_targets_pp": [0, 2, 5],
+ "success_target": "Aim for >=30% net savings near 1pp loss; report paired uncertainty, not a guarantee.",
+ "comparison": "Unchanged competition models with arithmetic recalibration versus same-recipe 500/2000 arithmetic fits.",
+ "sampling": "Question-only connected families; one fresh representative per family; fixed hash order.",
+ "settings_sha256": digest(solver.SETTINGS),
+ "files_sha256": {
+ name: file_hash(output / name)
+ for name in (
+ "inputs.json",
+ "rows.json",
+ "train-small-ids.json",
+ "gold-for-scoring-only.json",
+ "source-audit.json",
+ "near-duplicate-candidates.json",
+ )
+ },
+ "source_code_sha256": {
+ name: file_hash(Path(__file__).with_name(name))
+ for name in ("arithmetic_data.py", "data.py", "generalization_data.py", "solver.py")
+ },
+ }
+ write(output / "source-plan.json", plan)
+ fresh = inputs["calibration"] + inputs["test"]
+ write(output / "fresh/inputs.json", fresh)
+ write(output / "fresh/settings.json", solver.SETTINGS)
+ write(
+ output / "fresh/source-plan.json",
+ {
+ "format": FORMAT,
+ "parent_source_plan_sha256": file_hash(output / "source-plan.json"),
+ "n": len(fresh),
+ "counts": {split: COUNTS[split] for split in ("calibration", "test")},
+ "inputs_sha256": digest(fresh),
+ "settings_sha256": digest(solver.SETTINGS),
+ "reference_answers_staged": False,
+ },
+ )
+ return {
+ "counts": COUNTS,
+ "available_unused_families": audit["available_unused_families"],
+ "source_plan_sha256": file_hash(output / "source-plan.json"),
+ "near_duplicate_candidates": len(pairs),
+ }
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--source", type=Path, required=True)
+ parser.add_argument("--math-source", type=Path, required=True)
+ parser.add_argument("--exposed", type=Path, required=True)
+ parser.add_argument("--output", type=Path, required=True)
+ args = parser.parse_args()
+ print(json.dumps(prepare(args.source, args.math_source, args.exposed, args.output, progress=print), indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/arithmetic_labels.py b/examples/train-a-router/arithmetic_labels.py
new file mode 100644
index 000000000..5eb77a932
--- /dev/null
+++ b/examples/train-a-router/arithmetic_labels.py
@@ -0,0 +1,348 @@
+"""Compare independent teacher labels with benchmark labels on matched arithmetic rows."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from concurrent.futures import ProcessPoolExecutor
+from pathlib import Path
+
+import arithmetic_answers
+import arithmetic_optimizer
+import arithmetic_study as study
+import arithmetic_teacher
+import teacher_format
+from data import digest, file_hash, write
+from serving import vectors
+from solver_answers import extract_final_answer
+
+PROTOCOL = {
+ "label_source": "Independent question-only teacher answers; never substitute benchmark references.",
+ "eligibility": (
+ "Completed teacher answer satisfying its saved numeric or general answer contract and successful self-comparison. "
+ "This mask uses only question and teacher reference; apply the identical mask to both heads."
+ ),
+ "learner": "Unchanged two logistic heads, original source-family folds, fixed penalty grid and conditional cost formula.",
+ "loss_evaluation": arithmetic_optimizer.EVALUATION,
+ "unresolved_solver_comparison": "Fail the comparison; never drop a row or silently label an unresolved answer wrong.",
+ "teacher_accuracy": "Separate benchmark-reference audit, not an eligibility filter or a training target.",
+ "scope": "Training labels only; calibration and test remain benchmark-supervised.",
+}
+
+
+def grade_teacher(question, reference, response):
+ """Treat an unqualified clock hour as a concise answer, with no gold input."""
+ final = extract_final_answer(response)
+ expected, actual = arithmetic_answers.clock_value(reference), arithmetic_answers.clock_value(final)
+ if (
+ arithmetic_answers.clock_question(question)
+ and expected
+ and actual
+ and expected["period"] is not None
+ and actual["period"] is None
+ and actual["hour"] <= 12
+ ):
+ short = f"{expected['hour']}:{expected['minute']:02d}"
+ return arithmetic_answers.grade_reference(question, short, response) | {
+ "teacher_clock_reference": reference,
+ "normalization": "Bare clock-hour prediction leaves period unspecified; preserve explicit period conflicts.",
+ }
+ return arithmetic_answers.grade_reference(question, reference, response)
+
+
+def numeric_reference(reference):
+ """Validate the declared numeric-only answer without any benchmark input."""
+ return len(reference) <= 128 and arithmetic_teacher.protocol_for("numeric").numeric_answer(reference)
+
+
+def label_pair(question, reference, answers, *, answer_format="general"):
+ """Construct targets without receiving any benchmark reference or identity."""
+ try:
+ self_grade = grade_teacher(question, reference, r"\boxed{" + reference + "}")
+ if answer_format == "numeric":
+ unit = {
+ "unit_status": "numeric_requested_units"
+ if numeric_reference(reference)
+ else "numeric_contract_invalid",
+ "reference_only": True,
+ }
+ elif answer_format != "general":
+ raise ValueError("Unknown teacher answer format")
+ elif arithmetic_answers.clock_question(question) and arithmetic_answers.clock_value(reference):
+ unit = {"unit_status": "declared_clock"}
+ else:
+ unit = teacher_format.normalize_final(question, reference)
+ except (ValueError, TypeError, ArithmeticError, TimeoutError) as error:
+ return False, {"error_type": type(error).__name__, "reference_only": True}, {}
+ usable = bool(
+ self_grade["status"] == "graded"
+ and self_grade["correct"]
+ and unit["unit_status"] not in {"unknown_requested_unit", "incompatible", "numeric_contract_invalid"}
+ )
+ grades = {}
+ if usable:
+ for arm in study.outcomes.ARMS:
+ try:
+ grades[arm] = grade_teacher(question, reference, answers[arm])
+ except (ValueError, TypeError, ArithmeticError, TimeoutError) as error:
+ grades[arm] = {"correct": False, "status": "grading_error", "error_type": type(error).__name__}
+ return usable, {"self_grade": self_grade, "unit_contract": unit}, grades
+
+
+def _label_item(item):
+ identity, question, reference, answers, benchmark_reference = item[:5]
+ answer_format = item[5] if len(item) == 6 else "general"
+ usable, contract, grades = label_pair(question, reference, answers, answer_format=answer_format)
+ try:
+ audit = arithmetic_answers.grade_reference(question, benchmark_reference, r"\boxed{" + reference + "}")
+ except (ValueError, TypeError, ArithmeticError, TimeoutError) as error:
+ audit = {"status": "grading_error", "correct": False, "error_type": type(error).__name__}
+ return {"id": identity, "usable": usable, "contract": contract, "grades": grades, "audit": audit}
+
+
+def checked_closure(path, directory, inputs, collection):
+ """Accept an explicit end to acquisition without rewriting its receipts."""
+ path, directory = Path(path), Path(directory)
+ closure = study.read(path)
+ ids = [row["id"] for row in inputs]
+ completed = [identity for identity in ids if identity in collection["completed_attempts"]]
+ study.require(
+ closure.get("format") == "sie-arithmetic-teacher-closure-v1"
+ and closure.get("plan_sha256") == file_hash(directory / "plan.json")
+ and closure.get("collection_sha256") == file_hash(directory / "collection.json")
+ and closure.get("completed_ids") == completed
+ and closure.get("missing_ids") == [identity for identity in ids if identity not in completed]
+ and closure.get("physical_attempts") == collection["physical_attempts"]
+ and closure.get("acquisition_stopped") is True
+ and closure.get("additional_requests_authorized") is False
+ and closure.get("eligibility_basis")
+ == "Teacher completion and reference-only parsing; no solver or gold outcomes."
+ and isinstance(closure.get("reason"), str)
+ and bool(closure["reason"].strip()),
+ "Teacher acquisition closure does not match the preserved collection",
+ )
+ return closure
+
+
+def teacher_records(directory, inputs, *, closure_path=None):
+ directory = Path(directory)
+ plan, collection = (study.read(directory / name) for name in ("plan.json", "collection.json"))
+ protocol = arithmetic_teacher.protocol_for(plan.get("answer_format", "general"))
+ study.require(
+ plan["format"] == arithmetic_teacher.FORMAT
+ and plan["settings"] == protocol.SETTINGS
+ and plan["ids"] == [row["id"] for row in inputs]
+ and plan["questions_sha256"] == digest(inputs)
+ and plan["question_only"] is True
+ and plan["reference_answers_received"] is False
+ and plan["solver_answers_received"] is False
+ and collection["plan_sha256"] == file_hash(directory / "plan.json"),
+ "Need the settled teacher collection for the exact question-only training population",
+ )
+ if closure_path is not None:
+ checked_closure(closure_path, directory, inputs, collection)
+ study.require(
+ not collection["pending"] or closure_path is not None,
+ "Teacher collection has pending references; complete acquisition or bind an explicit closure receipt",
+ )
+ completed, missing = {}, []
+ for row in inputs:
+ history = arithmetic_teacher.attempt_history(directory, row)
+ valid = [(path, record) for path, _, record in history if record and record["status"] == "complete"]
+ if not valid:
+ study.require(row["id"] not in collection["completed_attempts"], "Teacher completion mapping differs")
+ missing.append(row["id"])
+ continue
+ path, record = valid[0]
+ study.require(
+ collection["completed_attempts"].get(row["id"]) == str(path.relative_to(directory)),
+ "Use the first schema-valid completed teacher answer, independent of correctness",
+ )
+ completed[row["id"]] = {
+ "answer": record["answer"],
+ "record_sha256": file_hash(path / "records" / f"{row['id']}.json"),
+ "raw_response_sha256": record["raw_response_sha256"],
+ }
+ study.require(len(completed) == collection["complete"], "Teacher collection count differs")
+ return completed, missing, plan, collection
+
+
+def fit(source, run, embeddings, answers, teacher, *, workers=8, teacher_closure=None):
+ source, run, embeddings, answers, teacher = map(Path, (source, run, embeddings, answers, teacher))
+ study.unopened(run)
+ for name in ("teacher-models.json", "teacher-labels.json", "frozen.json"):
+ if (run / name).exists():
+ raise FileExistsError(run / name)
+ study.require(type(workers) is int and 1 <= workers <= 16, "Use 1..16 grading processes")
+ study.checked_plan(source, run)
+ context = study.training(source, run, 2000)
+ records, missing, teacher_plan, collection = teacher_records(
+ teacher, context["inputs"], closure_path=teacher_closure
+ )
+ source_hashes = {
+ "arithmetic_labels.py": file_hash(Path(__file__)),
+ "arithmetic_answers.py": file_hash(Path(arithmetic_answers.__file__)),
+ "arithmetic_optimizer.py": file_hash(Path(arithmetic_optimizer.__file__)),
+ "arithmetic_teacher.py": file_hash(Path(arithmetic_teacher.__file__)),
+ Path(arithmetic_teacher.protocol_for(teacher_plan.get("answer_format", "general")).__file__).name: file_hash(
+ Path(arithmetic_teacher.protocol_for(teacher_plan.get("answer_format", "general")).__file__)
+ ),
+ "teacher_format.py": file_hash(Path(teacher_format.__file__)),
+ }
+ design = {
+ "format": "sie-arithmetic-teacher-comparison-plan-v1",
+ "protocol": PROTOCOL,
+ "study_plan_sha256": file_hash(run / "plan.json"),
+ "teacher_plan_sha256": file_hash(teacher / "plan.json"),
+ "teacher_collection_sha256": file_hash(teacher / "collection.json"),
+ "teacher_closure_sha256": file_hash(teacher_closure) if teacher_closure is not None else None,
+ "answer_format": teacher_plan.get("answer_format", "general"),
+ "source_sha256": source_hashes,
+ }
+ write(run / "teacher-comparison-plan.json", design)
+ study.checked_source(source, include_gold=True)
+ gold = study.read(source / "gold-for-scoring-only.json")
+ jobs = []
+ for row, benchmark in zip(context["inputs"], context["rows"], strict=True):
+ identity = row["id"]
+ if identity not in records:
+ continue
+ actual = {}
+ for arm in study.outcomes.ARMS:
+ path = answers / f"{identity}-{arm}.json"
+ study.require(file_hash(path) == benchmark[f"{arm}_record_sha256"], "Paired training answer changed")
+ actual[arm] = study.read(path)["answer"]
+ jobs.append(
+ (
+ identity,
+ row["request"],
+ records[identity]["answer"],
+ actual,
+ gold[identity],
+ teacher_plan.get("answer_format", "general"),
+ )
+ )
+ executor = ProcessPoolExecutor(max_workers=workers) if workers > 1 else None
+ try:
+ result = list(executor.map(_label_item, jobs, chunksize=8) if executor else map(_label_item, jobs))
+ finally:
+ if executor:
+ executor.shutdown()
+ unresolved = [
+ {"id": item["id"], "arm": arm, "grade": grade}
+ for item in result
+ if item["usable"]
+ for arm, grade in item["grades"].items()
+ if grade["status"] in study.UNRESOLVED
+ ]
+ write(
+ run / "teacher-labels.json",
+ {
+ **design,
+ "format": "sie-arithmetic-teacher-labels-v1",
+ "teacher_records": records,
+ "missing_teacher_ids": missing,
+ "rows": result,
+ "unresolved_solver_comparisons": unresolved,
+ },
+ )
+ study.require(not unresolved, "Unresolved teacher/solver comparison; retain all rows and review before fitting")
+ eligible = {item["id"]: item for item in result if item["usable"]}
+ positions = [i for i, row in enumerate(context["inputs"]) if row["id"] in eligible]
+ study.require(bool(positions), "No usable independent teacher references")
+ inputs = [context["inputs"][i] for i in positions]
+ baseline_rows = [context["rows"][i] for i in positions]
+ teacher_rows = [
+ {
+ "id": row["id"],
+ **{f"{arm}_correct": bool(eligible[row["id"]]["grades"][arm]["correct"]) for arm in study.outcomes.ARMS},
+ }
+ for row in inputs
+ ]
+ matrix = vectors(inputs, embeddings)
+ groups = [context["groups"][i] for i in positions]
+ folds = [context["folds"][i] for i in positions]
+ costs = [[row[f"{arm}_credits"] for arm in study.outcomes.ARMS] for row in baseline_rows]
+ heads = {}
+ for name, rows in zip(study.OPTIONAL_METHODS, (teacher_rows, baseline_rows), strict=True):
+ print(f"Fit {name} on {len(rows)} matched source-family rows", flush=True)
+ heads[name] = arithmetic_optimizer.fit_head(
+ matrix,
+ rows,
+ groups,
+ costs,
+ fold_assignments=folds,
+ progress=lambda value: print(value, flush=True),
+ )
+ audit_rows = [item for item in result if item["audit"]["status"] == "graded"]
+ summary = {
+ "teacher_model": teacher_plan["settings"]["model"],
+ "training_pool": len(context["inputs"]),
+ "completed_references": len(records),
+ "missing_references": len(missing),
+ "teacher_closure_sha256": design["teacher_closure_sha256"],
+ "usable": len(inputs),
+ "excluded": [{"id": identity, "reason": "missing_teacher_answer"} for identity in missing]
+ + [
+ {"id": item["id"], "reason": "reference_not_independently_gradeable", "contract": item["contract"]}
+ for item in result
+ if not item["usable"]
+ ],
+ "teacher_correct": sum(item["audit"]["correct"] for item in audit_rows),
+ "teacher_audit_gradeable": len(audit_rows),
+ "teacher_audit_unresolved": len(result) - len(audit_rows),
+ "label_agreement_by_arm": {
+ arm: sum(
+ a[f"{arm}_correct"] == b[f"{arm}_correct"] for a, b in zip(teacher_rows, baseline_rows, strict=True)
+ )
+ / len(inputs)
+ for arm in study.outcomes.ARMS
+ },
+ "signed_benefit_agreement": sum(
+ int(a["strong_correct"]) - int(a["cheap_correct"]) == int(b["strong_correct"]) - int(b["cheap_correct"])
+ for a, b in zip(teacher_rows, baseline_rows, strict=True)
+ )
+ / len(inputs),
+ "known_teacher_label_cost_usd": collection["known_list_price_usd"],
+ "budget_exposure": collection["budget"],
+ "cost_scope": collection["cost_scope"],
+ "scope": PROTOCOL["scope"],
+ }
+ output = {
+ "format": "sie-arithmetic-teacher-models-v1",
+ "study_plan_sha256": file_hash(run / "plan.json"),
+ "training_grades_sha256": file_hash(run / "grades-train.json"),
+ "teacher_comparison_plan_sha256": file_hash(run / "teacher-comparison-plan.json"),
+ "teacher_labels_sha256": file_hash(run / "teacher-labels.json"),
+ "source_sha256": source_hashes,
+ "heads": heads,
+ "summary": summary,
+ "embedding_files_sha256": {
+ name: file_hash(embeddings / name) for name in ("plan.json", "vectors.jsonl.gz", "collection.json")
+ },
+ }
+ write(run / "teacher-models.json", output)
+ return output
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ for name in ("source", "run", "embeddings", "answers", "teacher"):
+ parser.add_argument("--" + name, type=Path, required=True)
+ parser.add_argument("--workers", type=int, default=8)
+ parser.add_argument("--teacher-closure", type=Path)
+ args = parser.parse_args()
+ result = fit(
+ args.source,
+ args.run,
+ args.embeddings,
+ args.answers,
+ args.teacher,
+ workers=args.workers,
+ teacher_closure=args.teacher_closure,
+ )
+ print(json.dumps(result["summary"], indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/arithmetic_optimizer.py b/examples/train-a-router/arithmetic_optimizer.py
new file mode 100644
index 000000000..0503c7507
--- /dev/null
+++ b/examples/train-a-router/arithmetic_optimizer.py
@@ -0,0 +1,118 @@
+"""Stable evaluation of the existing arithmetic logistic-training objective."""
+
+from __future__ import annotations
+
+import numpy as np
+
+import benefit_outcomes as outcomes
+
+EVALUATION = {
+ "objective": "Sum softplus((1-2*y)*logit) + 0.5*lambda*||weights||^2",
+ "equivalence": "Algebraically identical Bernoulli NLL; avoids subtracting two large floating-point sums.",
+ "unchanged": "Penalty grid, regularization, intercept, folds, stopping tolerances, candidate selection and cost model.",
+}
+
+
+def fit_logistic(matrix, targets, penalty):
+ """Fit the unchanged objective using a cancellation-resistant value."""
+ import scipy
+ from scipy.optimize import minimize
+ from scipy.special import expit
+
+ outcomes._require(
+ scipy.__version__ == outcomes.PROTOCOL["optimizer"]["scipy"], "Fit requires the declared SciPy version"
+ )
+ matrix, targets = np.asarray(matrix, dtype=float), np.asarray(targets, dtype=float)
+ outcomes._require(
+ matrix.ndim == 2
+ and targets.shape == (len(matrix),)
+ and np.isfinite(matrix).all()
+ and np.isin(targets, (0, 1)).all()
+ and np.isfinite(penalty)
+ and penalty > 0,
+ "Need finite vectors, binary labels and a positive fixed penalty",
+ )
+ mean = matrix.mean(axis=0)
+ centered = np.ascontiguousarray(matrix - mean)
+ frequency = float(targets.mean())
+ outcomes._require(0 < frequency < 1, "Each training fold needs both correctness classes for each solver")
+ initial = np.zeros(matrix.shape[1] + 1)
+ initial[-1] = np.log(frequency / (1 - frequency))
+
+ def loss(theta):
+ weights, bias = theta[:-1], theta[-1]
+ logits = centered @ weights + bias
+ residual = expit(logits) - targets
+ value = np.logaddexp(0, (1 - 2 * targets) * logits).sum() + 0.5 * penalty * (weights @ weights)
+ return float(value), np.r_[centered.T @ residual + penalty * weights, residual.sum()]
+
+ options = {key: outcomes.PROTOCOL["optimizer"][key] for key in ("maxiter", "ftol", "gtol", "maxls")}
+ result = minimize(loss, initial, method="L-BFGS-B", jac=True, options=options)
+ outcomes._require(result.success and np.isfinite(result.x).all(), f"Logistic optimization failed: {result.message}")
+ return {
+ "weights": result.x[:-1].tolist(),
+ "bias": float(result.x[-1] - mean @ result.x[:-1]),
+ "optimizer": {
+ "success": bool(result.success),
+ "message": str(result.message),
+ "iterations": int(result.nit),
+ "function_evaluations": int(result.nfev),
+ "objective": float(result.fun),
+ "gradient_absolute_max": float(np.abs(result.jac).max()),
+ "loss_evaluation": EVALUATION["objective"],
+ },
+ }
+
+
+def fit_head(matrix, rows, groups, costs, *, fold_assignments, progress=None):
+ """Use the original fixed search and costs with the stable loss value."""
+ outcomes.solver_report.summarize_quality(rows, "always_strong")
+ matrix = np.asarray(matrix, dtype=float)
+ targets = np.array([[int(row[f"{arm}_correct"]) for arm in outcomes.ARMS] for row in rows], dtype=float)
+ outcomes._require(
+ matrix.ndim == 2 and len(matrix) == len(rows) == len(groups) and np.isfinite(matrix).all(),
+ "Training vectors, paired targets and paper groups must align",
+ )
+ folds = np.asarray(fold_assignments)
+ outcomes._require(
+ folds.shape == (len(rows),)
+ and set(folds.tolist()) == set(range(outcomes.PROTOCOL["folds"]))
+ and all(len({int(folds[i]) for i, item in enumerate(groups) if item == group}) == 1 for group in set(groups)),
+ "Every training family must stay in one of the declared folds",
+ )
+ statistics = outcomes.conditional_costs(targets, costs)
+ candidates = []
+ for penalty in outcomes.PROTOCOL["penalties"]:
+ predictions = np.empty_like(targets)
+ optimizers = []
+ for fold in range(outcomes.PROTOCOL["folds"]):
+ held = folds == fold
+ for arm, name in enumerate(outcomes.ARMS):
+ model = fit_logistic(matrix[~held], targets[~held, arm], penalty)
+ predictions[held, arm] = outcomes.probability(matrix[held], model)
+ optimizers.append({"fold": fold, "arm": name, **model["optimizer"]})
+ losses = outcomes._nll(targets, predictions)
+ candidates.append(
+ {
+ "penalty": penalty,
+ "cheap_nll": float(losses[0]),
+ "strong_nll": float(losses[1]),
+ "summed_binary_logloss": float(losses.sum()),
+ "oof_probabilities": predictions.tolist(),
+ "optimizer_runs": optimizers,
+ }
+ )
+ if progress is not None:
+ progress(f"Training family-fold CV lambda={penalty:g}: summed NLL={losses.sum():.6f}")
+ winner = min(candidates, key=lambda item: (item["summed_binary_logloss"], -item["penalty"]))
+ models = {name: fit_logistic(matrix, targets[:, arm], winner["penalty"]) for arm, name in enumerate(outcomes.ARMS)}
+ return {
+ "training_ids": [row["id"] for row in rows],
+ "targets": targets.astype(int).tolist(),
+ "folds": folds.tolist(),
+ "cv": candidates,
+ "penalty": winner["penalty"],
+ "models": models,
+ "training_cost_proxies": costs,
+ "conditional_cost_statistics": statistics,
+ }
diff --git a/examples/train-a-router/arithmetic_replay.py b/examples/train-a-router/arithmetic_replay.py
new file mode 100644
index 000000000..b09bcac74
--- /dev/null
+++ b/examples/train-a-router/arithmetic_replay.py
@@ -0,0 +1,418 @@
+"""Reproduce the public arithmetic study and train a router from its saved labels.
+
+This is an exposed reproduction: the published test results are already known.
+The original acquisition, model freeze and test-access records stay unchanged.
+"""
+
+from __future__ import annotations
+
+import argparse
+import copy
+import json
+from pathlib import Path
+
+import numpy as np
+
+import arithmetic_controls
+import arithmetic_optimizer
+import arithmetic_study as study_code
+import benefit_outcomes as outcomes
+import benefit_outcomes_native as native
+import evidence
+import solver_report
+from data import digest, file_hash, write
+from plot_arithmetic import LABELS, ORDER, label_for
+from serving import vectors
+
+POPULATIONS = {"original": 500, "expanded": 2000}
+MODE = "Public reproduction using previously exposed outcomes; this is not a new blind test."
+
+
+def read(path):
+ return json.loads(Path(path).read_text())
+
+
+def reference(study):
+ study = Path(study)
+ evidence.verify(study, manifest_sha256=evidence.ARITHMETIC_MANIFEST_SHA256)
+ root = study / "arithmetic"
+ study_code.checked_source(root / "source")
+ return root
+
+
+def prepare(study, output):
+ root, output = reference(study), Path(output)
+ plan = study_code.prepare(
+ root / "source", output, root / "reference/embedding", root / "reference/native"
+ )
+ records = {}
+ for split in ("train", "calibration"):
+ path = root / "study" / f"grades-{split}.json"
+ record = read(path)
+ original = record["study_plan_sha256"]
+ record.update(
+ study_plan_sha256=file_hash(output / "plan.json"),
+ reproduction={
+ "mode": MODE,
+ "source_grades_sha256": file_hash(path),
+ "source_study_plan_sha256": original,
+ "grades_recomputed": False,
+ },
+ )
+ write(output / path.name, record)
+ records[path.name] = file_hash(output / path.name)
+ provenance = {
+ "format": "sie-arithmetic-public-reproduction-v1",
+ "mode": MODE,
+ "evidence_manifest_sha256": file_hash(Path(study) / "manifest.json"),
+ "study_plan_sha256": file_hash(output / "plan.json"),
+ "copied_grades_sha256": records,
+ "solver_settings_sha256": digest(study_code.solver.SETTINGS),
+ "training_rows": len(plan["training_ids"]),
+ "policy_files_sha256": {},
+ }
+ write(output / "reproduction.json", provenance)
+ return provenance
+
+
+def checked_run(run, study):
+ root, run = reference(study), Path(run)
+ saved = read(run / "reproduction.json")
+ if (
+ saved.get("format") != "sie-arithmetic-public-reproduction-v1"
+ or saved.get("evidence_manifest_sha256") != file_hash(Path(study) / "manifest.json")
+ or saved.get("study_plan_sha256") != file_hash(run / "plan.json")
+ or saved.get("solver_settings_sha256") != digest(study_code.solver.SETTINGS)
+ or set(saved.get("copied_grades_sha256", {})) != {"grades-train.json", "grades-calibration.json"}
+ or any(file_hash(run / name) != sha for name, sha in saved["copied_grades_sha256"].items())
+ ):
+ raise ValueError("Prepared reproduction inputs or model protocol changed")
+ study_code.checked_plan(root / "source", run)
+ return root
+
+
+def score_head(matrix, head):
+ probabilities = np.column_stack([outcomes.probability(matrix, head["models"][arm]) for arm in outcomes.ARMS])
+ return outcomes.normalized_scores(probabilities, head["conditional_cost_statistics"])[0].tolist()
+
+
+def save_policy(root, run, name, scores, *, design, model_path, training_rows):
+ """Choose cutoffs from calibration, without opening published test outcomes."""
+ _, inputs, _ = study_code.checked_plan(root / "source", run)
+ calibration = read(run / "grades-calibration.json")["rows"]
+ ids = {split: [row["id"] for row in inputs[split]] for split in ("calibration", "test")}
+ if [row["id"] for row in calibration] != ids["calibration"]:
+ raise ValueError("Calibration outcomes and question scores have different order")
+ if any(len(scores[split]) != len(ids[split]) or not np.isfinite(scores[split]).all() for split in ids):
+ raise ValueError("Model scores do not cover the saved question order")
+ charges = {
+ split: study_code.overhead(root / "embeddings/fresh", ids[split]) if design == "embedding" else None
+ for split in ids
+ }
+ result = {
+ "format": "sie-arithmetic-public-policy-v1",
+ "mode": MODE,
+ "method": name,
+ "design": design,
+ "training_rows": training_rows,
+ "study_plan_sha256": file_hash(run / "plan.json"),
+ "model_path": str(model_path.relative_to(run)),
+ "model_sha256": file_hash(model_path),
+ "calibration_grades_sha256": file_hash(run / "grades-calibration.json"),
+ "ids": ids,
+ "scores": scores,
+ "router_credits": charges,
+ "policies": {
+ str(target): solver_report.select_cutoff(
+ calibration, scores["calibration"], target, router_credits=charges["calibration"], credit_usd=1e-5
+ )
+ for target in study_code.TARGETS
+ },
+ }
+ policy_path = run / f"policy-{name}.json"
+ write(policy_path, result)
+ provenance = read(run / "reproduction.json")
+ provenance["policy_files_sha256"][policy_path.name] = file_hash(policy_path)
+ temporary = run / ".reproduction.tmp"
+ temporary.write_text(json.dumps(provenance, indent=2) + "\n")
+ temporary.replace(run / "reproduction.json")
+ return result
+
+
+def fit_embedding(run, study, population):
+ run = Path(run)
+ root = checked_run(run, study)
+ size = POPULATIONS[population]
+ model = study_code.fit_embedding(root / "source", run, root / "embeddings/train", size)
+ _, inputs, _ = study_code.checked_plan(root / "source", run)
+ scores = {
+ split: score_head(vectors(inputs[split], root / "embeddings/fresh"), model["head"])
+ for split in ("calibration", "test")
+ }
+ save_policy(
+ root, run, f"embedding_{size}", scores,
+ design="embedding", model_path=run / f"embedding-{size}.json", training_rows=size,
+ )
+ return {"method": "embedding", "training_rows": size, "selected_penalty": model["head"]["penalty"]}
+
+
+def fit_native(run, study, population, *, local_files_only=False):
+ run = Path(run)
+ root = checked_run(run, study)
+ size = POPULATIONS[population]
+ trained = study_code.train_native(root / "source", run, size, local_files_only=local_files_only)
+ study_code.checked_native(root / "source", run, size)
+ _, inputs, _ = study_code.checked_plan(root / "source", run)
+ checkpoint = run / f"native-{size}/lora"
+ manifest = trained["checkpoints"]["lora"]["files_sha256"]
+ if any(file_hash(checkpoint / name) != expected for name, expected in manifest.items()):
+ raise ValueError("Native checkpoint differs from its saved export")
+ scores = {
+ split: native.score_checkpoint(checkpoint, inputs[split], "cuda", trained["conditional_cost_statistics"])
+ for split in ("calibration", "test")
+ }
+ save_policy(
+ root, run, f"lora_{size}", scores,
+ design="native", model_path=run / f"native-{size}/trained.json", training_rows=size,
+ )
+ return {"method": "native", "training_rows": size, "selected_epoch": trained["selected_epoch"]}
+
+
+def fit_teacher(run, study):
+ """Refit the same two heads from the published, matched training targets."""
+ run = Path(run)
+ root = checked_run(run, study)
+ teacher = read(root / "study/teacher-models.json")
+ labels_path = root / "study/teacher-labels.json"
+ labels = read(labels_path)
+ if teacher["teacher_labels_sha256"] != file_hash(labels_path):
+ raise ValueError("Saved teacher labels differ from their published provenance")
+ plan, inputs, metadata = study_code.checked_plan(root / "source", run)
+ source_heads = teacher["heads"]
+ names = study_code.OPTIONAL_METHODS
+ teacher_rows = {row["id"]: row for row in labels["rows"] if row["usable"]}
+ selected = set(teacher_rows)
+ positions = [i for i, row in enumerate(inputs["train"]) if row["id"] in selected]
+ ids = [inputs["train"][i]["id"] for i in positions]
+ if (
+ set(source_heads) != set(names)
+ or set(ids) != selected
+ or any(head["training_ids"] != ids for head in source_heads.values())
+ ):
+ raise ValueError("Teacher and reference targets must cover the same ordered training questions")
+ training_inputs = [inputs["train"][i] for i in positions]
+ matrix = vectors(training_inputs, root / "embeddings/train")
+ groups = [metadata["train"][i]["family"] for i in positions]
+ folds = [plan["training_folds"][i] for i in positions]
+ reference_rows = {row["id"]: row for row in read(run / "grades-train.json")["rows"]}
+ costs = [[reference_rows[identity][f"{arm}_credits"] for arm in outcomes.ARMS] for identity in ids]
+ heads = {}
+ for name in names:
+ original = source_heads[name]
+ targets = [
+ [
+ int(teacher_rows[identity]["grades"][arm]["correct"])
+ if name == names[0] else int(reference_rows[identity][f"{arm}_correct"])
+ for arm in outcomes.ARMS
+ ]
+ for identity in ids
+ ]
+ if original["folds"] != folds or original["targets"] != targets or original["training_cost_proxies"] != costs:
+ raise ValueError("Saved teacher targets or folds differ from the prepared source")
+ rows = [
+ {"id": identity, **{f"{arm}_correct": bool(value) for arm, value in zip(outcomes.ARMS, target, strict=True)}}
+ for identity, target in zip(ids, targets, strict=True)
+ ]
+ heads[name] = arithmetic_optimizer.fit_head(
+ matrix, rows, groups, costs, fold_assignments=folds, progress=print
+ )
+ result = {
+ "heads": heads,
+ "summary": teacher["summary"],
+ "study_plan_sha256": file_hash(run / "plan.json"),
+ "training_grades_sha256": file_hash(run / "grades-train.json"),
+ "source_teacher_models_sha256": file_hash(root / "study/teacher-models.json"),
+ "source_teacher_labels_sha256": file_hash(labels_path),
+ "mode": MODE,
+ }
+ write(run / "teacher-models.json", result)
+ matrices = {split: vectors(inputs[split], root / "embeddings/fresh") for split in ("calibration", "test")}
+ for name, head in heads.items():
+ scores = {split: score_head(features, head) for split, features in matrices.items()}
+ save_policy(
+ root, run, name, scores,
+ design="embedding", model_path=run / "teacher-models.json", training_rows=len(ids),
+ )
+ return {"method": "teacher", "training_rows": len(ids), "matched_reference_rows": len(ids)}
+
+
+def method_report(model, rows):
+ if [row["id"] for row in rows] != model["ids"]["test"]:
+ raise ValueError("Published outcomes and frozen decisions have different question order")
+ scores = np.asarray(model["scores"]["test"])
+ policies = {}
+ for target, policy in model["policies"].items():
+ mode = policy["mode"]
+ decisions = (scores >= policy["cutoff"]).tolist() if mode == "threshold" else mode
+ charges = model["router_credits"]["test"] if mode == "threshold" else 0
+ policies[target] = {
+ "mode": mode, "cutoff": policy["cutoff"],
+ "metrics": solver_report.summarize(rows, decisions, router_credits=charges),
+ }
+ return {
+ "label": LABELS.get(model.get("method"), model.get("method", "")),
+ "design": model["design"], "training_rows": model["training_rows"], "policies": policies,
+ }
+
+
+def evaluate(run, study, population, method, output):
+ run, output = Path(run), Path(output)
+ root = checked_run(run, study)
+ size = POPULATIONS[population]
+ names = study_code.OPTIONAL_METHODS if method == "teacher" else [f"{'lora' if method == 'native' else 'embedding'}_{size}"]
+ models = {}
+ provenance = read(run / "reproduction.json")
+ for name in names:
+ policy_path = run / f"policy-{name}.json"
+ model = read(policy_path)
+ model_path = Path(model["model_path"])
+ if (
+ model_path.is_absolute() or ".." in model_path.parts
+ or provenance["policy_files_sha256"].get(policy_path.name) != file_hash(policy_path)
+ or model.get("method") != name
+ or model.get("design") != ("native" if method == "native" else "embedding")
+ or (method != "teacher" and model.get("training_rows") != size)
+ or model["study_plan_sha256"] != file_hash(run / "plan.json")
+ or model["model_sha256"] != file_hash(run / model_path)
+ or model["calibration_grades_sha256"] != file_hash(run / "grades-calibration.json")
+ ):
+ raise ValueError("Trained model or calibration policy changed after selection")
+ models[name] = model
+ # All cutoffs above are already saved. These are public, previously exposed outcomes.
+ grades = read(root / "study/grades-test.json")
+ result = {
+ "format": "sie-arithmetic-public-evaluation-v1", "mode": MODE, "n": len(grades["rows"]),
+ "test_grades_sha256": file_hash(root / "study/grades-test.json"),
+ "methods": {name: method_report(model, grades["rows"]) for name, model in models.items()},
+ }
+ write(output, result)
+ return result
+
+
+def markdown(report, target="1"):
+ lines = [
+ "# Routing cost and answer quality", "", MODE, "",
+ f"{report['n']:,} test questions; cutoffs selected for a {target}-point calibration gap.", "",
+ "| Method | Answer accuracy | Quality gap | Cost per 10,000 | Savings |",
+ "| --- | ---: | ---: | ---: | ---: |",
+ ]
+ rows = [(name, metric) for name, metric in report["baselines"].items()]
+ rows += [(name, report["methods"][name]["policies"][target]["metrics"]) for name in ORDER if name in report["methods"]]
+ for name, metric in rows:
+ known = metric.get("net_total_cost_usd_per1000") is not None
+ cost = 10 * metric["net_total_cost_usd_per1000" if known else "total_solver_cost_usd_per1000"]
+ saving = metric["net_savings_pct" if known else "gross_solver_savings_pct"]
+ lines.append(
+ f"| {label_for(report, name)} | {100 * metric['answer_accuracy']:.2f}% | "
+ f"{metric['gap_vs_strong_pp']:.2f} points | ${cost:.2f}{'' if known else ' + hosting'} | "
+ f"{saving:.2f}%{'' if known else ' before hosting'} |"
+ )
+ lines += ["", report["cost_scope"], "", "The JSON retains every declared cutoff target and paired 95% uncertainty interval."]
+ return "\n".join(lines) + "\n"
+
+
+def replay_cost_controls(root, grades):
+ directory = root / "study-cost-controls"
+ frozen = arithmetic_controls.checked_frozen(root / "source", root / "study", directory)
+ recorded, access = (read(directory / name) for name in ("report.json", "test-access.json"))
+ if (
+ recorded["frozen_sha256"] != file_hash(directory / "frozen.json")
+ or recorded["core_frozen_sha256"] != frozen["core_frozen_sha256"]
+ or recorded["test_grades_sha256"] != file_hash(root / "study/grades-test.json")
+ or access["frozen_sha256"] != recorded["frozen_sha256"]
+ or access["core_test_access_sha256"] != file_hash(root / "study/test-access.json")
+ or grades["test_access_sha256"] != access["core_test_access_sha256"]
+ or set(recorded["methods"]) != set(frozen["methods"])
+ ):
+ raise ValueError("Cost controls differ from their saved decisions or shared test outcomes")
+ recomputed = copy.deepcopy(recorded)
+ for name, model in frozen["methods"].items():
+ selected = method_report(
+ model | {"ids": frozen["ids"], "method": name, "router_credits": {"test": None}},
+ grades["rows"],
+ )
+ recomputed["methods"][name]["policies"] = selected["policies"]
+ if digest(recomputed) != digest(recorded):
+ raise ValueError("Recomputed cost-control metrics differ from the published result")
+ return recomputed
+
+
+def replay(study, output):
+ root, output = reference(study), Path(output)
+ if output.exists():
+ raise FileExistsError(output)
+ recorded, frozen, grades = (read(root / "study" / name) for name in ("report.json", "frozen.json", "grades-test.json"))
+ if (
+ recorded["frozen_sha256"] != file_hash(root / "study/frozen.json")
+ or recorded["test_grades_sha256"] != file_hash(root / "study/grades-test.json")
+ ):
+ raise ValueError("Recorded comparison and source decisions differ")
+ recomputed = copy.deepcopy(recorded)
+ recomputed["baselines"] = {
+ mode: solver_report.summarize(grades["rows"], mode, router_credits=0) for mode in solver_report.BASELINES
+ }
+ for name, model in frozen["methods"].items():
+ selected = method_report(model | {"ids": frozen["ids"], "method": name}, grades["rows"])
+ recomputed["methods"][name]["policies"] = selected["policies"]
+ if digest(recomputed) != digest(recorded):
+ raise ValueError("Recomputed cost or quality metrics differ from the published result")
+ controls = replay_cost_controls(root, grades)
+ write(output / "comparison.json", recomputed)
+ write(output / "cost-controls.json", controls)
+ presentation = copy.deepcopy(recomputed)
+ presentation["methods"].update(controls["methods"])
+ (output / "comparison.md").write_text(
+ markdown(presentation)
+ + "\nUnchanged-Laya controls use the same cost estimates as each LoRA run, isolating changes to model weights. "
+ "Their separate pre-test freeze and metrics are retained in `cost-controls.json`.\n"
+ )
+ write(output / "replay.json", {
+ "mode": MODE, "metrics_match": True,
+ "source_report_sha256": file_hash(root / "study/report.json"),
+ "source_cost_controls_sha256": file_hash(root / "study-cost-controls/report.json"),
+ })
+ return {"n": recomputed["n"], "metrics_match": True, "output": str(output)}
+
+
+def main(argv=None):
+ parser = argparse.ArgumentParser(description=__doc__)
+ commands = parser.add_subparsers(dest="command", required=True)
+ for name in ("replay", "prepare", "fit-native", "fit-embedding", "fit-teacher", "evaluate"):
+ child = commands.add_parser(name)
+ child.add_argument("--study", type=Path, default=Path("evidence"))
+ if name not in ("prepare", "replay"):
+ child.add_argument("--run", type=Path, required=True)
+ if name in ("fit-native", "fit-embedding", "evaluate"):
+ child.add_argument("--population", choices=POPULATIONS, default="expanded")
+ if name in ("prepare", "replay", "evaluate"):
+ child.add_argument("--output", type=Path, required=True)
+ if name == "fit-native":
+ child.add_argument("--local-files-only", action="store_true")
+ if name == "evaluate":
+ child.add_argument("--method", choices=("embedding", "native", "teacher"), default="embedding")
+ args = parser.parse_args(argv)
+ if args.command in ("replay", "prepare"):
+ result = (replay if args.command == "replay" else prepare)(args.study, args.output)
+ elif args.command == "fit-embedding":
+ result = fit_embedding(args.run, args.study, args.population)
+ elif args.command == "fit-native":
+ result = fit_native(args.run, args.study, args.population, local_files_only=args.local_files_only)
+ elif args.command == "fit-teacher":
+ result = fit_teacher(args.run, args.study)
+ else:
+ result = evaluate(args.run, args.study, args.population, args.method, args.output)
+ print(json.dumps(result, indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/arithmetic_route.py b/examples/train-a-router/arithmetic_route.py
new file mode 100644
index 000000000..0d500b403
--- /dev/null
+++ b/examples/train-a-router/arithmetic_route.py
@@ -0,0 +1,149 @@
+"""Answer a new arithmetic question with a saved router and the public SIE SDK."""
+
+from __future__ import annotations
+
+import math
+import time
+from pathlib import Path
+
+import arithmetic_replay as public
+import benefit_outcomes_native as native
+import benefit_route
+import solver
+from data import digest, file_hash, render_request, write
+from serving import SingleSendClient, error_receipt
+
+
+def selected_model(study, run, population, method):
+ root = public.checked_run(run, study) if run else public.reference(study)
+ size = public.POPULATIONS[population]
+ name = {
+ "embedding": f"embedding_{size}", "lora": f"lora_{size}",
+ "laya": "base", "teacher": "embedding_teacher_2000",
+ }[method]
+ if run and method != "laya":
+ directory = Path(run)
+ policy_path = directory / f"policy-{name}.json"
+ policy = public.read(policy_path)
+ model_path = Path(policy["model_path"])
+ if (
+ model_path.is_absolute() or ".." in model_path.parts
+ or public.read(directory / "reproduction.json")["policy_files_sha256"].get(policy_path.name)
+ != file_hash(policy_path)
+ or policy.get("method") != name
+ or policy.get("design") != ("native" if method == "lora" else "embedding")
+ or (method != "teacher" and policy.get("training_rows") != size)
+ or policy["study_plan_sha256"] != file_hash(directory / "plan.json")
+ or policy["model_sha256"] != file_hash(directory / model_path)
+ or policy["calibration_grades_sha256"] != file_hash(directory / "grades-calibration.json")
+ ):
+ raise ValueError("Saved live model or calibration policy changed")
+ else:
+ directory = root / "study"
+ policy_path = root / "study-cost-controls/frozen.json" if method == "laya" else directory / "frozen.json"
+ frozen = public.read(policy_path)
+ policy = frozen["methods"][f"base_arithmetic_{size}" if method == "laya" else name]
+ if method in ("embedding", "teacher"):
+ path = directory / (f"embedding-{size}.json" if method == "embedding" else "teacher-models.json")
+ record = public.read(path)
+ head = record["head"] if method == "embedding" else record["heads"][name]
+ return policy, policy_path, head, None, None
+ directory = root / "reference/native" if method == "laya" else directory / f"native-{size}"
+ trained = public.read(directory / "trained.json")
+ variant = "base" if method == "laya" else "lora"
+ if method == "laya":
+ trained = trained | {"conditional_cost_statistics": policy["conditional_cost_statistics"]}
+ return policy, policy_path, None, directory / variant, trained
+
+
+def choose_native(policy, checkpoint, trained, request, target, device):
+ selected = policy["policies"][str(target)]
+ score, load_seconds, verification_seconds = None, None, None
+ started = time.perf_counter()
+ if selected["mode"] == "threshold":
+ verification_started = time.perf_counter()
+ for name, expected in trained["checkpoints"][checkpoint.name]["files_sha256"].items():
+ relative = Path(name)
+ if relative.is_absolute() or ".." in relative.parts or file_hash(checkpoint / relative) != expected:
+ raise ValueError("Selected native checkpoint differs; download the optional weights first")
+ verification_seconds = time.perf_counter() - verification_started
+ scoring_started = time.perf_counter()
+ score = float(native.score_checkpoint(
+ checkpoint, [{"request": request}], device, trained["conditional_cost_statistics"]
+ )[0])
+ if not math.isfinite(score):
+ raise ValueError("Native router returned an invalid score")
+ load_seconds = time.perf_counter() - scoring_started
+ arm = "strong" if score >= selected["cutoff"] else "cheap"
+ elif selected["mode"] in ("always_strong", "always_cheap"):
+ arm = selected["mode"].removeprefix("always_")
+ else:
+ raise ValueError("Unknown saved routing rule")
+ return {
+ "router": "laya_outcomes_" + checkpoint.name,
+ "arm": arm, "model": solver.SETTINGS["arms"][arm]["model"],
+ "mode": selected["mode"], "score": score, "cutoff": selected["cutoff"],
+ "calibration_quality_loss_target_pp": target,
+ "embedding_receipt": None, "native_model_calls": int(load_seconds is not None),
+ "local_checkpoint_verification_seconds": verification_seconds,
+ "local_model_load_and_score_seconds": load_seconds,
+ "local_router_end_to_end_seconds": time.perf_counter() - started,
+ "latency_note": "Fresh CLI timings include checkpoint verification and loading; they are not warm server latency.",
+ "cost_note": "Local Laya compute is unpriced; API credits cover answer generation only.",
+ }
+
+
+def answer(
+ study, output, question, *, run=None, population="expanded", method="embedding",
+ target=1, self_hosted=False, device="cpu",
+):
+ output = Path(output)
+ if output.exists():
+ raise FileExistsError(output)
+ if not question.strip() or len(question) > 20000 or target not in (0, 1, 2, 5):
+ raise ValueError("Use a nonempty question up to 20,000 characters and a declared calibration target")
+ if method not in ("embedding", "lora", "laya", "teacher") or device not in ("cpu", "cuda"):
+ raise ValueError("Choose a supported router and CPU or CUDA device")
+ policy, policy_path, head, checkpoint, trained = selected_model(study, run, population, method)
+ request = render_request("GSM8K", question)
+ row = {"id": "new-" + digest(request)[:16], "request": request}
+ write(output / "intent.json", {
+ "id": row["id"], "request_sha256": digest(request), "policy_sha256": file_hash(policy_path),
+ "solver_settings_sha256": digest(solver.SETTINGS), "target_pp": target,
+ "method": method, "self_hosted": self_hosted, "automatic_retries": False,
+ })
+ try:
+ routing = (
+ benefit_route.choose(
+ {"selections": policy["policies"]}, request, target, self_hosted=self_hosted,
+ receipt_path=output / "embedding-receipt.json", outcome_head=head,
+ )
+ if head is not None
+ else choose_native(policy, checkpoint, trained, request, target, device)
+ )
+ write(output / "routing.json", routing)
+ with SingleSendClient(timeout_s=60) as client:
+ metadata = client.get_model(routing["model"])
+ if (
+ metadata.get("revision") != solver.SETTINGS["arms"][routing["arm"]]["revision"]
+ or metadata.get("name", metadata.get("id")) != routing["model"]
+ ):
+ raise ValueError("Live answer model differs from the evaluated model")
+ record = solver.solve(row, routing["arm"], solver.SETTINGS, self_hosted=self_hosted)
+ solver.validate_record(record, row, routing["arm"], solver.SETTINGS)
+ write(output / "answer.json", record)
+ if record["status"] != "complete":
+ raise RuntimeError("Answer request failed; inspect the saved receipt before another attempt")
+ except Exception as error:
+ write(output / "incomplete.json", error_receipt(error))
+ raise
+ summary = {
+ "model": routing["model"], "answer": record["answer"], "solver_calls": len(record["calls"]),
+ "embedding_calls": int(routing["embedding_receipt"] is not None),
+ "native_model_calls": routing.get("native_model_calls", 0),
+ "total_api_credits": None if self_hosted else record["credits"] + (routing["embedding_receipt"] or {}).get("credits", 0),
+ "local_compute_cost": "unpriced",
+ "answer_quality": "Not graded; no reference answer was supplied for this new question.",
+ }
+ write(output / "result.json", summary)
+ return summary
diff --git a/examples/train-a-router/arithmetic_study.py b/examples/train-a-router/arithmetic_study.py
new file mode 100644
index 000000000..ea15f7b37
--- /dev/null
+++ b/examples/train-a-router/arithmetic_study.py
@@ -0,0 +1,861 @@
+"""Fit and evaluate the fixed arithmetic routing recipe on disjoint source families."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from concurrent.futures import ProcessPoolExecutor
+from pathlib import Path
+
+import numpy as np
+
+import arithmetic_answers
+import benefit
+import benefit_costs
+import benefit_outcomes as outcomes
+import benefit_outcomes_native as native
+import solver
+import solver_report
+from arithmetic_data import COUNTS
+from arithmetic_data import FORMAT as SOURCE_FORMAT
+from data import digest, file_hash, write
+from serving import vectors
+
+TARGETS = (1, 0, 2, 5)
+METHODS = (
+ "base",
+ "embedding_original_cutoff",
+ "lora_original_cutoff",
+ "embedding_calibration_only",
+ "lora_calibration_only",
+ "embedding_500",
+ "embedding_2000",
+ "lora_500",
+ "lora_2000",
+)
+OPTIONAL_METHODS = ("embedding_teacher_2000", "embedding_teacher_matched_reference")
+UNRESOLVED = {"gold_unparseable", "grading_error", "unit_contract_unresolved"}
+PROTOCOL = {
+ "population": "All exposed2000 arithmetic rows train/develop; fresh1000 calibration and3000 test families.",
+ "training_sizes": [500, 2000],
+ "embedding": {
+ key: outcomes.PROTOCOL[key]
+ for key in (
+ "encoder",
+ "revision",
+ "instruction",
+ "dimensions",
+ "penalties",
+ "folds",
+ "seed",
+ "fit",
+ "selection",
+ "optimizer",
+ "cost_model",
+ "score",
+ "incremental_cost_floor_credits",
+ )
+ },
+ "native": native.PROTOCOL,
+ "folds": "One assignment from all2000 source families; nested500 keeps its original family folds.",
+ "primary_target_pp": 1,
+ "secondary_targets_pp": [0, 2, 5],
+ "test_boundary": "Save every model, input-only test score and calibration cutoff before reading fresh test answers.",
+ "desired_result": "At least30% net saving near1pp quality loss is a goal, not a promised or guaranteed result.",
+ "optional_teacher_comparison": (
+ "Question-only teacher answers may replace training references in a matched pair of embedding heads. "
+ "Both use the same reference-usable subset, original family folds, penalty grid and physical costs; "
+ "conditional cost means are recomputed under each label source. No benchmark fallback. "
+ "Include both only if their heads are saved before test access."
+ ),
+}
+
+
+def read(path):
+ return json.loads(Path(path).read_text())
+
+
+def require(condition, message):
+ if not condition:
+ raise ValueError(message)
+
+
+def code_hashes():
+ names = (
+ "arithmetic_study.py",
+ "arithmetic_answers.py",
+ "teacher_format.py",
+ "benefit_outcomes.py",
+ "benefit_outcomes_native.py",
+ "finetune.py",
+ "solver.py",
+ "solver_answers.py",
+ "solver_gold.py",
+ "solver_report.py",
+ "benefit_costs.py",
+ "serving.py",
+ "data.py",
+ )
+ return {name: file_hash(Path(__file__).with_name(name)) for name in names}
+
+
+def checked_source(directory, *, include_gold=False):
+ directory = Path(directory)
+ plan = read(directory / "source-plan.json")
+ require(plan.get("format") == SOURCE_FORMAT and plan.get("counts") == COUNTS, "Arithmetic source protocol differs")
+ require(
+ all(
+ file_hash(directory / name) == expected
+ for name, expected in plan["files_sha256"].items()
+ if include_gold or name != "gold-for-scoring-only.json"
+ ),
+ "Prepared arithmetic source changed",
+ )
+ inputs, metadata = read(directory / "inputs.json"), read(directory / "rows.json")
+ families = {}
+ all_ids = []
+ for split in ("train", "calibration", "test"):
+ ids = [row["id"] for row in inputs[split]]
+ require(
+ len(ids) == COUNTS[split]
+ and ids == [row["id"] for row in metadata[split]]
+ and all(set(row) == {"id", "request"} for row in inputs[split]),
+ "Input order, metadata or counts changed",
+ )
+ all_ids.extend(ids)
+ for row in metadata[split]:
+ previous = families.setdefault(row["family"], split)
+ require(previous == split, "A question family crosses study roles")
+ require(len(set(all_ids)) == len(all_ids), "Question identities cross study roles")
+ small = read(directory / "train-small-ids.json")
+ small_ids = set(small)
+ train_ids = {row["id"] for row in inputs["train"]}
+ require(
+ len(small) == len(small_ids) == COUNTS["train_small"] and small_ids <= train_ids,
+ "The nested training subset changed",
+ )
+ for family in {row["family"] for row in metadata["train"]}:
+ members = {row["id"] for row in metadata["train"] if row["family"] == family}
+ require(not members & small_ids or members <= small_ids, "Nested training splits a source family")
+ return plan, inputs, metadata
+
+
+def prepare(source, run, reference_embedding, reference_native):
+ source, run, reference_embedding, reference_native = map(Path, (source, run, reference_embedding, reference_native))
+ if run.exists():
+ raise FileExistsError(run)
+ _, inputs, metadata = checked_source(source)
+ groups = [row["family"] for row in metadata["train"]]
+ folds = benefit.group_folds(groups)
+ plan = {
+ "format": "sie-arithmetic-adaptation-plan-v1",
+ "created_at": benefit.now(),
+ "protocol": PROTOCOL,
+ "methods": list(METHODS),
+ "optional_methods": list(OPTIONAL_METHODS),
+ "source_plan_sha256": file_hash(source / "source-plan.json"),
+ "training_ids": [row["id"] for row in inputs["train"]],
+ "training_families": groups,
+ "training_folds": folds,
+ "small_training_ids": read(source / "train-small-ids.json"),
+ "warm_probe_ids": [
+ row["id"] for row in sorted(inputs["train"], key=lambda row: digest(["arithmetic-warm-v1", row["id"]]))[:21]
+ ],
+ "reference_embedding_files_sha256": {
+ name: file_hash(reference_embedding / name) for name in ("selected-model.json", "frozen.json")
+ },
+ "reference_native_files_sha256": {
+ name: file_hash(reference_native / name) for name in ("trained.json", "frozen.json")
+ },
+ "source_code_sha256": code_hashes(),
+ "exposure": "The previous2000 arithmetic evaluation outcomes are development data; none are fresh test examples.",
+ }
+ write(run / "plan.json", plan)
+ return plan
+
+
+def checked_plan(source, run):
+ plan = read(Path(run) / "plan.json")
+ _, inputs, metadata = checked_source(source)
+ require(
+ plan.get("format") == "sie-arithmetic-adaptation-plan-v1"
+ and plan.get("protocol") == PROTOCOL
+ and plan.get("methods") == list(METHODS)
+ and plan.get("optional_methods") == list(OPTIONAL_METHODS)
+ and plan.get("source_plan_sha256") == file_hash(Path(source) / "source-plan.json")
+ and plan.get("training_ids") == [row["id"] for row in inputs["train"]]
+ and plan.get("training_families") == [row["family"] for row in metadata["train"]]
+ and plan.get("small_training_ids") == read(Path(source) / "train-small-ids.json")
+ and plan.get("source_code_sha256") == code_hashes(),
+ "Prepared experiment, source or protocol changed",
+ )
+ require(
+ plan["training_folds"] == benefit.group_folds(plan["training_families"]),
+ "Training family folds changed",
+ )
+ return plan, inputs, metadata
+
+
+def unopened(run):
+ for name in ("test-access.json", "grades-test.json", "report.json"):
+ if (Path(run) / name).exists():
+ raise FileExistsError(Path(run) / name)
+
+
+def _grade_pair(item):
+ question, meta, reference, records, hashes, rates = item
+ row, unresolved = {"id": question["id"], "family": meta["family"]}, []
+ for arm in outcomes.ARMS:
+ record = records[arm]
+ try:
+ measured = arithmetic_answers.grade_reference(question["request"], reference, record["answer"])
+ except Exception as error: # noqa: BLE001 - collect every unresolved row, then fail the whole grading phase.
+ measured = {"correct": False, "status": "grading_error", "error_type": type(error).__name__}
+ if measured["status"] in UNRESOLVED:
+ unresolved.append(
+ {
+ "id": question["id"],
+ "arm": arm,
+ "status": measured["status"],
+ "question": question["request"],
+ "reference": reference,
+ "final_answer": measured.get("final_answer"),
+ "record_sha256": hashes[arm],
+ }
+ )
+ calls = benefit_costs.physical_costs(record, rates[arm])
+ row.update(
+ {
+ f"{arm}_correct": measured["correct"],
+ f"{arm}_grade": measured,
+ f"{arm}_record_sha256": hashes[arm],
+ f"{arm}_credits": sum(call["credits_proxy"] for call in calls),
+ f"{arm}_cost_calls": calls,
+ }
+ )
+ return row, unresolved
+
+
+def grade(source, run, collection, prices, split, *, workers=8):
+ source, run, collection, prices = map(Path, (source, run, collection, prices))
+ plan, inputs, metadata = checked_plan(source, run)
+ if split not in ("train", "calibration", "test"):
+ raise ValueError("Choose train, calibration or test")
+ require(type(workers) is int and 1 <= workers <= 16, "Use 1..16 grading processes")
+ output = run / f"grades-{split}.json"
+ if output.exists():
+ raise FileExistsError(output)
+ if split == "test":
+ access = read(run / "test-access.json")
+ require(
+ access["frozen_sha256"] == file_hash(run / "frozen.json")
+ and access["study_plan_sha256"] == file_hash(run / "plan.json"),
+ "Freeze every model and cutoff before opening fresh test answers",
+ )
+ else:
+ unopened(run)
+ checked_source(source, include_gold=True)
+ acquisition_plan = read(collection / "collection-plan.json")
+ acquisition = read(collection / "collection.json")
+ source_plan = read(source / "source-plan.json")
+ expected_source = (
+ source_plan["exposed_source_plan_sha256"] if split == "train" else file_hash(source / "fresh/source-plan.json")
+ )
+ expected_count = COUNTS["train"] if split == "train" else COUNTS["calibration"] + COUNTS["test"]
+ collection_inputs = inputs["train"] if split == "train" else inputs["calibration"] + inputs["test"]
+ require(
+ acquisition.get("status") == "complete"
+ and acquisition.get("completed") == 2 * expected_count
+ and acquisition.get("missing") == 0
+ and acquisition.get("unknown_execution_attempts") == 0
+ and isinstance(acquisition.get("receipt_sha256"), dict)
+ and acquisition_plan.get("source_plan_sha256") == expected_source
+ and acquisition_plan.get("settings_sha256") == digest(solver.SETTINGS)
+ and set(acquisition_plan["records_sha256"])
+ == {f"{row['id']}-{arm}.json" for row in collection_inputs for arm in outcomes.ARMS},
+ "Need the complete pinned paired collection",
+ )
+ for relative, expected in acquisition["receipt_sha256"].items():
+ path = Path(relative)
+ require(
+ not path.is_absolute()
+ and ".." not in path.parts
+ and isinstance(expected, str)
+ and len(expected) == 64
+ and all(character in "0123456789abcdef" for character in expected),
+ "Invalid acquisition lineage mapping",
+ )
+ gold = read(source / "gold-for-scoring-only.json")
+ rates, public_rows = benefit_costs.rates_from_snapshot(read(prices))
+ jobs = []
+ for question, meta in zip(inputs[split], metadata[split], strict=True):
+ records, hashes = {}, {}
+ for arm in outcomes.ARMS:
+ path = collection / f"{question['id']}-{arm}.json"
+ hashes[arm] = file_hash(path)
+ require(
+ hashes[arm] == acquisition_plan["records_sha256"].get(path.name),
+ "A collected answer differs from the gathered manifest",
+ )
+ record = read(path)
+ solver.validate_record(record, question, arm, solver.SETTINGS)
+ records[arm] = record
+ jobs.append((question, meta, gold[question["id"]], records, hashes, rates))
+ rows, unresolved = [], []
+ executor = ProcessPoolExecutor(max_workers=workers) if workers > 1 else None
+ try:
+ results = executor.map(_grade_pair, jobs, chunksize=8) if executor else map(_grade_pair, jobs)
+ for i, (row, flags) in enumerate(results, 1):
+ rows.append(row)
+ unresolved.extend(flags)
+ if i % 100 == 0:
+ print(f"Graded {i}/{len(jobs)} {split} pairs; unresolved={len(unresolved)}", flush=True)
+ finally:
+ if executor:
+ executor.shutdown()
+ if unresolved:
+ write(run / f"unresolved-{split}.json", {"cases": unresolved, "source_code_sha256": code_hashes()})
+ raise ValueError(f"{len(unresolved)} unresolved comparisons; retain every row and review the complete list")
+ result = {
+ "format": "sie-arithmetic-adaptation-grades-v1",
+ "split": split,
+ "study_plan_sha256": file_hash(run / "plan.json"),
+ "source_plan_sha256": plan["source_plan_sha256"],
+ "collection_plan_sha256": file_hash(collection / "collection-plan.json"),
+ "collection_summary_sha256": file_hash(collection / "collection.json"),
+ "acquisition_receipt_mapping_sha256": digest(acquisition["receipt_sha256"]),
+ "acquisition_verification": (
+ "Every selected final record is hash-verified and validated against the exact request/model/settings. "
+ "The unchanged complete acquisition-receipt mapping is bound; optional raw execution lineage "
+ "is not required to recompute answer grades."
+ ),
+ "gold_sha256": file_hash(source / "gold-for-scoring-only.json"),
+ "prices_sha256": file_hash(prices),
+ "public_price_rows": public_rows,
+ "grading_source_sha256": {
+ name: code_hashes()[name] for name in ("arithmetic_answers.py", "teacher_format.py", "solver_answers.py")
+ },
+ "test_access_sha256": file_hash(run / "test-access.json") if split == "test" else None,
+ "cost_scope": "Every physical call priced at pinned uncached Cloud rates; not observed Cloud bills or Modal spend.",
+ "rows": rows,
+ }
+ write(output, result)
+ return result
+
+
+def training(source, run, size):
+ plan, inputs, metadata = checked_plan(source, run)
+ require(size in PROTOCOL["training_sizes"], "Choose the declared500 or2000 training stage")
+ grades = read(Path(run) / "grades-train.json")
+ require(
+ grades["study_plan_sha256"] == file_hash(Path(run) / "plan.json")
+ and [row["id"] for row in grades["rows"]] == plan["training_ids"],
+ "Training outcomes changed or do not match source order",
+ )
+ ids = set(plan["small_training_ids"] if size == 500 else plan["training_ids"])
+ positions = [i for i, row in enumerate(inputs["train"]) if row["id"] in ids]
+ require(len(positions) == size, "Training subset differs")
+ return {
+ "inputs": [inputs["train"][i] for i in positions],
+ "rows": [grades["rows"][i] for i in positions],
+ "groups": [metadata["train"][i]["family"] for i in positions],
+ "folds": [plan["training_folds"][i] for i in positions],
+ "grades": grades,
+ "positions": positions,
+ }
+
+
+def fit_embedding(source, run, embeddings, size):
+ run, embeddings = Path(run), Path(embeddings)
+ unopened(run)
+ output = run / f"embedding-{size}.json"
+ if output.exists():
+ raise FileExistsError(output)
+ context = training(source, run, size)
+ matrix = vectors(context["inputs"], embeddings)
+ require(matrix.shape == (size, PROTOCOL["embedding"]["dimensions"]), "Pinned embedding dimensions changed")
+ costs = [[row[f"{arm}_credits"] for arm in outcomes.ARMS] for row in context["rows"]]
+ head = outcomes.fit_head(
+ matrix,
+ context["rows"],
+ context["groups"],
+ costs,
+ fold_assignments=context["folds"],
+ progress=lambda value: print(value, flush=True),
+ )
+ result = {
+ "format": "sie-arithmetic-embedding-model-v1",
+ "training_rows": size,
+ "study_plan_sha256": file_hash(run / "plan.json"),
+ "training_grades_sha256": file_hash(run / "grades-train.json"),
+ "embedding_files_sha256": {
+ name: file_hash(embeddings / name) for name in ("plan.json", "vectors.jsonl.gz", "collection.json")
+ },
+ "head": head,
+ "source_code_sha256": code_hashes(),
+ }
+ write(output, result)
+ return result
+
+
+def train_native(source, run, size, *, local_files_only=False):
+ run = Path(run)
+ unopened(run)
+ output = run / f"native-{size}"
+ if output.exists():
+ raise FileExistsError(output)
+ context = training(source, run, size)
+ targets = np.array([[int(row[f"{arm}_correct"]) for arm in outcomes.ARMS] for row in context["rows"]])
+ costs = [[row[f"{arm}_credits"] for arm in outcomes.ARMS] for row in context["rows"]]
+ held = np.asarray(context["folds"]) == 0
+ plan = {
+ "format": "sie-arithmetic-native-plan-v1",
+ "protocol": native.PROTOCOL,
+ "head_plan_sha256": file_hash(run / "plan.json"),
+ "selection_validation_ids": [row["id"] for row, flag in zip(context["inputs"], held, strict=True) if flag],
+ "training_ids": [row["id"] for row in context["inputs"]],
+ "training_folds": context["folds"],
+ "source_code_sha256": code_hashes(),
+ }
+ write(output / "plan.json", plan)
+ return native._train_cuda(
+ output,
+ run,
+ {
+ "plan": plan,
+ "rows": context["inputs"],
+ "targets": targets,
+ "held": held,
+ "costs": costs,
+ "statistics": outcomes.conditional_costs(targets, costs),
+ "cost_proxy_provenance": {
+ "prices_sha256": context["grades"]["prices_sha256"],
+ "cost_scope": context["grades"]["cost_scope"],
+ },
+ "training_grade_provenance": {key: value for key, value in context["grades"].items() if key != "rows"},
+ "training_artifacts_sha256": {
+ "study_plan": file_hash(run / "plan.json"),
+ "grades": file_hash(run / "grades-train.json"),
+ "source_plan": file_hash(Path(source) / "source-plan.json"),
+ },
+ },
+ local_files_only,
+ )
+
+
+def score_native(source, run, reference_native, *, device="cuda"):
+ run, reference_native = Path(run), Path(reference_native)
+ unopened(run)
+ output = run / "native-scores.json"
+ if output.exists():
+ raise FileExistsError(output)
+ plan, inputs, _ = checked_plan(source, run)
+ require(
+ all(
+ file_hash(reference_native / name) == expected
+ for name, expected in plan["reference_native_files_sha256"].items()
+ ),
+ "Reference native model metadata changed",
+ )
+ populations = {split: inputs[split] for split in ("calibration", "test")}
+ models = {}
+ for label, directory, variants in (
+ ("reference", reference_native, ("base", "lora")),
+ ("500", run / "native-500", ("lora",)),
+ ("2000", run / "native-2000", ("lora",)),
+ ):
+ trained = read(directory / "trained.json") if label == "reference" else checked_native(source, run, int(label))
+ statistics = trained["conditional_cost_statistics"]
+ for variant in variants:
+ manifest = trained["checkpoints"][variant]["files_sha256"]
+ require(
+ all(file_hash(directory / variant / name) == expected for name, expected in manifest.items()),
+ "Native checkpoint differs from its training export",
+ )
+ key = f"{label}_{variant}"
+ models[key] = {"trained_sha256": file_hash(directory / "trained.json"), "splits": {}}
+ for split, rows in populations.items():
+ margins = native.checkpoint_margins(directory / variant, rows, device)
+ probabilities = native.probabilities(margins)
+ scores, expected = outcomes.normalized_scores(probabilities, statistics)
+ models[key]["splits"][split] = {
+ "ids": [row["id"] for row in rows],
+ "margins": margins.tolist(),
+ "probabilities": probabilities.tolist(),
+ "scores": scores.tolist(),
+ "expected_costs_proxy": expected.tolist(),
+ }
+ result = {
+ "format": "sie-arithmetic-native-scores-v1",
+ "study_plan_sha256": file_hash(run / "plan.json"),
+ "source_plan_sha256": file_hash(Path(source) / "source-plan.json"),
+ "source_code_sha256": code_hashes(),
+ "models": models,
+ "solver_test_answers_read": False,
+ }
+ write(output, result)
+ return result
+
+
+def checked_native(source, run, size):
+ run = Path(run)
+ context = training(source, run, size)
+ directory = run / f"native-{size}"
+ plan = read(directory / "plan.json")
+ trained = benefit.checked(read(directory / "trained.json"), native.TRAINED_FORMAT)
+ targets = [[int(row[f"{arm}_correct"]) for arm in outcomes.ARMS] for row in context["rows"]]
+ costs = [[row[f"{arm}_credits"] for arm in outcomes.ARMS] for row in context["rows"]]
+ held = np.asarray(context["folds"]) == 0
+ selected = native.selected_epoch(trained["selection_history"], np.asarray(targets)[held])
+ start = read(directory / "training-start.json")
+ require(
+ plan["protocol"] == native.PROTOCOL
+ and plan["source_code_sha256"] == code_hashes()
+ and plan["head_plan_sha256"] == file_hash(run / "plan.json")
+ and plan["training_ids"] == [row["id"] for row in context["inputs"]]
+ and plan["training_folds"] == context["folds"]
+ and trained["plan_sha256"] == file_hash(directory / "plan.json")
+ and trained["head_plan_sha256"] == file_hash(run / "plan.json")
+ and trained["training_artifacts_sha256"]["grades"] == file_hash(run / "grades-train.json")
+ and trained["training_start_sha256"] == file_hash(directory / "training-start.json")
+ and start["plan_sha256"] == file_hash(directory / "plan.json")
+ and start["runtime"] == native.PROTOCOL["runtime"]
+ and start["eager_flags"] == native.PROTOCOL["eager_flags"]
+ and {name: version.split("+", 1)[0] for name, version in start["versions"].items()}
+ == native.PROTOCOL["versions"]
+ and plan["selection_validation_ids"]
+ == [row["id"] for row, flag in zip(context["inputs"], held, strict=True) if flag]
+ and trained["selected_epoch"] == selected
+ and len(trained["refit_losses"]) == selected
+ and set(trained["checkpoints"]) == {"base", "lora"}
+ and trained["training_targets"] == targets
+ and trained["training_cost_proxies"] == costs
+ and trained["conditional_cost_statistics"] == outcomes.conditional_costs(targets, costs)
+ and all(file_hash(directory / name) == sha for name, sha in trained["receipts_sha256"].items()),
+ "Native checkpoint training labels, protocol or receipts changed",
+ )
+ return trained
+
+
+def checked_embedding(source, run, size):
+ run = Path(run)
+ context = training(source, run, size)
+ model = read(run / f"embedding-{size}.json")
+ head = model["head"]
+ targets = [[int(row[f"{arm}_correct"]) for arm in outcomes.ARMS] for row in context["rows"]]
+ costs = [[row[f"{arm}_credits"] for arm in outcomes.ARMS] for row in context["rows"]]
+ require(
+ model["training_rows"] == size
+ and model["study_plan_sha256"] == file_hash(run / "plan.json")
+ and model["training_grades_sha256"] == file_hash(run / "grades-train.json")
+ and model["source_code_sha256"] == code_hashes()
+ and head["training_ids"] == [row["id"] for row in context["inputs"]]
+ and head["folds"] == context["folds"]
+ and head["targets"] == targets
+ and head["training_cost_proxies"] == costs
+ and head["conditional_cost_statistics"] == outcomes.conditional_costs(targets, costs),
+ "Embedding model training labels, folds or protocol changed",
+ )
+ return model
+
+
+def overhead(directory, ids):
+ receipt = read(Path(directory) / "collection.json")
+ costs = {}
+ for call in receipt["calls"]:
+ require(bool(call["ids"]) and solver.valid_credit(call.get("credits")), "Need measured embedding charges")
+ for identity in call["ids"]:
+ require(identity not in costs, "Embedding input charged in multiple batches")
+ costs[identity] = call["credits"] / len(call["ids"])
+ require(set(ids) <= set(costs), "Embedding charges do not cover selected inputs")
+ return [costs[identity] for identity in ids]
+
+
+def freeze(source, run, embeddings, reference_embedding, reference_native):
+ run, embeddings, reference_embedding, reference_native = map(
+ Path, (run, embeddings, reference_embedding, reference_native)
+ )
+ unopened(run)
+ if (run / "frozen.json").exists():
+ raise FileExistsError(run / "frozen.json")
+ plan, inputs, _ = checked_plan(source, run)
+ calibration = read(run / "grades-calibration.json")
+ require(
+ calibration["study_plan_sha256"] == file_hash(run / "plan.json")
+ and [row["id"] for row in calibration["rows"]] == [row["id"] for row in inputs["calibration"]],
+ "Calibration outcomes differ from saved source order",
+ )
+ require(
+ all(
+ file_hash(reference_embedding / name) == expected
+ for name, expected in plan["reference_embedding_files_sha256"].items()
+ )
+ and all(
+ file_hash(reference_native / name) == expected
+ for name, expected in plan["reference_native_files_sha256"].items()
+ ),
+ "Reference model metadata changed",
+ )
+ matrices = {split: vectors(inputs[split], embeddings) for split in ("calibration", "test")}
+ embedding_costs = {split: overhead(embeddings, [row["id"] for row in inputs[split]]) for split in matrices}
+ methods = {}
+ reference = read(reference_embedding / "selected-model.json")
+ old_embedding = read(reference_embedding / "frozen.json")
+ embedding_models = [
+ ("embedding_calibration_only", reference),
+ ("embedding_500", checked_embedding(source, run, 500)),
+ ("embedding_2000", checked_embedding(source, run, 2000)),
+ ]
+ teacher_path = run / "teacher-models.json"
+ if teacher_path.exists():
+ teacher = read(teacher_path)
+ require(
+ teacher["study_plan_sha256"] == file_hash(run / "plan.json")
+ and teacher["training_grades_sha256"] == file_hash(run / "grades-train.json")
+ and set(teacher["heads"]) == set(OPTIONAL_METHODS),
+ "Teacher comparison must bind this study and both matched heads",
+ )
+ ids = teacher["heads"][OPTIONAL_METHODS[0]]["training_ids"]
+ require(
+ ids
+ and set(ids) <= set(plan["training_ids"])
+ and ids == [identity for identity in plan["training_ids"] if identity in set(ids)]
+ and all(head["training_ids"] == ids for head in teacher["heads"].values())
+ and all(
+ head["folds"]
+ == [
+ fold
+ for identity, fold in zip(plan["training_ids"], plan["training_folds"], strict=True)
+ if identity in set(ids)
+ ]
+ for head in teacher["heads"].values()
+ ),
+ "Teacher and reference heads must use the same source-ordered rows and original folds",
+ )
+ embedding_models.extend((name, {"head": head}) for name, head in teacher["heads"].items())
+ for name, model in embedding_models:
+ head = model["head"]
+ scores = {}
+ for split, matrix in matrices.items():
+ probabilities = np.column_stack(
+ [outcomes.probability(matrix, head["models"][arm]) for arm in outcomes.ARMS]
+ )
+ scores[split] = outcomes.normalized_scores(probabilities, head["conditional_cost_statistics"])[0].tolist()
+ methods[name] = {
+ "design": "embedding",
+ "training_rows": len(head["training_ids"]),
+ "scores": scores,
+ "router_credits": embedding_costs,
+ }
+ methods["embedding_original_cutoff"] = {
+ **methods["embedding_calibration_only"],
+ "original_policies": old_embedding["selections"],
+ }
+ native_scores = read(run / "native-scores.json")
+ require(
+ native_scores["study_plan_sha256"] == file_hash(run / "plan.json")
+ and native_scores["source_code_sha256"] == code_hashes(),
+ "Native scores belong to a different study",
+ )
+ old_native = read(reference_native / "frozen.json")
+ for name, key, size in (
+ ("base", "reference_base", 0),
+ ("lora_calibration_only", "reference_lora", 2503),
+ ("lora_500", "500_lora", 500),
+ ("lora_2000", "2000_lora", 2000),
+ ):
+ model = native_scores["models"][key]
+ trained_path = (
+ reference_native / "trained.json" if key.startswith("reference_") else run / f"native-{size}/trained.json"
+ )
+ if not key.startswith("reference_"):
+ checked_native(source, run, size)
+ require(
+ model["trained_sha256"] == file_hash(trained_path)
+ and all(model["splits"][split]["ids"] == [row["id"] for row in inputs[split]] for split in matrices),
+ "Native scores differ from calibration/test question order",
+ )
+ methods[name] = {
+ "design": "native",
+ "training_rows": size,
+ "cost_model_source": "competition training" if key.startswith("reference_") else "arithmetic training",
+ "scores": {split: model["splits"][split]["scores"] for split in matrices},
+ "router_credits": {split: None for split in matrices},
+ }
+ methods["lora_original_cutoff"] = {
+ **methods["lora_calibration_only"],
+ "original_policies": old_native["models"]["lora"]["policies"],
+ }
+ for model in methods.values():
+ model["policies"] = model.pop("original_policies", None) or {
+ str(target): solver_report.select_cutoff(
+ calibration["rows"],
+ model["scores"]["calibration"],
+ target,
+ router_credits=model["router_credits"]["calibration"],
+ credit_usd=1e-5,
+ )
+ for target in TARGETS
+ }
+ expected_methods = set(METHODS) | (set(OPTIONAL_METHODS) if teacher_path.exists() else set())
+ require(set(methods) == expected_methods, "The declared method set changed")
+ result = {
+ "format": "sie-arithmetic-adaptation-frozen-v1",
+ "study_plan_sha256": file_hash(run / "plan.json"),
+ "calibration_grades_sha256": file_hash(run / "grades-calibration.json"),
+ "model_artifacts_sha256": {
+ name: file_hash(run / name)
+ for name in [
+ "embedding-500.json",
+ "embedding-2000.json",
+ "native-500/trained.json",
+ "native-2000/trained.json",
+ "native-scores.json",
+ ]
+ + (["teacher-models.json"] if teacher_path.exists() else [])
+ },
+ "embedding_files_sha256": {
+ name: file_hash(embeddings / name) for name in ("plan.json", "vectors.jsonl.gz", "collection.json")
+ },
+ "source_code_sha256": code_hashes(),
+ "ids": {split: [row["id"] for row in inputs[split]] for split in matrices},
+ "methods": methods,
+ "teacher_comparison": teacher.get("summary") if teacher_path.exists() else None,
+ }
+ write(run / "frozen.json", result)
+ return result
+
+
+def authorize_test(source, run):
+ run = Path(run)
+ unopened(run)
+ _, inputs, _ = checked_plan(source, run)
+ frozen = read(run / "frozen.json")
+ require(
+ frozen["study_plan_sha256"] == file_hash(run / "plan.json")
+ and frozen["ids"]["test"] == [row["id"] for row in inputs["test"]]
+ and set(frozen["methods"]) in (set(METHODS), set(METHODS) | set(OPTIONAL_METHODS))
+ and all(file_hash(run / name) == expected for name, expected in frozen["model_artifacts_sha256"].items()),
+ "Models, predictions or cutoff freeze changed",
+ )
+ result = {
+ "format": "sie-arithmetic-adaptation-test-access-v1",
+ "frozen_sha256": file_hash(run / "frozen.json"),
+ "study_plan_sha256": file_hash(run / "plan.json"),
+ "authorized_at": benefit.now(),
+ "statement": "Every declared model and cutoff is saved before fresh test solver answers are read.",
+ }
+ write(run / "test-access.json", result)
+ return result
+
+
+def report(source, run):
+ run = Path(run)
+ if (run / "report.json").exists():
+ raise FileExistsError(run / "report.json")
+ _, inputs, _ = checked_plan(source, run)
+ frozen, access, grades = (read(run / name) for name in ("frozen.json", "test-access.json", "grades-test.json"))
+ require(
+ access["frozen_sha256"] == file_hash(run / "frozen.json")
+ and grades["test_access_sha256"] == file_hash(run / "test-access.json")
+ and [row["id"] for row in grades["rows"]] == [row["id"] for row in inputs["test"]],
+ "Test outcomes, model freeze or access changed",
+ )
+ rows, methods = grades["rows"], {}
+ for name, model in frozen["methods"].items():
+ policies = {}
+ values = np.asarray(model["scores"]["test"])
+ for target, policy in model["policies"].items():
+ mode = policy["mode"]
+ decisions = (values >= policy["cutoff"]).tolist() if mode == "threshold" else mode
+ credits = model["router_credits"]["test"] if mode == "threshold" else 0
+ policies[target] = {
+ "mode": mode,
+ "cutoff": policy["cutoff"],
+ "metrics": solver_report.summarize(rows, decisions, router_credits=credits),
+ }
+ methods[name] = {
+ "label": name.replace("_", " "),
+ "design": model["design"],
+ "training_rows": model["training_rows"],
+ "policies": policies,
+ "show_by_default": not name.endswith("original_cutoff"),
+ "cost_model_source": model.get(
+ "cost_model_source",
+ "competition training"
+ if "calibration_only" in name or "original_cutoff" in name
+ else "arithmetic training",
+ ),
+ }
+ result = {
+ "format": "sie-arithmetic-adaptation-report-v1",
+ "n": len(rows),
+ "ids": [row["id"] for row in rows],
+ "source_plan_sha256": file_hash(Path(source) / "source-plan.json"),
+ "test_grades_sha256": file_hash(run / "grades-test.json"),
+ "frozen_sha256": file_hash(run / "frozen.json"),
+ "primary_target_pp": 1,
+ "baselines": {mode: solver_report.summarize(rows, mode, router_credits=0) for mode in solver_report.BASELINES},
+ "methods": methods,
+ "teacher_comparison": frozen.get("teacher_comparison"),
+ "cost_scope": grades["cost_scope"] + " Embeddings use observed Cloud charges; native hosting remains unpriced.",
+ "population": "Fresh3000 source-family-disjoint GSM8K source-train questions; known2000 used only for training/development.",
+ "uncertainty": "Targets are calibration choices, not guarantees; retain every declared target and paired interval.",
+ }
+ write(run / "report.json", result)
+ return result
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--source", type=Path, required=True)
+ parser.add_argument("--run", type=Path, required=True)
+ commands = parser.add_subparsers(dest="command", required=True)
+ preparation = commands.add_parser("prepare")
+ preparation.add_argument("--reference-embedding", type=Path, required=True)
+ preparation.add_argument("--reference-native", type=Path, required=True)
+ grading = commands.add_parser("grade")
+ grading.add_argument("--split", choices=("train", "calibration", "test"), required=True)
+ grading.add_argument("--collection", type=Path, required=True)
+ grading.add_argument("--prices", type=Path, required=True)
+ grading.add_argument("--workers", type=int, default=8)
+ embedding = commands.add_parser("fit-embedding")
+ embedding.add_argument("--size", type=int, choices=(500, 2000), required=True)
+ embedding.add_argument("--embeddings", type=Path, required=True)
+ training = commands.add_parser("train-native")
+ training.add_argument("--size", type=int, choices=(500, 2000), required=True)
+ training.add_argument("--local-files-only", action="store_true")
+ scoring = commands.add_parser("score-native")
+ scoring.add_argument("--reference-native", type=Path, required=True)
+ scoring.add_argument("--device", choices=("cuda", "mps", "cpu"), default="cuda")
+ freezing = commands.add_parser("freeze")
+ freezing.add_argument("--embeddings", type=Path, required=True)
+ freezing.add_argument("--reference-embedding", type=Path, required=True)
+ freezing.add_argument("--reference-native", type=Path, required=True)
+ commands.add_parser("authorize-test")
+ commands.add_parser("report")
+ args = parser.parse_args()
+ common = (args.source, args.run)
+ if args.command == "prepare":
+ result = prepare(*common, args.reference_embedding, args.reference_native)
+ elif args.command == "grade":
+ result = grade(*common, args.collection, args.prices, args.split, workers=args.workers)
+ elif args.command == "fit-embedding":
+ result = fit_embedding(*common, args.embeddings, args.size)
+ elif args.command == "train-native":
+ result = train_native(*common, args.size, local_files_only=args.local_files_only)
+ elif args.command == "score-native":
+ result = score_native(*common, args.reference_native, device=args.device)
+ elif args.command == "freeze":
+ result = freeze(*common, args.embeddings, args.reference_embedding, args.reference_native)
+ elif args.command == "authorize-test":
+ result = authorize_test(*common)
+ else:
+ result = report(*common)
+ print(
+ json.dumps({key: value for key, value in result.items() if key in ("format", "n", "training_rows")}, indent=2)
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/arithmetic_teacher.py b/examples/train-a-router/arithmetic_teacher.py
new file mode 100644
index 000000000..f412ecd7d
--- /dev/null
+++ b/examples/train-a-router/arithmetic_teacher.py
@@ -0,0 +1,293 @@
+"""Collect independent teacher answers for question-only training inputs.
+
+The teacher never receives benchmark answers or either answer model's output.
+Completed answers are reused on resume. Transport failures have a fixed retry
+limit; incorrect or unparseable answers are never resampled.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import inspect
+import json
+import os
+import re
+import time
+from decimal import Decimal
+from pathlib import Path
+
+import httpx
+
+import arithmetic_teacher_protocol as numeric_teacher
+import teacher_collect as teacher
+from data import digest, file_hash, write
+
+FORMAT = "sie-arithmetic-teacher-v1"
+MAX_ATTEMPTS = 3
+RETRY_KINDS = {"ConnectError", "ReadError", "WriteError", "ReadTimeout", "WriteTimeout", "ConnectTimeout", "TimeoutError"}
+
+
+def read(path):
+ return json.loads(Path(path).read_text())
+
+
+def questions(path):
+ rows = read(path)
+ if (
+ not isinstance(rows, list)
+ or not rows
+ or any(
+ not isinstance(row, dict)
+ or set(row) != {"id", "request"}
+ or not isinstance(row["id"], str)
+ or re.fullmatch(r"[A-Za-z0-9_-]{1,120}", row["id"]) is None
+ or not isinstance(row["request"], str)
+ or not row["request"].strip()
+ for row in rows
+ )
+ or len({row["id"] for row in rows}) != len(rows)
+ ):
+ raise ValueError("Expected unique question-only {id, request} rows; no labels or other fields")
+ return rows
+
+
+def protocol_for(answer_format):
+ if answer_format == "numeric":
+ return numeric_teacher
+ if answer_format == "general":
+ return teacher
+ raise ValueError("Unknown teacher answer format")
+
+
+def source_hashes():
+ return {
+ "arithmetic_teacher.py": file_hash(Path(__file__)),
+ "arithmetic_teacher_protocol.py": file_hash(Path(numeric_teacher.__file__)),
+ "teacher_collect.py": file_hash(Path(teacher.__file__)),
+ }
+
+
+def prepare(inputs, output, *, max_usd=30, workers=16, answer_format="numeric"):
+ inputs, output = Path(inputs), Path(output)
+ rows = questions(inputs)
+ budget = Decimal(str(max_usd))
+ if not budget.is_finite() or budget <= 0 or not 1 <= workers <= 32:
+ raise ValueError("Use a finite positive budget and 1..32 workers")
+ if output.exists():
+ raise FileExistsError(output)
+ protocol = protocol_for(answer_format)
+ plan = {
+ "format": FORMAT,
+ "input_sha256": file_hash(inputs),
+ "questions_sha256": digest(rows),
+ "ids": [row["id"] for row in rows],
+ "answer_format": answer_format,
+ "settings": protocol.SETTINGS,
+ "prices_usd_per_million": teacher.PRICES,
+ "price_source": "https://developers.openai.com/api/docs/pricing",
+ "price_checked_date": "2026-10-19",
+ "max_usd": str(budget),
+ "workers": workers,
+ "max_attempts_per_question": MAX_ATTEMPTS,
+ "request_timeout_seconds": teacher.REQUEST_SECONDS,
+ "retry_policy": "Up to three identical attempts for 429/5xx or transport failure; exponential backoff. Never retry answer content, refusals, or output-token exhaustion.",
+ "question_only": True,
+ "reference_answers_received": False,
+ "solver_answers_received": False,
+ "source_sha256": source_hashes(),
+ }
+ write(output / "plan.json", plan)
+ write(output / "inputs.json", rows)
+ return plan
+
+
+def retryable(record):
+ error = record.get("error") or {}
+ return error.get("kind") in RETRY_KINDS or (
+ error.get("kind") == "http_error" and (error.get("status") == 429 or 500 <= error.get("status", 0) <= 599)
+ )
+
+
+def attempt_history(run, row, protocol=None):
+ protocol = protocol or protocol_for(read(run / "plan.json").get("answer_format", "general"))
+ result = []
+ for directory in sorted((run / "attempts" / row["id"]).glob("attempt-*")):
+ intent = read(directory / "intent.json")
+ if intent.get("request_sha256") != digest(protocol.body_for(row)):
+ raise ValueError("A saved teacher attempt belongs to a different question or settings")
+ path = directory / "records" / f"{row['id']}.json"
+ record = read(path) if path.exists() else None
+ if record and record.get("request_sha256") != intent["request_sha256"]:
+ raise ValueError("Saved request and response bindings differ")
+ if record and record.get("raw_response_sha256"):
+ raw = directory / "responses" / f"{row['id']}.json"
+ if not raw.is_file() or file_hash(raw) != record["raw_response_sha256"]:
+ raise ValueError("Saved raw teacher response is missing or changed")
+ if record and record.get("status") == "complete":
+ if not record.get("raw_response_sha256"):
+ raise ValueError("Completed teacher answer has no raw response receipt")
+ protocol._check_teacher_record(record, row)
+ result.append((directory, intent, record))
+ return result
+
+
+def save_summary(run, result):
+ history = run / "summaries"
+ number = len(list(history.glob("summary-*.json"))) + 1
+ write(history / f"summary-{number:04d}.json", result)
+ temporary = run / ".collection.tmp"
+ temporary.write_text(json.dumps(result, indent=2) + "\n")
+ temporary.replace(run / "collection.json")
+
+
+async def collect(run, *, secret=None, request_fn=teacher._http_request, checkpoint=None):
+ run = Path(run)
+ plan, rows = read(run / "plan.json"), questions(run / "inputs.json")
+ protocol = protocol_for(plan.get("answer_format", "general"))
+ if (
+ plan.get("format") != FORMAT
+ or plan.get("questions_sha256") != digest(rows)
+ or plan.get("ids") != [row["id"] for row in rows]
+ or plan.get("settings") != protocol.SETTINGS
+ or plan.get("max_attempts_per_question") != MAX_ATTEMPTS
+ or plan.get("source_sha256") != source_hashes()
+ ):
+ raise ValueError("Prepared teacher inputs, source or settings changed")
+ secret = secret if secret is not None else os.environ["OPENAI_API_KEY"]
+ budget = teacher.Budget(plan["max_usd"], 0)
+ histories, completed, terminal = {}, {}, {}
+ for row in rows:
+ history = attempt_history(run, row, protocol)
+ histories[row["id"]] = history
+ for directory, intent, record in history:
+ charge = (
+ Decimal(record["conservative_list_price_usd"])
+ if record and record.get("usage_verified")
+ else Decimal(intent["reservation_usd"])
+ )
+ budget.settled += charge
+ if record and record["status"] == "complete":
+ completed[row["id"]] = str(directory.relative_to(run))
+ if history and row["id"] not in completed:
+ last = history[-1][2]
+ if last is None or not retryable(last) or len(history) >= MAX_ATTEMPTS:
+ terminal[row["id"]] = "uncertain_interrupted_attempt" if last is None else "no_more_retries"
+ if budget.exposure > budget.maximum:
+ raise ValueError("Historical exposure already exceeds the saved budget")
+ started, counters = time.monotonic(), {"new_attempts": 0, "finished_questions": len(completed) + len(terminal)}
+ queue = asyncio.Queue()
+ for row in rows:
+ if row["id"] not in completed and row["id"] not in terminal:
+ queue.put_nowait(row)
+ lock = asyncio.Lock()
+
+ async def persist():
+ if checkpoint is not None:
+ result = checkpoint()
+ if inspect.isawaitable(result):
+ await result
+
+ def summary():
+ known = Decimal(0)
+ attempts = 0
+ for row in rows:
+ for _, _, record in attempt_history(run, row, protocol):
+ attempts += 1
+ if record and record.get("usage_verified"):
+ known += Decimal(str(record["list_price_estimate_usd"]))
+ return {
+ "format": FORMAT,
+ "plan_sha256": file_hash(run / "plan.json"),
+ "n": len(rows),
+ "complete": len(completed),
+ "status": "complete" if len(completed) == len(rows) else "incomplete",
+ "completed_attempts": completed,
+ "unavailable": terminal,
+ "pending": [row["id"] for row in rows if row["id"] not in completed and row["id"] not in terminal],
+ "known_list_price_usd": str(known),
+ "budget": budget.receipt(),
+ "physical_attempts": attempts,
+ "new_attempts": counters["new_attempts"],
+ "elapsed_seconds_this_run": time.monotonic() - started,
+ "cost_scope": "Known token usage at pinned list prices plus separate conservative unknown-attempt exposure; not an invoice.",
+ "question_only": True,
+ }
+
+ async with httpx.AsyncClient(timeout=teacher.REQUEST_SECONDS, follow_redirects=False, trust_env=False) as client:
+
+ async def worker():
+ while not queue.empty():
+ row = queue.get_nowait()
+ identity = row["id"]
+ for number in range(len(histories[identity]) + 1, MAX_ATTEMPTS + 1):
+ reservation = protocol.reservation(row)
+ async with lock:
+ if not budget.reserve(identity, reservation):
+ break
+ destination = run / "attempts" / identity / f"attempt-{number:02d}"
+ write(
+ destination / "intent.json",
+ {
+ "id": identity,
+ "number": number,
+ "request_sha256": digest(protocol.body_for(row)),
+ "reservation_usd": str(reservation),
+ },
+ )
+ counters["new_attempts"] += 1
+ # A remote checkpoint must finish before this request can be
+ # sent, so a restart can identify uncertain executions.
+ await persist()
+ if number > 1:
+ await asyncio.sleep(2**number)
+ record = await protocol._call(destination, row, secret, client, request_fn)
+ async with lock:
+ budget.settle(identity, record.get("conservative_list_price_usd"))
+ if record["status"] == "complete":
+ completed[identity] = str(destination.relative_to(run))
+ break
+ if not retryable(record) or number == MAX_ATTEMPTS:
+ terminal[identity] = "no_more_retries"
+ break
+ counters["finished_questions"] += 1
+ if counters["finished_questions"] % 100 == 0:
+ async with lock:
+ progress = summary()
+ save_summary(run, progress)
+ print(
+ f"Teacher: {len(completed)}/{len(rows)} complete; "
+ f"known usage ${progress['known_list_price_usd']}; exposure ${budget.exposure}",
+ flush=True,
+ )
+ await persist()
+
+ await asyncio.gather(*(worker() for _ in range(plan["workers"])))
+ result = summary()
+ save_summary(run, result)
+ await persist()
+ return result
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("command", choices=("prepare", "collect"))
+ parser.add_argument("--inputs", type=Path)
+ parser.add_argument("--run", type=Path, required=True)
+ parser.add_argument("--max-usd", type=Decimal, default=Decimal(30))
+ parser.add_argument("--workers", type=int, default=16)
+ parser.add_argument("--answer-format", choices=("numeric", "general"), default="numeric")
+ args = parser.parse_args()
+ if args.command == "prepare":
+ if args.inputs is None:
+ parser.error("prepare requires --inputs")
+ result = prepare(
+ args.inputs, args.run, max_usd=args.max_usd, workers=args.workers, answer_format=args.answer_format
+ )
+ else:
+ result = asyncio.run(collect(args.run))
+ print(json.dumps(result, indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/arithmetic_teacher_protocol.py b/examples/train-a-router/arithmetic_teacher_protocol.py
new file mode 100644
index 000000000..136c8867c
--- /dev/null
+++ b/examples/train-a-router/arithmetic_teacher_protocol.py
@@ -0,0 +1,131 @@
+"""Question-only arithmetic teacher with a numeric final-answer contract."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import re
+import time
+from decimal import Decimal
+
+import teacher_collect as general
+from data import digest, file_hash, write
+
+MODEL = general.MODEL
+REQUEST_SECONDS = general.REQUEST_SECONDS
+PRICES = general.PRICES
+SETTINGS = general.SETTINGS | {
+ "instructions": (
+ "Solve the arithmetic word problem independently. Return only its final numeric answer in the answer "
+ "field: one signed integer, decimal, or fraction such as 42, -3.5, or 7/2. Express the number in the "
+ "units requested by the question, with any requested rounding. Do not include units, currency symbols, "
+ "thousands separators, percent signs, LaTeX, or explanatory words. For a clock-time question, give the "
+ "requested numeric clock hour without AM/PM. Do not use external tools."
+ )
+}
+NUMBER = re.compile(r"[+-]?(?:\d+(?:\.\d*)?|\.\d+)(?:/[+-]?(?:\d+(?:\.\d*)?|\.\d+))?")
+
+
+def numeric_answer(value):
+ if not isinstance(value, str) or NUMBER.fullmatch(value.strip()) is None:
+ return False
+ pieces = value.strip().split("/")
+ return len(pieces) == 1 or Decimal(pieces[1]) != 0
+
+
+def body_for(row):
+ return SETTINGS | {"input": row["request"]}
+
+
+def input_allowance(row):
+ allowance = len(json.dumps(body_for(row)).encode()) + 2048
+ if allowance >= 272_000:
+ raise ValueError("Teacher input exceeds the declared short-context budget")
+ return allowance
+
+
+def reservation(row):
+ return (Decimal(input_allowance(row)) * Decimal("12.5") + Decimal(8192 * 50)) / 1_000_000
+
+
+def _check_teacher_record(record, row):
+ if (
+ record.get("id") != row["id"]
+ or record.get("request_sha256") != digest(body_for(row))
+ or record.get("status") != "complete"
+ or record.get("response_status") != "completed"
+ or record.get("returned_model") != MODEL
+ or not isinstance(record.get("answer"), str)
+ or json.loads(record["output_text"]) != {"answer": record["answer"]}
+ ):
+ raise ValueError("Reused teacher record differs from the exact numeric question-only request")
+ return general.usage_cost(record.get("usage"), input_allowance(row))
+
+
+async def _call(run, row, secret, client, request_fn):
+ """Keep every returned string; numeric eligibility is assessed without gold."""
+ started = time.monotonic()
+ record = {
+ "id": row["id"],
+ "request_sha256": digest(body_for(row)),
+ "status": "incomplete",
+ "answer": None,
+ "usage_verified": False,
+ }
+ try:
+ async with asyncio.timeout(REQUEST_SECONDS):
+ result = await request_fn(client, body_for(row), secret, REQUEST_SECONDS)
+ if "error" in result:
+ record["error"] = result["error"]
+ else:
+ payload = result["payload"]
+ raw = run / "responses" / f"{row['id']}.json"
+ write(raw, payload)
+ record["raw_response_sha256"] = file_hash(raw)
+ record.update(
+ {
+ "response_id": payload.get("id"),
+ "returned_model": payload.get("model"),
+ "response_status": payload.get("status"),
+ "usage": payload.get("usage"),
+ "service_tier": payload.get("service_tier"),
+ "incomplete_details": payload.get("incomplete_details"),
+ }
+ )
+ try:
+ if payload.get("model") != MODEL or payload.get("service_tier") not in (None, "default"):
+ raise ValueError("Returned model or service tier differs from reserved pricing")
+ invoice, conservative = general.usage_cost(payload.get("usage"), input_allowance(row))
+ record.update(
+ {
+ "usage_verified": True,
+ "list_price_estimate_usd": float(invoice),
+ "conservative_list_price_usd": str(conservative),
+ }
+ )
+ except (TypeError, ValueError):
+ record["error"] = {"kind": "invalid_or_missing_usage", "execution_unknown": True}
+ output = "".join(
+ content.get("text", "")
+ for item in payload.get("output", [])
+ if item.get("type") == "message"
+ for content in item.get("content", [])
+ if content.get("type") == "output_text"
+ )
+ record["output_text"] = output
+ parsed = json.loads(output)
+ if not isinstance(parsed, dict) or set(parsed) != {"answer"} or not isinstance(parsed["answer"], str):
+ raise ValueError("Teacher response differs from its answer schema")
+ record["answer"] = parsed["answer"]
+ record["numeric_answer"] = numeric_answer(parsed["answer"])
+ if record["usage_verified"] and payload.get("status") == "completed" and payload.get("model") == MODEL:
+ record["status"] = "complete"
+ else:
+ record.setdefault(
+ "error", {"kind": "incomplete_or_changed_model", "execution_unknown": not record["usage_verified"]}
+ )
+ except (Exception, asyncio.CancelledError) as error: # noqa: BLE001 — retain uncertain execution.
+ record["error"] = {"kind": type(error).__name__, "execution_unknown": not record["usage_verified"]}
+ record["elapsed_seconds"] = time.monotonic() - started
+ write(run / "records" / f"{row['id']}.json", record)
+ return record
diff --git a/examples/train-a-router/benchmark_arithmetic.py b/examples/train-a-router/benchmark_arithmetic.py
new file mode 100644
index 000000000..e571cd8f5
--- /dev/null
+++ b/examples/train-a-router/benchmark_arithmetic.py
@@ -0,0 +1,124 @@
+"""Time 20 warm Cloud embedding decisions on the native probe's training inputs.
+
+This makes 21 embedding calls: a separate warmup and 20 measured requests.
+It makes no answer-model calls. Network time is included, and model selection
+does not depend on this timing probe.
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib.metadata
+import json
+import platform
+import time
+from pathlib import Path
+
+import numpy as np
+from sie_sdk.types import Item
+
+import benefit_outcomes as outcomes
+import solver
+from data import file_hash, write
+from serving import INSTRUCTION, MODEL, REVISION, SingleSendClient, error_receipt, normalize, usage
+
+
+def run(inputs, model, output, *, self_hosted=False):
+ inputs, model, output = Path(inputs), Path(model), Path(output)
+ sample, head = json.loads(inputs.read_text()), json.loads(model.read_text())["head"]
+ rows = sample["rows"]
+ if (
+ sample.get("salt") != "arithmetic-warm-v1" or sample.get("warmup") != 1 or sample.get("timed") != 20
+ or len(rows) != 21 or len({row["id"] for row in rows}) != 21
+ or any(set(row) != {"id", "request"} for row in rows)
+ or not {row["id"] for row in rows} <= set(head["training_ids"])
+ ):
+ raise ValueError("Need the saved 20 training requests and separate 21st warmup request")
+ timed, warmup = rows[:20], rows[20]
+ result = {
+ "format": "sie-arithmetic-warm-embedding-v1", "status": "incomplete",
+ "inputs_sha256": file_hash(inputs), "model_sha256": file_hash(model),
+ "source_sha256": file_hash(Path(__file__)), "model": MODEL, "revision": REVISION,
+ "instruction": INSTRUCTION, "is_query": True, "dimensions": 2560,
+ "self_hosted": self_hosted, "timed_ids": [row["id"] for row in timed], "warmup_id": warmup["id"],
+ "client_runtime": {
+ "machine": platform.machine(), "system": platform.system(), "python": platform.python_version(),
+ "sdk": importlib.metadata.version("sie-sdk"),
+ },
+ "protocol": {
+ "concurrency": 1, "batch_size": 1, "automatic_retries": 0,
+ "scope": "Resident SDK client, HTTP, normalization, both classifiers and a timing-only comparison with zero.",
+ "excludes": "Artifact verification, metadata, warmup and answer generation.",
+ "limitation": "Small sequential probe, not throughput or an equal-hardware comparison with direct Laya. Static policies would skip this work.",
+ },
+ "calls": [],
+ }
+ write(output, result)
+
+ def save():
+ temporary = output.with_name(output.name + ".tmp")
+ temporary.write_text(json.dumps(result, indent=2, allow_nan=False) + "\n")
+ temporary.replace(output)
+
+ deadline = time.monotonic() + 600
+ try:
+ with SingleSendClient(timeout_s=120, read_timeout_s=120) as client:
+ metadata = client.get_model(MODEL)
+ if metadata.get("revision") != REVISION or metadata.get("dims", {}).get("dense") != 2560:
+ raise ValueError("Live encoder differs from the pinned model")
+ for index, row in enumerate([warmup, *timed]):
+ if time.monotonic() >= deadline:
+ raise TimeoutError("Embedding timing deadline reached")
+ started = time.perf_counter()
+ reply = client.encode(
+ MODEL, Item(text=row["request"]), output_types=["dense"], instruction=INSTRUCTION,
+ is_query=True, wait_for_capacity=False, max_oom_retries=0, provision_timeout_s=60,
+ )
+ arrived = time.perf_counter()
+ record = {"id": row["id"], "warmup": index == 0, "receipt": usage(reply)}
+ result["calls"].append(record)
+ vector = normalize(reply["dense"])
+ if vector.shape != (2560,):
+ raise ValueError("Wrong vector dimensions")
+ score = float(outcomes.score(vector[None, :], head)[0])
+ if not np.isfinite(score):
+ raise ValueError("Invalid router score")
+ comparison = score >= 0
+ finished = time.perf_counter()
+ record.update(
+ seconds=finished - started, network_and_embedding_seconds=arrived - started,
+ local_scoring_seconds=finished - arrived, score=score, timing_only_comparison=comparison,
+ )
+ save()
+ if not self_hosted and not solver.valid_credit(record["receipt"]["credits"]):
+ raise ValueError("Cloud call has no valid charge receipt")
+ measured = result["calls"][1:]
+ seconds = [row["seconds"] for row in measured]
+ result.update(
+ status="complete", n=len(measured), warmup_seconds_excluded=result["calls"][0]["seconds"],
+ p50_ms=float(np.percentile(seconds, 50) * 1000), p95_ms=float(np.percentile(seconds, 95) * 1000),
+ local_scoring_p50_ms=float(np.median([row["local_scoring_seconds"] for row in measured]) * 1000),
+ total_observed_credits=None if self_hosted else sum(row["receipt"]["credits"] for row in result["calls"]),
+ timed_observed_credits=None if self_hosted else sum(row["receipt"]["credits"] for row in measured),
+ )
+ except Exception as error:
+ result["error"] = error_receipt(error)
+ raise
+ finally:
+ save()
+ return result
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--inputs", type=Path, required=True)
+ parser.add_argument("--model", type=Path, required=True)
+ parser.add_argument("--output", type=Path, required=True)
+ parser.add_argument("--self-hosted", action="store_true")
+ args = parser.parse_args()
+ result = run(args.inputs, args.model, args.output, self_hosted=args.self_hosted)
+ print(json.dumps({key: result[key] for key in ("n", "p50_ms", "p95_ms", "local_scoring_p50_ms", "total_observed_credits")}, indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/benchmark_arithmetic_native.py b/examples/train-a-router/benchmark_arithmetic_native.py
new file mode 100644
index 000000000..ba9be1bc8
--- /dev/null
+++ b/examples/train-a-router/benchmark_arithmetic_native.py
@@ -0,0 +1,192 @@
+"""Time a resident arithmetic native router on the saved 20 training requests.
+
+One separate warmup is excluded. This measures two question forwards per
+request, score normalization and a timing-only comparison with zero. It makes
+no answer calls and does not select or change a calibrated routing policy.
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib.metadata
+import json
+import os
+import platform
+import time
+from pathlib import Path
+
+import benefit
+import benefit_outcomes_native as native
+import finetune
+from benchmark_outcomes_native import WarmRouter, measure_requests
+from data import digest, file_hash, write
+
+
+def inputs_and_checkpoint(inputs, directory, variants):
+ """Check the question-only sample and selected exports before loading Torch."""
+ inputs, directory = Path(inputs), Path(directory)
+ sample = json.loads(inputs.read_text())
+ plan_path, study_path = directory / "plan.json", directory.parent / "plan.json"
+ plan, study = json.loads(plan_path.read_text()), json.loads(study_path.read_text())
+ trained = benefit.checked(json.loads((directory / "trained.json").read_text()), native.TRAINED_FORMAT)
+ rows = sample.get("rows", [])
+ if (
+ sample.get("salt") != "arithmetic-warm-v1"
+ or sample.get("warmup") != 1
+ or sample.get("timed") != 20
+ or len(rows) != 21
+ or any(
+ not isinstance(row, dict)
+ or set(row) != {"id", "request"}
+ or not isinstance(row["id"], str)
+ or not row["id"]
+ or not isinstance(row["request"], str)
+ or not row["request"].strip()
+ for row in rows
+ )
+ or len({row["id"] for row in rows}) != 21
+ or [row["id"] for row in rows] != study["warm_probe_ids"]
+ or not {row["id"] for row in rows} <= set(study["training_ids"])
+ or sample.get("source_plan_sha256") != study["source_plan_sha256"]
+ ):
+ raise ValueError("Need the saved21 question-only training inputs in the study's fixed order")
+ if (
+ plan["protocol"] != native.PROTOCOL
+ or trained["plan_sha256"] != file_hash(plan_path)
+ or trained["head_plan_sha256"] != file_hash(study_path)
+ or plan["head_plan_sha256"] != file_hash(study_path)
+ or not set(plan["training_ids"]) <= set(study["training_ids"])
+ ):
+ raise ValueError("Native checkpoint and arithmetic study bindings differ")
+ verification = {}
+ for variant in variants:
+ started = time.perf_counter()
+ native._manifest(directory, variant, trained["checkpoints"][variant]["files_sha256"], True)
+ verification[variant] = time.perf_counter() - started
+ return sample, trained, verification
+
+
+def _runtime(device):
+ # Optional training dependencies; core imports and --help need no Torch.
+ import laya.train as lt
+ import torch
+
+ versions = {name: importlib.metadata.version(name) for name in (*native.PROTOCOL["versions"], "numpy")}
+ if any(versions[name].split("+", 1)[0] != expected for name, expected in native.PROTOCOL["versions"].items()):
+ raise ValueError("Use the pinned native inference dependencies")
+ if device == "cuda" and not torch.cuda.is_available():
+ raise ValueError("CUDA is unavailable")
+ runtime_device = torch.device(device)
+ metadata = {
+ "device": device,
+ "gpu": torch.cuda.get_device_name(runtime_device) if device == "cuda" else None,
+ "cuda_version": torch.version.cuda,
+ "machine": platform.machine(),
+ "processor": platform.processor(),
+ "system": platform.system(),
+ "python": platform.python_version(),
+ "versions": versions,
+ "torch_threads": torch.get_num_threads(),
+ "eager_flags": native.PROTOCOL["eager_flags"],
+ "forward_amp": False,
+ }
+ return lt, torch, runtime_device, metadata
+
+
+def _save(output, receipt):
+ temporary = output.with_name(output.name + ".tmp")
+ temporary.write_text(json.dumps(receipt, indent=2, allow_nan=False) + "\n")
+ temporary.replace(output)
+
+
+def run(inputs, directory, output, *, device, variant="lora"):
+ inputs, directory, output = map(Path, (inputs, directory, output))
+ if output.exists():
+ raise FileExistsError(output)
+ if device not in ("cpu", "cuda") or variant not in ("base", "lora", "both"):
+ raise ValueError("Choose cpu or cuda and base, lora or both")
+ variants = ("base", "lora") if variant == "both" else (variant,)
+ sample, trained, verification = inputs_and_checkpoint(inputs, directory, variants)
+ rows, warmup = sample["rows"][:20], sample["rows"][20]
+ receipt = {
+ "format": "sie-arithmetic-warm-native-v1",
+ "status": "incomplete",
+ "source_sha256": file_hash(Path(__file__)),
+ "timing_helper_sha256": file_hash(Path(__file__).with_name("benchmark_outcomes_native.py")),
+ "inputs_sha256": file_hash(inputs),
+ "requests_sha256": digest(sample["rows"]),
+ "native_plan_sha256": file_hash(directory / "plan.json"),
+ "trained_sha256": file_hash(directory / "trained.json"),
+ "study_plan_sha256": file_hash(directory.parent / "plan.json"),
+ "timed_ids": [row["id"] for row in rows],
+ "warmup_id": warmup["id"],
+ "questions_per_request": 2,
+ "request_batch_size": 1,
+ "question_item_batch_size": 2,
+ "excluded_warmup_requests_per_model": 1,
+ "quantile_method": "linear",
+ "scope": "Resident direct inference: tokenization, two question forwards, normalization and timing-only score>=0 comparison; synchronized before and after each request.",
+ "load_scope": "Checkpoint load and device transfer after manifest verification, measured separately; not fresh-process or cold-disk startup.",
+ "limitation": "20 training requests, no HTTP, concurrency or answer calls. Different-hardware Cloud timings are not a matched comparison. A static bypass policy skips this model work. Hosting dollars remain unknown.",
+ "models": {},
+ }
+ write(output, receipt)
+ os.environ.update(native.PROTOCOL["eager_flags"])
+ try:
+ started = time.perf_counter()
+ lt, torch, runtime_device, runtime = _runtime(device)
+ runtime["import_seconds"] = time.perf_counter() - started
+ receipt["runtime"] = runtime
+ for choice in variants:
+ finetune.empty_cache(torch, runtime_device)
+ finetune.synchronize(torch, runtime_device)
+ started = time.perf_counter()
+ router = WarmRouter(
+ lt, directory / choice, runtime_device, trained["conditional_cost_statistics"],
+ {"mode": "threshold", "cutoff": 0.0},
+ )
+ finetune.synchronize(torch, runtime_device)
+ load_seconds = time.perf_counter() - started
+ try:
+ measurement = measure_requests(
+ router.score_one, rows, warmup, lambda: finetune.synchronize(torch, runtime_device)
+ )
+ measurement.update(
+ artifact_verification_seconds=verification[choice],
+ model_load_seconds_after_hash_verification=load_seconds,
+ parameters=trained["checkpoints"][choice]["parameters"],
+ weight_file_bytes=(directory / choice / "model.safetensors").stat().st_size,
+ )
+ receipt["models"][choice] = measurement
+ _save(output, receipt)
+ print(
+ f"{choice} ({device}): warm median {measurement['p50_ms']:.2f}ms; "
+ f"p95 {measurement['p95_ms']:.2f}ms; load {load_seconds:.2f}s",
+ flush=True,
+ )
+ finally:
+ del router
+ finetune.empty_cache(torch, runtime_device)
+ receipt["status"] = "complete"
+ except BaseException as error:
+ receipt["error_type"] = type(error).__name__
+ raise
+ finally:
+ _save(output, receipt)
+ return receipt
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--inputs", type=Path, required=True)
+ parser.add_argument("--native", type=Path, required=True, help="Arithmetic native-500 or native-2000 export directory")
+ parser.add_argument("--output", type=Path, required=True)
+ parser.add_argument("--device", choices=("cpu", "cuda"), required=True)
+ parser.add_argument("--variant", choices=("base", "lora", "both"), default="lora")
+ args = parser.parse_args()
+ run(args.inputs, args.native, args.output, device=args.device, variant=args.variant)
+ print(json.dumps({"output": str(args.output), "sha256": file_hash(args.output)}))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/benchmark_outcomes_native.py b/examples/train-a-router/benchmark_outcomes_native.py
new file mode 100644
index 000000000..17b2f0fcf
--- /dev/null
+++ b/examples/train-a-router/benchmark_outcomes_native.py
@@ -0,0 +1,279 @@
+"""Measure warm, single-request inference for a frozen two-question native router.
+
+Hash-ordered training inputs are shared by the base and LoRA measurements. This
+is direct local inference; model loading, artifact checks, HTTP transport and
+Cloud generation charges are separate from the warm inference measurement.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.metadata
+import json
+import os
+import platform
+import time
+from pathlib import Path
+
+import numpy as np
+
+import benefit_outcomes as outcomes
+import benefit_outcomes_native as native
+import finetune
+from data import digest, file_hash
+
+FORMAT = "sie-native-outcome-warm-inference-v1"
+SAMPLE_SALT = "sie-native-outcome-warm-inference-v1"
+N = 20
+REQUEST_BATCH_SIZE = 1
+QUESTION_ITEM_BATCH_SIZE = 2
+
+
+def select_inputs(original_rows, eligible_ids):
+ """Sample reviewed original inputs shared by either training size."""
+ identities = [row["id"] for row in original_rows]
+ if len(identities) != 512 or len(set(identities)) != len(identities):
+ raise ValueError("Expected the exact original512 source pool with unique IDs")
+ eligible = set(eligible_ids)
+ if len(eligible) != len(eligible_ids):
+ raise ValueError("Reviewed training IDs must be unique")
+ rows = [row for row in original_rows if row["id"] in eligible]
+ if len(rows) < N + 1:
+ raise ValueError("Benchmark needs at least21 reviewed original training inputs")
+ ordered = sorted(
+ rows,
+ key=lambda row: (hashlib.sha256(f"{SAMPLE_SALT}\0{row['id']}".encode()).hexdigest(), row["id"]),
+ )
+ return ordered[:N], ordered[N]
+
+
+def _measurement(value):
+ if (
+ not isinstance(value, dict)
+ or not np.isfinite(value.get("score", np.nan))
+ or np.asarray(value.get("margins"), dtype=float).shape != (2,)
+ or not np.isfinite(value["margins"]).all()
+ or value.get("route") not in ("cheap", "strong")
+ ):
+ raise ValueError("Warm inference must return a finite two-outcome score and route")
+ return value
+
+
+def measure_requests(score_one, rows, warmup, synchronize, *, clock=time.perf_counter):
+ if len(rows) != N or warmup["id"] in {row["id"] for row in rows}:
+ raise ValueError("Need20 timed requests and one separate warmup request")
+ synchronize()
+ started = clock()
+ _measurement(score_one(warmup))
+ synchronize()
+ warmup_seconds = clock() - started
+ timings = []
+ for row in rows:
+ synchronize()
+ started = clock()
+ result = _measurement(score_one(row))
+ synchronize()
+ elapsed = clock() - started
+ if not np.isfinite(elapsed) or elapsed < 0:
+ raise ValueError("Invalid monotonic-clock duration")
+ timings.append({"id": row["id"], "seconds": elapsed, **result})
+ if not np.isfinite(warmup_seconds) or warmup_seconds < 0:
+ raise ValueError("Invalid warmup duration")
+ seconds = np.asarray([row["seconds"] for row in timings])
+ return {
+ "warmup_id": warmup["id"],
+ "warmup_seconds_excluded": warmup_seconds,
+ "requests": timings,
+ "n": len(timings),
+ "p50_ms": float(np.percentile(seconds, 50, method="linear") * 1000),
+ "p95_ms": float(np.percentile(seconds, 95, method="linear") * 1000),
+ "mean_ms": float(seconds.mean() * 1000),
+ "total_timed_seconds": float(seconds.sum()),
+ }
+
+
+class WarmRouter:
+ """Keep one checkpoint loaded across requests; reuse the frozen scoring path."""
+
+ def __init__(self, lt, checkpoint, device, statistics, policy):
+ self.lt, self.device, self.statistics, self.policy = lt, device, statistics, policy
+ self.model, self.tokenizer, _ = lt.load_checkpoint(str(checkpoint))
+ self.model.encoder.config.reference_compile = False
+ self.model.to(device).eval()
+
+ def score_one(self, row):
+ items = native.encode_rows(self.lt, self.tokenizer, [row])
+ records = self.lt.calibration_records(
+ self.model,
+ self.tokenizer,
+ items,
+ self.device,
+ finetune.MAX_LEN,
+ finetune.HEAD_LEN,
+ batch_size=QUESTION_ITEM_BATCH_SIZE,
+ )
+ margins = native.records_to_margins(records, np.zeros((1, 2)))
+ scores, _ = outcomes.normalized_scores(native.probabilities(margins), self.statistics)
+ score = float(scores[0])
+ mode = self.policy["mode"]
+ if mode == "threshold":
+ strong = score >= self.policy["cutoff"]
+ elif mode in ("always_strong", "always_cheap"):
+ strong = mode == "always_strong"
+ else:
+ raise ValueError("Unknown frozen policy mode")
+ return {
+ "score": score,
+ "margins": margins[0].tolist(),
+ "route": "strong" if strong else "cheap",
+ "state_tokens": len(items[0]["state_ids"]),
+ }
+
+
+def _save(path, value):
+ temporary = path.with_name(path.name + ".tmp")
+ temporary.write_text(json.dumps(value, indent=2, allow_nan=False) + "\n")
+ temporary.replace(path)
+
+
+def run(run_directory, head_run, baseline, output, *, device="cuda"):
+ run_directory, head_run, baseline, output = map(Path, (run_directory, head_run, baseline, output))
+ if output.exists():
+ raise FileExistsError(output)
+ if device not in ("cuda", "mps", "cpu"):
+ raise ValueError("Choose an explicit supported inference device")
+ plan, trained, frozen = native.checked_frozen(run_directory, head_run)
+ dataset = outcomes._baseline(plan["head_plan"], baseline)
+ rows, warmup = select_inputs(
+ dataset["splits"]["train"], [row["id"] for row in plan["head_plan"]["populations"]["train"]]
+ )
+ receipt = {
+ "format": FORMAT,
+ "status": "incomplete",
+ "source_sha256": file_hash(Path(__file__)),
+ "native_source_sha256": native.code_hashes(),
+ "native_plan_sha256": file_hash(run_directory / "plan.json"),
+ "native_trained_sha256": file_hash(run_directory / "trained.json"),
+ "native_frozen_sha256": file_hash(run_directory / "frozen.json"),
+ "dataset_sha256": plan["head_plan"]["dataset_sha256"],
+ "sampling": {
+ "eligible_original_rows": len(
+ {row["id"] for row in dataset["splits"]["train"]}
+ & {row["id"] for row in plan["head_plan"]["populations"]["train"]}
+ ),
+ "ordering": f"sha256({SAMPLE_SALT!r} + NUL + id), then id; first20 timed,21st warmup",
+ "timed_ids": [row["id"] for row in rows],
+ "warmup_id": warmup["id"],
+ "requests_sha256": digest([{key: row[key] for key in ("id", "request")} for row in [*rows, warmup]]),
+ "paper_groups": [row["paper_group"] for row in rows],
+ "request_characters": [len(row["request"]) for row in rows],
+ },
+ "questions_per_request": 2,
+ "request_batch_size": REQUEST_BATCH_SIZE,
+ "question_item_batch_size": QUESTION_ITEM_BATCH_SIZE,
+ "excluded_warmup_requests_per_model": 1,
+ "quantile_method": "linear",
+ "timed_scope": "Tokenization, both native question forwards, CPU score conversion, predicted-cost normalization and primary frozen cutoff; synchronize device before and after each request",
+ "load_scope": "Checkpoint load and device transfer, measured separately after weight-file hash verification; this is not a cold-disk or fresh-process startup measurement",
+ "limitation": "A small direct-inference probe on20 training inputs, with one resident model and no HTTP or concurrent requests. Different-hardware embedding or Cloud timings are not matched comparisons. No serving dollars are inferred. A bypass policy would skip this model work in deployment.",
+ "models": {},
+ }
+ output.parent.mkdir(parents=True, exist_ok=True)
+ with output.open("x") as handle:
+ json.dump(receipt, handle, indent=2, allow_nan=False)
+ handle.write("\n")
+ os.environ.update(native.PROTOCOL["eager_flags"])
+ started = time.perf_counter()
+ try:
+ import laya.train as lt
+ import torch
+
+ runtime_device = torch.device(device)
+ versions = {name: importlib.metadata.version(name) for name in (*native.PROTOCOL["versions"], "numpy")}
+ if any(versions[name].split("+", 1)[0] != expected for name, expected in native.PROTOCOL["versions"].items()):
+ raise ValueError("Native benchmark requires the pinned training/inference dependencies")
+ if device == "cuda" and not torch.cuda.is_available():
+ raise ValueError("CUDA is unavailable")
+ if device == "mps" and not torch.backends.mps.is_available():
+ raise ValueError("MPS is unavailable")
+ receipt["runtime"] = {
+ "device": device,
+ "gpu": torch.cuda.get_device_name(runtime_device) if device == "cuda" else None,
+ "cuda_version": torch.version.cuda,
+ "machine": platform.machine(),
+ "system": platform.system(),
+ "python": platform.python_version(),
+ "versions": versions,
+ "import_seconds": time.perf_counter() - started,
+ "eager_flags": native.PROTOCOL["eager_flags"],
+ "forward_amp": False,
+ }
+ for variant in ("base", "lora"):
+ verification_started = time.perf_counter()
+ native._manifest(run_directory, variant, trained["checkpoints"][variant]["files_sha256"], True)
+ verification_seconds = time.perf_counter() - verification_started
+ finetune.empty_cache(torch, runtime_device)
+ finetune.synchronize(torch, runtime_device)
+ started = time.perf_counter()
+ router = WarmRouter(
+ lt,
+ run_directory / variant,
+ runtime_device,
+ trained["conditional_cost_statistics"],
+ frozen["models"][variant]["policies"]["1"],
+ )
+ finetune.synchronize(torch, runtime_device)
+ load_seconds = time.perf_counter() - started
+ try:
+ if device == "cuda":
+ torch.cuda.reset_peak_memory_stats(runtime_device)
+ measurement = measure_requests(
+ router.score_one, rows, warmup, lambda: finetune.synchronize(torch, runtime_device)
+ )
+ measurement.update(
+ artifact_verification_seconds=verification_seconds,
+ model_load_seconds_after_hash_verification=load_seconds,
+ parameters=sum(parameter.numel() for parameter in router.model.parameters()),
+ parameter_dtypes=sorted({str(parameter.dtype) for parameter in router.model.parameters()}),
+ weight_file_bytes=(run_directory / variant / "model.safetensors").stat().st_size,
+ primary_policy_mode=router.policy["mode"],
+ cuda_peak_allocated_mib=(
+ torch.cuda.max_memory_allocated(runtime_device) / 2**20 if device == "cuda" else None
+ ),
+ cuda_peak_reserved_mib=(
+ torch.cuda.max_memory_reserved(runtime_device) / 2**20 if device == "cuda" else None
+ ),
+ )
+ receipt["models"][variant] = measurement
+ _save(output, receipt)
+ print(
+ f"{variant}: warm p50={measurement['p50_ms']:.2f}ms p95={measurement['p95_ms']:.2f}ms; load after hash={load_seconds:.2f}s",
+ flush=True,
+ )
+ finally:
+ del router
+ finetune.empty_cache(torch, runtime_device)
+ receipt["status"] = "complete"
+ except BaseException as error:
+ receipt["error_type"] = type(error).__name__
+ raise
+ finally:
+ _save(output, receipt)
+ return receipt
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--run", type=Path, required=True)
+ parser.add_argument("--head-run", type=Path, required=True)
+ parser.add_argument("--baseline", type=Path, required=True)
+ parser.add_argument("--output", type=Path, required=True)
+ parser.add_argument("--device", choices=("cuda", "mps", "cpu"), default="cuda")
+ args = parser.parse_args()
+ run(args.run, args.head_run, args.baseline, args.output, device=args.device)
+ print(json.dumps({"output": str(args.output), "sha256": file_hash(args.output)}))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/benefit.py b/examples/train-a-router/benefit.py
new file mode 100644
index 000000000..63ebffff0
--- /dev/null
+++ b/examples/train-a-router/benefit.py
@@ -0,0 +1,565 @@
+"""Learn the benefit of the strong solver from paired training outcomes.
+
+This separate CPU experiment audits the original 512-input training pool and
+uses its 511 source-eligible questions with cached Qwen vectors. Human difficulty
+labels are not training targets. Calibration
+answers select cost cutoffs; solver test answers may be opened only after the
+new model, cutoffs and test scores are frozen and access is recorded.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+from datetime import UTC, datetime
+from pathlib import Path
+
+import numpy as np
+
+import evidence
+import policies
+import solver_policies
+import solver_report
+from data import SOURCE_HASHES, SOURCE_REVISION, audit, digest, file_hash, write
+from serving import INSTRUCTION, MODEL, REVISION, vectors
+
+PROTOCOL = {
+ "target": "strong_correct - cheap_correct; -1 = cheap-only success, 0 = equal outcomes, +1 = strong-only success",
+ "training_pool_rows": 512,
+ "training_rows": 511,
+ "excluded_training_ids": ["e2h-amc-eval-661"],
+ "eligibility": "Exclude the predeclared missing-choice interval question by source review before training outcomes; retain every other original input in order",
+ "encoder": MODEL,
+ "revision": REVISION,
+ "instruction": INSTRUCTION,
+ "dimensions": 2560,
+ "penalties": [0.01, 0.1, 1.0, 10.0],
+ "folds": 5,
+ "seed": 89,
+ "fold_assignment": "Sort paper groups, permute with NumPy seed89, assign whole groups round-robin to five folds",
+ "selection": "Minimum pooled training out-of-fold mean squared error; exact ties prefer larger regularization",
+ "fit": "Unpenalized intercept and L2-regularized linear ridge head on normalized cached vectors; refit all training rows",
+ "primary_max_quality_loss_pp": 1,
+ "secondary_max_quality_loss_pp": [0, 2, 5],
+ "motivation": "New experiment prompted by calibration-only mismatch between human difficulty and paired solver benefit",
+ "test_boundary": "Freeze before solver test OUTCOME ACCESS, not before test collection; no test outcomes used in selection",
+}
+CODE_FILES = ("benefit.py", "policies.py", "serving.py", "solver_policies.py", "solver_report.py", "data.py")
+PLAN_FORMAT = "sie-solver-benefit-plan-v1"
+FROZEN_FORMAT = "sie-solver-benefit-frozen-v1"
+ACCESS_FORMAT = "sie-solver-benefit-test-access-v1"
+TARGETS = (1, 0, 2, 5)
+
+
+def read(path):
+ return json.loads(Path(path).read_text())
+
+
+def seal(value):
+ return value | {"sha256": digest(value)}
+
+
+def checked(value, format_name):
+ if value.get("format") != format_name or value.get("sha256") != digest(
+ {key: item for key, item in value.items() if key != "sha256"}
+ ):
+ raise ValueError("Benefit artifact format or content hash differs")
+ return value
+
+
+def code_hashes():
+ return {name: file_hash(Path(__file__).with_name(name)) for name in CODE_FILES}
+
+
+def now():
+ return datetime.now(UTC).isoformat()
+
+
+def group_folds(groups):
+ """Assign whole papers to deterministic folds without inspecting outcomes."""
+ if not groups or any(not isinstance(group, str) or not group for group in groups):
+ raise ValueError("Need a paper group for every training input")
+ unique = sorted(set(groups))
+ if len(unique) < PROTOCOL["folds"]:
+ raise ValueError("Need at least five distinct training paper groups")
+ order = np.random.default_rng(PROTOCOL["seed"]).permutation(unique).tolist()
+ assignments = {group: index % PROTOCOL["folds"] for index, group in enumerate(order)}
+ return [assignments[group] for group in groups]
+
+
+def signed_targets(rows):
+ """Validate paired correctness without requiring training inference prices."""
+ solver_report.summarize_quality(rows, "always_strong")
+ return np.array([int(row["strong_correct"]) - int(row["cheap_correct"]) for row in rows], dtype=float)
+
+
+def fit_head(matrix, rows, groups):
+ """Select regularization from training papers only, then refit every row."""
+ matrix = np.asarray(matrix, dtype=float)
+ targets = signed_targets(rows)
+ if matrix.ndim != 2 or len(matrix) != len(rows) or len(groups) != len(rows) or not np.isfinite(matrix).all():
+ raise ValueError("Training vectors, outcomes and groups must be aligned and finite")
+ folds = np.asarray(group_folds(groups))
+ candidates = []
+ for penalty in PROTOCOL["penalties"]:
+ predictions = np.empty(len(rows))
+ fold_mse = []
+ for fold in range(PROTOCOL["folds"]):
+ held = folds == fold
+ if (~held).sum() < 2:
+ raise ValueError("Each fold needs at least two fitting rows")
+ weights, bias = policies.ridge(matrix[~held], targets[~held], penalty)
+ predictions[held] = matrix[held] @ weights + bias
+ fold_mse.append(float(np.mean((predictions[held] - targets[held]) ** 2)))
+ candidates.append(
+ {"penalty": penalty, "pooled_mse": float(np.mean((predictions - targets) ** 2)), "fold_mse": fold_mse}
+ )
+ winner = min(candidates, key=lambda item: (item["pooled_mse"], -item["penalty"]))
+ weights, bias = policies.ridge(matrix, targets, winner["penalty"])
+ return {
+ "weights": weights.tolist(),
+ "bias": bias,
+ "penalty": winner["penalty"],
+ "training_ids": [row["id"] for row in rows],
+ "targets": targets.astype(int).tolist(),
+ "folds": folds.tolist(),
+ "cv": candidates,
+ }
+
+
+def _inputs(baseline, original_path):
+ baseline, original_path = Path(baseline), Path(original_path)
+ evidence.verify(baseline)
+ dataset = read(baseline / "data.json")
+ audit(dataset)
+ original = read(original_path)
+ solver_policies.check_seal(original, solver_policies.FROZEN_FORMAT)
+ solver_policies._grading_provenance(original)
+ solver_policies._checked_scores(original["source_scores"])
+ if (
+ original["analysis_mode"] != "cost"
+ or original["embedding_calibration_credits"] is None
+ or original["source_scores"]["dataset_sha256"] != digest(dataset)
+ or len(dataset["splits"]["train"]) != PROTOCOL["training_pool_rows"]
+ or original["source_code_sha256"]
+ != {name: file_hash(Path(__file__).with_name(name)) for name in ("solver_policies.py", "solver_report.py")}
+ ):
+ raise ValueError("Need the same frozen priced study, original 512 training inputs and reporting code")
+ for split in ("calibration", "test"):
+ if original["source_scores"]["ids"][split] != [row["id"] for row in dataset["splits"][split]]:
+ raise ValueError("Benefit and original policy populations differ")
+ solver_plan_path = original_path.with_name("plan.json")
+ if file_hash(solver_plan_path) != original["plan_sha256"]:
+ raise ValueError("Original solver plan differs from its freeze")
+ solver_plan = read(solver_plan_path)
+ artifacts = {
+ name: file_hash(baseline / name)
+ for name in (
+ "manifest.json",
+ "data.json",
+ "development-embeddings/plan.json",
+ "development-embeddings/vectors.jsonl.gz",
+ "test-embeddings/plan.json",
+ "test-embeddings/vectors.jsonl.gz",
+ )
+ }
+ return dataset, original, solver_plan, artifacts
+
+
+def _eligibility(receipt, dataset_sha, pool_ids):
+ excluded = PROTOCOL["excluded_training_ids"]
+ expected = [identity for identity in pool_ids if identity not in excluded]
+ exclusions = receipt.get("exclusions", [])
+ source_sha = receipt.get("training_gold_review_source_sha256")
+ if (
+ receipt.get("format") != "sie-benefit-training-eligibility-v1"
+ or receipt.get("source_only") is not True
+ or receipt.get("decided_before_training_outcomes") is not True
+ or receipt.get("source_revision") != SOURCE_REVISION
+ or receipt.get("source_hashes") != SOURCE_HASHES
+ or receipt.get("dataset_sha256") != dataset_sha
+ or receipt.get("original_pool_ids") != pool_ids
+ or len(pool_ids) != PROTOCOL["training_pool_rows"]
+ or len(set(pool_ids)) != len(pool_ids)
+ or receipt.get("eligible_train_ids") != expected
+ or len(expected) != PROTOCOL["training_rows"]
+ or [row.get("id") for row in exclusions] != excluded
+ or any(not isinstance(row.get("reason"), str) or not row["reason"].strip() for row in exclusions)
+ or not isinstance(source_sha, str)
+ or len(source_sha) != 64
+ or any(char not in "0123456789abcdef" for char in source_sha)
+ ):
+ raise ValueError(
+ "Eligibility must bind the original pool and exact source-only exclusion before training outcomes"
+ )
+ return expected
+
+
+def prepare(baseline, original_path, eligibility_path, *, attest_test_outcomes_unopened):
+ if attest_test_outcomes_unopened is not True:
+ raise ValueError("An explicit assertion that solver test outcomes remain unopened is required")
+ dataset, original, solver_plan, artifacts = _inputs(baseline, original_path)
+ eligibility = read(eligibility_path)
+ eligible_ids = _eligibility(eligibility, digest(dataset), [row["id"] for row in dataset["splits"]["train"]])
+ populations = {
+ split: [
+ {"id": row["id"], "request_sha256": digest(row["request"]), "paper_group": row["paper_group"]}
+ for row in dataset["splits"][split]
+ ]
+ for split in ("train", "calibration", "test")
+ }
+ populations["train_pool"] = populations["train"]
+ populations["train"] = [row for row in populations["train_pool"] if row["id"] in eligible_ids]
+ return seal(
+ {
+ "format": PLAN_FORMAT,
+ "created_at": now(),
+ "protocol": PROTOCOL,
+ "dataset_sha256": digest(dataset),
+ "original_frozen_sha256": file_hash(original_path),
+ "original_solver_plan_sha256": original["plan_sha256"],
+ "solver_settings_sha256": solver_plan["settings_sha256"],
+ "eligibility": eligibility,
+ "eligibility_sha256": file_hash(eligibility_path),
+ "populations": populations,
+ "training_folds": group_folds([row["paper_group"] for row in populations["train"]]),
+ "baseline_artifacts_sha256": artifacts,
+ "source_code_sha256": code_hashes(),
+ "attest_test_outcomes_unopened": True,
+ "access_limitation": "Local receipts record the declared access boundary; they cannot prove nobody read an external copy",
+ }
+ )
+
+
+def _checked_plan(plan):
+ checked(plan, PLAN_FORMAT)
+ eligible_ids = _eligibility(
+ plan["eligibility"], plan["dataset_sha256"], [row["id"] for row in plan["populations"]["train_pool"]]
+ )
+ if (
+ plan["protocol"] != PROTOCOL
+ or plan["source_code_sha256"] != code_hashes()
+ or plan["attest_test_outcomes_unopened"] is not True
+ or plan["populations"]["train"]
+ != [row for row in plan["populations"]["train_pool"] if row["id"] in eligible_ids]
+ or plan["training_folds"] != group_folds([row["paper_group"] for row in plan["populations"]["train"]])
+ ):
+ raise ValueError("Benefit protocol, training folds or source code changed after preparation")
+ return plan
+
+
+def _training(plan, training_plan_path, training_inputs_path, training_grades_path, original):
+ training_plan, inputs, grades = map(read, (training_plan_path, training_inputs_path, training_grades_path))
+ expected = plan["populations"]["train"]
+ if (
+ training_plan["dataset_sha256"] != plan["dataset_sha256"]
+ or training_plan["settings_sha256"] != plan["solver_settings_sha256"]
+ or training_plan["inputs_sha256"] != digest(inputs)
+ or training_plan["source_counts"]["train"] != PROTOCOL["training_pool_rows"]
+ or training_plan["counts"]["train"] != PROTOCOL["training_rows"]
+ or training_plan["eligibility_sha256"] != plan["eligibility_sha256"]
+ or training_plan["eligible_train_ids"] != [row["id"] for row in expected]
+ or training_plan["excluded_train_ids"] != PROTOCOL["excluded_training_ids"]
+ or [{"id": row["id"], "request_sha256": digest(row["request"])} for row in inputs["train"]]
+ != [{key: row[key] for key in ("id", "request_sha256")} for row in expected]
+ or grades["split"] != "train"
+ or grades["plan_sha256"] != file_hash(training_plan_path)
+ or [row["id"] for row in grades["rows"]] != [row["id"] for row in expected]
+ ):
+ raise ValueError("Paired training outcomes differ from the original training population or solver settings")
+ provenance = solver_policies._grading_provenance(grades)
+ if (
+ any(provenance[name] != original[name] for name in ("grader_source_sha256", "gold_review_source_sha256"))
+ or grades.get("training_gold_review_source_sha256") != plan["eligibility"]["training_gold_review_source_sha256"]
+ or any(
+ grades.get(name) != training_plan.get(name)
+ for name in (
+ "grader_source_sha256",
+ "gold_review_source_sha256",
+ "gold_review_sha256",
+ "training_gold_rules_sha256",
+ "training_gold_review_source_sha256",
+ )
+ )
+ ):
+ raise ValueError("Training grades must match their reviewed source plan and the unchanged base grader")
+ review_hashes = ("training_gold_rules_sha256", "training_gold_review_source_sha256")
+ if any(name in grades for name in review_hashes) and any(
+ not isinstance(grades.get(name), str)
+ or len(grades[name]) != 64
+ or any(char not in "0123456789abcdef" for char in grades[name])
+ for name in review_hashes
+ ):
+ raise ValueError("Training-specific gold review requires both source and rule hashes")
+ signed_targets(grades["rows"])
+ return grades
+
+
+def _selections(rows, scores, credits, rate):
+ return {
+ str(target): solver_report.select_cutoff(rows, scores, target, router_credits=credits, credit_usd=rate)
+ for target in TARGETS
+ }
+
+
+def freeze(run, baseline, original_path, training_plan_path, training_inputs_path, training_grades_path):
+ run, baseline = Path(run), Path(baseline)
+ for name in ("frozen.json", "test-access.json", "report.json"):
+ if (run / name).exists():
+ raise FileExistsError(run / name)
+ plan = _checked_plan(read(run / "plan.json"))
+ dataset, original, solver_plan, artifacts = _inputs(baseline, original_path)
+ if (
+ digest(dataset) != plan["dataset_sha256"]
+ or file_hash(original_path) != plan["original_frozen_sha256"]
+ or artifacts != plan["baseline_artifacts_sha256"]
+ or solver_plan["settings_sha256"] != plan["solver_settings_sha256"]
+ ):
+ raise ValueError("Original experiment or vectors changed after benefit preparation")
+ grades = _training(plan, training_plan_path, training_inputs_path, training_grades_path, original)
+ eligible_ids = {row["id"] for row in plan["populations"]["train"]}
+ populations = dataset["splits"] | {
+ "train": [row for row in dataset["splits"]["train"] if row["id"] in eligible_ids]
+ }
+ matrices = {
+ split: vectors(
+ populations[split], baseline / ("test-embeddings" if split == "test" else "development-embeddings")
+ )
+ for split in ("train", "calibration", "test")
+ }
+ if any(
+ matrix.shape != (len(plan["populations"][split]), PROTOCOL["dimensions"]) for split, matrix in matrices.items()
+ ):
+ raise ValueError("Cached vector dimensions differ from the pinned encoder")
+ head = fit_head(matrices["train"], grades["rows"], [row["paper_group"] for row in plan["populations"]["train"]])
+ scores = {
+ split: (matrix @ np.array(head["weights"]) + head["bias"]).tolist()
+ for split, matrix in matrices.items()
+ if split != "train"
+ }
+ selections = _selections(
+ original["calibration_rows"],
+ scores["calibration"],
+ original["embedding_calibration_credits"],
+ original["credit_usd"],
+ )
+ return seal(
+ {
+ "format": FROZEN_FORMAT,
+ "frozen_at": now(),
+ "plan": plan,
+ "plan_file_sha256": file_hash(run / "plan.json"),
+ "training_pool_rows": PROTOCOL["training_pool_rows"],
+ "training_rows": len(head["training_ids"]),
+ "head": head,
+ "scores": scores,
+ "training_artifacts_sha256": {
+ "plan": file_hash(training_plan_path),
+ "inputs": file_hash(training_inputs_path),
+ "grades": file_hash(training_grades_path),
+ },
+ "training_grade_provenance": {key: value for key, value in grades.items() if key != "rows"},
+ "grading_provenance": solver_policies._grading_provenance(original),
+ "calibration_grades_sha256": original["calibration_grades_sha256"],
+ "calibration_rows": original["calibration_rows"],
+ "embedding_calibration_credits": original["embedding_calibration_credits"],
+ "embedding_calibration_credits_sha256": original["embedding_calibration_credits_sha256"],
+ "credit_usd": original["credit_usd"],
+ "selections": selections,
+ "test_access": PROTOCOL["test_boundary"],
+ }
+ )
+
+
+def _checked_frozen(run):
+ run = Path(run)
+ frozen = checked(read(run / "frozen.json"), FROZEN_FORMAT)
+ plan = _checked_plan(frozen["plan"])
+ if frozen["plan_file_sha256"] != file_hash(run / "plan.json") or read(run / "plan.json") != plan:
+ raise ValueError("Prepared benefit plan changed after freezing")
+ head = frozen["head"]
+ expected_ids = [row["id"] for row in plan["populations"]["train"]]
+ if (
+ head["training_ids"] != expected_ids
+ or frozen["training_pool_rows"] != PROTOCOL["training_pool_rows"]
+ or frozen["training_rows"] != PROTOCOL["training_rows"]
+ or head["folds"] != plan["training_folds"]
+ or len(head["targets"]) != len(expected_ids)
+ or any(target not in (-1, 0, 1) for target in head["targets"])
+ or np.asarray(head["weights"]).shape != (PROTOCOL["dimensions"],)
+ or not np.isfinite(head["weights"]).all()
+ or not np.isfinite(head["bias"])
+ or [item["penalty"] for item in head["cv"]] != PROTOCOL["penalties"]
+ or any(
+ np.asarray(item["fold_mse"]).shape != (PROTOCOL["folds"],)
+ or not np.isfinite([item["pooled_mse"], *item["fold_mse"]]).all()
+ or min(item["pooled_mse"], *item["fold_mse"]) < 0
+ for item in head["cv"]
+ )
+ or head["penalty"] != min(head["cv"], key=lambda item: (item["pooled_mse"], -item["penalty"]))["penalty"]
+ ):
+ raise ValueError("Frozen benefit model differs from the declared training/CV protocol")
+ for split in ("calibration", "test"):
+ solver_policies.finite_scores(frozen["scores"][split], len(plan["populations"][split]))
+ if [row["id"] for row in frozen["calibration_rows"]] != [
+ row["id"] for row in plan["populations"]["calibration"]
+ ] or frozen["selections"] != _selections(
+ frozen["calibration_rows"],
+ frozen["scores"]["calibration"],
+ frozen["embedding_calibration_credits"],
+ frozen["credit_usd"],
+ ):
+ raise ValueError("Frozen benefit cutoffs differ from calibration-only selection")
+ solver_policies._grading_provenance(frozen["grading_provenance"])
+ return frozen
+
+
+def authorize_test(run):
+ """Record the new freeze before callers remove any held-output guard."""
+ run = Path(run)
+ if (run / "test-access.json").exists() or (run / "report.json").exists():
+ raise FileExistsError(run / "test-access.json")
+ frozen = _checked_frozen(run)
+ return seal(
+ {
+ "format": ACCESS_FORMAT,
+ "authorized_at": now(),
+ "frozen_sha256": file_hash(run / "frozen.json"),
+ "plan_sha256": frozen["plan_file_sha256"],
+ "source_code_sha256": code_hashes(),
+ "statement": "This receipt was created after the benefit model and all cutoffs froze, before opening solver test outcomes",
+ }
+ )
+
+
+def report(run, test_grades_path, router_credits_path=None):
+ """Read test outcomes only after validating the frozen model/access chain."""
+ run = Path(run)
+ frozen = _checked_frozen(run)
+ access = checked(read(run / "test-access.json"), ACCESS_FORMAT)
+ if (
+ access["frozen_sha256"] != file_hash(run / "frozen.json")
+ or access["plan_sha256"] != frozen["plan_file_sha256"]
+ or access["source_code_sha256"] != code_hashes()
+ ):
+ raise ValueError("Test-access receipt does not bind this frozen benefit model")
+ plan = frozen["plan"]
+ ids = [row["id"] for row in plan["populations"]["test"]]
+ grades = solver_policies._grade_rows(test_grades_path, "test", plan["original_solver_plan_sha256"], ids)
+ if (
+ grades["policies_frozen_sha256"] != plan["original_frozen_sha256"]
+ or solver_policies._grading_provenance(grades) != frozen["grading_provenance"]
+ ):
+ raise ValueError("Test answers differ from the original frozen solver/grading experiment")
+ credits, credit_sha = solver_policies._credits(
+ router_credits_path,
+ "test",
+ plan["original_solver_plan_sha256"],
+ ids,
+ {row["id"]: row["request_sha256"] for row in plan["populations"]["test"]},
+ )
+ rows, rate = grades["rows"], frozen["credit_usd"]
+ measured = {}
+ for target in TARGETS:
+ selection = frozen["selections"][str(target)]
+ mode = selection["mode"]
+ routes = (np.array(frozen["scores"]["test"]) >= selection["cutoff"]).tolist() if mode == "threshold" else mode
+ measured[str(target)] = {
+ "mode": mode,
+ "cutoff": selection["cutoff"],
+ "metrics": solver_report.summarize(
+ rows, routes, router_credits=credits if mode == "threshold" else 0, credit_usd=rate
+ ),
+ }
+ return {
+ "format": "sie-solver-benefit-report-v1",
+ "n": len(ids),
+ "training_pool_rows": frozen["training_pool_rows"],
+ "training_rows": frozen["training_rows"],
+ "eligibility_sha256": plan["eligibility_sha256"],
+ "ids": ids,
+ "frozen_sha256": file_hash(run / "frozen.json"),
+ "test_access_sha256": file_hash(run / "test-access.json"),
+ "test_grades_sha256": file_hash(test_grades_path),
+ "embedding_test_credits_sha256": credit_sha,
+ "primary_max_quality_loss_pp": 1,
+ "baselines": {
+ name: solver_report.summarize(rows, name, router_credits=0, credit_usd=rate)
+ for name in ("always_strong", "always_cheap")
+ },
+ "policies": measured,
+ "interpretation": "Outcome-targeted head prompted by calibration analysis and frozen before solver test outcome access. Difficulty benchmark was already exposed. Calibration loss targets are empirical, not test guarantees. Acquisition/training costs are separate from per-request serving costs.",
+ }
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ commands = parser.add_subparsers(dest="command", required=True)
+ for command in ("prepare", "freeze", "authorize-test", "report"):
+ child = commands.add_parser(command)
+ child.add_argument("--run", type=Path, required=True)
+ if command in ("prepare", "freeze"):
+ child.add_argument("--baseline", type=Path, required=True)
+ child.add_argument("--original-frozen", type=Path, required=True)
+ if command == "prepare":
+ child.add_argument("--eligibility", type=Path, required=True)
+ child.add_argument("--attest-test-outcomes-unopened", action="store_true", required=True)
+ if command == "freeze":
+ for name in ("training-plan", "training-inputs", "training-grades"):
+ child.add_argument(f"--{name}", type=Path, required=True)
+ if command == "report":
+ child.add_argument("--test-grades", type=Path, required=True)
+ child.add_argument("--router-credits", type=Path)
+ args = parser.parse_args()
+ destination = (
+ args.run
+ / {
+ "prepare": "plan.json",
+ "freeze": "frozen.json",
+ "authorize-test": "test-access.json",
+ "report": "report.json",
+ }[args.command]
+ )
+ if destination.exists():
+ raise FileExistsError(destination)
+ if args.command == "prepare":
+ if args.run.exists():
+ raise FileExistsError(args.run)
+ result = prepare(
+ args.baseline,
+ args.original_frozen,
+ args.eligibility,
+ attest_test_outcomes_unopened=args.attest_test_outcomes_unopened,
+ )
+ elif args.command == "freeze":
+ result = freeze(
+ args.run,
+ args.baseline,
+ args.original_frozen,
+ args.training_plan,
+ args.training_inputs,
+ args.training_grades,
+ )
+ elif args.command == "authorize-test":
+ result = authorize_test(args.run)
+ else:
+ result = report(args.run, args.test_grades, args.router_credits)
+ write(destination, result)
+ if args.command == "report":
+ metrics = result["policies"]["1"]["metrics"]
+ gross = "n/a" if metrics["gross_solver_savings_pct"] is None else f"{metrics['gross_solver_savings_pct']:.2f}%"
+ print(
+ f"Benefit router: answer accuracy {100 * metrics['answer_accuracy']:.2f}%; gap {metrics['gap_vs_strong_pp']:+.2f}pp; gross savings {gross}"
+ )
+ if "net_savings_pct" in metrics:
+ net = "n/a" if metrics["net_savings_pct"] is None else f"{metrics['net_savings_pct']:.2f}%"
+ print(
+ f"Including encoder: {metrics['net_total_cost_usd_per1000']:.4f} USD/1,000 requests; net savings {net}"
+ )
+ else:
+ print("Router overhead unpriced; net savings unknown")
+ print(json.dumps({"output": str(destination), "sha256": file_hash(destination)}))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/benefit_collect.py b/examples/train-a-router/benefit_collect.py
new file mode 100644
index 000000000..2baf800e2
--- /dev/null
+++ b/examples/train-a-router/benefit_collect.py
@@ -0,0 +1,448 @@
+"""Collect training solver outcomes through a Cloud or self-hosted SIE endpoint.
+
+Each arm saves its initial answers before collecting conditional finalizers.
+This changes scheduling only: messages, sampling, answer budgets and finalizer
+conditions come from solver.py. Completed calls are never repeated on resume.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+import shutil
+import time
+from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
+from pathlib import Path
+
+import solver
+from benefit_data import checked_training
+from data import digest, file_hash, write
+from serving import SingleSendClient, error_receipt
+from solver_answers import extract_final_answer
+
+PHASES = ("initial", "final")
+
+
+def messages_for(row, arm, settings, first=None):
+ if first is None:
+ return [
+ {"role": "system", "content": settings["arms"][arm]["system"]},
+ {"role": "user", "content": row["request"]},
+ ]
+ return [
+ {"role": "system", "content": solver.FINAL_SYSTEM},
+ {"role": "user", "content": json.dumps({"problem": row["request"], "partial_work": first["answer"]})},
+ ]
+
+
+def needs_final(first):
+ return first["finish_reason"] == "length" or not extract_final_answer(first["answer"])
+
+
+def stage_call(row, arm, settings, phase, messages, self_hosted):
+ result = {
+ "id": row["id"],
+ "arm": arm,
+ "phase": phase,
+ "request_sha256": digest(row["request"]),
+ "settings_sha256": digest(settings),
+ "messages_sha256": digest(messages),
+ "status": "incomplete",
+ }
+ client = None
+ try:
+ with solver.RecordingClient(timeout_s=180, read_timeout_s=180) as client:
+ result["call"] = solver.call(
+ client, settings["arms"][arm], messages, final=phase == "final", self_hosted=self_hosted
+ )
+ if not result["call"]["response_valid"] or not result["call"]["billing_known"]:
+ raise ValueError("Successful response has no usable answer or required charge receipt")
+ result["status"] = "complete"
+ except Exception as error: # noqa: BLE001 -- retain sanitized failures and any successful receipt
+ result["error"] = error_receipt(error)
+ failures = list(client.http_failures) if client is not None else []
+ result["http_failures"] = failures
+ if failures and "error" in result:
+ result["error"].update(failures[-1])
+ receipt = result.get("call", {})
+ credits = receipt.get("receipt", {}).get("credits")
+ result["observed_credits"] = (credits if solver.valid_credit(credits) else 0) + sum(
+ failure["credits"] for failure in failures if solver.valid_credit(failure["credits"])
+ )
+ known_rejection = bool(failures) and failures[-1]["known_pre_execution_rejection"]
+ result["execution_unknown"] = result["status"] != "complete" and not known_rejection
+ result["safe_to_resume"] = known_rejection and not receipt
+ return result
+
+
+def checked_stage(record, row, arm, settings, phase, messages):
+ expected = {
+ "id": row["id"],
+ "arm": arm,
+ "phase": phase,
+ "request_sha256": digest(row["request"]),
+ "settings_sha256": digest(settings),
+ "messages_sha256": digest(messages),
+ }
+ if any(record.get(key) != value for key, value in expected.items()) or record.get("status") not in (
+ "complete",
+ "incomplete",
+ ):
+ raise ValueError("Saved stage differs from its request or solver protocol")
+ if not solver.valid_credit(record.get("observed_credits")):
+ raise ValueError("Saved stage has invalid observed charges")
+ if record["status"] == "complete":
+ call = record.get("call", {})
+ if (
+ call.get("messages_sha256") != digest(messages)
+ or not call.get("response_valid")
+ or not call.get("billing_known")
+ or not isinstance(call.get("answer"), str)
+ ):
+ raise ValueError("Saved successful stage has no valid bound call receipt")
+
+
+def history(output, row, arm, settings, phase, messages):
+ folder = output / "attempts" / row["id"]
+ receipts = sorted(folder.glob(f"{phase}-*.json"))
+ receipts = [path for path in receipts if not path.name.endswith(".intent.json")]
+ records = []
+ for path in receipts:
+ record = solver.read(path)
+ checked_stage(record, row, arm, settings, phase, messages)
+ records.append((path, record))
+ intents = sorted(folder.glob(f"{phase}-*.intent.json"))
+ interrupted = [path for path in intents if not path.with_name(path.name.replace(".intent", "")).exists()]
+ successful = [(path, record) for path, record in records if record["status"] == "complete"]
+ if len(successful) > 1:
+ raise ValueError("An already completed stage was submitted again")
+ return records, interrupted, successful[0] if successful else None
+
+
+def materialize(output, row, arm, settings, self_hosted):
+ initial_messages = messages_for(row, arm, settings)
+ _, _, initial = history(output, row, arm, settings, "initial", initial_messages)
+ if initial is None:
+ return None
+ first = initial[1]["call"]
+ calls = [first]
+ answer = first["answer"]
+ format_error = False
+ if needs_final(first):
+ _, _, final = history(output, row, arm, settings, "final", messages_for(row, arm, settings, first))
+ if final is None:
+ return None
+ calls.append(final[1]["call"])
+ try:
+ parsed = json.loads(calls[-1]["answer"])
+ if not isinstance(parsed, dict) or set(parsed) != {"answer"} or not isinstance(parsed["answer"], str):
+ raise ValueError("Malformed final answer")
+ answer = "\\boxed{" + parsed["answer"] + "}"
+ except (ValueError, TypeError):
+ answer, format_error = "", True
+ observed = sum(call["receipt"]["credits"] for call in calls if solver.valid_credit(call["receipt"]["credits"]))
+ record = solver.initial_record(row, arm, settings) | {
+ "status": "complete",
+ "calls": calls,
+ "answer": answer,
+ "observed_credits": observed,
+ "credits": None if self_hosted else observed,
+ "has_unknown_charges": False,
+ "safe_to_resume": False,
+ }
+ if format_error:
+ record["final_format_error"] = True
+ solver.validate_record(record, row, arm, settings)
+ target = output / "records" / f"{row['id']}-{arm}.json"
+ if target.exists() and solver.read(target) != record:
+ raise ValueError("A completed answer changed after acquisition")
+ if not target.exists():
+ write(target, record)
+ return record
+
+
+def collect(
+ directory,
+ arm,
+ phase,
+ *,
+ workers=4,
+ limit=None,
+ max_seconds=600,
+ max_credits=None,
+ self_hosted=False,
+ resume=False,
+ retry_unknown=False,
+ max_attempts=1,
+ runtime_receipt=None,
+):
+ directory = Path(directory)
+ _, inputs, settings = checked_training(directory)
+ rows = inputs["train"]
+ if arm not in settings["arms"] or phase not in PHASES:
+ raise ValueError("Select a declared arm and collection phase")
+ if not 1 <= workers <= 16 or not math.isfinite(max_seconds) or max_seconds <= 0:
+ raise ValueError("Use 1..16 workers and a positive finite time limit")
+ if limit is not None and (type(limit) is not int or limit <= 0):
+ raise ValueError("The pilot limit must be a positive integer")
+ if type(max_attempts) is not int or max_attempts <= 0 or (retry_unknown and not resume):
+ raise ValueError("Use positive bounded attempts; unknown repeats require explicit resume")
+ if not self_hosted and (max_credits is None or not solver.valid_credit(max_credits) or max_credits <= 0):
+ raise ValueError("Cloud collection requires a positive observed-credit ceiling")
+ output = directory / "benefit-arms" / arm
+ binding = {
+ "ids": [row["id"] for row in rows],
+ "arm": arm,
+ "plan_sha256": file_hash(directory / "plan.json"),
+ "settings_sha256": digest(settings),
+ "self_hosted": self_hosted,
+ "endpoint_sha256": digest(os.environ.get("SIE_BASE_URL", "https://api.superlinked.com")),
+ "runtime_receipt_sha256": file_hash(runtime_receipt) if runtime_receipt else None,
+ "scheduling": "Initial calls first, then conditional finalizers; no automatic generation retries",
+ }
+ plan_path = output / "arm-plan.json"
+ if plan_path.exists():
+ if not resume or solver.read(plan_path) != binding:
+ raise ValueError("Existing arm requires resume with identical inputs, endpoint and runtime")
+ else:
+ if resume:
+ raise ValueError("No existing arm to resume")
+ write(plan_path, binding)
+ if runtime_receipt:
+ shutil.copyfile(runtime_receipt, output / "runtime-receipt.json")
+ todo = []
+ for row in rows[:limit] if limit else rows:
+ first = None
+ if phase == "final":
+ _, _, initial = history(output, row, arm, settings, "initial", messages_for(row, arm, settings))
+ if initial is None:
+ raise ValueError("Finalization requires a successful initial receipt for every selected row")
+ first = initial[1]["call"]
+ if not needs_final(first):
+ materialize(output, row, arm, settings, self_hosted)
+ continue
+ messages = messages_for(row, arm, settings, first)
+ records, interrupted, success = history(output, row, arm, settings, phase, messages)
+ if success:
+ materialize(output, row, arm, settings, self_hosted)
+ continue
+ if len(records) + len(interrupted) >= max_attempts:
+ raise ValueError("Attempt ceiling reached; inspect immutable receipts before raising it")
+ if (interrupted or (records and not records[-1][1].get("safe_to_resume"))) and not retry_unknown:
+ raise ValueError("Prior execution is unknown; inspect receipts before explicit --retry-unknown")
+ todo.append((row, messages, len(records) + len(interrupted) + 1))
+ sessions = output / "sessions"
+ session = sessions / f"session-{len(list(sessions.glob('session-*'))) + 1:04d}"
+ write(
+ session / "plan.json",
+ {
+ "arm_plan_sha256": file_hash(plan_path),
+ "phase": phase,
+ "workers": workers,
+ "pending_ids": [row["id"] for row, _, _ in todo],
+ "max_seconds": max_seconds,
+ "max_credits": max_credits,
+ "max_attempts": max_attempts,
+ "retry_unknown": retry_unknown,
+ "limit": limit,
+ "budget_scope": "New observed charges; already running calls are drained",
+ },
+ )
+ started = time.monotonic()
+ sent, failed, observed = 0, 0, 0.0
+ fatal = None
+ try:
+ if todo:
+ with SingleSendClient(timeout_s=60) as client:
+ model = settings["arms"][arm]
+ metadata = client.get_model(model["model"])
+ if (
+ metadata.get("revision") != model["revision"]
+ or metadata.get("name", metadata.get("id")) != model["model"]
+ ):
+ raise ValueError("Endpoint model revision differs from the frozen solver")
+ write(session / "model.json", {key: metadata.get(key) for key in ("name", "revision", "profiles")})
+ tasks = iter(todo)
+ with ThreadPoolExecutor(max_workers=workers) as pool:
+ pending = {}
+
+ def schedule():
+ nonlocal sent
+ if (
+ failed
+ or time.monotonic() - started >= max_seconds
+ or (max_credits is not None and observed >= max_credits)
+ ):
+ return
+ task = next(tasks, None)
+ if task is None:
+ return
+ row, messages, attempt = task
+ target = output / "attempts" / row["id"] / f"{phase}-{attempt:04d}.json"
+ write(
+ target.with_name(target.stem + ".intent.json"),
+ {
+ "id": row["id"],
+ "arm": arm,
+ "phase": phase,
+ "session": session.name,
+ "request_sha256": digest(row["request"]),
+ "settings_sha256": digest(settings),
+ "messages_sha256": digest(messages),
+ },
+ )
+ future = pool.submit(stage_call, row, arm, settings, phase, messages, self_hosted)
+ pending[future] = row, messages, target
+ sent += 1
+
+ for _ in range(workers):
+ schedule()
+ while pending:
+ done, _ = wait(pending, timeout=10, return_when=FIRST_COMPLETED)
+ for future in done:
+ row, messages, target = pending.pop(future)
+ record = future.result()
+ checked_stage(record, row, arm, settings, phase, messages)
+ write(target, record)
+ observed += record["observed_credits"]
+ failed += record["status"] != "complete"
+ materialize(output, row, arm, settings, self_hosted)
+ print(f"{arm}/{phase}: {sent - len(pending)}/{len(todo)} returned, {failed} failed", flush=True)
+ for _ in done:
+ schedule()
+ except Exception as error: # noqa: BLE001 -- keep stage receipts after any orchestration failure
+ fatal = error_receipt(error)
+ summary = {
+ "status": "complete" if sent == len(todo) and not failed and fatal is None else "incomplete",
+ "phase": phase,
+ "sent": sent,
+ "failed": failed,
+ "observed_credits": observed,
+ "elapsed_seconds": time.monotonic() - started,
+ "fatal_error": fatal,
+ }
+ write(session / "collection.json", summary)
+ if summary["status"] != "complete":
+ raise RuntimeError("Collection phase incomplete; preserve receipts and inspect failures before manual resume")
+ return summary
+
+
+def gather(directory, arm_directories=None):
+ directory = Path(directory)
+ _, inputs, settings = checked_training(directory)
+ rows = inputs["train"]
+ locations = arm_directories or {arm: directory / "benefit-arms" / arm for arm in settings["arms"]}
+ if set(locations) != set(settings["arms"]):
+ raise ValueError("Gather requires exactly the declared solver arms")
+ plans = {arm: solver.read(Path(path) / "arm-plan.json") for arm, path in locations.items()}
+ self_hosted = plans[next(iter(plans))]["self_hosted"]
+ for arm, plan in plans.items():
+ if (
+ plan["ids"] != [r["id"] for r in rows]
+ or plan["arm"] != arm
+ or plan["plan_sha256"] != file_hash(directory / "plan.json")
+ or plan["settings_sha256"] != digest(settings)
+ or plan["self_hosted"] != self_hosted
+ ):
+ raise ValueError("Arm evidence differs from the prepared training population or hosting mode")
+ output = directory / "answers-train"
+ plan = {
+ "ids": [r["id"] for r in rows],
+ "split": "train",
+ "plan_sha256": file_hash(directory / "plan.json"),
+ "settings_sha256": digest(settings),
+ "policies_frozen_sha256": None,
+ "self_hosted": self_hosted,
+ "arms": plans,
+ "acquisition_scheduling": "Initial calls before conditional finalizers, separately per arm",
+ }
+ plan_path = output / "collection-plan.json"
+ if plan_path.exists() and solver.read(plan_path) != plan:
+ raise ValueError("Existing combined collection uses a different acquisition plan")
+ if not plan_path.exists():
+ write(plan_path, plan)
+ completed, observed, failed_attempts, unknown = 0, 0.0, 0, 0
+ receipts = {}
+ for arm, location in locations.items():
+ location = Path(location)
+ for row in rows:
+ record = materialize(location, row, arm, settings, self_hosted)
+ if record is not None:
+ target = output / f"{row['id']}-{arm}.json"
+ if target.exists() and solver.read(target) != record:
+ raise ValueError("Combined answer changed after collection")
+ if not target.exists():
+ write(target, record)
+ completed += 1
+ for phase in PHASES:
+ folder = location / "attempts" / row["id"]
+ for path in sorted(folder.glob(f"{phase}-*.json")):
+ relative = Path(arm) / path.relative_to(location)
+ target = output / "acquisition" / relative
+ if target.exists() and file_hash(target) != file_hash(path):
+ raise ValueError("Archived acquisition receipt changed")
+ if not target.exists():
+ target.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copyfile(path, target)
+ receipts[relative.as_posix()] = file_hash(path)
+ if path.name.endswith(".intent.json"):
+ unknown += not path.with_name(path.name.replace(".intent", "")).exists()
+ else:
+ attempt = solver.read(path)
+ observed += attempt["observed_credits"]
+ failed_attempts += attempt["status"] != "complete"
+ unknown += attempt.get("execution_unknown", True)
+ summary = {
+ "status": "complete" if completed == 2 * len(rows) else "incomplete",
+ "completed": completed,
+ "failed": 0,
+ "missing": 2 * len(rows) - completed,
+ "observed_credits": observed,
+ "failed_attempts": failed_attempts,
+ "unknown_execution_attempts": unknown,
+ "acquisition_cost_complete": not self_hosted and unknown == 0,
+ "infrastructure_cost_status": "unpriced" if self_hosted else "not_applicable",
+ "receipt_sha256": receipts,
+ }
+ solver.save_latest(output / "collection.json", summary)
+ if summary["status"] != "complete":
+ raise RuntimeError("Paired training collection is incomplete")
+ return summary
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ commands = parser.add_subparsers(dest="command", required=True)
+ acquire = commands.add_parser("collect")
+ acquire.add_argument("--directory", type=Path, required=True)
+ acquire.add_argument("--arm", choices=tuple(solver.ARMS), required=True)
+ acquire.add_argument("--phase", choices=PHASES, required=True)
+ acquire.add_argument("--workers", type=int, default=4)
+ acquire.add_argument("--limit", type=int)
+ acquire.add_argument("--max-seconds", type=float, required=True)
+ acquire.add_argument("--max-credits", type=float)
+ acquire.add_argument("--self-hosted", action="store_true")
+ acquire.add_argument("--resume", action="store_true")
+ acquire.add_argument("--retry-unknown", action="store_true")
+ acquire.add_argument("--max-attempts", type=int, default=1)
+ acquire.add_argument("--runtime-receipt", type=Path)
+ combine = commands.add_parser("gather")
+ combine.add_argument("--directory", type=Path, required=True)
+ combine.add_argument("--cheap", type=Path)
+ combine.add_argument("--strong", type=Path)
+ args = parser.parse_args()
+ if args.command == "collect":
+ values = vars(args).copy()
+ values.pop("command")
+ result = collect(**values)
+ else:
+ if bool(args.cheap) != bool(args.strong):
+ parser.error("Provide both --cheap and --strong, or neither")
+ result = gather(args.directory, {"cheap": args.cheap, "strong": args.strong} if args.cheap else None)
+ print(json.dumps({key: value for key, value in result.items() if key != "receipt_sha256"}, sort_keys=True))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/benefit_costs.py b/examples/train-a-router/benefit_costs.py
new file mode 100644
index 000000000..18dac3be8
--- /dev/null
+++ b/examples/train-a-router/benefit_costs.py
@@ -0,0 +1,208 @@
+"""Estimate Cloud token charges for paired answers collected on your own server.
+
+These estimates train the router's cost model. They are not Modal bills or
+observed Cloud charges: every input token is priced without a cache discount.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+from collections import defaultdict
+from datetime import UTC, datetime
+from fractions import Fraction
+from pathlib import Path
+
+import solver
+from data import digest, file_hash, write
+
+MODELS = {"cheap": "Qwen/Qwen3.5-4B", "strong": "Qwen/Qwen3.8-27B-FP8"}
+
+
+def read(path):
+ return json.loads(Path(path).read_text())
+
+
+def _fraction(value):
+ return Fraction(int(value["numerator"]), int(value["denominator"]))
+
+
+def rates_from_snapshot(snapshot):
+ if snapshot.get("currency") != "usd" or not snapshot.get("pricing_version"):
+ raise ValueError("Need a versioned USD price snapshot")
+ rates, selected = {}, []
+ for arm, model in MODELS.items():
+ rates[arm] = {}
+ for unit in ("input_tokens", "output_tokens"):
+ matches = [
+ row
+ for row in snapshot.get("items", [])
+ if row.get("operation") == "generate"
+ and row.get("model") == model
+ and row.get("profile") == "default"
+ and row.get("unit") == unit
+ and row.get("region") == "us"
+ and row.get("currency") == "usd"
+ ]
+ if len(matches) != 1:
+ raise ValueError(f"Need exactly one default/us price for {model}/{unit}")
+ value = _fraction(matches[0]["usd_per_unit"]) * 100000
+ if value < 0:
+ raise ValueError("Token prices cannot be negative")
+ rates[arm][unit] = {"numerator": value.numerator, "denominator": value.denominator}
+ selected.append(matches[0])
+ return rates, selected
+
+
+def physical_costs(record, rates):
+ if record.get("status") != "complete" or record.get("has_unknown_charges"):
+ raise ValueError("Cost estimates require a complete, unambiguous answer record")
+ calls = record.get("calls")
+ if not isinstance(calls, list) or not calls:
+ raise ValueError("Every answer must retain every physical call")
+ priced = []
+ for call in calls:
+ tokens = call.get("tokens", {})
+ if any(type(tokens.get(key)) is not int or tokens[key] < 0 for key in ("prompt_tokens", "completion_tokens")):
+ raise ValueError("Physical calls need nonnegative integer token counts")
+ price = math.ceil(
+ tokens["prompt_tokens"] * _fraction(rates["input_tokens"])
+ + tokens["completion_tokens"] * _fraction(rates["output_tokens"])
+ )
+ priced.append(
+ {
+ "prompt_tokens": tokens["prompt_tokens"],
+ "completion_tokens": tokens["completion_tokens"],
+ "credits_proxy": price,
+ "finish_reason": call.get("finish_reason"),
+ }
+ )
+ return priced
+
+
+def build(training, prices_path, *, grades_path=None):
+ training, prices_path = Path(training), Path(prices_path)
+ grades_path = training / "grades-train.json" if grades_path is None else Path(grades_path)
+ plan_path = training / "plan.json"
+ collection_path = training / "answers-train" / "collection.json"
+ binding_path = training / "answers-train" / "collection-plan.json"
+ grades, collection = read(grades_path), read(collection_path)
+ plan, inputs, settings, binding = map(
+ read, (plan_path, training / "inputs.json", training / "settings.json", binding_path)
+ )
+ if plan.get("inputs_sha256") != digest(inputs) or plan.get("settings_sha256") != digest(settings):
+ raise ValueError("Training inputs or solver settings changed")
+ solver.collection_binding(binding, training, "train", inputs["train"], settings, None, binding["self_hosted"])
+ collected_ids = [row["id"] for row in inputs["train"]]
+ graded_ids = [row["id"] for row in grades["rows"]]
+ filtered = "collected_ids" in grades or "excluded_ids" in grades
+ excluded_ids = grades.get("excluded_ids", [])
+ if (
+ not isinstance(excluded_ids, list)
+ or any(not isinstance(identity, str) for identity in excluded_ids)
+ or len(set(excluded_ids)) != len(excluded_ids)
+ or (filtered and (grades.get("collected_ids") != collected_ids or "excluded_ids" not in grades))
+ or not set(excluded_ids).issubset(collected_ids)
+ or excluded_ids != [identity for identity in collected_ids if identity in set(excluded_ids)]
+ or graded_ids != [identity for identity in collected_ids if identity not in set(excluded_ids)]
+ ):
+ raise ValueError("Retained and excluded IDs must partition the original collected training population in order")
+ if (
+ grades.get("split") != "train"
+ or grades.get("plan_sha256") != file_hash(plan_path)
+ or grades.get("collection_plan_sha256") != file_hash(binding_path)
+ or grades.get("collection_summary_sha256") != file_hash(collection_path)
+ or collection.get("status") != "complete"
+ or collection.get("completed") != 2 * len(collected_ids)
+ or collection.get("failed") != 0
+ or collection.get("missing", 0) != 0
+ or collection.get("unknown_execution_attempts", 0) != 0
+ ):
+ raise ValueError("Training grades must bind the complete paired collection and plan")
+ prices = read(prices_path)
+ rates, public_rows = rates_from_snapshot(prices)
+ input_by_id = {row["id"]: row for row in inputs["train"]}
+ rows, groups, seen = [], defaultdict(list), set()
+ for grade in grades["rows"]:
+ identity = grade["id"]
+ if identity in seen or not identity or Path(identity).name != identity:
+ raise ValueError("Training IDs must be unique filenames")
+ seen.add(identity)
+ row = {"id": identity}
+ for arm, model in MODELS.items():
+ path = training / "answers-train" / f"{identity}-{arm}.json"
+ if file_hash(path) != grade[f"{arm}_record_sha256"]:
+ raise ValueError("Raw solver answer changed after grading")
+ record = read(path)
+ solver.validate_record(record, input_by_id[identity], arm, settings)
+ if record.get("id") != identity or record.get("arm") != arm or record.get("model") != model:
+ raise ValueError("Raw solver answer uses a different question, arm or model")
+ correct = grade[f"{arm}_correct"]
+ if type(correct) is not bool:
+ raise ValueError("Correctness labels must be booleans")
+ calls = physical_costs(record, rates[arm])
+ total = sum(call["credits_proxy"] for call in calls)
+ row.update(
+ {
+ f"{arm}_credits_proxy": total,
+ f"{arm}_correct": correct,
+ f"{arm}_record_sha256": file_hash(path),
+ f"{arm}_calls": calls,
+ }
+ )
+ groups[f"{arm}_{'correct' if correct else 'wrong'}"].append(total)
+ rows.append(row)
+ if set(groups) != {f"{arm}_{state}" for arm in MODELS for state in ("correct", "wrong")}:
+ raise ValueError("Need correct and incorrect examples for both solvers")
+ return {
+ "format": "sie-training-uncached-cloud-cost-proxy-v1",
+ "created_at": datetime.now(UTC).isoformat(),
+ "split": "train",
+ "n": len(rows),
+ "credit_usd": 1e-5,
+ "training_plan_sha256": file_hash(plan_path),
+ "training_grades_sha256": file_hash(grades_path),
+ "training_collection_sha256": file_hash(collection_path),
+ "training_collection_plan_sha256": file_hash(binding_path),
+ "collected_ids": collected_ids,
+ "excluded_ids": excluded_ids,
+ "builder_source_sha256": file_hash(Path(__file__)),
+ "price_source": {
+ "sha256": file_hash(prices_path),
+ "pricing_version": prices["pricing_version"],
+ "rate_book_sha256": prices["rate_book_sha256"],
+ "public_rows": public_rows,
+ },
+ "rates_credits_per_token": rates,
+ "formula": "Sum every physical call, including finalizers: ceil(prompt_tokens*input_rate + completion_tokens*output_rate), using exact rational rates.",
+ "cache_assumption": "All input tokens use the uncached rate; saved token summaries do not retain cached-token counts.",
+ "cost_interpretation": "Uncached Cloud list-price estimates for the observed output lengths, not measured Cloud charges or self-hosting costs.",
+ "conditional_means": {
+ name: {"n": len(values), "sum_credits_proxy": sum(values), "mean_credits_proxy": sum(values) / len(values)}
+ for name, values in groups.items()
+ },
+ "rows": rows,
+ }
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--training", type=Path, required=True)
+ parser.add_argument("--prices", type=Path, required=True)
+ parser.add_argument(
+ "--grades",
+ type=Path,
+ help="Separate source-reviewed training grades; defaults to the collection's grades-train.json",
+ )
+ parser.add_argument("--output", type=Path, required=True)
+ args = parser.parse_args()
+ if args.output.exists():
+ raise FileExistsError(args.output)
+ result = build(args.training, args.prices, grades_path=args.grades)
+ write(args.output, result)
+ print(json.dumps({"n": result["n"], "conditional_means": result["conditional_means"]}, indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/benefit_data.py b/examples/train-a-router/benefit_data.py
new file mode 100644
index 000000000..662867046
--- /dev/null
+++ b/examples/train-a-router/benefit_data.py
@@ -0,0 +1,545 @@
+"""Prepare and grade paired solver outcomes on the original 512 training rows.
+
+This is a separate source-only review protocol. The published validation/test
+grader is unchanged. Its labels are automated solver-outcome grades against
+reviewed benchmark reference answers, not human difficulty or teacher labels.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+from decimal import ROUND_HALF_UP, Decimal, localcontext
+from functools import lru_cache
+from pathlib import Path
+
+import numpy as np
+
+import evidence
+import solver
+from data import (
+ SOURCE_HASHES,
+ SOURCE_REVISION,
+ audit,
+ canonical,
+ digest,
+ download_source,
+ file_hash,
+ render_request,
+ write,
+)
+from serving import INSTRUCTION, MODEL, REVISION, vectors
+from solver_answers import extract_final_answer, grade_answer, grader_metadata, preflight_gold
+from solver_gold import grade_reviewed
+
+PROTOCOL = "source-only-benefit-training-gold-v1"
+# ID and original text are checked against the pinned Parquet before use. These
+# rules were decided from source questions/solutions before solver collection.
+RULES = {
+ "e2h-amc-eval-287": ("1,-1, i,-i", r"\{1,-1,i,-i\}", "finite_set"),
+ "e2h-amc-train-966": ("251,750,0", "251750000", "source_correction"),
+ "e2h-amc-eval-605": (
+ r"\left(\frac{1-\sqrt{3}}{2}, \frac{3-\sqrt{3}}{2}\right)",
+ r"(\frac{1-\sqrt{3}}{2},\frac{3-\sqrt{3}}{2})",
+ "ordered_tuple",
+ ),
+ "e2h-amc-eval-2451": (
+ "614.519...",
+ "1129.41806919024220383020154465861873859432518364370131999664",
+ "prime_approximation",
+ ),
+ "e2h-amc-train-574": ("1,2,4", r"\{1,2,4\}", "finite_set"),
+ "e2h-amc-train-230": (
+ r"(1,2),(4,2),\left(\frac{5}{2}, 2 \pm \frac{3}{2} i\right)",
+ r"\{(1,2),(4,2),(\frac52,2+\frac32 i),(\frac52,2-\frac32 i)\}",
+ "set_of_ordered_tuples",
+ ),
+ "e2h-amc-eval-2684": ("(13,-22)", "(13,-22)", "ordered_tuple"),
+ "e2h-amc-eval-2430": (r"\sqrt{\frac{2}{7}}", r"\frac27", "source_correction"),
+ "e2h-amc-eval-1984": ("80", r"\frac{3400}{41}\%", "rounded_percentage"),
+ "e2h-amc-eval-2931": (r"1 \%", r"1\%", "percentage"),
+ "e2h-amc-eval-1324": ("20", r"20\%", "percentage"),
+ "e2h-amc-eval-759": ("75", r"75\%", "percentage"),
+ "e2h-amc-eval-225": ("40", r"40\%", "percentage"),
+ "e2h-amc-train-649": ("70", r"70\%", "percentage"),
+ "e2h-amc-train-76": ("70", r"70\%", "percentage"),
+}
+REASONS = {
+ "e2h-amc-eval-287": "Subtracting k times the first polynomial from the second gives a*(1-k^4)=0; all four fourth roots are required.",
+ "e2h-amc-train-966": "The first five divisors are 1,2,4,5,8; 2014000000/8=251750000. The source prose agrees and its box lost zeros.",
+ "e2h-amc-eval-605": "The equilateral-triangle construction returns an ordered point; swapping coordinates changes the answer.",
+ "e2h-amc-eval-2451": "Exact integer DP W[0]=1, W[n]=sum(W[n-p] for prime p<=n) counts ordered prime tuples. log2(W[2013])=1129.418069190242..., contradicting the source heuristic 614.519.... The source awards partial credit; binary success here means at least 24 of its 25 points: min(C/A,A/C)>=123/125. This is a declared approximation rubric, not exact-answer grading.",
+ "e2h-amc-train-574": "Direct primality checks for all n=1..15 give exactly {1,2,4} for n^n+1 prime.",
+ "e2h-amc-train-230": "Subtracting the coupled equations gives (2a-5)*(b-2)=0. The four explicit ordered pairs satisfy both original equations; plus/minus denotes two points.",
+ "e2h-amc-eval-2684": "The square spiral reaches (-22,-22) at step1980 and moves35 east to (13,-22) at step2015.",
+ "e2h-amc-eval-2430": "The source equation is .7*(1-p)=.5, so p=2/7. Its derivation agrees; the boxed square root is erroneous.",
+ "e2h-amc-eval-1984": "The exact decrease from41 to7 is3400/41 percent. Accept the source's coarse80 percent, exact equivalents, and decimal values correctly rounded to their stated precision (for example83,82.9,82.93 percent). Other approximate answers are not silently accepted.",
+}
+CORRECTIONS = {"e2h-amc-train-966", "e2h-amc-eval-2430", "e2h-amc-eval-2451"}
+EXCLUSIONS = {
+ "e2h-amc-eval-661": {
+ "original_gold": "(3, 4)",
+ "reason": "The question asks which interval contains a root, but the original multiple-choice options are missing. Infinitely many intervals qualify; requiring the reference (3,4) would reject mathematically valid answers. Excluded from training before any solver outcomes were collected or inspected.",
+ },
+}
+
+
+def read(path):
+ return json.loads(Path(path).read_text())
+
+
+def source_hashes():
+ return {
+ "grader_source_sha256": file_hash(Path(__file__).with_name("solver_answers.py")),
+ "gold_review_source_sha256": file_hash(Path(__file__).with_name("solver_gold.py")),
+ "training_gold_review_source_sha256": file_hash(Path(__file__)),
+ "collector_source_sha256": file_hash(Path(__file__).with_name("solver.py")),
+ }
+
+
+@lru_cache(maxsize=1)
+def prime_compositions():
+ """Independently reconstruct the exceptional source answer with integer DP."""
+ primes = []
+ for number in range(2, 2014):
+ if all(number % prime for prime in primes if prime * prime <= number):
+ primes.append(number)
+ ways = [1] + [0] * 2013
+ for total in range(1, 2014):
+ ways[total] = sum(ways[total - prime] for prime in primes if prime <= total)
+ with localcontext() as context:
+ context.prec = 60
+ logarithm = Decimal(ways[2013]).ln() / Decimal(2).ln()
+ return str(ways[2013]), str(logarithm)
+
+
+def scalar_decimal(final):
+ """Read a bounded decimal or literal fraction, never an arbitrary expression."""
+ text = re.sub(r"\\[,;! ]|\s+", "", final.replace(r"\left", "").replace(r"\right", ""))
+ text = re.sub(r"^[a-zA-Z]=", "", text)
+ if len(text) > 160:
+ return None, None
+ number = r"[+-]?(?:\d{1,60}(?:\.\d{1,60})?|\.\d{1,60})"
+ if re.fullmatch(number, text):
+ return Decimal(text), text
+ fraction = re.fullmatch(r"\\(?:dfrac|frac)\{(" + number + r")\}\{(" + number + r")\}", text)
+ if fraction is None:
+ fraction = re.fullmatch(r"(" + number + r")/(" + number + r")", text)
+ if fraction is not None and Decimal(fraction[2]):
+ with localcontext() as context:
+ context.prec = 70
+ return Decimal(fraction[1]) / Decimal(fraction[2]), None
+ return None, None
+
+
+def special_result(correct, final, kind, expected, observed):
+ return {
+ "correct": bool(correct),
+ "status": "graded" if final else "no_final_answer",
+ "final_answer": final,
+ "gold": {"kind": kind, "value": str(expected)},
+ "prediction": {"kind": kind, "value": str(observed) if observed is not None else None},
+ }
+
+
+def grade_training(identity, original, response):
+ """Apply only this predeclared training-source review to an explicit answer."""
+ if identity not in RULES:
+ return grade_reviewed(identity, original, response)
+ expected_original, answer, kind = RULES[identity]
+ if original != expected_original:
+ raise ValueError("Training adjudication differs from the original source answer")
+ final = extract_final_answer(response)
+ if kind == "prime_approximation":
+ observed, _ = scalar_decimal(final)
+ target = Decimal(answer)
+ with localcontext() as context:
+ context.prec = 70
+ ratio = min(observed / target, target / observed) if observed is not None and observed > 0 else None
+ result = special_result(ratio is not None and ratio >= Decimal(123) / 125, final, kind, answer, observed)
+ result["minimum_source_points"] = 24
+ elif kind == "rounded_percentage":
+ number = final.removesuffix(r"\%").removesuffix("%")
+ observed, literal = scalar_decimal(number)
+ with localcontext() as context:
+ context.prec = 80
+ target = Decimal(3400) / 41
+ rounded = (
+ target.quantize(Decimal(1).scaleb(Decimal(literal).as_tuple().exponent), rounding=ROUND_HALF_UP)
+ if literal
+ else None
+ )
+ expected = answer if "%" in final else answer.removesuffix(r"\%")
+ exact = grade_answer(expected, response)
+ result = special_result(
+ exact["correct"] or observed == 80 or (literal is not None and observed == rounded),
+ final,
+ kind,
+ answer,
+ observed,
+ )
+ elif kind == "percentage":
+ expected = answer if "%" in final else answer.removesuffix(r"\%")
+ result = grade_answer(expected, response)
+ else:
+ if kind == "set_of_ordered_tuples" and r"\pm" in final:
+ # In this source, one plus/minus inside one point denotes two
+ # solutions. A repeated point is still checked by the base grader.
+ compact = final.replace(r"\left", "").replace(r"\right", "")
+ pairs = re.findall(r"\([^()]*\)", compact)
+ if len(pairs) == 3 and sum(r"\pm" in pair for pair in pairs) == 1 and compact.count(r"\pm") == 1:
+ expanded = [
+ value
+ for pair in pairs
+ for value in ([pair.replace(r"\pm", "+"), pair.replace(r"\pm", "-")] if r"\pm" in pair else [pair])
+ ]
+ remainder = compact
+ for pair in pairs:
+ remainder = remainder.replace(pair, "", 1)
+ if re.fullmatch(r"[\s,{}\\]*", remainder):
+ response = "\\boxed{\\{" + ",".join(expanded) + "\\}}"
+ result = grade_answer(answer, response)
+ return result | {"adjudication": {"protocol": PROTOCOL, "id": identity, "kind": kind}}
+
+
+def join_training(dataset, originals):
+ inputs, gold, sources = {"train": []}, {}, {}
+ for row in dataset["splits"]["train"]:
+ match = re.fullmatch(r"e2h-amc-(train|eval)-(0|[1-9]\d*)", row["id"])
+ if match is None or int(match[2]) >= len(originals[match[1]]):
+ raise ValueError("Unknown training source ID")
+ source = originals[match[1]][int(match[2])]
+ if (
+ row["request"] != render_request(source["tag"], source["problem"])
+ or row["contest"] != source["tag"]
+ or row["year"] != source["year"]
+ or row["id"] in gold
+ ):
+ raise ValueError("Training source/request join differs or is duplicated")
+ if not isinstance(source["answer"], str) or not source["answer"].strip():
+ raise ValueError("Training source has no answer")
+ inputs["train"].append({"id": row["id"], "request": row["request"]})
+ gold[row["id"]] = source["answer"]
+ sources[row["id"]] = source
+ return inputs, gold, sources
+
+
+def review_training(gold, sources):
+ preflight = preflight_gold(gold)
+ corrected, records = dict(gold), []
+ for identity, (original, answer, kind) in RULES.items():
+ if identity not in gold:
+ continue
+ if gold[identity] != original or not grade_training(identity, original, "\\boxed{" + answer + "}")["correct"]:
+ raise ValueError("Training source review is not bound or cannot grade its canonical answer")
+ corrected[identity] = answer
+ source = sources[identity]
+ records.append(
+ {
+ "id": identity,
+ "original": original,
+ "canonical": answer,
+ "kind": kind,
+ "source_answer_correction": identity in CORRECTIONS,
+ "reason": REASONS.get(
+ identity,
+ "The source explicitly asks for a percentage; allow an omitted percent sign, but do not treat a count as a percentage.",
+ ),
+ "problem_sha256": digest(source["problem"]),
+ "solution_sha256": digest(source["solution"]),
+ }
+ )
+ unresolved = {identity: value for identity, value in preflight["needs_review"].items() if identity not in RULES}
+ review = {
+ "protocol": PROTOCOL,
+ "source_hashes": SOURCE_HASHES,
+ "original_gold_sha256": digest(gold),
+ "canonical_gold_sha256": digest(corrected),
+ "approved_ids": sorted(set(gold) - set(unresolved)),
+ "adjudicated_ids": sorted(set(gold) & set(RULES)),
+ "unresolved_ids": sorted(unresolved),
+ "needs_review": unresolved,
+ "ready": not unresolved,
+ "n": len(gold),
+ "excluded": [],
+ "correction_records": records,
+ "label_meaning": "Automated solver-outcome grades against reviewed benchmark reference answers; not human difficulty labels or a no-reference teacher experiment",
+ "review_scope": "Parser-flagged and percentage-context training answers reviewed from original source only; this is not an independent solution audit of all512 problems",
+ "prime_composition_check": {"ordered_tuple_count": prime_compositions()[0], "log2": prime_compositions()[1]},
+ }
+ if prime_compositions()[1] != RULES["e2h-amc-eval-2451"][1]:
+ raise ValueError("Independent prime-composition reconstruction differs")
+ return corrected, preflight, review
+
+
+def prepare(reference, source, output):
+ if output.exists():
+ raise FileExistsError(output)
+ evidence.verify(reference)
+ dataset = read(reference / "data.json")
+ audit(dataset)
+ if dataset["source_revision"] != SOURCE_REVISION or len(dataset["splits"]["train"]) != 512:
+ raise ValueError("Need the pinned original512 training population")
+ download_source(source)
+ import pyarrow.parquet as pq
+
+ originals = {}
+ for name, expected in SOURCE_HASHES.items():
+ if file_hash(source / name) != expected:
+ raise ValueError("Training source hash differs")
+ originals[name.removesuffix(".parquet")] = pq.read_table(
+ source / name, columns=["problem", "answer", "tag", "year", "solution"]
+ ).to_pylist()
+ pool_inputs, pool_gold, sources = join_training(dataset, originals)
+ if (set(RULES) | set(EXCLUSIONS)) - set(pool_gold):
+ raise ValueError("The pinned training population is missing reviewed rows")
+ for identity, rule in EXCLUSIONS.items():
+ if pool_gold[identity] != rule["original_gold"]:
+ raise ValueError("Training exclusion does not match its original source")
+ inputs = {"train": [row for row in pool_inputs["train"] if row["id"] not in EXCLUSIONS]}
+ gold = {row["id"]: pool_gold[row["id"]] for row in inputs["train"]}
+ eligibility = {
+ "format": "sie-benefit-training-eligibility-v1",
+ "original_pool_ids": [row["id"] for row in pool_inputs["train"]],
+ "eligible_train_ids": [row["id"] for row in inputs["train"]],
+ "exclusions": [
+ {"id": identity, **rule, "problem_sha256": digest(sources[identity]["problem"])}
+ for identity, rule in EXCLUSIONS.items()
+ ],
+ "source_only": True,
+ "decided_before_training_outcomes": True,
+ "source_revision": SOURCE_REVISION,
+ "source_hashes": SOURCE_HASHES,
+ "dataset_sha256": digest(dataset),
+ "training_gold_review_source_sha256": file_hash(Path(__file__)),
+ }
+ corrected, preflight, review = review_training(gold, sources)
+ train, held_out = dataset["splits"]["train"], {}
+ for split in ("validation", "calibration", "test"):
+ rows = dataset["splits"][split]
+ overlaps = {
+ key: len({row[key] for row in train} & {row[key] for row in rows})
+ for key in ("id", "family", "paper_group")
+ }
+ overlaps["canonical_request"] = len(
+ {canonical(row["request"]) for row in train} & {canonical(row["request"]) for row in rows}
+ )
+ if any(overlaps.values()):
+ raise ValueError("Training overlap with a held-out population")
+ held_out[split] = {"n": len(rows), "overlaps": overlaps, "ids_sha256": digest([row["id"] for row in rows])}
+ feature_directory = reference / "development-embeddings"
+ matrix = vectors(train, feature_directory)
+ if matrix.shape != (512, 2560) or not np.isfinite(matrix).all():
+ raise ValueError("Frozen training embedding shape or values differ")
+ write(output / "inputs.json", inputs)
+ write(output / "inputs-original-pool.json", pool_inputs)
+ write(output / "gold-original-pool.json", pool_gold)
+ write(output / "eligibility.json", eligibility)
+ write(output / "gold-for-scoring-only.json", gold)
+ write(output / "gold-reviewed.json", corrected)
+ write(output / "settings.json", solver.SETTINGS)
+ write(output / "gold-preflight.json", preflight)
+ review["preflight_sha256"] = file_hash(output / "gold-preflight.json")
+ review["eligibility_sha256"] = file_hash(output / "eligibility.json")
+ review["original_pool_gold_sha256"] = digest(pool_gold)
+ review["excluded"] = eligibility["exclusions"]
+ write(output / "training-gold-adjudications.json", review)
+ rules_hash = file_hash(output / "training-gold-adjudications.json")
+ write(
+ output / "gold-review.json",
+ {
+ key: review[key]
+ for key in (
+ "preflight_sha256",
+ "original_gold_sha256",
+ "canonical_gold_sha256",
+ "approved_ids",
+ "adjudicated_ids",
+ "unresolved_ids",
+ "ready",
+ )
+ }
+ | {"training_gold_rules_sha256": rules_hash},
+ )
+ write(
+ output / "plan.json",
+ {
+ "format": "sie-benefit-training-source-v1",
+ "dataset_sha256": digest(dataset),
+ "inputs_sha256": digest(inputs),
+ "gold_sha256": digest(gold),
+ "settings_sha256": digest(solver.SETTINGS),
+ "counts": {"train": len(inputs["train"])},
+ "source_counts": {"train": 512},
+ "eligible_train_ids": eligibility["eligible_train_ids"],
+ "excluded_train_ids": list(EXCLUSIONS),
+ "eligibility_sha256": file_hash(output / "eligibility.json"),
+ "source": {"revision": SOURCE_REVISION, "hashes": SOURCE_HASHES},
+ "baseline_manifest_sha256": file_hash(reference / "manifest.json"),
+ "baseline_data_sha256": file_hash(reference / "data.json"),
+ "train_ids_sha256": digest([row["id"] for row in train]),
+ "held_out": held_out,
+ "features": {
+ "model": MODEL,
+ "revision": REVISION,
+ "instruction": INSTRUCTION,
+ "shape": list(matrix.shape),
+ "plan_sha256": file_hash(feature_directory / "plan.json"),
+ "vectors_sha256": file_hash(feature_directory / "vectors.jsonl.gz"),
+ },
+ "inference_reads_gold": False,
+ "selection": "Original training IDs only; no calibration/test solver outcomes for fitting",
+ "grader": grader_metadata(),
+ **source_hashes(),
+ "training_gold_rules_sha256": rules_hash,
+ "gold_review_sha256": file_hash(output / "gold-review.json"),
+ },
+ )
+ if not review["ready"]:
+ raise ValueError("Training gold review remains unresolved; inspect prepared artifacts before collection")
+ print(
+ json.dumps(
+ {
+ "prepared": str(output),
+ "original_pool": 512,
+ "eligible": len(gold),
+ "reviewed": len(review["adjudicated_ids"]),
+ "source_corrections": len(CORRECTIONS),
+ "plan_sha256": file_hash(output / "plan.json"),
+ }
+ )
+ )
+
+
+def checked_training(directory):
+ """Check input/protocol bindings without opening either answer-key file."""
+ plan, inputs, settings = solver.checked_inputs(directory)
+ if (
+ plan.get("format") != "sie-benefit-training-source-v1"
+ or plan.get("source_counts") != {"train": 512}
+ or plan.get("counts") != {"train": 511}
+ or set(inputs) != {"train"}
+ or len(inputs["train"]) != 511
+ ):
+ raise ValueError("Need the original512 pool with its declared511 eligible rows")
+ if (
+ any(set(row) != {"id", "request"} for row in inputs["train"])
+ or len({row["id"] for row in inputs["train"]}) != 511
+ ):
+ raise ValueError("Training inputs contain labels or duplicate identities")
+ for key, value in source_hashes().items():
+ if plan.get(key) != value:
+ raise ValueError("Training collection/grading source changed after preparation")
+ for key, name in (
+ ("training_gold_rules_sha256", "training-gold-adjudications.json"),
+ ("gold_review_sha256", "gold-review.json"),
+ ("eligibility_sha256", "eligibility.json"),
+ ):
+ if plan.get(key) != file_hash(directory / name):
+ raise ValueError("Training source review changed after preparation")
+ eligibility = read(directory / "eligibility.json")
+ if (
+ eligibility.get("format") != "sie-benefit-training-eligibility-v1"
+ or eligibility.get("source_only") is not True
+ or eligibility.get("decided_before_training_outcomes") is not True
+ or eligibility.get("eligible_train_ids") != [row["id"] for row in inputs["train"]]
+ or plan.get("eligible_train_ids") != eligibility["eligible_train_ids"]
+ or [row["id"] for row in eligibility["exclusions"]] != list(EXCLUSIONS)
+ or plan.get("excluded_train_ids") != list(EXCLUSIONS)
+ or eligibility["eligible_train_ids"]
+ != [identity for identity in eligibility["original_pool_ids"] if identity not in EXCLUSIONS]
+ or len(set(eligibility["original_pool_ids"])) != 512
+ ):
+ raise ValueError("Training eligibility differs from the predeclared source-only exclusion")
+ solver.check_review(directory, [row["id"] for row in inputs["train"]])
+ return plan, inputs, settings
+
+
+def grade(directory, output=None):
+ output = output or directory / "grades-train.json"
+ if output.exists():
+ raise FileExistsError(output)
+ plan, inputs, settings = checked_training(directory)
+ if plan["grader"] != grader_metadata():
+ raise ValueError("Training grading dependency or parser protocol changed")
+ gold = read(directory / "gold-for-scoring-only.json")
+ if digest(gold) != plan["gold_sha256"] or set(gold) != {row["id"] for row in inputs["train"]}:
+ raise ValueError("Training answer key changed after preparation")
+ collection = directory / "answers-train"
+ summary, collection_plan = read(collection / "collection.json"), read(collection / "collection-plan.json")
+ selected = collection_plan["ids"]
+ rows = inputs["train"][: len(selected)]
+ solver.collection_binding(collection_plan, directory, "train", rows, settings, None, collection_plan["self_hosted"])
+ if (
+ not selected
+ or summary["status"] != "complete"
+ or summary["completed"] != 2 * len(rows)
+ or summary["failed"] != 0
+ ):
+ raise ValueError("Training collection is incomplete or its count differs")
+ results = []
+ for row in rows:
+ outcome = {"id": row["id"]}
+ for arm in settings["arms"]:
+ path = collection / f"{row['id']}-{arm}.json"
+ answer = read(path)
+ solver.validate_record(answer, row, arm, settings)
+ if answer["status"] != "complete":
+ raise ValueError("Incomplete training answer")
+ measured = grade_training(row["id"], gold[row["id"]], answer["answer"])
+ if measured["status"] in ("gold_unparseable", "grading_error"):
+ raise ValueError("Training grader could not safely score a reviewed answer")
+ outcome.update(
+ {
+ f"{arm}_correct": measured["correct"],
+ f"{arm}_credits": answer["credits"],
+ f"{arm}_grade": measured,
+ f"{arm}_record_sha256": file_hash(path),
+ }
+ )
+ results.append(outcome)
+ write(
+ output,
+ {
+ "split": "train",
+ "plan_sha256": file_hash(directory / "plan.json"),
+ "collection_plan_sha256": file_hash(collection / "collection-plan.json"),
+ "collection_summary_sha256": file_hash(collection / "collection.json"),
+ "policies_frozen_sha256": None,
+ **source_hashes(),
+ "gold_review_sha256": file_hash(directory / "gold-review.json"),
+ "training_gold_rules_sha256": plan["training_gold_rules_sha256"],
+ "rows": results,
+ },
+ )
+ print(
+ json.dumps(
+ {
+ "graded": str(output),
+ "n": len(results),
+ "label_meaning": "Automated solver-outcome grades against reviewed benchmark reference answers",
+ }
+ )
+ )
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ sub = parser.add_subparsers(dest="command", required=True)
+ child = sub.add_parser("prepare")
+ child.add_argument("--evidence", type=Path, default=Path("evidence"))
+ child.add_argument("--source", type=Path, default=Path("run-output/solver-source"))
+ child.add_argument("--output", type=Path, required=True)
+ child = sub.add_parser("grade")
+ child.add_argument("--run", type=Path, required=True)
+ child.add_argument("--output", type=Path)
+ args = parser.parse_args()
+ if args.command == "prepare":
+ prepare(args.evidence, args.source, args.output)
+ else:
+ grade(args.run, args.output)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/benefit_expanded.py b/examples/train-a-router/benefit_expanded.py
new file mode 100644
index 000000000..6b66a7785
--- /dev/null
+++ b/examples/train-a-router/benefit_expanded.py
@@ -0,0 +1,430 @@
+"""Prepare a larger, source-reviewed training population and grade its answers.
+
+The original 511 completed pairs are reused byte for byte. Additional source
+exclusions are settled before collecting more answers; held-out solver answers
+are never opened by this module.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import shutil
+from pathlib import Path
+
+import benefit_data
+import solver
+from data import SOURCE_HASHES, SOURCE_REVISION, audit, canonical, digest, file_hash, render_request, write
+from serving import INSTRUCTION, MODEL, REVISION
+from solver_answers import grader_metadata
+
+FORMAT = "sie-expanded-benefit-training-source-v2"
+CODE_FILES = (
+ "benefit_expanded.py",
+ "benefit_expanded_collect.py",
+ "benefit_expanded_gold.py",
+ "benefit_expanded_gold_v2.py",
+ "benefit_data.py",
+ "benefit_collect.py",
+ "solver.py",
+ "solver_answers.py",
+ "solver_gold.py",
+ "data.py",
+ "serving.py",
+)
+
+
+def read(path):
+ return solver.read(Path(path))
+
+
+def source_hashes():
+ from benefit_expanded_gold_v2 import outcome_grader_metadata
+
+ here = Path(__file__).parent
+ return {
+ "grader_source_sha256": file_hash(here / "solver_answers.py"),
+ "gold_review_source_sha256": file_hash(here / "solver_gold.py"),
+ "training_gold_review_source_sha256": file_hash(here / "benefit_expanded_gold_v2.py"),
+ "collector_source_sha256": file_hash(here / "solver.py"),
+ "expanded_source_sha256": {name: file_hash(here / name) for name in CODE_FILES},
+ **outcome_grader_metadata(),
+ }
+
+
+def pool_rows(reference, source, expansion):
+ import pyarrow.parquet as pq
+
+ dataset, receipt = read(reference / "data.json"), read(expansion / "training-expansion-audit.json")
+ audit(dataset)
+ if (
+ dataset["source_revision"] != SOURCE_REVISION
+ or receipt["original_dataset_sha256"] != digest(dataset)
+ or receipt["source_revision"] != SOURCE_REVISION
+ or receipt["source_hashes"] != SOURCE_HASHES
+ or receipt["training_rows"] != 2581
+ or receipt["original_prefix_rows"] != 512
+ or file_hash(expansion / "train-expanded.jsonl") != receipt["train_expanded_jsonl_sha256"]
+ ):
+ raise ValueError("Expansion differs from the pinned source and original training prefix")
+ originals = {}
+ for name, expected in SOURCE_HASHES.items():
+ if file_hash(source / name) != expected:
+ raise ValueError("Pinned source hash differs")
+ originals[name.removesuffix(".parquet")] = pq.read_table(source / name).to_pylist()
+ rows = []
+ for identity, family in zip(receipt["ids"], receipt["families"], strict=True):
+ split, index = identity.removeprefix("e2h-amc-").rsplit("-", 1)
+ original = originals[split][int(index)]
+ rows.append(
+ {
+ "id": identity,
+ "request": render_request(original["tag"], original["problem"]),
+ "label": int(original["rating"] >= dataset["label_thresholds"][original["tag"]]),
+ "contest": original["tag"],
+ "year": original["year"],
+ "family": family,
+ "paper_group": f"{original['contest']}:{original['year']}",
+ }
+ )
+ if digest(rows) != receipt["expanded_rows_sha256"] or rows[:512] != dataset["splits"]["train"]:
+ raise ValueError("Reconstructed expansion differs from its source audit")
+ audit(dict(dataset, splits=dict(dataset["splits"], train=rows)))
+ return dataset, rows, originals, receipt
+
+
+def reviewed_exclusions(gold, sources, original_ids, leakage, mandatory):
+ """Combine frozen source judgments and deduplicate connected components."""
+ if (
+ leakage.get("format") != "sie-expanded-source-leakage-review-v1"
+ or leakage.get("source_only") is not True
+ or leakage.get("solver_outcomes_read_for_decisions") is not False
+ or leakage.get("decided_before_new_training_outcomes") is not True
+ or leakage.get("reused_training_outcomes_already_available") is not True
+ ):
+ raise ValueError("Leakage review must acknowledge existing outcomes and use source inputs only")
+ excluded = {}
+ for kind, rules in (
+ ("original_source_context", benefit_data.EXCLUSIONS),
+ ("expanded_source_context", mandatory),
+ ("cross_split_family", leakage["exclusions"]),
+ ):
+ for identity, rule in rules.items():
+ if identity not in gold or identity in excluded or not rule.get("reason"):
+ raise ValueError("Invalid or repeated source-only exclusion")
+ for key, value in (
+ ("original_gold", gold[identity]),
+ ("problem_sha256", digest(sources[identity]["problem"])),
+ ("solution_sha256", digest(sources[identity]["solution"])),
+ ):
+ if key in rule and rule[key] != value:
+ raise ValueError("Exclusion differs from its pinned source")
+ excluded[identity] = {"id": identity, "kind": kind, **rule}
+ for review in leakage["candidate_reviews"]:
+ for identity, expected in review["source_problem_sha256"].items():
+ if identity in sources and hashlib.sha256(sources[identity]["problem"].encode()).hexdigest() != expected:
+ raise ValueError("Leakage review source problem differs")
+ parent = {identity: identity for identity in gold}
+
+ def root(identity):
+ while parent[identity] != identity:
+ identity = parent[identity]
+ return identity
+
+ for group in leakage["duplicate_groups"]:
+ if len(group) < 2 or len(set(group)) != len(group) or any(identity not in gold for identity in group):
+ raise ValueError("Duplicate review must identify distinct source training rows")
+ for identity in group[1:]:
+ parent[root(identity)] = root(group[0])
+ components = {}
+ for identity in gold:
+ components.setdefault(root(identity), []).append(identity)
+ duplicate_decisions = []
+ for component in components.values():
+ if len(component) < 2:
+ continue
+ available = [identity for identity in component if identity not in excluded]
+ if not available:
+ continue
+ candidates = [identity for identity in available if identity in original_ids] or available
+ retained = min(candidates, key=lambda identity: digest(["expanded-source-duplicate-v1", identity]))
+ duplicate_decisions.append({"ids": component, "retained_id": retained})
+ for identity in available:
+ if identity != retained:
+ excluded[identity] = {
+ "id": identity,
+ "kind": "within_training_duplicate",
+ "reason": f"Source-reviewed duplicate of retained training request {retained}",
+ "retained_id": retained,
+ }
+ return excluded, duplicate_decisions
+
+
+def prepare(reference, source, expansion, prior, leakage_path, output, population="expanded"):
+ from benefit_expanded_gold_v2 import EXCLUSIONS, review_expanded
+
+ reference, source, expansion, prior, output = map(Path, (reference, source, expansion, prior, output))
+ if output.exists():
+ raise FileExistsError(output)
+ dataset, pool, originals, expansion_audit = pool_rows(reference, source, expansion)
+ if population not in ("original", "expanded"):
+ raise ValueError("Choose the cleaned original or expanded population")
+ prior_plan, prior_inputs, settings = benefit_data.checked_training(prior)
+ old_ids = [row["id"] for row in prior_inputs["train"]]
+ pool_inputs, pool_gold, sources = benefit_data.join_training(dict(dataset, splits={"train": pool}), originals)
+ leakage = read(leakage_path)
+ excluded, duplicates = reviewed_exclusions(
+ pool_gold, sources, {row["id"] for row in pool[:512]}, leakage, EXCLUSIONS
+ )
+ if population == "original":
+ pool = pool[:512]
+ pool_ids = {row["id"] for row in pool}
+ excluded = {identity: rule for identity, rule in excluded.items() if identity in pool_ids}
+ inputs = {"train": [row for row in pool_inputs["train"] if row["id"] in pool_ids and row["id"] not in excluded]}
+ eligible = [row["id"] for row in inputs["train"]]
+ reuse_ids = [identity for identity in old_ids if identity not in excluded]
+ if eligible[: len(reuse_ids)] != reuse_ids or prior_plan["dataset_sha256"] != digest(dataset):
+ raise ValueError("Expansion must retain the original eligible prefix")
+ gold = {identity: pool_gold[identity] for identity in eligible}
+ corrected, preflight, review = review_expanded(gold, sources)
+ if not review["ready"]:
+ raise ValueError("Expanded source review remains unresolved")
+ prior_collection = prior / "answers-train"
+ old_summary, old_binding = (
+ read(prior_collection / "collection.json"),
+ read(prior_collection / "collection-plan.json"),
+ )
+ solver.collection_binding(old_binding, prior, "train", prior_inputs["train"], settings, None, True)
+ if old_summary["status"] != "complete" or old_summary["completed"] != 2 * len(old_ids):
+ raise ValueError("Original paired collection is incomplete")
+ reused_hashes = {}
+ for row in prior_inputs["train"]:
+ if row["id"] not in reuse_ids:
+ continue
+ for arm in settings["arms"]:
+ name = f"{row['id']}-{arm}.json"
+ record = read(prior_collection / name)
+ solver.validate_record(record, row, arm, settings)
+ if record["status"] != "complete":
+ raise ValueError("Only completed original answers can be reused")
+ reused_hashes[name] = file_hash(prior_collection / name)
+ eligibility = {
+ "format": "sie-benefit-training-eligibility-v1",
+ "original_pool_ids": [row["id"] for row in pool],
+ "eligible_train_ids": eligible,
+ "exclusions": [
+ {
+ "id": identity,
+ "kind": record["kind"],
+ "reason": record["reason"],
+ "problem_sha256": digest(sources[identity]["problem"]),
+ }
+ for identity, record in excluded.items()
+ ],
+ "source_only": True,
+ "decided_before_new_training_outcomes": True,
+ "reused_training_outcomes_already_available": True,
+ "timing": "Historical outcomes already exist. This revised source-only audit precedes the new controlled fit and additional acquisition; no removals use model outcomes.",
+ "leakage_review_sha256": file_hash(leakage_path),
+ "duplicate_decisions": duplicates,
+ "source_revision": SOURCE_REVISION,
+ "source_hashes": SOURCE_HASHES,
+ "dataset_sha256": digest(dataset),
+ "training_gold_review_source_sha256": source_hashes()["training_gold_review_source_sha256"],
+ }
+ for name, value in (
+ ("training-rows.json", pool),
+ ("inputs.json", inputs),
+ ("eligibility.json", eligibility),
+ ("expansion-audit.json", expansion_audit),
+ ("gold-for-scoring-only.json", gold),
+ ("gold-reviewed.json", corrected),
+ ("settings.json", settings),
+ ("gold-preflight.json", preflight),
+ ):
+ write(output / name, value)
+ shutil.copyfile(leakage_path, output / "source-leakage-review.json")
+ review["preflight_sha256"] = file_hash(output / "gold-preflight.json")
+ review["eligibility_sha256"] = file_hash(output / "eligibility.json")
+ write(output / "training-gold-adjudications.json", review)
+ review_gate = {
+ key: review[key]
+ for key in ("preflight_sha256", "original_gold_sha256", "canonical_gold_sha256", "approved_ids", "ready")
+ }
+ review_gate.update(
+ unresolved_ids=review.get("unresolved_ids", []),
+ training_gold_rules_sha256=file_hash(output / "training-gold-adjudications.json"),
+ )
+ write(output / "gold-review.json", review_gate)
+ reuse = {
+ "ids": reuse_ids,
+ "records_sha256": reused_hashes,
+ "plan_sha256": file_hash(prior / "plan.json"),
+ "collection_plan_sha256": file_hash(prior_collection / "collection-plan.json"),
+ "collection_sha256": file_hash(prior_collection / "collection.json"),
+ "same_completed_record_bytes": True,
+ "historical_ids_excluded_by_new_source_audit": [identity for identity in old_ids if identity in excluded],
+ }
+ write(output / "reuse.json", reuse)
+ overlaps = {}
+ for split in ("validation", "calibration", "test"):
+ held = dataset["splits"][split]
+ overlaps[split] = {
+ key: len({row[key] for row in pool} & {row[key] for row in held}) for key in ("id", "family", "paper_group")
+ }
+ overlaps[split]["canonical_request"] = len(
+ {canonical(row["request"]) for row in pool} & {canonical(row["request"]) for row in held}
+ )
+ if any(overlaps[split].values()):
+ raise ValueError("Expansion overlaps held-out source inputs")
+ plan = {
+ "format": FORMAT,
+ "population": population,
+ "dataset_sha256": digest(dataset),
+ "inputs_sha256": digest(inputs),
+ "gold_sha256": digest(gold),
+ "settings_sha256": digest(settings),
+ "counts": {"train": len(eligible)},
+ "source_counts": {"train": len(pool)},
+ "eligible_train_ids": eligible,
+ "excluded_train_ids": list(excluded),
+ "new_train_ids": [identity for identity in eligible if identity not in set(reuse_ids)],
+ "eligibility_sha256": file_hash(output / "eligibility.json"),
+ "leakage_review_sha256": file_hash(output / "source-leakage-review.json"),
+ "training_rows_sha256": file_hash(output / "training-rows.json"),
+ "reuse_sha256": file_hash(output / "reuse.json"),
+ "expansion_audit_sha256": file_hash(output / "expansion-audit.json"),
+ "source": {"revision": SOURCE_REVISION, "hashes": SOURCE_HASHES},
+ "source_revision": SOURCE_REVISION,
+ "source_hashes": SOURCE_HASHES,
+ "held_out_source_overlap": overlaps,
+ "paper_groups": len({row["paper_group"] for row in pool}),
+ "features": {"model": MODEL, "revision": REVISION, "instruction": INSTRUCTION, "dimensions": 2560},
+ "inference_reads_gold": False,
+ "grader": grader_metadata(),
+ **source_hashes(),
+ "training_gold_rules_sha256": file_hash(output / "training-gold-adjudications.json"),
+ "gold_review_sha256": file_hash(output / "gold-review.json"),
+ }
+ write(output / "plan.json", plan)
+ return plan
+
+
+def checked_training(directory):
+ directory = Path(directory)
+ plan, inputs, settings = solver.checked_inputs(directory)
+ ids = [row["id"] for row in inputs.get("train", [])]
+ if (
+ plan.get("format") != FORMAT
+ or set(inputs) != {"train"}
+ or any(set(row) != {"id", "request"} for row in inputs["train"])
+ or len(ids) != len(set(ids))
+ or plan.get("counts") != {"train": len(ids)}
+ or plan.get("eligible_train_ids") != ids
+ or any(plan.get(key) != value for key, value in source_hashes().items())
+ ):
+ raise ValueError("Expanded inputs, population or frozen source differs")
+ for key, name in (
+ ("eligibility_sha256", "eligibility.json"),
+ ("leakage_review_sha256", "source-leakage-review.json"),
+ ("reuse_sha256", "reuse.json"),
+ ("training_rows_sha256", "training-rows.json"),
+ ("expansion_audit_sha256", "expansion-audit.json"),
+ ("training_gold_rules_sha256", "training-gold-adjudications.json"),
+ ("gold_review_sha256", "gold-review.json"),
+ ):
+ if plan.get(key) != file_hash(directory / name):
+ raise ValueError("Expanded population or source review evidence changed")
+ eligibility, reuse = read(directory / "eligibility.json"), read(directory / "reuse.json")
+ excluded = [row["id"] for row in eligibility["exclusions"]]
+ if (
+ eligibility.get("source_only") is not True
+ or eligibility.get("decided_before_new_training_outcomes") is not True
+ or eligibility.get("reused_training_outcomes_already_available") is not True
+ or eligibility.get("leakage_review_sha256") != plan["leakage_review_sha256"]
+ or eligibility["eligible_train_ids"] != ids
+ or excluded != plan["excluded_train_ids"]
+ or ids != [identity for identity in eligibility["original_pool_ids"] if identity not in set(excluded)]
+ or len(eligibility["original_pool_ids"]) != plan["source_counts"]["train"]
+ or len(set(eligibility["original_pool_ids"])) != plan["source_counts"]["train"]
+ or ids[: len(reuse["ids"])] != reuse["ids"]
+ or plan["new_train_ids"] != [identity for identity in ids if identity not in set(reuse["ids"])]
+ ):
+ raise ValueError("Expanded eligibility or reused prefix differs")
+ solver.check_review(directory, ids)
+ return plan, inputs, settings
+
+
+def grade(directory, output=None):
+ from benefit_expanded_gold_v2 import grade_expanded
+
+ directory = Path(directory)
+ output = Path(output) if output else directory / "grades-train.json"
+ if output.exists():
+ raise FileExistsError(output)
+ plan, inputs, settings = checked_training(directory)
+ gold = read(directory / "gold-for-scoring-only.json")
+ if digest(gold) != plan["gold_sha256"] or plan["grader"] != grader_metadata():
+ raise ValueError("Expanded reference answers or grading dependency changed")
+ collection = directory / "answers-train"
+ summary, binding = read(collection / "collection.json"), read(collection / "collection-plan.json")
+ solver.collection_binding(binding, directory, "train", inputs["train"], settings, None, True)
+ if summary["status"] != "complete" or summary["completed"] != 2 * len(inputs["train"]) or summary["failed"]:
+ raise ValueError("Expanded paired collection is incomplete")
+ results = []
+ for row in inputs["train"]:
+ outcome = {"id": row["id"]}
+ for arm in settings["arms"]:
+ path = collection / f"{row['id']}-{arm}.json"
+ record = read(path)
+ solver.validate_record(record, row, arm, settings)
+ if record["status"] != "complete":
+ raise ValueError("Incomplete expanded training answer")
+ measured = grade_expanded(row["id"], gold[row["id"]], record["answer"])
+ if measured["status"] in ("gold_unparseable", "grading_error"):
+ raise ValueError("Source-reviewed training answer could not be graded")
+ outcome.update(
+ {
+ f"{arm}_correct": measured["correct"],
+ f"{arm}_credits": record["credits"],
+ f"{arm}_grade": measured,
+ f"{arm}_record_sha256": file_hash(path),
+ }
+ )
+ results.append(outcome)
+ result = {
+ "split": "train",
+ "plan_sha256": file_hash(directory / "plan.json"),
+ "collection_plan_sha256": file_hash(collection / "collection-plan.json"),
+ "collection_summary_sha256": file_hash(collection / "collection.json"),
+ "policies_frozen_sha256": None,
+ **source_hashes(),
+ "gold_review_sha256": plan["gold_review_sha256"],
+ "training_gold_rules_sha256": plan["training_gold_rules_sha256"],
+ "rows": results,
+ }
+ write(output, result)
+ return result
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ commands = parser.add_subparsers(dest="command", required=True)
+ prepare_parser = commands.add_parser("prepare")
+ for name in ("reference", "source", "expansion", "prior", "leakage-review", "output"):
+ prepare_parser.add_argument(f"--{name}", type=Path, required=True)
+ prepare_parser.add_argument("--population", choices=("original", "expanded"), default="expanded")
+ grade_parser = commands.add_parser("grade")
+ grade_parser.add_argument("--directory", type=Path, required=True)
+ grade_parser.add_argument("--output", type=Path)
+ args = parser.parse_args()
+ if args.command == "prepare":
+ prepare(
+ args.reference, args.source, args.expansion, args.prior, args.leakage_review, args.output, args.population
+ )
+ else:
+ grade(args.directory, args.output)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/benefit_expanded_collect.py b/examples/train-a-router/benefit_expanded_collect.py
new file mode 100644
index 000000000..8d20dc0dc
--- /dev/null
+++ b/examples/train-a-router/benefit_expanded_collect.py
@@ -0,0 +1,413 @@
+"""Collect only new training pairs, with optional disjoint strong-model jobs.
+
+Prepare one immutable job plan, run each job's initial and final phases through
+the same SIE SDK, then gather with the original completed records. Each physical
+call has an intent and receipt. Resume never repeats a submitted call.
+"""
+
+from __future__ import annotations
+
+import argparse
+import math
+import os
+import shutil
+import time
+from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
+from pathlib import Path
+
+import benefit_collect as stages
+import solver
+from benefit_expanded import checked_training
+from data import digest, file_hash, write
+from serving import SingleSendClient, error_receipt
+
+FORMAT = "sie-expanded-benefit-jobs-v1"
+
+
+def prepare_jobs(directory, strong_shards=2):
+ directory = Path(directory)
+ plan, _, settings = checked_training(directory)
+ if type(strong_shards) is not int or not 1 <= strong_shards <= 2:
+ raise ValueError("Use one or two disjoint strong-model jobs")
+ ids = plan["new_train_ids"]
+ if ids and strong_shards > len(ids):
+ raise ValueError("Need at least one new row per job")
+ jobs = {"cheap": {"arm": "cheap", "ids": ids}} if ids else {}
+ if ids:
+ jobs.update(
+ {f"strong-{index}": {"arm": "strong", "ids": ids[index::strong_shards]} for index in range(strong_shards)}
+ )
+ result = {
+ "format": FORMAT,
+ "plan_sha256": file_hash(directory / "plan.json"),
+ "settings_sha256": digest(settings),
+ "reuse_sha256": plan["reuse_sha256"],
+ "jobs": jobs,
+ "scheduling": "Source-order round-robin shards; initial calls before conditional finalizers",
+ "max_generation_attempts": 1,
+ "collector_sha256": file_hash(Path(__file__)),
+ }
+ path = directory / "collection-jobs.json"
+ if path.exists():
+ raise FileExistsError(path)
+ write(path, result)
+ return result
+
+
+def checked_jobs(directory):
+ plan, inputs, settings = checked_training(directory)
+ jobs = solver.read(directory / "collection-jobs.json")
+ if (
+ jobs.get("format") != FORMAT
+ or jobs.get("plan_sha256") != file_hash(directory / "plan.json")
+ or jobs.get("settings_sha256") != digest(settings)
+ or jobs.get("reuse_sha256") != plan["reuse_sha256"]
+ or jobs.get("collector_sha256") != file_hash(Path(__file__))
+ or jobs.get("max_generation_attempts") != 1
+ ):
+ raise ValueError("Collection jobs differ from the frozen source and protocol")
+ selected = jobs["jobs"]
+ allowed = ({"cheap", "strong-0"}, {"cheap", "strong-0", "strong-1"}) if plan["new_train_ids"] else (set(),)
+ if set(selected) not in allowed:
+ raise ValueError("Unexpected collection job names")
+ for arm in settings["arms"]:
+ ids = [identity for job in selected.values() if job["arm"] == arm for identity in job["ids"]]
+ if len(ids) != len(set(ids)) or set(ids) != set(plan["new_train_ids"]):
+ raise ValueError("Each arm must cover each new request exactly once")
+ by_id = {row["id"]: row for row in inputs["train"]}
+ return jobs, {name: [by_id[identity] for identity in job["ids"]] for name, job in selected.items()}, settings
+
+
+def collect(
+ directory,
+ job,
+ phase,
+ *,
+ workers=4,
+ limit=None,
+ max_seconds=600,
+ max_credits=None,
+ self_hosted=False,
+ resume=False,
+ runtime_receipt=None,
+):
+ directory = Path(directory)
+ jobs, populations, settings = checked_jobs(directory)
+ if job not in populations or phase not in stages.PHASES:
+ raise ValueError("Choose a prepared job and initial/final phase")
+ if not 1 <= workers <= 16 or not math.isfinite(max_seconds) or max_seconds <= 0:
+ raise ValueError("Use 1..16 workers and a positive finite time limit")
+ if limit is not None and (type(limit) is not int or limit <= 0):
+ raise ValueError("Pilot limit must be a positive integer")
+ if not self_hosted and (max_credits is None or not solver.valid_credit(max_credits) or max_credits <= 0):
+ raise ValueError("Cloud collection needs a positive observed-credit ceiling")
+ arm, rows = jobs["jobs"][job]["arm"], populations[job]
+ output = directory / "collection-jobs" / job
+ binding = {
+ "job": job,
+ "arm": arm,
+ "ids": [row["id"] for row in rows],
+ "jobs_sha256": file_hash(directory / "collection-jobs.json"),
+ "self_hosted": self_hosted,
+ "endpoint_sha256": digest(os.environ.get("SIE_BASE_URL", "https://api.superlinked.com")),
+ "runtime_receipt_sha256": file_hash(runtime_receipt) if runtime_receipt else None,
+ }
+ plan_path = output / "job-plan.json"
+ if plan_path.exists():
+ if not resume or solver.read(plan_path) != binding:
+ raise ValueError("Resume needs an identical job, endpoint and runtime receipt")
+ else:
+ if resume:
+ raise ValueError("No existing job to resume")
+ write(plan_path, binding)
+ if runtime_receipt:
+ shutil.copyfile(runtime_receipt, output / "runtime-receipt.json")
+ todo = []
+ for row in rows[:limit] if limit else rows:
+ first = None
+ if phase == "final":
+ _, _, initial = stages.history(
+ output, row, arm, settings, "initial", stages.messages_for(row, arm, settings)
+ )
+ if initial is None:
+ raise ValueError("Final phase requires a successful initial receipt for every selected request")
+ first = initial[1]["call"]
+ if not stages.needs_final(first):
+ stages.materialize(output, row, arm, settings, self_hosted)
+ continue
+ messages = stages.messages_for(row, arm, settings, first)
+ previous, interrupted, success = stages.history(output, row, arm, settings, phase, messages)
+ if success:
+ stages.materialize(output, row, arm, settings, self_hosted)
+ continue
+ if previous or interrupted:
+ raise ValueError("A submitted call has no successful receipt; no retries are permitted")
+ todo.append((row, messages))
+ sessions = output / "sessions"
+ session = sessions / f"session-{len(list(sessions.glob('session-*'))) + 1:04d}"
+ write(
+ session / "plan.json",
+ {
+ "job_plan_sha256": file_hash(plan_path),
+ "phase": phase,
+ "workers": workers,
+ "pending_ids": [row["id"] for row, _ in todo],
+ "max_seconds": max_seconds,
+ "max_credits": max_credits,
+ "limit": limit,
+ "max_generation_attempts": 1,
+ "budget_scope": "New observed charges; in-flight calls drain; external watchdog bounds infrastructure",
+ },
+ )
+ started, sent, failed, observed, fatal = time.monotonic(), 0, 0, 0.0, None
+ try:
+ if todo:
+ with SingleSendClient(timeout_s=60) as client:
+ model = settings["arms"][arm]
+ metadata = client.get_model(model["model"])
+ if (
+ metadata.get("revision") != model["revision"]
+ or metadata.get("name", metadata.get("id")) != model["model"]
+ ):
+ raise ValueError("Endpoint model identity or HF revision differs")
+ write(session / "model.json", {key: metadata.get(key) for key in ("name", "revision", "profiles")})
+ tasks = iter(todo)
+ with ThreadPoolExecutor(max_workers=workers) as pool:
+ pending = {}
+
+ def schedule():
+ nonlocal sent
+ if (
+ failed
+ or time.monotonic() - started >= max_seconds
+ or (max_credits is not None and observed >= max_credits)
+ ):
+ return
+ task = next(tasks, None)
+ if task is None:
+ return
+ row, messages = task
+ target = output / "attempts" / row["id"] / f"{phase}-0001.json"
+ write(
+ target.with_name(target.stem + ".intent.json"),
+ {
+ "id": row["id"],
+ "arm": arm,
+ "phase": phase,
+ "job": job,
+ "session": session.name,
+ "request_sha256": digest(row["request"]),
+ "settings_sha256": digest(settings),
+ "messages_sha256": digest(messages),
+ },
+ )
+ future = pool.submit(stages.stage_call, row, arm, settings, phase, messages, self_hosted)
+ pending[future] = row, messages, target
+ sent += 1
+
+ for _ in range(workers):
+ schedule()
+ while pending:
+ done, _ = wait(pending, timeout=10, return_when=FIRST_COMPLETED)
+ for future in done:
+ row, messages, target = pending.pop(future)
+ result = future.result()
+ stages.checked_stage(result, row, arm, settings, phase, messages)
+ write(target, result)
+ observed += result["observed_credits"]
+ failed += result["status"] != "complete"
+ stages.materialize(output, row, arm, settings, self_hosted)
+ print(f"{job}/{phase}: {sent - len(pending)}/{len(todo)} returned, {failed} failed", flush=True)
+ for _ in done:
+ schedule()
+ except Exception as error: # noqa: BLE001 -- retain all intent and response evidence
+ fatal = error_receipt(error)
+ summary = {
+ "status": "complete" if sent == len(todo) and not failed and fatal is None else "incomplete",
+ "phase": phase,
+ "sent": sent,
+ "failed": failed,
+ "observed_credits": observed,
+ "elapsed_seconds": time.monotonic() - started,
+ "fatal_error": fatal,
+ }
+ write(session / "collection.json", summary)
+ if summary["status"] != "complete":
+ raise RuntimeError("Collection phase incomplete; preserve receipts and inspect before continuing")
+ return summary
+
+
+def copy_exact(source, destination):
+ if destination.exists():
+ if file_hash(source) != file_hash(destination):
+ raise ValueError("Archived evidence bytes differ")
+ else:
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copyfile(source, destination)
+
+
+def gather(directory, prior, job_directories=None):
+ directory, prior = Path(directory), Path(prior)
+ plan, inputs, settings = checked_training(directory)
+ jobs, populations, _ = checked_jobs(directory)
+ locations = (
+ {name: directory / "collection-jobs" / name for name in jobs["jobs"]}
+ if job_directories is None
+ else {name: Path(path) for name, path in job_directories.items()}
+ )
+ if set(locations) != set(jobs["jobs"]):
+ raise ValueError("Gather needs exactly the prepared jobs")
+ reuse = solver.read(directory / "reuse.json")
+ old_collection = prior / "answers-train"
+ if any(
+ file_hash(path) != expected
+ for path, expected in (
+ (prior / "plan.json", reuse["plan_sha256"]),
+ (old_collection / "collection-plan.json", reuse["collection_plan_sha256"]),
+ (old_collection / "collection.json", reuse["collection_sha256"]),
+ )
+ ):
+ raise ValueError("Original collection binding changed")
+ output = directory / "answers-train"
+ reused = set(reuse["ids"])
+ for row in inputs["train"]:
+ if row["id"] in reused:
+ for arm in settings["arms"]:
+ name = f"{row['id']}-{arm}.json"
+ if file_hash(old_collection / name) != reuse["records_sha256"][name]:
+ raise ValueError("Original completed answer changed")
+ record = solver.read(old_collection / name)
+ solver.validate_record(record, row, arm, settings)
+ copy_exact(old_collection / name, output / name)
+ failures, unknown, observed, missing = 0, 0, 0.0, 0
+ receipts, bindings = {}, {}
+ for name, location in locations.items():
+ binding = solver.read(location / "job-plan.json")
+ expected = jobs["jobs"][name]
+ if (
+ binding["job"] != name
+ or binding["arm"] != expected["arm"]
+ or binding["ids"] != expected["ids"]
+ or binding["jobs_sha256"] != file_hash(directory / "collection-jobs.json")
+ or not binding["self_hosted"]
+ ):
+ raise ValueError("Job binding differs from prepared IDs, arm or hosting")
+ bindings[name] = binding
+ if (
+ binding["runtime_receipt_sha256"]
+ and file_hash(location / "runtime-receipt.json") != binding["runtime_receipt_sha256"]
+ ):
+ raise ValueError("Runtime receipt changed")
+ for row in populations[name]:
+ record = stages.materialize(location, row, expected["arm"], settings, True)
+ if record is None:
+ missing += 1
+ else:
+ copy_exact(
+ location / "records" / f"{row['id']}-{expected['arm']}.json",
+ output / f"{row['id']}-{expected['arm']}.json",
+ )
+ for path in location.rglob("*"):
+ if (
+ not path.is_file()
+ or path.suffix not in (".json", ".yaml")
+ or "records" in path.relative_to(location).parts
+ ):
+ continue
+ relative = Path(name) / path.relative_to(location)
+ copy_exact(path, output / "acquisition" / relative)
+ receipts[relative.as_posix()] = file_hash(path)
+ if "attempts" not in path.relative_to(location).parts:
+ continue
+ if path.name.endswith(".intent.json"):
+ unknown += not path.with_name(path.name.replace(".intent", "")).exists()
+ else:
+ attempt = solver.read(path)
+ observed += attempt["observed_credits"]
+ failures += attempt["status"] != "complete"
+ unknown += attempt.get("execution_unknown", True)
+ binding = {
+ "ids": [row["id"] for row in inputs["train"]],
+ "split": "train",
+ "plan_sha256": file_hash(directory / "plan.json"),
+ "settings_sha256": digest(settings),
+ "policies_frozen_sha256": None,
+ "self_hosted": True,
+ "reuse_sha256": plan["reuse_sha256"],
+ "jobs_sha256": file_hash(directory / "collection-jobs.json"),
+ "jobs": bindings,
+ }
+ target = output / "collection-plan.json"
+ if target.exists() and solver.read(target) != binding:
+ raise ValueError("Combined collection binding changed")
+ if not target.exists():
+ write(target, binding)
+ for name in ("plan.json", "collection-jobs.json", "reuse.json"):
+ copy_exact(directory / name, output / "acquisition" / name)
+ for name in ("collection-plan.json", "collection.json"):
+ copy_exact(old_collection / name, output / "acquisition" / "reused" / name)
+ for path in (old_collection / "acquisition").rglob("*"):
+ if path.is_file():
+ copy_exact(path, output / "acquisition" / "reused" / path.relative_to(old_collection / "acquisition"))
+ summary = {
+ "status": "complete" if not missing and not failures and not unknown else "incomplete",
+ "completed": 2 * len(inputs["train"]) - missing,
+ "failed": failures,
+ "missing": missing,
+ "reused_records": 2 * len(reused),
+ "new_completed_records": 2 * len(plan["new_train_ids"]) - missing,
+ "observed_credits": observed,
+ "unknown_execution_attempts": unknown,
+ "acquisition_cost_complete": False,
+ "infrastructure_cost_status": "unpriced",
+ "receipt_sha256": receipts,
+ }
+ solver.save_latest(output / "collection.json", summary)
+ if summary["status"] != "complete":
+ raise RuntimeError("Expanded collection is incomplete")
+ return summary
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ commands = parser.add_subparsers(dest="command", required=True)
+ prepare = commands.add_parser("prepare-jobs")
+ prepare.add_argument("--directory", type=Path, required=True)
+ prepare.add_argument("--strong-shards", type=int, default=2)
+ acquire = commands.add_parser("collect")
+ acquire.add_argument("--directory", type=Path, required=True)
+ acquire.add_argument("--job", required=True)
+ acquire.add_argument("--phase", choices=stages.PHASES, required=True)
+ acquire.add_argument("--workers", type=int, default=4)
+ acquire.add_argument("--limit", type=int)
+ acquire.add_argument("--max-seconds", type=float, required=True)
+ acquire.add_argument("--max-credits", type=float)
+ acquire.add_argument("--self-hosted", action="store_true")
+ acquire.add_argument("--resume", action="store_true")
+ acquire.add_argument("--runtime-receipt", type=Path)
+ combine = commands.add_parser("gather")
+ combine.add_argument("--directory", type=Path, required=True)
+ combine.add_argument("--prior", type=Path, required=True)
+ combine.add_argument("--job-directory", action="append", default=[], help="Prepared job name=downloaded directory")
+ args = parser.parse_args()
+ if args.command == "prepare-jobs":
+ prepare_jobs(args.directory, args.strong_shards)
+ elif args.command == "collect":
+ collect(
+ args.directory,
+ args.job,
+ args.phase,
+ workers=args.workers,
+ limit=args.limit,
+ max_seconds=args.max_seconds,
+ max_credits=args.max_credits,
+ self_hosted=args.self_hosted,
+ resume=args.resume,
+ runtime_receipt=args.runtime_receipt,
+ )
+ else:
+ gather(args.directory, args.prior, dict(value.split("=", 1) for value in args.job_directory) or None)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/benefit_expanded_gold.py b/examples/train-a-router/benefit_expanded_gold.py
new file mode 100644
index 000000000..8c2fc6490
--- /dev/null
+++ b/examples/train-a-router/benefit_expanded_gold.py
@@ -0,0 +1,528 @@
+"""Source-only answer review for the expanded paired-training population.
+
+The original 511-example grading protocol is delegated unchanged. New rules
+were decided from benchmark questions and solutions before collecting any new
+solver answers. Incomplete questions are excluded, never silently repaired.
+"""
+
+from __future__ import annotations
+
+import re
+from math import factorial
+from pathlib import Path
+
+import benefit_data
+from data import digest, file_hash
+from solver_answers import _parts, extract_final_answer, grade_answer
+
+PROTOCOL = "source-only-expanded-training-gold-v1"
+
+
+def _plain(value):
+ value = re.sub(r"\\(?:text|mathrm)\{([^{}]*)\}", r"\1", value)
+ value = re.sub(r"\\[,;! ]|~", " ", value)
+ return " ".join(value.casefold().split()).strip()
+
+
+def _clock_minutes(value):
+ value = re.sub(r"\b([ap])\s*\.\s*m\.?", r"\1m", _plain(value))
+ match = re.fullmatch(r"(\d{1,2})(?::(\d{1,2}))?\s*(am|pm)?", value)
+ if match is None:
+ return None
+ hour, minute, meridiem = int(match[1]), int(match[2] or 0), match[3]
+ if hour > 23 or minute > 59 or (meridiem and not 1 <= hour <= 12):
+ return None
+ if meridiem:
+ hour = hour % 12 + (12 if meridiem == "pm" else 0)
+ return 60 * hour + minute
+
+
+def _expand_plus_minus(value):
+ """Expand one explicit plus/minus per finite-list member, with a small cap."""
+ value = value.replace(r"\left", "").replace(r"\right", "").replace("±", r"\pm")
+ if value.startswith(r"\{") and value.endswith(r"\}"):
+ value = value[2:-2]
+ elif value.startswith("{") and value.endswith("}"):
+ value = value[1:-1]
+ try:
+ parts = _parts(value)
+ except ValueError:
+ return None
+ if len(parts) > 16 or any(part.count(r"\pm") > 1 for part in parts):
+ return None
+ expanded = [
+ item
+ for part in parts
+ for item in ([part.replace(r"\pm", "+"), part.replace(r"\pm", "-")] if r"\pm" in part else [part])
+ ]
+ return r"\{" + ",".join(expanded) + r"\}"
+
+
+def _special(correct, final, kind, expected, observed):
+ return {
+ "correct": bool(correct),
+ "status": "graded" if final else "no_final_answer",
+ "final_answer": final,
+ "gold": {"kind": kind, "value": str(expected)},
+ "prediction": {"kind": kind, "value": str(observed) if observed is not None else None},
+ }
+
+
+def grade_expanded(identity, original, response):
+ """Grade one eligible row, using only a source-bound explicit final answer."""
+ if identity in EXCLUSIONS or identity in benefit_data.EXCLUSIONS:
+ raise ValueError("Source-invalid training question must be excluded before collection")
+ if identity not in RULES:
+ return benefit_data.grade_training(identity, original, response)
+ expected_original, canonical, kind = RULES[identity]
+ if original != expected_original:
+ raise ValueError("Expanded source adjudication differs from the original answer")
+ final = extract_final_answer(response)
+ if kind == "clock_time":
+ observed = _clock_minutes(final) if final else None
+ result = _special(observed == 660, final, kind, "11:00 AM", observed)
+ elif kind == "division_operator":
+ observed = _plain(final)
+ accepted = {r"\div", "÷", "/", "division", "divide", "division operator"}
+ result = _special(observed in accepted, final, kind, "division", observed)
+ elif kind == "ordered_labels":
+ observed = _plain(final)
+ for name, letter in (("medium", "m"), ("large", "l"), ("small", "s")):
+ observed = re.sub(r"\b" + name + r"\b", letter, observed)
+ observed = observed.strip("()[]{}").replace(r"\to", "> ").replace("→", ">")
+ correct = observed == "mls" or re.fullmatch(r"m[\s,>]+l[\s,>]+s", observed) is not None
+ result = _special(correct, final, kind, "medium, large, small", observed)
+ elif kind == "weekday":
+ observed = _plain(final)
+ result = _special(
+ observed in {"friday", "day 5", "fifth day", "the fifth day"}, final, kind, "Friday", observed
+ )
+ else:
+ normalized = final
+ if kind == "percentage":
+ canonical = canonical if "%" in final else canonical.removesuffix(r"\%")
+ elif kind == "ratio" and final.count(":") == 1:
+ left, right = final.split(":")
+ if left.strip() and right.strip():
+ normalized = r"\frac{" + left + "}{" + right + "}"
+ elif kind in {"finite_set", "set_of_ordered_tuples"} and (r"\pm" in final or "±" in final):
+ normalized = _expand_plus_minus(final) or final
+ elif kind == "official_alternatives":
+ # The spaced source form15 !^2 is not parsed reliably by Math-Verify.
+ # Replace only this declared factorial, without evaluating arbitrary code.
+ normalized = re.sub(r"(? MAX_CHARS:
+ raise ValueError("Unsupported expression length")
+ text = text.strip()
+ for left, right in (("$$", "$$"), ("$", "$"), (r"\(", r"\)"), (r"\[", r"\]")):
+ if text.startswith(left) and text.endswith(right):
+ text = text[len(left) : -len(right)]
+ break
+ text = re.sub(r"^[A-Za-z]\s*=\s*", "", text)
+ for old, new in (
+ (r"\left", ""),
+ (r"\right", ""),
+ (r"\dfrac", r"\frac"),
+ (r"\tfrac", r"\frac"),
+ (r"\cdot", "*"),
+ (r"\times", "*"),
+ (r"\div", "/"),
+ (r"\%", "%"),
+ ("×", "*"),
+ ("÷", "/"),
+ ("−", "-"),
+ ("[", "("),
+ ("]", ")"),
+ ):
+ text = text.replace(old, new)
+ text = re.sub(r"\\(?:qquad|quad|,|;|!| )", " ", text)
+ # Preserve token boundaries. Joining digits can turn a mixed number or
+ # grouped integer into a different value (for example1 1/5 ->11/5).
+ # Likewise1\frac15 conventionally denotes a mixed number, not a product.
+ # These ambiguous forms retain the previous parser's declared behavior.
+ if re.search(r"\d\s+\d", text):
+ raise ValueError("Ambiguous separated numeric literals")
+ text = re.sub(r"\s+", "", text)
+ if re.search(r"\d\\frac", text):
+ raise ValueError("Ambiguous mixed-number adjacency")
+ self.tokens = _TOKEN.findall(text)
+ if "".join(self.tokens) != text or not self.tokens or len(self.tokens) > MAX_TOKENS:
+ raise ValueError("Unsupported numeric grammar")
+ self.index = 0
+ self.depth = 0
+
+ def peek(self):
+ return self.tokens[self.index] if self.index < len(self.tokens) else ""
+
+ def take(self):
+ token = self.peek()
+ self.index += 1
+ return token
+
+ @staticmethod
+ def bounded(value):
+ if max(abs(value.numerator).bit_length(), value.denominator.bit_length()) > MAX_BITS:
+ raise ValueError("Exact arithmetic size limit")
+ return value
+
+ def expression(self):
+ self.depth += 1
+ if self.depth > MAX_DEPTH:
+ raise ValueError("Expression nesting limit")
+ value = self.product()
+ while self.peek() in ("+", "-"):
+ operator = self.take()
+ other = self.product()
+ value = self.bounded(value + other if operator == "+" else value - other)
+ self.depth -= 1
+ return value
+
+ def product(self):
+ value = self.unary()
+ while self.peek() and self.peek() not in ("+", "-", ")", "}"):
+ operator = self.peek()
+ if operator in ("*", "/"):
+ self.take()
+ elif operator not in ("(", "{", r"\frac", r"\sqrt"):
+ raise ValueError("Ambiguous implicit multiplication")
+ other = self.unary()
+ value = self.bounded(value / other if operator == "/" else value * other)
+ return value
+
+ def unary(self):
+ self.depth += 1
+ if self.depth > MAX_DEPTH:
+ raise ValueError("Expression nesting limit")
+ if self.peek() in ("+", "-"):
+ sign = self.take()
+ value = self.unary()
+ self.depth -= 1
+ return -value if sign == "-" else value
+ value = self.atom()
+ if self.peek() == "^":
+ self.take()
+ exponent = self.unary()
+ if exponent.denominator != 1 or abs(exponent.numerator) > MAX_EXPONENT:
+ raise ValueError("Unsupported exponent")
+ exponent = exponent.numerator
+ if not value and exponent <= 0:
+ raise ValueError("Undefined zero power")
+ size = max(abs(value.numerator).bit_length(), value.denominator.bit_length())
+ if size * abs(exponent) > MAX_BITS:
+ raise ValueError("Exponent size limit")
+ value = self.bounded(value**exponent)
+ while self.peek() == "%":
+ self.take()
+ value = self.bounded(value / 100)
+ self.depth -= 1
+ return value
+
+ def argument(self):
+ # An unbraced LaTeX fraction argument is one digit, even in \frac34.
+ token = self.peek()
+ if re.fullmatch(r"\d+", token):
+ self.tokens[self.index] = token[1:]
+ if len(token) == 1:
+ self.index += 1
+ return Fraction(token[0])
+ return self.atom()
+
+ def atom(self):
+ token = self.take()
+ if token in ("(", "{"):
+ value = self.expression()
+ if self.take() != (")" if token == "(" else "}"):
+ raise ValueError("Unbalanced expression")
+ return value
+ if token == r"\frac":
+ numerator, denominator = self.argument(), self.argument()
+ return self.bounded(numerator / denominator)
+ if token == r"\sqrt":
+ value = self.argument()
+ if value < 0:
+ raise ValueError("Non-rational square root")
+ numerator, denominator = isqrt(value.numerator), isqrt(value.denominator)
+ if numerator**2 != value.numerator or denominator**2 != value.denominator:
+ raise ValueError("Non-rational square root")
+ return Fraction(numerator, denominator)
+ if re.fullmatch(r"\d+(?:\.\d*)?|\.\d+", token):
+ return self.bounded(Fraction(token))
+ raise ValueError("Expected a numeric atom")
+
+ def parse(self):
+ value = self.expression()
+ if self.index != len(self.tokens):
+ raise ValueError("Unconsumed expression")
+ return value
+
+
+def exact_rational(final):
+ """Return an exact Fraction, or None when the bounded grammar is unsupported."""
+ try:
+ return _RationalParser(final).parse()
+ except (ValueError, ZeroDivisionError, OverflowError, RecursionError):
+ return None
+
+
+def _numeric_summary(value):
+ # Hash the complete value while keeping large powers readable in reports.
+ numerator, denominator = str(value.numerator), str(value.denominator)
+ return {
+ "kind": "exact_rational",
+ "numerator": numerator if len(numerator) <= 120 else numerator[:60] + "…",
+ "denominator": denominator if len(denominator) <= 120 else denominator[:60] + "…",
+ "value_sha256": digest([numerator, denominator]),
+ }
+
+
+def exact_comparison(expected_final, observed_final):
+ """Compare full final expressions exactly; None explicitly requests fallback."""
+ expected, observed = exact_rational(expected_final), exact_rational(observed_final)
+ if expected is None or observed is None:
+ return None
+ return {
+ "correct": expected == observed,
+ "status": "graded",
+ "final_answer": observed_final,
+ "gold": _numeric_summary(expected),
+ "prediction": _numeric_summary(observed),
+ "exact_numeric_guard": "applied",
+ }
+
+
+def outcome_grader_metadata():
+ return {
+ "outcome_grader_source_sha256": file_hash(Path(__file__)),
+ "outcome_grader_protocol_sha256": digest(OUTCOME_PROTOCOL),
+ }
+
+
+def _ratio(value):
+ if value.count(":") == 1:
+ left, right = value.split(":")
+ return r"\frac{" + left + "}{" + right + "}"
+ return value
+
+
+def _game_count(value):
+ plain = re.sub(r"\\(?:text|mathrm)\{([^{}]*)\}", r"\1", value)
+ plain = re.sub(r"\\[,;! ]|~", " ", plain).strip()
+ match = re.fullmatch(r"(.+?)\s*(?:conference\s+)?games?", plain, flags=re.IGNORECASE)
+ return match[1] if match and exact_rational(match[1]) is not None else value
+
+
+def _guard(expected, final, fallback):
+ compared = exact_comparison(expected, final)
+ if compared is None:
+ return fallback | {"exact_numeric_guard": "unsupported; previous reviewed grader retained"}
+ return fallback | compared
+
+
+def grade_expanded(identity, original, response):
+ """Apply new source amendments, then exactness to eligible scalar answers."""
+ final = extract_final_answer(response)
+ rule = RULES.get(identity)
+ if rule:
+ expected_original, canonical, kind = rule
+ if original != expected_original:
+ raise ValueError("Amended source adjudication differs from original answer")
+ if not final:
+ return grade_answer(canonical, response) | {"exact_numeric_guard": "no final answer"}
+ normalized = _ratio(final) if kind == "ratio" else _game_count(final) if kind == "game_count" else final
+ result = _guard(canonical, normalized, grade_answer(canonical, r"\boxed{" + normalized + "}"))
+ if kind == "exact_rational" and exact_rational(normalized) is None:
+ result = result | {
+ "correct": False,
+ "status": "prediction_unparseable",
+ "exact_numeric_guard": "required; unsupported expression",
+ }
+ return result | {"final_answer": final, "adjudication": {"protocol": PROTOCOL, "id": identity, "kind": kind}}
+ result = previous.grade_expanded(identity, original, response)
+ canonical, kind = original, "scalar"
+ for rules in (previous.RULES, benefit_data.RULES, solver_gold.RULES):
+ if identity in rules:
+ _, canonical, kind = rules[identity]
+ break
+ if kind not in {"scalar", "source_correction", "notation", "ratio", "percentage"}:
+ return result | {"exact_numeric_guard": "not applicable to reviewed non-exact or non-scalar semantics"}
+ if kind == "percentage":
+ canonical = canonical if "%" in final else canonical.removesuffix(r"\%")
+ normalized = _ratio(final) if kind == "ratio" else final
+ return _guard(canonical, normalized, result) | {"final_answer": final}
+
+
+def grade_teacher_reference(question, teacher_final, solver_response):
+ """Question-only formatting; no IDs, benchmark references, or corrections.
+
+ teacher_final is the teacher's declared answer field. solver_response is the
+ solver's full response and must contain an extractable final answer.
+ """
+ final = extract_final_answer(solver_response)
+ expected = extract_final_answer(r"\boxed{" + teacher_final + "}")
+ observed = final
+ contexts = []
+ percentage = re.search(
+ r"\b(?:what|which)\s+(?:is\s+(?:the\s+)?)?percent(?:age)?\b|\b(?:find|determine|compute|give)\s+(?:the\s+)?percent(?:age)?\b",
+ question,
+ re.IGNORECASE,
+ )
+ ratio = re.search(r"\b(?:what\s+is|find|determine|compute|give)\s+(?:the\s+)?ratio\b", question, re.IGNORECASE)
+ games = re.search(r"\bhow\s+many(?:\s+\w+){0,5}\s+games\b", question, re.IGNORECASE)
+ if percentage:
+ expected = expected if "%" in expected else "(" + expected + r")\%"
+ observed = observed if "%" in observed else "(" + observed + r")\%"
+ contexts.append("explicit_percentage_request")
+ if ratio:
+ expected, observed = _ratio(expected), _ratio(observed)
+ contexts.append("explicit_ratio_request")
+ if games:
+ expected, observed = _game_count(expected), _game_count(observed)
+ contexts.append("explicit_game_count_request")
+ fallback = grade_answer(expected, r"\boxed{" + observed + "}" if final else solver_response)
+ result = _guard(expected, observed, fallback)
+ return result | {
+ "final_answer": final,
+ "reference_mode": "teacher_final_and_question_context_only",
+ "context_normalizations": contexts,
+ }
+
+
+def review_expanded(gold, sources):
+ """Keep old review receipts intact; emit an explicitly amended new review."""
+ corrected, preflight, old_review = previous.review_expanded(gold, sources)
+ records = list(old_review["correction_records"])
+ for identity, (original, canonical, kind) in RULES.items():
+ if identity not in gold:
+ continue
+ source = sources[identity]
+ if gold[identity] != original or source["answer"] != original:
+ raise ValueError("Amended source review differs from the pinned answer")
+ if not grade_expanded(identity, original, r"\boxed{" + canonical + "}")["correct"]:
+ raise ValueError("Amended canonical answer does not grade")
+ corrected[identity] = canonical
+ records.append(
+ {
+ "id": identity,
+ "original": original,
+ "canonical": canonical,
+ "kind": kind,
+ "source_answer_correction": kind == "source_correction",
+ "reason": REASONS[identity],
+ "problem_sha256": digest(source["problem"]),
+ "solution_sha256": digest(source["solution"]),
+ }
+ )
+ unresolved = {identity: value for identity, value in old_review["needs_review"].items() if identity not in RULES}
+ review = (
+ old_review
+ | {
+ "protocol": PROTOCOL,
+ "previous_review_protocol": previous.PROTOCOL,
+ "previous_review_source_sha256": file_hash(Path(previous.__file__)),
+ "canonical_gold_sha256": digest(corrected),
+ "approved_ids": sorted(set(gold) - set(unresolved)),
+ "adjudicated_ids": sorted(set(old_review["adjudicated_ids"]) | (set(gold) & set(RULES))),
+ "unresolved_ids": sorted(unresolved),
+ "needs_review": unresolved,
+ "ready": not unresolved,
+ "correction_records": records,
+ "expanded_gold_rules_sha256": digest(
+ {
+ "previous_rules": previous.RULES,
+ "rules": RULES,
+ "exclusions": EXCLUSIONS,
+ "outcome_protocol": OUTCOME_PROTOCOL,
+ }
+ ),
+ "expanded_gold_review_source_sha256": file_hash(Path(__file__)),
+ "review_scope": "Four amendments independently checked against source and exact arithmetic after a teacher audit on original training rows; earlier reports are preserved. No expanded solver outcomes were consulted. The generic rational exactness guard is not an independent solution audit of every question.",
+ }
+ | outcome_grader_metadata()
+ )
+ return corrected, preflight, review
diff --git a/examples/train-a-router/benefit_native.py b/examples/train-a-router/benefit_native.py
new file mode 100644
index 000000000..4bb492978
--- /dev/null
+++ b/examples/train-a-router/benefit_native.py
@@ -0,0 +1,726 @@
+"""Train native Laya on paired solver benefit, with a separate test-access gate.
+
+Preparation and reporting need only core dependencies. Training uses the pinned
+optional training environment on MPS or CUDA; no solver API is called here.
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib.metadata
+import json
+import os
+import random
+import shutil
+import tempfile
+import time
+from dataclasses import asdict
+from pathlib import Path
+
+import numpy as np
+
+import benefit
+import finetune
+import solver_policies
+import solver_report
+from data import digest, file_hash, write
+
+QUESTIONS = {
+ "route": {
+ "type": "choice",
+ "instructions": "Predict which solver is more likely to answer this mathematics problem correctly. Compare the cheaper Qwen3.5-4B solver with the stronger Qwen3.8-27B solver. Judge the expected benefit of the stronger solver; do not solve the problem.",
+ "criteria": {
+ "cheap": "The cheaper solver is at least as likely to produce the correct answer.",
+ "strong": "The stronger solver is more likely to produce the correct answer.",
+ },
+ }
+}
+PROTOCOL = {
+ "target": "Signed paired benefit y=strong_correct-cheap_correct; soft target p_strong=(1+y)/2, including ties=0.5",
+ "base": finetune.BASE,
+ "revision": finetune.REVISION,
+ "recipe": finetune.LORA_RECIPES["original"],
+ "seed": 89,
+ "selection_fold": 0,
+ "selection_epochs": 8,
+ "selection": "Minimum training-fold validation MSE of tanh((strong_logit-cheap_logit)/2) versus signed benefit; include epoch0; ties retain earlier epoch",
+ "refit": "Fresh base and seed89, all eligible training rows, selected epoch count; cosine schedule spans the selected refit duration; epoch0 skips training",
+ "score_order": ["cheap", "strong"],
+ "max_len": finetune.MAX_LEN,
+ "head_max_len": finetune.HEAD_LEN,
+ "export_dtype": "float32",
+ "max_merge_margin_error": 0.001,
+ "max_export_margin_error": 0.03,
+ "primary_max_quality_loss_pp": 1,
+ "secondary_max_quality_loss_pp": [0, 2, 5],
+ "runtime_profiles": {device: finetune.runtime_settings(device) for device in ("mps", "cuda")},
+ "eager_flags": {"TORCH_COMPILE_DISABLE": "1", "TORCHDYNAMO_DISABLE": "1"},
+ "versions": {"laya": "0.3.29", "torch": "2.9.1", "transformers": "4.57.6", "peft": "0.18.1"},
+ "test_boundary": "Freeze before solver test OUTCOME ACCESS; collection may already exist. Solver test answers never enter native training or scoring.",
+ "hosting_cost": "Unpriced native serving; report gross solver savings and break-even router cost only",
+}
+PLAN_FORMAT = "sie-solver-benefit-native-plan-v1"
+FROZEN_FORMAT = "sie-solver-benefit-native-frozen-v1"
+ACCESS_FORMAT = "sie-solver-benefit-native-access-v1"
+
+
+def code_hashes():
+ return benefit.code_hashes() | {
+ name: file_hash(Path(__file__).with_name(name)) for name in ("benefit_native.py", "finetune.py")
+ }
+
+
+def _unopened(head_run):
+ if (Path(head_run) / "test-access.json").exists():
+ raise ValueError("Shared solver test access is already authorized; native preparation/training must precede it")
+
+
+def prepare(head_run):
+ head_run = Path(head_run)
+ _unopened(head_run)
+ head = benefit._checked_plan(benefit.read(head_run / "plan.json"))
+ fold = np.asarray(head["training_folds"])
+ rows = head["populations"]["train"]
+ return benefit.seal(
+ {
+ "format": PLAN_FORMAT,
+ "created_at": benefit.now(),
+ "protocol": PROTOCOL,
+ "questions": QUESTIONS,
+ "head_plan": head,
+ "head_plan_sha256": file_hash(head_run / "plan.json"),
+ "selection_fit_ids": [
+ row["id"]
+ for row, assignment in zip(rows, fold, strict=True)
+ if assignment != PROTOCOL["selection_fold"]
+ ],
+ "selection_validation_ids": [
+ row["id"]
+ for row, assignment in zip(rows, fold, strict=True)
+ if assignment == PROTOCOL["selection_fold"]
+ ],
+ "training_pool_rows": len(head["populations"]["train_pool"]),
+ "training_rows": len(rows),
+ "source_code_sha256": code_hashes(),
+ }
+ )
+
+
+def checked_plan(run, head_run):
+ plan = benefit.checked(benefit.read(Path(run) / "plan.json"), PLAN_FORMAT)
+ head = benefit._checked_plan(plan["head_plan"])
+ rows = head["populations"]["train"]
+ if (
+ plan["protocol"] != PROTOCOL
+ or plan["questions"] != QUESTIONS
+ or plan["source_code_sha256"] != code_hashes()
+ or plan["head_plan_sha256"] != file_hash(Path(head_run) / "plan.json")
+ or benefit.read(Path(head_run) / "plan.json") != head
+ or plan["training_pool_rows"] != len(head["populations"]["train_pool"])
+ or plan["training_rows"] != len(rows)
+ or any(
+ plan[key]
+ != [row["id"] for row, fold in zip(rows, head["training_folds"], strict=True) if (fold == 0) == held]
+ for key, held in (("selection_fit_ids", False), ("selection_validation_ids", True))
+ )
+ ):
+ raise ValueError("Native protocol, paper split, head plan or source code changed")
+ return plan
+
+
+def soft_rows(inputs, targets=None):
+ targets = np.zeros(len(inputs)) if targets is None else np.asarray(targets, dtype=float)
+ if targets.shape != (len(inputs),) or not np.isfinite(targets).all() or not np.isin(targets, [-1, 0, 1]).all():
+ raise ValueError("Need aligned signed paired outcomes; ties must remain zero")
+ return [
+ {
+ "state": row["request"],
+ "questions": QUESTIONS,
+ "gold": {"route": {"probabilities": {"cheap": float((1 - target) / 2), "strong": float((1 + target) / 2)}}},
+ }
+ for row, target in zip(inputs, targets, strict=True)
+ ]
+
+
+def benefit_scores(margins):
+ margins = np.asarray(margins, dtype=float)
+ if margins.ndim != 1 or not len(margins) or not np.isfinite(margins).all():
+ raise ValueError("Need finite one-dimensional native margins")
+ # Numerically stable identity: 2 * sigmoid(margin) - 1.
+ return np.tanh(margins / 2)
+
+
+def records_to_scores(records, expected_targets):
+ logits = np.asarray([record[1] for record in records], dtype=float)
+ targets = np.asarray([record[2] for record in records], dtype=float)
+ if (
+ logits.shape != (len(expected_targets), 2)
+ or targets.shape != logits.shape
+ or any(record[3] != 2 for record in records)
+ or not np.isfinite(logits).all()
+ or not np.isfinite(targets).all()
+ or not np.allclose(targets.sum(axis=1), 1, rtol=0, atol=1e-12)
+ or np.any(targets < 0)
+ or not np.allclose(2 * targets[:, 1] - 1, expected_targets, rtol=0, atol=1e-12)
+ ):
+ raise ValueError("Native records changed the cheap/strong order, soft targets or row count")
+ margins = logits[:, 1] - logits[:, 0]
+ return margins, benefit_scores(margins)
+
+
+def selected_epoch(history, targets):
+ targets = np.asarray(targets, dtype=float)
+ if targets.ndim != 1 or not len(targets) or not np.isin(targets, [-1, 0, 1]).all():
+ raise ValueError("Selection requires signed paired targets, including zero ties")
+ if [row["epoch"] for row in history] != list(range(PROTOCOL["selection_epochs"] + 1)):
+ raise ValueError("Selection must retain epoch0 and every declared training epoch")
+ scores = [benefit_scores(row["margins"]) for row in history]
+ if any(values.shape != np.asarray(targets).shape for values in scores):
+ raise ValueError("Validation scores and signed targets are not aligned")
+ losses = [float(np.mean((values - targets) ** 2)) for values in scores]
+ if any(
+ not np.isclose(row["benefit_mse"], loss, rtol=0, atol=1e-12) for row, loss in zip(history, losses, strict=True)
+ ):
+ raise ValueError("Recorded signed-benefit MSE differs from preserved soft targets")
+ return int(min(range(len(losses)), key=lambda epoch: (losses[epoch], epoch)))
+
+
+def score_checkpoint(checkpoint, rows, device):
+ """Return signed benefit scores in input order, using a local merged checkpoint.
+
+ ``rows`` contains request dictionaries; any other keys are ignored. Callers
+ loading a frozen policy must verify its checkpoint hashes before calling.
+ This helper performs no solver requests or threshold fitting.
+ """
+ checkpoint = Path(checkpoint)
+ if device not in ("cpu", "mps", "cuda") or not rows:
+ raise ValueError("Need nonempty request rows and device cpu, mps or cuda")
+ if not all((checkpoint / name).is_file() for name in ("rl_agent_config.json", "model.safetensors")):
+ raise ValueError("Native inference requires a local complete checkpoint")
+ os.environ.update(PROTOCOL["eager_flags"])
+ if device == "mps" and os.environ.get("PYTORCH_ENABLE_MPS_FALLBACK", "0") != "0":
+ raise ValueError("This protocol requires MPS without CPU fallback")
+ import laya.train as lt
+ import torch
+
+ runtime_device = torch.device(device)
+ model, tok, _ = lt.load_checkpoint(str(checkpoint))
+ try:
+ model.encoder.config.reference_compile = False
+ model.to(runtime_device).eval()
+ encoded, skipped = lt.items_from_rows(
+ tok, soft_rows(rows), finetune.MAX_LEN, finetune.HEAD_LEN, label_smoothing=0.0
+ )
+ if skipped or len(encoded) != len(rows):
+ raise ValueError(f"Native inference must not drop rows: {skipped}")
+ records = lt.calibration_records(
+ model,
+ tok,
+ encoded,
+ runtime_device,
+ finetune.MAX_LEN,
+ finetune.HEAD_LEN,
+ batch_size=finetune.runtime_settings(device)["evaluation_batch"],
+ )
+ return records_to_scores(records, np.zeros(len(rows)))[1].tolist()
+ finally:
+ del model
+ finetune.empty_cache(torch, runtime_device)
+
+
+def preflight(run, head_run, baseline, original_path, training_plan_path, training_inputs_path, training_grades_path):
+ run = Path(run)
+ _unopened(head_run)
+ for name in ("training-start.json", "frozen.json", "test-access.json", "report.json"):
+ if (run / name).exists():
+ raise FileExistsError(run / name)
+ plan = checked_plan(run, head_run)
+ head = plan["head_plan"]
+ dataset, original, solver_plan, artifacts = benefit._inputs(baseline, original_path)
+ if (
+ digest(dataset) != head["dataset_sha256"]
+ or file_hash(original_path) != head["original_frozen_sha256"]
+ or artifacts != head["baseline_artifacts_sha256"]
+ or solver_plan["settings_sha256"] != head["solver_settings_sha256"]
+ ):
+ raise ValueError("Native data or original solver study changed after preparation")
+ grades = benefit._training(head, training_plan_path, training_inputs_path, training_grades_path, original)
+ training_plan = benefit.read(training_plan_path)
+ for key in ("gold_review_sha256", "training_gold_rules_sha256", "training_gold_review_source_sha256"):
+ if grades.get(key) != training_plan.get(key) or not isinstance(grades.get(key), str):
+ raise ValueError("Training grade review provenance differs from its source plan")
+ by_id = {row["id"]: row for row in dataset["splits"]["train"]}
+ training = [by_id[row["id"]] for row in head["populations"]["train"]]
+ targets = benefit.signed_targets(grades["rows"])
+ folds = np.asarray(head["training_folds"])
+ return {
+ "plan": plan,
+ "original": original,
+ "training": training,
+ "targets": targets,
+ "held": folds == PROTOCOL["selection_fold"],
+ "calibration": dataset["splits"]["calibration"],
+ "test": dataset["splits"]["test"],
+ "training_grade_provenance": {key: value for key, value in grades.items() if key != "rows"},
+ "training_artifacts_sha256": {
+ "plan": file_hash(training_plan_path),
+ "inputs": file_hash(training_inputs_path),
+ "grades": file_hash(training_grades_path),
+ },
+ }
+
+
+def _calibrated(rows, scores, rate):
+ return {
+ str(target): solver_report.select_cutoff(rows, scores, target, router_credits=None, credit_usd=rate)
+ for target in benefit.TARGETS
+ }
+
+
+def train(
+ run,
+ head_run,
+ baseline,
+ original_path,
+ training_plan_path,
+ training_inputs_path,
+ training_grades_path,
+ *,
+ device,
+ local_files_only=False,
+):
+ context = preflight(
+ run, head_run, baseline, original_path, training_plan_path, training_inputs_path, training_grades_path
+ )
+ if device not in PROTOCOL["runtime_profiles"]:
+ raise ValueError("Native training requires mps or cuda")
+ return _train_gpu(Path(run), Path(head_run), context, device, local_files_only)
+
+
+def _train_gpu(run, head_run, context, device_name, local_files_only):
+ os.environ.update(PROTOCOL["eager_flags"])
+ if device_name == "mps" and os.environ.get("PYTORCH_ENABLE_MPS_FALLBACK", "0") != "0":
+ raise ValueError("This protocol requires MPS without CPU fallback")
+ import laya.train as lt
+ import torch
+ from huggingface_hub import snapshot_download
+ from peft import LoraConfig, get_peft_model
+ from safetensors.torch import save_file
+
+ versions = {name: importlib.metadata.version(name) for name in PROTOCOL["versions"]}
+ if {name: version.split("+", 1)[0] for name, version in versions.items()} != PROTOCOL["versions"]:
+ raise ValueError("Native training dependencies differ from the pinned protocol")
+ device = torch.device(device_name)
+ if not (torch.backends.mps.is_available() if device_name == "mps" else torch.cuda.is_available()):
+ raise ValueError("Requested GPU is unavailable")
+ runtime, recipe = PROTOCOL["runtime_profiles"][device_name], PROTOCOL["recipe"]
+ start = {
+ "plan_sha256": file_hash(run / "plan.json"),
+ "started_at": benefit.now(),
+ "versions": versions,
+ "device": device_name,
+ "runtime": runtime,
+ "eager_flags": PROTOCOL["eager_flags"],
+ "training_artifacts_sha256": context["training_artifacts_sha256"],
+ }
+ write(run / "training-start.json", start)
+ started = time.monotonic()
+ snapshot = snapshot_download(
+ finetune.BASE,
+ revision=finetune.REVISION,
+ allow_patterns=["rl_agent_config.json", "model.safetensors", "encoder/*", "tokenizer/*"],
+ local_files_only=local_files_only,
+ )
+ targets, held = context["targets"], context["held"]
+ histories, refit_losses = [], []
+
+ def items(tok, inputs, y=None):
+ result, skipped = lt.items_from_rows(
+ tok, soft_rows(inputs, y), finetune.MAX_LEN, finetune.HEAD_LEN, label_smoothing=0.0
+ )
+ if skipped or len(result) != len(inputs):
+ raise ValueError(f"Native preprocessing must not drop rows: {skipped}")
+ return result
+
+ def score(model, tok, encoded, y):
+ records = lt.calibration_records(
+ model, tok, encoded, device, finetune.MAX_LEN, finetune.HEAD_LEN, batch_size=runtime["evaluation_batch"]
+ )
+ return records_to_scores(records, y)
+
+ def config(epochs):
+ return lt.TrainConfig(
+ epochs=epochs,
+ micro_batch=runtime["micro_batch"],
+ grad_accum=runtime["grad_accum"],
+ encoder_lr=recipe["encoder_lr"],
+ head_lr=recipe["head_lr"],
+ weight_decay=recipe["weight_decay"],
+ seed=PROTOCOL["seed"],
+ max_len=finetune.MAX_LEN,
+ head_max_len=finetune.HEAD_LEN,
+ loss="soft-ce",
+ shuffle_options=("choice",),
+ gradient_checkpointing=runtime["gradient_checkpointing"],
+ label_smoothing=0.0,
+ log_every=32,
+ )
+
+ with tempfile.TemporaryDirectory(prefix=".native-initial-", dir=run) as temporary:
+ initial = Path(temporary)
+ for name in ("rl_agent_config.json", "model.safetensors"):
+ shutil.copy2(Path(snapshot) / name, initial / name)
+ for name in ("encoder", "tokenizer"):
+ shutil.copytree(Path(snapshot) / name, initial / name, symlinks=False)
+ initial_sha = file_hash(initial / "model.safetensors")
+
+ def load(with_lora):
+ torch.manual_seed(PROTOCOL["seed"])
+ np.random.seed(PROTOCOL["seed"])
+ random.seed(PROTOCOL["seed"])
+ model, tok, cfg = lt.load_checkpoint(str(initial))
+ cfg.update(max_len=finetune.MAX_LEN, head_max_len=finetune.HEAD_LEN)
+ model.encoder.config.reference_compile = False
+ if with_lora:
+ model.encoder = get_peft_model(
+ model.encoder,
+ LoraConfig(
+ r=recipe["rank"],
+ lora_alpha=2 * recipe["rank"],
+ lora_dropout=recipe["dropout"],
+ target_modules=finetune.lora_targets(recipe),
+ bias="none",
+ ),
+ )
+ return model.to(device), tok, cfg
+
+ model, tok, cfg = load(True)
+ fit_rows = [row for row, is_held in zip(context["training"], held, strict=True) if not is_held]
+ held_rows = [row for row, is_held in zip(context["training"], held, strict=True) if is_held]
+ fitting, validation = items(tok, fit_rows, targets[~held]), items(tok, held_rows, targets[held])
+ selection_config = config(PROTOCOL["selection_epochs"])
+ write(run / "selection-config.json", asdict(selection_config))
+
+ def observe(epoch, loss, model=model, tok=tok, validation=validation):
+ margins, values = score(model, tok, validation, targets[held])
+ row = {
+ "epoch": epoch + 1,
+ "train_loss": loss,
+ "margins": margins.tolist(),
+ "benefit_mse": float(np.mean((values - targets[held]) ** 2)),
+ }
+ histories.append(row)
+ finetune.write(run / "selection-history.json", histories)
+ print(json.dumps({key: row[key] for key in ("epoch", "train_loss", "benefit_mse")}), flush=True)
+ model.train()
+
+ observe(-1, None)
+ lt.train_model(
+ model, tok, fitting, selection_config, device, finetune.MAX_LEN, finetune.HEAD_LEN, on_epoch_end=observe
+ )
+ observe = None
+ epoch = selected_epoch(histories, targets[held])
+ write(
+ run / "epoch-selected.json",
+ {
+ "epoch": epoch,
+ "history_sha256": file_hash(run / "selection-history.json"),
+ "validation_ids": context["plan"]["selection_validation_ids"],
+ "signed_targets": targets[held].astype(int).tolist(),
+ "criterion": PROTOCOL["selection"],
+ },
+ )
+ del model, fitting, validation, tok
+ finetune.empty_cache(torch, device)
+ models = {}
+ for variant in ("base", "lora"):
+ model, tok, cfg = load(variant == "lora")
+ validation = items(tok, held_rows, targets[held])
+ if variant == "lora":
+ refit_config = config(epoch)
+ write(run / "refit-config.json", asdict(refit_config))
+ if epoch:
+ fitting = items(tok, context["training"], targets)
+ refit_losses = lt.train_model(
+ model, tok, fitting, refit_config, device, finetune.MAX_LEN, finetune.HEAD_LEN
+ )
+ del fitting
+ before, _ = score(model, tok, validation, targets[held])
+ merge_error = 0.0
+ if variant == "lora":
+ model.encoder = model.encoder.merge_and_unload()
+ merged, _ = score(model, tok, validation, targets[held])
+ merge_error = float(np.max(np.abs(merged - before)))
+ if merge_error > PROTOCOL["max_merge_margin_error"]:
+ raise ValueError("LoRA merge changed native margins beyond the predeclared tolerance")
+ destination = run / variant
+ lt.save_checkpoint(model, tok, cfg, str(destination))
+ save_file(
+ {name: tensor.detach().float().cpu().contiguous() for name, tensor in model.state_dict().items()},
+ str(destination / "model.safetensors"),
+ )
+ del model
+ finetune.empty_cache(torch, device)
+ model, tok, _ = lt.load_checkpoint(str(destination))
+ model.encoder.config.reference_compile = False
+ model.to(device).eval()
+ exported, _ = score(model, tok, validation, targets[held])
+ export_error = float(np.max(np.abs(exported - before)))
+ if export_error > PROTOCOL["max_export_margin_error"]:
+ raise ValueError("FP32 export changed native margins beyond the predeclared tolerance")
+ margins, scores = {}, {}
+ for split in ("calibration", "test"):
+ unlabeled = context[split]
+ values, expected_benefit = score(model, tok, items(tok, unlabeled), np.zeros(len(unlabeled)))
+ margins[split], scores[split] = values.tolist(), expected_benefit.tolist()
+ models[variant] = {
+ "checkpoint_files_sha256": {
+ str(path.relative_to(destination)): file_hash(path)
+ for path in sorted(destination.rglob("*"))
+ if path.is_file()
+ },
+ "parameters": sum(parameter.numel() for parameter in model.parameters()),
+ "margins": margins,
+ "scores": scores,
+ "max_merge_margin_error": merge_error,
+ "max_export_margin_error": export_error,
+ "policies": _calibrated(
+ context["original"]["calibration_rows"], scores["calibration"], context["original"]["credit_usd"]
+ ),
+ }
+ write(run / f"{variant}-scores.json", models[variant])
+ del model, tok, validation
+ finetune.empty_cache(torch, device)
+ _unopened(head_run)
+ finetune.synchronize(torch, device)
+ frozen = benefit.seal(
+ {
+ "format": FROZEN_FORMAT,
+ "frozen_at": benefit.now(),
+ "plan_sha256": file_hash(run / "plan.json"),
+ "head_plan_sha256": context["plan"]["head_plan_sha256"],
+ "training_start_sha256": file_hash(run / "training-start.json"),
+ "initial_checkpoint_sha256": initial_sha,
+ "training_artifacts_sha256": context["training_artifacts_sha256"],
+ "training_grade_provenance": context["training_grade_provenance"],
+ "selection_history": histories,
+ "validation_signed_targets": targets[held].astype(int).tolist(),
+ "selected_epoch": epoch,
+ "refit_losses": refit_losses,
+ "grading_provenance": solver_policies._grading_provenance(context["original"]),
+ "calibration_rows": context["original"]["calibration_rows"],
+ "calibration_grades_sha256": context["original"]["calibration_grades_sha256"],
+ "credit_usd": context["original"]["credit_usd"],
+ "models": models,
+ "elapsed_seconds": time.monotonic() - started,
+ }
+ )
+ write(run / "frozen.json", frozen)
+ print(
+ json.dumps(
+ {"frozen": str(run / "frozen.json"), "selected_epoch": epoch, "elapsed_seconds": frozen["elapsed_seconds"]}
+ ),
+ flush=True,
+ )
+ return frozen
+
+
+def checked_frozen(run, head_run, *, weights=False):
+ run = Path(run)
+ plan = checked_plan(run, head_run)
+ frozen = benefit.checked(benefit.read(run / "frozen.json"), FROZEN_FORMAT)
+ start = benefit.read(run / "training-start.json")
+ if (
+ frozen["plan_sha256"] != file_hash(run / "plan.json")
+ or frozen["head_plan_sha256"] != plan["head_plan_sha256"]
+ or frozen["training_start_sha256"] != file_hash(run / "training-start.json")
+ or frozen["selected_epoch"]
+ != selected_epoch(frozen["selection_history"], np.asarray(frozen["validation_signed_targets"]))
+ or len(frozen["refit_losses"]) != frozen["selected_epoch"]
+ or not np.isfinite(frozen["refit_losses"]).all()
+ or len(frozen["validation_signed_targets"]) != len(plan["selection_validation_ids"])
+ or set(frozen["models"]) != {"base", "lora"}
+ or start["plan_sha256"] != frozen["plan_sha256"]
+ or start["training_artifacts_sha256"] != frozen["training_artifacts_sha256"]
+ or start["eager_flags"] != PROTOCOL["eager_flags"]
+ or start["runtime"] != PROTOCOL["runtime_profiles"].get(start["device"])
+ or {name: version.split("+", 1)[0] for name, version in start["versions"].items()} != PROTOCOL["versions"]
+ ):
+ raise ValueError("Native freeze does not match its training-only epoch selection and refit")
+ solver_policies._grading_provenance(frozen["grading_provenance"])
+ if [row["id"] for row in frozen["calibration_rows"]] != [
+ row["id"] for row in plan["head_plan"]["populations"]["calibration"]
+ ]:
+ raise ValueError("Native calibration rows differ from the fixed population")
+ for variant, model in frozen["models"].items():
+ if (
+ not 0 <= model["max_merge_margin_error"] <= PROTOCOL["max_merge_margin_error"]
+ or not 0 <= model["max_export_margin_error"] <= PROTOCOL["max_export_margin_error"]
+ or "model.safetensors" not in model["checkpoint_files_sha256"]
+ ):
+ raise ValueError("Native checkpoint/export parity receipt is invalid")
+ for split in ("calibration", "test"):
+ count = len(plan["head_plan"]["populations"][split])
+ margins = solver_policies.finite_scores(model["margins"][split], count)
+ scores = solver_policies.finite_scores(model["scores"][split], count)
+ if not np.allclose(benefit_scores(margins), scores, rtol=0, atol=1e-12):
+ raise ValueError("Native benefit scores differ from canonical cheap/strong margins")
+ if model["policies"] != _calibrated(
+ frozen["calibration_rows"], model["scores"]["calibration"], frozen["credit_usd"]
+ ):
+ raise ValueError("Native cutoffs differ from calibration-only cost selection")
+ for relative, expected in model["checkpoint_files_sha256"].items():
+ path = Path(relative)
+ if (
+ path.is_absolute()
+ or ".." in path.parts
+ or not isinstance(expected, str)
+ or len(expected) != 64
+ or any(character not in "0123456789abcdef" for character in expected)
+ ):
+ raise ValueError("Native checkpoint manifest contains an invalid path or hash")
+ if weights and file_hash(run / variant / path) != expected:
+ raise ValueError("Native checkpoint bytes differ from their frozen hashes")
+ return plan, frozen
+
+
+def checkpoint_manifest_hash(frozen):
+ return digest({name: model["checkpoint_files_sha256"] for name, model in frozen["models"].items()})
+
+
+def authorize_test(run, head_run, *, verify_weights=False):
+ run, head_run = Path(run), Path(head_run)
+ if (run / "test-access.json").exists():
+ raise FileExistsError(run / "test-access.json")
+ plan, frozen = checked_frozen(run, head_run, weights=verify_weights)
+ benefit._checked_frozen(head_run)
+ return benefit.seal(
+ {
+ "format": ACCESS_FORMAT,
+ "authorized_at": benefit.now(),
+ "native_frozen_sha256": file_hash(run / "frozen.json"),
+ "head_frozen_sha256": file_hash(head_run / "frozen.json"),
+ "head_plan_sha256": plan["head_plan_sha256"],
+ "checkpoint_manifest_sha256": checkpoint_manifest_hash(frozen),
+ "checkpoint_verification": "local-bytes" if verify_weights else "manifest-only",
+ "source_code_sha256": code_hashes(),
+ }
+ )
+
+
+def report(run, head_run, test_grades_path):
+ run, head_run = Path(run), Path(head_run)
+ plan, frozen = checked_frozen(run, head_run)
+ access = benefit.checked(benefit.read(run / "test-access.json"), ACCESS_FORMAT)
+ shared = benefit.checked(benefit.read(head_run / "test-access.json"), benefit.ACCESS_FORMAT)
+ if (
+ access["native_frozen_sha256"] != file_hash(run / "frozen.json")
+ or access["head_frozen_sha256"] != file_hash(head_run / "frozen.json")
+ or access["head_plan_sha256"] != plan["head_plan_sha256"]
+ or access["source_code_sha256"] != code_hashes()
+ or access["checkpoint_manifest_sha256"] != checkpoint_manifest_hash(frozen)
+ or access["checkpoint_verification"] not in ("local-bytes", "manifest-only")
+ or shared["frozen_sha256"] != access["head_frozen_sha256"]
+ or shared["plan_sha256"] != plan["head_plan_sha256"]
+ or shared["source_code_sha256"] != benefit.code_hashes()
+ ):
+ raise ValueError("Native/shared test access does not bind the frozen models")
+ head = plan["head_plan"]
+ ids = [row["id"] for row in head["populations"]["test"]]
+ grades = solver_policies._grade_rows(test_grades_path, "test", head["original_solver_plan_sha256"], ids)
+ if (
+ grades["policies_frozen_sha256"] != head["original_frozen_sha256"]
+ or solver_policies._grading_provenance(grades) != frozen["grading_provenance"]
+ ):
+ raise ValueError("Native test grades differ from the original paired solver/grading experiment")
+ measured = {}
+ for variant, model in frozen["models"].items():
+ measured[variant] = {}
+ for target in benefit.TARGETS:
+ selection = model["policies"][str(target)]
+ mode = selection["mode"]
+ routes = (
+ (np.asarray(model["scores"]["test"]) >= selection["cutoff"]).tolist() if mode == "threshold" else mode
+ )
+ measured[variant][str(target)] = {
+ "mode": mode,
+ "cutoff": selection["cutoff"],
+ "metrics": solver_report.summarize(
+ grades["rows"],
+ routes,
+ router_credits=None if mode == "threshold" else 0,
+ credit_usd=frozen["credit_usd"],
+ ),
+ }
+ return {
+ "format": "sie-solver-benefit-native-report-v1",
+ "n": len(ids),
+ "ids": ids,
+ "training_pool_rows": plan["training_pool_rows"],
+ "training_rows": plan["training_rows"],
+ "selected_epoch": frozen["selected_epoch"],
+ "frozen_sha256": file_hash(run / "frozen.json"),
+ "native_access_sha256": file_hash(run / "test-access.json"),
+ "shared_access_sha256": file_hash(head_run / "test-access.json"),
+ "test_grades_sha256": file_hash(test_grades_path),
+ "primary_max_quality_loss_pp": 1,
+ "models": measured,
+ "interpretation": "Training-only paper fold selected epoch; refit all eligible rows. Calibration loss targets are not test guarantees. Native hosting remains unpriced; gross solver savings exclude router cost. Frozen before solver test outcome access, not collection.",
+ }
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ commands = parser.add_subparsers(dest="command", required=True)
+ for command in ("prepare", "train", "authorize-test", "report"):
+ child = commands.add_parser(command)
+ child.add_argument("--run", type=Path, required=True)
+ child.add_argument("--head-run", type=Path, required=True)
+ if command == "train":
+ for name in ("baseline", "original-frozen", "training-plan", "training-inputs", "training-grades"):
+ child.add_argument(f"--{name}", type=Path, required=True)
+ child.add_argument("--device", choices=("mps", "cuda"), required=True)
+ child.add_argument("--local-files-only", action="store_true")
+ if command == "report":
+ child.add_argument("--test-grades", type=Path, required=True)
+ if command == "authorize-test":
+ child.add_argument("--verify-weights", action="store_true", help="Also verify local checkpoint bytes")
+ args = parser.parse_args()
+ destination = (
+ args.run
+ / {
+ "prepare": "plan.json",
+ "train": "frozen.json",
+ "authorize-test": "test-access.json",
+ "report": "report.json",
+ }[args.command]
+ )
+ if destination.exists():
+ raise FileExistsError(destination)
+ if args.command == "prepare":
+ if args.run.exists():
+ raise FileExistsError(args.run)
+ result = prepare(args.head_run)
+ elif args.command == "train":
+ train(
+ args.run,
+ args.head_run,
+ args.baseline,
+ args.original_frozen,
+ args.training_plan,
+ args.training_inputs,
+ args.training_grades,
+ device=args.device,
+ local_files_only=args.local_files_only,
+ )
+ return
+ elif args.command == "authorize-test":
+ result = authorize_test(args.run, args.head_run, verify_weights=args.verify_weights)
+ else:
+ result = report(args.run, args.head_run, args.test_grades)
+ write(destination, result)
+ print(json.dumps({"output": str(destination), "sha256": file_hash(destination)}))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/benefit_outcomes.py b/examples/train-a-router/benefit_outcomes.py
new file mode 100644
index 000000000..e92fd8ba9
--- /dev/null
+++ b/examples/train-a-router/benefit_outcomes.py
@@ -0,0 +1,1028 @@
+"""Predict each solver's correctness and route by benefit per expected credit.
+
+Prepare the source-only population before acquiring new outcomes. Fit the two
+heads using paper-group training folds, then freeze calibration cutoffs in a
+separate step. Replay and live scoring need NumPy; fitting also needs SciPy.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+from fractions import Fraction
+from pathlib import Path
+
+import numpy as np
+
+import benefit
+import solver_policies
+import solver_report
+import source_review
+from data import SOURCE_HASHES, SOURCE_REVISION, audit, digest, file_hash, write
+from serving import INSTRUCTION, MODEL, REVISION, vectors
+
+PROTOCOL = {
+ "target": "Two separate Bernoulli targets: cheap_correct and strong_correct",
+ "training_pool_sizes": [512, 2581],
+ "required_original_exclusions": ["e2h-amc-eval-661"],
+ "encoder": MODEL,
+ "revision": REVISION,
+ "instruction": INSTRUCTION,
+ "dimensions": 2560,
+ "penalties": [0.01, 0.1, 1.0, 10.0],
+ "folds": 5,
+ "seed": 89,
+ "fold_assignment": "Sort paper groups, permute with NumPy seed89, assign whole groups round-robin to five folds",
+ "fit": "Independent logistic heads: summed Bernoulli NLL plus 0.5*lambda*L2(weights); unpenalized intercept; no class weighting",
+ "selection": "One shared lambda minimizes pooled training-fold cheap NLL plus strong NLL; exact ties prefer larger lambda",
+ "optimizer": {"scipy": "1.17.0", "method": "L-BFGS-B", "maxiter": 2000, "ftol": 1e-12, "gtol": 1e-7, "maxls": 40},
+ "cost_model": "Four training-only means of uncached Cloud token-cost proxies, conditioned on each solver's correctness; expected cost is the probability-weighted mixture",
+ "score": "(p_strong_correct-p_cheap_correct)/max(expected_strong_credits-expected_cheap_credits,1.0)",
+ "incremental_cost_floor_credits": 1.0,
+ "cost_interpretation": "Training token-cost proxies are not observed Cloud charges or self-hosting costs. Calibration and test accounting use measured credits.",
+ "primary_max_quality_loss_pp": 1,
+ "secondary_max_quality_loss_pp": [0, 2, 5],
+ "expansion": "The larger pool adds examples within the same32 training paper groups, not new papers or years",
+ "eligibility": "Original collection eligibility is preserved; a separately bound source-reference audit may exclude malformed questions after training outcomes exist, before corrected fitting and held-out solver outcome access",
+ "test_boundary": "Freeze every model and cutoff before solver test OUTCOME ACCESS, not before test collection",
+ "motivation": "Calibration diagnostics and training-only learning curves motivated this bounded original-versus-expanded comparison; no additional architecture search",
+}
+CODE_FILES = (
+ "benefit_outcomes.py",
+ "benefit.py",
+ "policies.py",
+ "serving.py",
+ "solver_policies.py",
+ "solver_report.py",
+ "data.py",
+ "source_review.py",
+ "teacher_format.py",
+)
+PLAN_FORMAT = "sie-solver-outcome-plan-v2"
+MODEL_FORMAT = "sie-solver-outcome-selected-v1"
+FROZEN_FORMAT = "sie-solver-outcome-frozen-v1"
+ACCESS_FORMAT = "sie-solver-outcome-test-access-v1"
+TARGETS = (1, 0, 2, 5)
+ARMS = ("cheap", "strong")
+GRADE_HASHES = (
+ "grader_source_sha256",
+ "gold_review_source_sha256",
+ "gold_review_sha256",
+ "training_gold_rules_sha256",
+ "training_gold_review_source_sha256",
+ "outcome_grader_source_sha256",
+ "outcome_grader_protocol_sha256",
+)
+SHARED_GRADE_HASHES = (
+ "grader_source_sha256",
+ "gold_review_source_sha256",
+ "outcome_grader_source_sha256",
+ "outcome_grader_protocol_sha256",
+)
+
+
+def read(path):
+ return json.loads(Path(path).read_text())
+
+
+def code_hashes():
+ return {name: file_hash(Path(__file__).with_name(name)) for name in CODE_FILES}
+
+
+def _require(condition, message):
+ if not condition:
+ raise ValueError(message)
+
+
+def _sha(value):
+ return isinstance(value, str) and len(value) == 64 and all(char in "0123456789abcdef" for char in value)
+
+
+def grading_provenance(record):
+ """Bind base semantics plus the same explicit source audit on every split."""
+ legacy = solver_policies._grading_provenance(record)
+ extra = {name: record.get(name) for name in SHARED_GRADE_HASHES[2:]}
+ _require(
+ all(_sha(value) for value in extra.values()), "Outcome grades require the shared v2 source and protocol hashes"
+ )
+ reviewed = {name: record[name] for name in source_review.METADATA_KEYS if name in record}
+ _require(
+ not reviewed or (len(reviewed) == len(source_review.METADATA_KEYS) and all(_sha(v) for v in reviewed.values())),
+ "Source-review provenance must be complete",
+ )
+ return legacy | extra | reviewed
+
+
+def _unopened(run):
+ for name in ("test-access.json", "report.json"):
+ if (Path(run) / name).exists():
+ raise FileExistsError(Path(run) / name)
+
+
+def _population(rows):
+ return [
+ {
+ "id": row["id"],
+ "request_sha256": digest(row["request"]),
+ "paper_group": row["paper_group"],
+ "family": row["family"],
+ "year": row["year"],
+ }
+ for row in rows
+ ]
+
+
+def _eligibility(receipt, pool, dataset_sha):
+ ids = [row["id"] for row in pool]
+ exclusions = receipt.get("exclusions", [])
+ excluded = [row.get("id") for row in exclusions]
+ eligible = [identity for identity in ids if identity not in excluded]
+ _require(
+ receipt.get("format") == "sie-benefit-training-eligibility-v1"
+ and receipt.get("source_only") is True
+ and (
+ receipt.get("decided_before_training_outcomes") is True
+ or (
+ receipt.get("decided_before_new_training_outcomes") is True
+ and receipt.get("reused_training_outcomes_already_available") is True
+ )
+ )
+ and receipt.get("source_revision") == SOURCE_REVISION
+ and receipt.get("source_hashes") == SOURCE_HASHES
+ and receipt.get("dataset_sha256") == dataset_sha
+ and receipt.get("original_pool_ids") == ids
+ and len(ids) in PROTOCOL["training_pool_sizes"]
+ and len(set(ids)) == len(ids)
+ and len(set(excluded)) == len(excluded)
+ and set(excluded).issubset(ids)
+ and set(PROTOCOL["required_original_exclusions"]).issubset(excluded)
+ and all(isinstance(row.get("reason"), str) and row["reason"].strip() for row in exclusions)
+ and receipt.get("eligible_train_ids") == eligible
+ and _sha(receipt.get("training_gold_review_source_sha256")),
+ "Eligibility must preserve every nonexcluded source input in order, before new outcomes",
+ )
+ return eligible
+
+
+def _training_contract(plan, inputs, eligibility, eligibility_sha, population, pool_size, dataset_sha, settings_sha):
+ expected = [{key: row[key] for key in ("id", "request_sha256")} for row in population]
+ _require(
+ plan.get("dataset_sha256") == dataset_sha
+ and plan.get("settings_sha256") == settings_sha
+ and plan.get("inputs_sha256") == digest(inputs)
+ and plan.get("source_counts", {}).get("train") == pool_size
+ and plan.get("counts", {}).get("train") == len(population)
+ and plan.get("eligible_train_ids") == [row["id"] for row in population]
+ and plan.get("excluded_train_ids") == [row["id"] for row in eligibility["exclusions"]]
+ and plan.get("eligibility_sha256") == eligibility_sha
+ and [{"id": row["id"], "request_sha256": digest(row["request"])} for row in inputs["train"]] == expected
+ and all(_sha(plan.get(name)) for name in GRADE_HASHES)
+ and plan["training_gold_review_source_sha256"] == eligibility["training_gold_review_source_sha256"],
+ "Training collection plan, inputs, source review and eligible population differ",
+ )
+
+
+def prepare(
+ baseline,
+ original_path,
+ training_plan_path,
+ training_inputs_path,
+ training_rows_path,
+ eligibility_path,
+ *,
+ expansion_audit_path=None,
+ source_review_path=None,
+ attest_test_outcomes_unopened=False,
+):
+ if attest_test_outcomes_unopened is not True:
+ raise ValueError("An explicit assertion that solver test outcomes remain unopened is required")
+ dataset, original, solver_plan, artifacts = benefit._inputs(baseline, original_path)
+ pool, eligibility, training_plan, inputs = map(
+ read, (training_rows_path, eligibility_path, training_plan_path, training_inputs_path)
+ )
+ original_rows = dataset["splits"]["train"]
+ _require(pool[: len(original_rows)] == original_rows, "Training pool must preserve the original512 prefix exactly")
+ # The existing audit checks IDs, canonical requests, families, papers and
+ # temporal partitions. Human difficulty labels are never fitting targets.
+ audit(dataset | {"splits": dataset["splits"] | {"train": pool}})
+ _require(
+ {row["paper_group"] for row in pool} == {row["paper_group"] for row in original_rows},
+ "This bounded expansion must retain exactly the original training paper groups",
+ )
+ expansion = None
+ if len(pool) > len(original_rows):
+ if expansion_audit_path is None:
+ raise ValueError("Expanded training requires the pinned source expansion audit")
+ expansion = read(expansion_audit_path)
+ _require(
+ expansion.get("source_revision") == SOURCE_REVISION
+ and expansion.get("source_hashes") == SOURCE_HASHES
+ and expansion.get("original_dataset_sha256") == digest(dataset)
+ and expansion.get("expanded_rows_sha256") == digest(pool)
+ and expansion.get("original_prefix_rows") == len(original_rows)
+ and expansion.get("training_rows") == len(pool)
+ and expansion.get("ids") == [row["id"] for row in pool]
+ and expansion.get("families") == [row["family"] for row in pool],
+ "Expanded source rows differ from the original pinned expansion",
+ )
+ ids = _eligibility(eligibility, pool, digest(dataset))
+ population = _population([row for row in pool if row["id"] in set(ids)])
+ _training_contract(
+ training_plan,
+ inputs,
+ eligibility,
+ file_hash(eligibility_path),
+ population,
+ len(pool),
+ digest(dataset),
+ solver_plan["settings_sha256"],
+ )
+ _require(
+ all(training_plan.get(name) == original.get(name) for name in SHARED_GRADE_HASHES),
+ "Training and calibration must use the same v2 grader and base dependencies",
+ )
+ source_populations = {
+ "train": population,
+ **{split: _population(dataset["splits"][split]) for split in ("calibration", "test")},
+ }
+ populations, review_binding = source_populations, None
+ if source_review_path is not None:
+ review = source_review.checked_review(read(source_review_path))
+ populations = {
+ split: source_review.eligible_rows(rows, split, review) for split, rows in source_populations.items()
+ }
+ review_binding = source_review.metadata(source_review_path) | {
+ "exclusions": {
+ split: [row["id"] for row in rows if row["id"] not in {item["id"] for item in populations[split]}]
+ for split, rows in source_populations.items()
+ },
+ }
+ _require(
+ all(original.get(key) == review_binding[key] for key in source_review.METADATA_KEYS),
+ "Calibration and planned source review differ",
+ )
+ else:
+ _require(
+ not any(key in original for key in source_review.METADATA_KEYS),
+ "Reviewed calibration needs its source review",
+ )
+ return benefit.seal(
+ {
+ "format": PLAN_FORMAT,
+ "created_at": benefit.now(),
+ "protocol": PROTOCOL,
+ "source_code_sha256": code_hashes(),
+ "dataset_sha256": digest(dataset),
+ "original_frozen_sha256": file_hash(original_path),
+ "original_solver_plan_sha256": original["plan_sha256"],
+ "solver_settings_sha256": solver_plan["settings_sha256"],
+ "grading_provenance": grading_provenance(original),
+ "training_grade_provenance": {name: training_plan[name] for name in GRADE_HASHES},
+ "training_artifacts_sha256": {
+ "plan": file_hash(training_plan_path),
+ "inputs": file_hash(training_inputs_path),
+ "rows": file_hash(training_rows_path),
+ "eligibility": file_hash(eligibility_path),
+ "expansion_audit": None if expansion_audit_path is None else file_hash(expansion_audit_path),
+ },
+ "eligibility": eligibility,
+ "expansion_audit": expansion,
+ "training_pool": _population(pool),
+ "source_populations": source_populations,
+ "source_review": review_binding,
+ "populations": populations,
+ "training_folds": benefit.group_folds([row["paper_group"] for row in populations["train"]]),
+ "baseline_artifacts_sha256": artifacts,
+ "attest_test_outcomes_unopened": True,
+ "access_limitation": "Receipts record the declared boundary; they cannot prove nobody read another copy",
+ }
+ )
+
+
+def checked_plan(plan):
+ benefit.checked(plan, PLAN_FORMAT)
+ grading_provenance(plan["grading_provenance"])
+ eligible = _eligibility(plan["eligibility"], plan["training_pool"], plan["dataset_sha256"])
+ _require(
+ plan["protocol"] == PROTOCOL
+ and plan["source_code_sha256"] == code_hashes()
+ and plan["attest_test_outcomes_unopened"] is True
+ and plan["source_populations"]["train"] == [row for row in plan["training_pool"] if row["id"] in set(eligible)]
+ and plan["training_folds"] == benefit.group_folds([row["paper_group"] for row in plan["populations"]["train"]])
+ and all(_sha(plan["training_grade_provenance"].get(name)) for name in GRADE_HASHES),
+ "Prepared outcome protocol, source, eligible rows or paper folds changed",
+ )
+ review = plan["source_review"]
+ if review is not None:
+ _require(
+ all(
+ _sha(review.get(key)) and plan["grading_provenance"].get(key) == review[key]
+ for key in source_review.METADATA_KEYS
+ )
+ and review["source_review_grader_sha256"] == file_hash(Path(source_review.__file__))
+ and review["format_normalizer_sha256"] == file_hash(Path(__file__).with_name("teacher_format.py")),
+ "Source review or formatting implementation changed after preparation",
+ )
+ for split, rows in plan["source_populations"].items():
+ excluded = [] if review is None else review["exclusions"][split]
+ _require(
+ len(excluded) == len(set(excluded))
+ and set(excluded).issubset(row["id"] for row in rows)
+ and plan["populations"][split] == [row for row in rows if row["id"] not in set(excluded)],
+ "Reviewed population must preserve every nonexcluded source input in order",
+ )
+ _require(
+ all(
+ plan["training_grade_provenance"][name] == plan["grading_provenance"][name] for name in SHARED_GRADE_HASHES
+ ),
+ "Prepared training and held-out v2 grading contracts differ",
+ )
+ for split in ("calibration", "test"):
+ for key in ("id", "request_sha256", "paper_group", "family"):
+ _require(
+ not {row[key] for row in plan["training_pool"]} & {row[key] for row in plan["populations"][split]},
+ "Training overlaps a held population",
+ )
+ return plan
+
+
+def _baseline(plan, baseline):
+ baseline = Path(baseline)
+ _require(
+ all(file_hash(baseline / name) == sha for name, sha in plan["baseline_artifacts_sha256"].items()),
+ "Pinned baseline inputs or vectors changed",
+ )
+ dataset = read(baseline / "data.json")
+ _require(digest(dataset) == plan["dataset_sha256"], "Baseline dataset changed")
+ return dataset
+
+
+def held_rows(plan, dataset, split):
+ rows = dataset["splits"][split]
+ _require(_population(rows) == plan["source_populations"][split], "Held acquisition input order changed")
+ ids = {row["id"] for row in plan["populations"][split]}
+ selected = [row for row in rows if row["id"] in ids]
+ _require(_population(selected) == plan["populations"][split], "Reviewed held input order changed")
+ return selected
+
+
+def test_credits(plan, path):
+ """Validate the complete measured receipt, then select reviewed test IDs."""
+ full = plan["source_populations"]["test"]
+ credits, sha = solver_policies._credits(
+ path,
+ "test",
+ plan["original_solver_plan_sha256"],
+ [row["id"] for row in full],
+ {row["id"]: row["request_sha256"] for row in full},
+ )
+ if credits is None:
+ return None, sha
+ eligible = {row["id"] for row in plan["populations"]["test"]}
+ return [value for row, value in zip(full, credits, strict=True) if row["id"] in eligible], sha
+
+
+def _training(plan, training_plan_path, training_inputs_path, training_rows_path, grades_path):
+ for name, path in (("plan", training_plan_path), ("inputs", training_inputs_path), ("rows", training_rows_path)):
+ _require(
+ file_hash(path) == plan["training_artifacts_sha256"][name], "Training inputs changed after preparation"
+ )
+ collection, inputs, pool, grades = map(
+ read, (training_plan_path, training_inputs_path, training_rows_path, grades_path)
+ )
+ _require(_population(pool) == plan["training_pool"], "Training source metadata changed")
+ _training_contract(
+ collection,
+ inputs,
+ plan["eligibility"],
+ plan["training_artifacts_sha256"]["eligibility"],
+ plan["source_populations"]["train"],
+ len(pool),
+ plan["dataset_sha256"],
+ plan["solver_settings_sha256"],
+ )
+ _require(
+ grades.get("split") == "train"
+ and grades.get("plan_sha256") == file_hash(training_plan_path)
+ and [row["id"] for row in grades["rows"]] == [row["id"] for row in plan["populations"]["train"]]
+ and all(
+ grades.get(name) == collection.get(name) == plan["training_grade_provenance"][name] for name in GRADE_HASHES
+ ),
+ "Training grades differ from the frozen population or reviewed source provenance",
+ )
+ grading_provenance(grades)
+ if plan["source_review"] is not None:
+ _require(
+ grades.get("collected_ids") == [row["id"] for row in plan["source_populations"]["train"]]
+ and grades.get("excluded_ids") == plan["source_review"]["exclusions"]["train"]
+ and all(grades.get(key) == plan["source_review"][key] for key in source_review.METADATA_KEYS),
+ "Training grades must retain the complete collection and explicit source-only exclusions",
+ )
+ solver_report.summarize_quality(grades["rows"], "always_strong")
+ eligible = {row["id"] for row in plan["populations"]["train"]}
+ return [row for row in inputs["train"] if row["id"] in eligible], grades
+
+
+def _fraction(value):
+ return Fraction(int(value["numerator"]), int(value["denominator"]))
+
+
+def conditional_costs(targets, costs):
+ targets, costs = np.asarray(targets), np.asarray(costs, dtype=float)
+ _require(
+ targets.shape == costs.shape
+ and targets.ndim == 2
+ and targets.shape[1] == 2
+ and np.isin(targets, (0, 1)).all()
+ and np.isfinite(costs).all()
+ and (costs >= 0).all(),
+ "Cost targets must be finite, nonnegative and aligned with both outcomes",
+ )
+ result = {}
+ for arm, name in enumerate(ARMS):
+ for correct, state in ((0, "wrong"), (1, "correct")):
+ values = costs[targets[:, arm] == correct, arm]
+ _require(len(values) > 0, "Each outcome head needs correct and incorrect training examples")
+ result[f"{name}_{state}"] = {
+ "n": len(values),
+ "sum_credits_proxy": float(values.sum()),
+ "mean_credits_proxy": float(values.mean()),
+ }
+ return result
+
+
+def cost_proxy(path, price_path, training_plan_path, grades_path, grades):
+ record, prices = read(path), read(price_path)
+ _require(
+ record.get("format") == "sie-training-uncached-cloud-cost-proxy-v1"
+ and record.get("split") == "train"
+ and record.get("n") == len(grades["rows"])
+ and record.get("credit_usd") == 1e-5
+ and record.get("training_plan_sha256") == file_hash(training_plan_path)
+ and record.get("training_grades_sha256") == file_hash(grades_path)
+ and record.get("training_collection_sha256") == grades.get("collection_summary_sha256")
+ and record.get("price_source", {}).get("sha256") == file_hash(price_path)
+ and [row["id"] for row in record["rows"]] == [row["id"] for row in grades["rows"]],
+ "Training cost proxy must bind the same paired rows, collection and public prices",
+ )
+ rates = record["rates_credits_per_token"]
+ public_rows = record["price_source"]["public_rows"]
+ _require(
+ prices.get("currency") == "usd"
+ and record["price_source"].get("pricing_version") == prices.get("pricing_version")
+ and record["price_source"].get("rate_book_sha256") == prices.get("rate_book_sha256")
+ and all(row in prices.get("items", []) for row in public_rows),
+ "Embedded price rows must match the bound public snapshot",
+ )
+ models = {"cheap": "Qwen/Qwen3.5-4B", "strong": "Qwen/Qwen3.8-27B-FP8"}
+ for arm in ARMS:
+ for unit in ("input_tokens", "output_tokens"):
+ matching = [
+ row
+ for row in public_rows
+ if row["model"] == models[arm]
+ and row["unit"] == unit
+ and row["profile"] == "default"
+ and row["region"] == "us"
+ ]
+ _require(
+ len(matching) == 1
+ and _fraction(rates[arm][unit]) >= 0
+ and _fraction(rates[arm][unit]) == _fraction(matching[0]["usd_per_unit"]) * 100000,
+ "Credit rates disagree with the public USD prices",
+ )
+ costs = []
+ for proxy, grade in zip(record["rows"], grades["rows"], strict=True):
+ row_costs = []
+ for arm in ARMS:
+ _require(
+ type(proxy[f"{arm}_correct"]) is bool
+ and proxy[f"{arm}_correct"] == grade[f"{arm}_correct"]
+ and _sha(proxy[f"{arm}_record_sha256"])
+ and proxy[f"{arm}_record_sha256"] == grade[f"{arm}_record_sha256"],
+ "Cost proxy correctness or raw-record binding differs",
+ )
+ calls = proxy[f"{arm}_calls"]
+ _require(bool(calls), "Every training answer needs at least one priced physical call")
+ total = 0
+ for call in calls:
+ _require(
+ all(
+ type(call[key]) is int and call[key] >= 0
+ for key in ("prompt_tokens", "completion_tokens", "credits_proxy")
+ ),
+ "Token counts and cost proxies must be nonnegative integers",
+ )
+ charge = math.ceil(
+ call["prompt_tokens"] * _fraction(rates[arm]["input_tokens"])
+ + call["completion_tokens"] * _fraction(rates[arm]["output_tokens"])
+ )
+ _require(charge == call["credits_proxy"], "Physical-call cost formula or rounding differs")
+ total += charge
+ _require(
+ type(proxy[f"{arm}_credits_proxy"]) is int and total == proxy[f"{arm}_credits_proxy"],
+ "Question cost must sum every initial and finalizer call",
+ )
+ row_costs.append(total)
+ costs.append(row_costs)
+ targets = [[int(row[f"{arm}_correct"]) for arm in ARMS] for row in grades["rows"]]
+ statistics = conditional_costs(targets, costs)
+ _require(statistics == record["conditional_means"], "Training cost means disagree with recomputation")
+ return costs, statistics, record
+
+
+def probability(matrix, model):
+ matrix = np.asarray(matrix, dtype=float)
+ weights = np.asarray(model["weights"], dtype=float)
+ _require(
+ matrix.ndim == 2
+ and weights.shape == (matrix.shape[1],)
+ and np.isfinite(matrix).all()
+ and np.isfinite(weights).all()
+ and np.isfinite(model["bias"]),
+ "Outcome features or weights are invalid",
+ )
+ logits = matrix @ weights + model["bias"]
+ return np.exp(-np.logaddexp(0, -logits))
+
+
+def normalized_scores(probabilities, statistics):
+ probabilities = np.asarray(probabilities, dtype=float)
+ _require(
+ probabilities.ndim == 2
+ and probabilities.shape[1] == 2
+ and np.isfinite(probabilities).all()
+ and ((probabilities >= 0) & (probabilities <= 1)).all(),
+ "Need two finite probabilities per request",
+ )
+ expected = np.column_stack(
+ [
+ probabilities[:, arm] * statistics[f"{name}_correct"]["mean_credits_proxy"]
+ + (1 - probabilities[:, arm]) * statistics[f"{name}_wrong"]["mean_credits_proxy"]
+ for arm, name in enumerate(ARMS)
+ ]
+ )
+ _require(np.isfinite(expected).all() and (expected >= 0).all(), "Invalid conditional cost means")
+ delta = expected[:, 1] - expected[:, 0]
+ scores = (probabilities[:, 1] - probabilities[:, 0]) / np.maximum(delta, PROTOCOL["incremental_cost_floor_credits"])
+ return scores, expected
+
+
+def score(matrix, head):
+ """Return the fixed benefit-per-credit score; no SciPy or model download."""
+ probabilities = np.column_stack([probability(matrix, head["models"][arm]) for arm in ARMS])
+ return normalized_scores(probabilities, head["conditional_cost_statistics"])[0]
+
+
+def fit_logistic(matrix, targets, penalty):
+ # Optional training dependency; replay/live inference remain NumPy-only.
+ import scipy
+ from scipy.optimize import minimize
+ from scipy.special import expit
+
+ _require(scipy.__version__ == PROTOCOL["optimizer"]["scipy"], "Fit requires the declared SciPy version")
+ mean = matrix.mean(axis=0)
+ centered = np.ascontiguousarray(matrix - mean)
+ frequency = float(targets.mean())
+ _require(0 < frequency < 1, "Each training fold needs both correctness classes for each solver")
+ initial = np.zeros(matrix.shape[1] + 1)
+ initial[-1] = np.log(frequency / (1 - frequency))
+
+ def loss(theta):
+ weights, bias = theta[:-1], theta[-1]
+ logits = centered @ weights + bias
+ residual = expit(logits) - targets
+ value = np.logaddexp(0, logits).sum() - targets @ logits + 0.5 * penalty * (weights @ weights)
+ return float(value), np.r_[centered.T @ residual + penalty * weights, residual.sum()]
+
+ options = {key: PROTOCOL["optimizer"][key] for key in ("maxiter", "ftol", "gtol", "maxls")}
+ result = minimize(loss, initial, method="L-BFGS-B", jac=True, options=options)
+ _require(result.success and np.isfinite(result.x).all(), f"Logistic optimization failed: {result.message}")
+ return {
+ "weights": result.x[:-1].tolist(),
+ "bias": float(result.x[-1] - mean @ result.x[:-1]),
+ "optimizer": {
+ "success": bool(result.success),
+ "message": str(result.message),
+ "iterations": int(result.nit),
+ "function_evaluations": int(result.nfev),
+ "objective": float(result.fun),
+ "gradient_absolute_max": float(np.abs(result.jac).max()),
+ },
+ }
+
+
+def _nll(targets, probabilities):
+ probabilities = np.clip(probabilities, 1e-15, 1 - 1e-15)
+ return np.mean(-targets * np.log(probabilities) - (1 - targets) * np.log1p(-probabilities), axis=0)
+
+
+def fit_head(matrix, rows, groups, costs, *, progress=None, fold_assignments=None):
+ solver_report.summarize_quality(rows, "always_strong")
+ matrix = np.asarray(matrix, dtype=float)
+ targets = np.array([[int(row[f"{arm}_correct"]) for arm in ARMS] for row in rows], dtype=float)
+ _require(
+ matrix.ndim == 2 and len(matrix) == len(rows) == len(groups) and np.isfinite(matrix).all(),
+ "Training vectors, paired targets and paper groups must align",
+ )
+ folds = np.array(benefit.group_folds(groups) if fold_assignments is None else fold_assignments)
+ _require(
+ folds.shape == (len(rows),)
+ and set(folds.tolist()) == set(range(PROTOCOL["folds"]))
+ and all(len({int(folds[i]) for i, item in enumerate(groups) if item == group}) == 1 for group in set(groups)),
+ "Every training paper must stay in one of the declared folds",
+ )
+ statistics = conditional_costs(targets, costs)
+ candidates = []
+ for penalty in PROTOCOL["penalties"]:
+ predictions = np.empty_like(targets)
+ optimizers = []
+ for fold in range(PROTOCOL["folds"]):
+ held = folds == fold
+ for arm, name in enumerate(ARMS):
+ model = fit_logistic(matrix[~held], targets[~held, arm], penalty)
+ predictions[held, arm] = probability(matrix[held], model)
+ optimizers.append({"fold": fold, "arm": name, **model["optimizer"]})
+ losses = _nll(targets, predictions)
+ candidates.append(
+ {
+ "penalty": penalty,
+ "cheap_nll": float(losses[0]),
+ "strong_nll": float(losses[1]),
+ "summed_binary_logloss": float(losses.sum()),
+ "oof_probabilities": predictions.tolist(),
+ "optimizer_runs": optimizers,
+ }
+ )
+ if progress is not None:
+ progress(f"Training paper-fold CV lambda={penalty:g}: summed NLL={losses.sum():.6f}")
+ winner = min(candidates, key=lambda item: (item["summed_binary_logloss"], -item["penalty"]))
+ models = {name: fit_logistic(matrix, targets[:, arm], winner["penalty"]) for arm, name in enumerate(ARMS)}
+ return {
+ "training_ids": [row["id"] for row in rows],
+ "targets": targets.astype(int).tolist(),
+ "folds": folds.tolist(),
+ "cv": candidates,
+ "penalty": winner["penalty"],
+ "models": models,
+ "training_cost_proxies": costs,
+ "conditional_cost_statistics": statistics,
+ }
+
+
+def fit(
+ run,
+ baseline,
+ training_plan_path,
+ training_inputs_path,
+ training_rows_path,
+ grades_path,
+ embeddings,
+ cost_path,
+ price_path,
+ *,
+ progress=None,
+):
+ run = Path(run)
+ _unopened(run)
+ for name in ("selected-model.json", "frozen.json"):
+ if (run / name).exists():
+ raise FileExistsError(run / name)
+ embeddings = Path(embeddings)
+ plan = checked_plan(read(run / "plan.json"))
+ _baseline(plan, baseline)
+ rows, grades = _training(plan, training_plan_path, training_inputs_path, training_rows_path, grades_path)
+ costs, statistics, cost_record = cost_proxy(cost_path, price_path, training_plan_path, grades_path, grades)
+ matrix = vectors(rows, embeddings)
+ _require(matrix.shape == (len(rows), PROTOCOL["dimensions"]), "Training vectors differ from the pinned dimensions")
+ head = fit_head(
+ matrix, grades["rows"], [row["paper_group"] for row in plan["populations"]["train"]], costs, progress=progress
+ )
+ _require(head["conditional_cost_statistics"] == statistics, "Training cost statistics changed while fitting")
+ return benefit.seal(
+ {
+ "format": MODEL_FORMAT,
+ "selected_at": benefit.now(),
+ "plan": plan,
+ "plan_file_sha256": file_hash(run / "plan.json"),
+ "head": head,
+ "training_grades_sha256": file_hash(grades_path),
+ "training_grade_provenance": {key: value for key, value in grades.items() if key != "rows"},
+ "training_embeddings_sha256": {
+ name: file_hash(embeddings / name) for name in ("plan.json", "vectors.jsonl.gz")
+ },
+ "cost_proxy_sha256": file_hash(cost_path),
+ "price_snapshot_sha256": file_hash(price_path),
+ "cost_proxy_provenance": {key: value for key, value in cost_record.items() if key != "rows"},
+ "versions": {"numpy": np.__version__, "scipy": PROTOCOL["optimizer"]["scipy"]},
+ "boundary": "Only training outcomes used; this selected model must be written before calibration is opened",
+ }
+ )
+
+
+def checked_model(run):
+ run = Path(run)
+ model = benefit.checked(read(run / "selected-model.json"), MODEL_FORMAT)
+ plan = checked_plan(model["plan"])
+ _require(
+ model["plan_file_sha256"] == file_hash(run / "plan.json") and read(run / "plan.json") == plan,
+ "Selected outcome model differs from its prepared plan",
+ )
+ head = model["head"]
+ n = len(plan["populations"]["train"])
+ targets = np.asarray(head["targets"])
+ _require(
+ head["training_ids"] == [row["id"] for row in plan["populations"]["train"]]
+ and head["folds"] == plan["training_folds"]
+ and targets.shape == (n, 2)
+ and np.isin(targets, (0, 1)).all()
+ and head["conditional_cost_statistics"] == conditional_costs(targets, head["training_cost_proxies"])
+ and [item["penalty"] for item in head["cv"]] == PROTOCOL["penalties"],
+ "Selected outcome model changes training targets, costs, folds or candidates",
+ )
+ for candidate in head["cv"]:
+ probabilities = np.asarray(candidate["oof_probabilities"], dtype=float)
+ _require(
+ probabilities.shape == (n, 2)
+ and np.isfinite(probabilities).all()
+ and ((probabilities >= 0) & (probabilities <= 1)).all(),
+ "Invalid training-fold probabilities",
+ )
+ losses = _nll(targets, probabilities)
+ _require(
+ [candidate["cheap_nll"], candidate["strong_nll"], candidate["summed_binary_logloss"]]
+ == [float(losses[0]), float(losses[1]), float(losses.sum())]
+ and [(item["fold"], item["arm"]) for item in candidate["optimizer_runs"]]
+ == [(fold, arm) for fold in range(PROTOCOL["folds"]) for arm in ARMS]
+ and all(item["success"] is True for item in candidate["optimizer_runs"]),
+ "Training-only CV losses or optimizer completion differ",
+ )
+ winner = min(head["cv"], key=lambda item: (item["summed_binary_logloss"], -item["penalty"]))
+ _require(
+ head["penalty"] == winner["penalty"] and set(head["models"]) == set(ARMS),
+ "Selected penalty does not minimize the declared training-only objective",
+ )
+ for item in head["models"].values():
+ probability(np.zeros((1, PROTOCOL["dimensions"])), item)
+ _require(item["optimizer"]["success"] is True, "Full training refit did not converge")
+ _require(
+ all(
+ model["training_grade_provenance"].get(name) == plan["training_grade_provenance"][name]
+ for name in GRADE_HASHES
+ )
+ and model["cost_proxy_provenance"]["training_grades_sha256"] == model["training_grades_sha256"]
+ and model["cost_proxy_provenance"]["conditional_means"] == head["conditional_cost_statistics"],
+ "Selected model grade or cost provenance changed",
+ )
+ return model
+
+
+def _selections(rows, scores, credits, rate):
+ return {
+ str(target): solver_report.select_cutoff(rows, scores, target, router_credits=credits, credit_usd=rate)
+ for target in TARGETS
+ }
+
+
+def freeze(run, baseline, original_path):
+ run = Path(run)
+ _unopened(run)
+ if (run / "frozen.json").exists():
+ raise FileExistsError(run / "frozen.json")
+ model = checked_model(run)
+ plan = model["plan"]
+ # The selected training-only model already exists on disk before this
+ # function opens the original frozen calibration outcomes.
+ _require(file_hash(original_path) == plan["original_frozen_sha256"], "Original calibration study changed")
+ dataset, original, _, artifacts = benefit._inputs(baseline, original_path)
+ _require(
+ digest(dataset) == plan["dataset_sha256"] and artifacts == plan["baseline_artifacts_sha256"],
+ "Held inputs or vectors changed after preparing the outcome experiment",
+ )
+ scores, probabilities, expected_costs = {}, {}, {}
+ for split in ("calibration", "test"):
+ rows = held_rows(plan, dataset, split)
+ matrix = vectors(rows, Path(baseline) / ("test-embeddings" if split == "test" else "development-embeddings"))
+ probabilities[split] = np.column_stack(
+ [probability(matrix, model["head"]["models"][arm]) for arm in ARMS]
+ ).tolist()
+ values, costs = normalized_scores(probabilities[split], model["head"]["conditional_cost_statistics"])
+ scores[split], expected_costs[split] = values.tolist(), costs.tolist()
+ selections = _selections(
+ original["calibration_rows"],
+ scores["calibration"],
+ original["embedding_calibration_credits"],
+ original["credit_usd"],
+ )
+ return benefit.seal(
+ {
+ "format": FROZEN_FORMAT,
+ "frozen_at": benefit.now(),
+ "plan_file_sha256": file_hash(run / "plan.json"),
+ "selected_model_sha256": file_hash(run / "selected-model.json"),
+ "scores": scores,
+ "probabilities": probabilities,
+ "expected_costs_proxy": expected_costs,
+ "calibration_rows": original["calibration_rows"],
+ "calibration_grades_sha256": original["calibration_grades_sha256"],
+ "grading_provenance": grading_provenance(original),
+ "embedding_calibration_credits": original["embedding_calibration_credits"],
+ "embedding_calibration_credits_sha256": original["embedding_calibration_credits_sha256"],
+ "credit_usd": original["credit_usd"],
+ "selections": selections,
+ "test_access": PROTOCOL["test_boundary"],
+ }
+ )
+
+
+def checked_frozen(run):
+ run = Path(run)
+ model = checked_model(run)
+ plan = model["plan"]
+ frozen = benefit.checked(read(run / "frozen.json"), FROZEN_FORMAT)
+ _require(
+ frozen["plan_file_sha256"] == model["plan_file_sha256"]
+ and frozen["selected_model_sha256"] == file_hash(run / "selected-model.json")
+ and frozen["grading_provenance"] == plan["grading_provenance"],
+ "Frozen outcome lifecycle hashes differ",
+ )
+ for split in ("calibration", "test"):
+ _require(
+ np.asarray(frozen["probabilities"][split]).shape == (len(plan["populations"][split]), 2),
+ "Frozen held probabilities are misaligned",
+ )
+ scores, costs = normalized_scores(frozen["probabilities"][split], model["head"]["conditional_cost_statistics"])
+ _require(
+ frozen["scores"][split] == scores.tolist() and frozen["expected_costs_proxy"][split] == costs.tolist(),
+ "Frozen cost-normalized scores differ from their probabilities and training costs",
+ )
+ _require(
+ [row["id"] for row in frozen["calibration_rows"]] == [row["id"] for row in plan["populations"]["calibration"]]
+ and frozen["selections"]
+ == _selections(
+ frozen["calibration_rows"],
+ frozen["scores"]["calibration"],
+ frozen["embedding_calibration_credits"],
+ frozen["credit_usd"],
+ ),
+ "Frozen cutoffs differ from calibration-only cost selection",
+ )
+ return frozen, model
+
+
+def authorize_test(run):
+ _unopened(run)
+ frozen, model = checked_frozen(run)
+ return benefit.seal(
+ {
+ "format": ACCESS_FORMAT,
+ "authorized_at": benefit.now(),
+ "frozen_sha256": file_hash(Path(run) / "frozen.json"),
+ "selected_model_sha256": frozen["selected_model_sha256"],
+ "plan_sha256": model["plan_file_sha256"],
+ "source_code_sha256": code_hashes(),
+ "statement": "All outcome models and cutoffs froze before this receipt authorizes solver test outcome access",
+ }
+ )
+
+
+def report(run, grades_path, router_credits_path=None):
+ run = Path(run)
+ frozen, model = checked_frozen(run)
+ access = benefit.checked(read(run / "test-access.json"), ACCESS_FORMAT)
+ _require(
+ access["frozen_sha256"] == file_hash(run / "frozen.json")
+ and access["selected_model_sha256"] == frozen["selected_model_sha256"]
+ and access["plan_sha256"] == model["plan_file_sha256"]
+ and access["source_code_sha256"] == code_hashes(),
+ "Test-access receipt does not bind this frozen outcome model",
+ )
+ plan = model["plan"]
+ ids = [row["id"] for row in plan["populations"]["test"]]
+ grades = solver_policies._grade_rows(grades_path, "test", plan["original_solver_plan_sha256"], ids)
+ _require(
+ grades["policies_frozen_sha256"] == plan["original_frozen_sha256"]
+ and grading_provenance(grades) == frozen["grading_provenance"],
+ "Test answers differ from the original solver and grading study",
+ )
+ credits, credit_sha = test_credits(plan, router_credits_path)
+ measured = {}
+ for target in TARGETS:
+ selection = frozen["selections"][str(target)]
+ mode = selection["mode"]
+ routes = (np.array(frozen["scores"]["test"]) >= selection["cutoff"]).tolist() if mode == "threshold" else mode
+ measured[str(target)] = {
+ "mode": mode,
+ "cutoff": selection["cutoff"],
+ "metrics": solver_report.summarize(
+ grades["rows"],
+ routes,
+ router_credits=credits if mode == "threshold" else 0,
+ credit_usd=frozen["credit_usd"],
+ ),
+ }
+ return {
+ "format": "sie-solver-outcome-report-v1",
+ "n": len(ids),
+ "ids": ids,
+ "training_pool_rows": len(plan["training_pool"]),
+ "training_rows": len(model["head"]["training_ids"]),
+ "training_papers": len({row["paper_group"] for row in plan["populations"]["train"]}),
+ "selected_penalty": model["head"]["penalty"],
+ "frozen_sha256": file_hash(run / "frozen.json"),
+ "test_access_sha256": file_hash(run / "test-access.json"),
+ "test_grades_sha256": file_hash(grades_path),
+ "embedding_test_credits_sha256": credit_sha,
+ "primary_max_quality_loss_pp": 1,
+ "baselines": {
+ name: solver_report.summarize(grades["rows"], name, router_credits=0, credit_usd=frozen["credit_usd"])
+ for name in ("always_strong", "always_cheap")
+ },
+ "policies": measured,
+ "interpretation": "Two outcome heads and training-only cost proxies; calibration targets are empirical, not test guarantees. Expansion adds within-paper examples. Acquisition/training costs are separate from serving costs; predicted training cost proxies are never reported as measured bills.",
+ }
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ commands = parser.add_subparsers(dest="command", required=True)
+ for command in ("prepare", "fit", "freeze", "authorize-test", "report"):
+ child = commands.add_parser(command)
+ child.add_argument("--run", type=Path, required=True)
+ if command in ("prepare", "fit", "freeze"):
+ child.add_argument("--baseline", type=Path, required=True)
+ if command in ("prepare", "freeze"):
+ child.add_argument("--original-frozen", type=Path, required=True)
+ if command in ("prepare", "fit"):
+ for name in ("training-plan", "training-inputs", "training-rows"):
+ child.add_argument(f"--{name}", type=Path, required=True)
+ if command == "prepare":
+ child.add_argument("--eligibility", type=Path, required=True)
+ child.add_argument("--expansion-audit", type=Path)
+ child.add_argument("--source-review", type=Path)
+ child.add_argument("--attest-test-outcomes-unopened", action="store_true", required=True)
+ if command == "fit":
+ for name in ("training-grades", "training-embeddings", "cost-proxy", "price-snapshot"):
+ child.add_argument(f"--{name}", type=Path, required=True)
+ if command == "report":
+ child.add_argument("--test-grades", type=Path, required=True)
+ child.add_argument("--router-credits", type=Path)
+ args = parser.parse_args()
+ names = {
+ "prepare": "plan.json",
+ "fit": "selected-model.json",
+ "freeze": "frozen.json",
+ "authorize-test": "test-access.json",
+ "report": "report.json",
+ }
+ destination = args.run / names[args.command]
+ if destination.exists() or (args.command == "prepare" and args.run.exists()):
+ raise FileExistsError(destination)
+ if args.command == "prepare":
+ result = prepare(
+ args.baseline,
+ args.original_frozen,
+ args.training_plan,
+ args.training_inputs,
+ args.training_rows,
+ args.eligibility,
+ expansion_audit_path=args.expansion_audit,
+ source_review_path=args.source_review,
+ attest_test_outcomes_unopened=args.attest_test_outcomes_unopened,
+ )
+ elif args.command == "fit":
+ result = fit(
+ args.run,
+ args.baseline,
+ args.training_plan,
+ args.training_inputs,
+ args.training_rows,
+ args.training_grades,
+ args.training_embeddings,
+ args.cost_proxy,
+ args.price_snapshot,
+ progress=lambda message: print(message, flush=True),
+ )
+ elif args.command == "freeze":
+ result = freeze(args.run, args.baseline, args.original_frozen)
+ elif args.command == "authorize-test":
+ result = authorize_test(args.run)
+ else:
+ result = report(args.run, args.test_grades, args.router_credits)
+ write(destination, result)
+ if args.command == "report":
+ metrics = result["policies"]["1"]["metrics"]
+ print(
+ f"Outcome router: accuracy {100 * metrics['answer_accuracy']:.2f}%; gap {metrics['gap_vs_strong_pp']:+.2f}pp"
+ )
+ if "net_savings_pct" in metrics:
+ net = "n/a" if metrics["net_savings_pct"] is None else f"{metrics['net_savings_pct']:.2f}%"
+ print(f"Including encoder: {metrics['net_total_cost_usd_per1000']:.4f} USD/1,000; net savings {net}")
+ else:
+ print("Router overhead unpriced; net savings unknown")
+ print(json.dumps({"output": str(destination), "sha256": file_hash(destination)}))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/benefit_outcomes_native.py b/examples/train-a-router/benefit_outcomes_native.py
new file mode 100644
index 000000000..a15ddb707
--- /dev/null
+++ b/examples/train-a-router/benefit_outcomes_native.py
@@ -0,0 +1,1000 @@
+"""Train one Laya checkpoint to predict both solvers' answer correctness.
+
+The two questions share encoder and decision-head weights. Each source request
+produces two items. CUDA training and calibration scoring are separate commands;
+preparation, receipt validation and recorded-result replay need only NumPy.
+"""
+
+from __future__ import annotations
+
+import argparse
+import importlib.metadata
+import json
+import os
+import random
+import shutil
+import tempfile
+import time
+from dataclasses import asdict
+from pathlib import Path
+
+import numpy as np
+
+import benefit
+import benefit_outcomes as outcomes
+import finetune
+import solver_policies
+import solver_report
+from data import digest, file_hash, write
+
+QUESTION_ORDER = ("cheap_correct", "strong_correct")
+OPTIONS = ("incorrect", "correct")
+QUESTIONS = {
+ "cheap_correct": {
+ "type": "choice",
+ "instructions": "Predict whether the cheaper Qwen3.5-4B solver will answer this mathematics problem correctly. Judge its likely answer quality; do not solve the problem.",
+ "criteria": {
+ "incorrect": "The solver will give an incorrect answer.",
+ "correct": "The solver will give the correct answer.",
+ },
+ },
+ "strong_correct": {
+ "type": "choice",
+ "instructions": "Predict whether the stronger Qwen3.8-27B solver will answer this mathematics problem correctly. Judge its likely answer quality; do not solve the problem.",
+ "criteria": {
+ "incorrect": "The solver will give an incorrect answer.",
+ "correct": "The solver will give the correct answer.",
+ },
+ },
+}
+PROTOCOL = {
+ "base": finetune.BASE,
+ "revision": finetune.REVISION,
+ "recipe": finetune.LORA_RECIPES["original"],
+ "seed": 89,
+ "question_order": list(QUESTION_ORDER),
+ "option_order": list(OPTIONS),
+ "target": "Two hard Bernoulli outcomes per request, with incorrect/correct probabilities [1-y,y]",
+ "architecture": "One shared Laya encoder and native decision head; rank16 attention LoRA plus the full native head",
+ "selection_fold": 0,
+ "selection_epochs": 8,
+ "selection": "Minimum mean per-request sum of cheap and strong binary NLL on the training-only paper fold0; include epoch0, ties earlier",
+ "refit": "Fresh base and seed89; all eligible requests for the selected epoch count; cosine schedule spans that refit duration; epoch0 skips training",
+ "batch_unit": "One question-item, two items per source request; effective batch16 items with the unchanged CUDA profile",
+ "runtime": finetune.runtime_settings("cuda"),
+ "training_device": "cuda",
+ "max_len": finetune.MAX_LEN,
+ "head_max_len": finetune.HEAD_LEN,
+ "export_dtype": "float32",
+ "max_merge_margin_error": 0.001,
+ "max_export_margin_error": 0.03,
+ "eager_flags": {"TORCH_COMPILE_DISABLE": "1", "TORCHDYNAMO_DISABLE": "1"},
+ "versions": {"laya": "0.3.29", "torch": "2.9.1", "transformers": "4.57.6", "peft": "0.18.1"},
+ "cost_model": outcomes.PROTOCOL["cost_model"],
+ "score": outcomes.PROTOCOL["score"],
+ "primary_max_quality_loss_pp": 1,
+ "secondary_max_quality_loss_pp": [0, 2, 5],
+ "test_boundary": outcomes.PROTOCOL["test_boundary"],
+ "hosting_cost": "Native hosting unpriced; compare gross solver savings and break-even router cost, never assume native inference is free",
+}
+PLAN_FORMAT = "sie-solver-outcome-native-plan-v2"
+TRAINED_FORMAT = "sie-solver-outcome-native-trained-v1"
+FROZEN_FORMAT = "sie-solver-outcome-native-frozen-v1"
+ACCESS_FORMAT = "sie-solver-outcome-native-access-v1"
+# This revision differs only in replay validation: NumPy CPU kernels can round
+# float64 exp/logaddexp by a few ulps. Historical model/score/threshold bytes and
+# every other source hash stay authoritative. Never use this exception to train
+# or freeze under a historical plan with different executing source bytes.
+REPLAY_COMPATIBLE_NATIVE_SOURCES = frozenset({"611bc5c09e5d854c5a1528b683ac7eeca583c00a1e41812ec1e8f902e3e4f68a"})
+REPLAY_RTOL = 1e-12
+REPLAY_ATOL = 1e-15
+
+
+def code_hashes():
+ return outcomes.code_hashes() | {
+ name: file_hash(Path(__file__).with_name(name)) for name in ("benefit_outcomes_native.py", "finetune.py")
+ }
+
+
+def _source_matches(recorded, *, replay=False):
+ current = code_hashes()
+ if recorded == current:
+ return True
+ name = "benefit_outcomes_native.py"
+ return (
+ replay
+ and isinstance(recorded, dict)
+ and set(recorded) == set(current)
+ and recorded[name] in REPLAY_COMPATIBLE_NATIVE_SOURCES
+ and all(recorded[key] == value for key, value in current.items() if key != name)
+ )
+
+
+def _matches_recorded(recorded, recomputed):
+ """Validate derived float64 arithmetic, without replacing saved values."""
+ try:
+ saved, checked = np.asarray(recorded, dtype=float), np.asarray(recomputed, dtype=float)
+ except (TypeError, ValueError):
+ return False
+ return (
+ saved.shape == checked.shape
+ and saved.size > 0
+ and np.isfinite(saved).all()
+ and np.isfinite(checked).all()
+ and np.allclose(saved, checked, rtol=REPLAY_RTOL, atol=REPLAY_ATOL)
+ )
+
+
+def replay_validation(plan):
+ return {
+ "format": "sie-native-float64-replay-validation-v1",
+ "training_source_code_sha256": plan["source_code_sha256"],
+ "validator_source_code_sha256": code_hashes(),
+ "known_historical_native_source": plan["source_code_sha256"] != code_hashes(),
+ "rtol": REPLAY_RTOL,
+ "atol": REPLAY_ATOL,
+ "authoritative_values": "Stored scores, cutoffs, selection losses and exact tie decisions; arithmetic replay never replaces them",
+ }
+
+
+def _unopened(run, head_run):
+ outcomes._unopened(run)
+ outcomes._unopened(head_run)
+
+
+def prepare(head_run):
+ head_run = Path(head_run)
+ outcomes._unopened(head_run)
+ head = outcomes.checked_plan(outcomes.read(head_run / "plan.json"))
+ rows = head["populations"]["train"]
+ return benefit.seal(
+ {
+ "format": PLAN_FORMAT,
+ "created_at": benefit.now(),
+ "protocol": PROTOCOL,
+ "questions": QUESTIONS,
+ "head_plan": head,
+ "head_plan_sha256": file_hash(head_run / "plan.json"),
+ "source_code_sha256": code_hashes(),
+ "selection_fit_ids": [
+ row["id"] for row, fold in zip(rows, head["training_folds"], strict=True) if fold != 0
+ ],
+ "selection_validation_ids": [
+ row["id"] for row, fold in zip(rows, head["training_folds"], strict=True) if fold == 0
+ ],
+ }
+ )
+
+
+def checked_plan(run, head_run, *, replay=False):
+ plan = benefit.checked(outcomes.read(Path(run) / "plan.json"), PLAN_FORMAT)
+ head = outcomes.checked_plan(plan["head_plan"])
+ outcomes._require(
+ plan["protocol"] == PROTOCOL
+ and plan["questions"] == QUESTIONS
+ and _source_matches(plan["source_code_sha256"], replay=replay)
+ and plan["head_plan_sha256"] == file_hash(Path(head_run) / "plan.json")
+ and outcomes.read(Path(head_run) / "plan.json") == head
+ and all(
+ plan[key]
+ == [
+ row["id"]
+ for row, fold in zip(head["populations"]["train"], head["training_folds"], strict=True)
+ if (fold == 0) == held
+ ]
+ for key, held in (("selection_fit_ids", False), ("selection_validation_ids", True))
+ ),
+ "Native protocol, source, outcome plan or whole-paper selection fold changed",
+ )
+ return plan
+
+
+def _targets(targets, count=None):
+ values = np.asarray(targets, dtype=float)
+ outcomes._require(
+ values.ndim == 2
+ and values.shape[1] == 2
+ and len(values) > 0
+ and (count is None or len(values) == count)
+ and np.isfinite(values).all()
+ and np.isin(values, (0, 1)).all(),
+ "Need aligned cheap/strong binary targets",
+ )
+ return values
+
+
+def question_rows(rows, targets=None):
+ y = np.zeros((len(rows), 2)) if targets is None else _targets(targets, len(rows))
+ return [
+ {
+ "state": row["request"],
+ "questions": QUESTIONS,
+ "gold": {
+ question: {"probabilities": {"incorrect": float(1 - labels[index]), "correct": float(labels[index])}}
+ for index, question in enumerate(QUESTION_ORDER)
+ },
+ }
+ for row, labels in zip(rows, y, strict=True)
+ ]
+
+
+def encode_rows(lt, tokenizer, rows, targets=None):
+ y = np.zeros((len(rows), 2)) if targets is None else _targets(targets, len(rows))
+ items, skipped = lt.items_from_rows(
+ tokenizer, question_rows(rows, y), finetune.MAX_LEN, finetune.HEAD_LEN, label_smoothing=0.0
+ )
+ outcomes._require(not skipped and len(items) == 2 * len(rows), f"Two native items per request required: {skipped}")
+ for row_index, row in enumerate(rows):
+ state = lt.encode_state(tokenizer, row["request"], finetune.MAX_LEN)
+ for arm, question in enumerate(QUESTION_ORDER):
+ item = items[2 * row_index + arm]
+ outcomes._require(
+ item["state_ids"] == state
+ and item["q"]["ins"] == QUESTIONS[question]["instructions"]
+ and list(item["q"]["crit"]) == list(OPTIONS)
+ and item["k"] == 2
+ and item["target"] == [float(1 - y[row_index, arm]), float(y[row_index, arm])],
+ "Native item request, question or incorrect/correct target order changed",
+ )
+ return items
+
+
+def records_to_margins(records, expected_targets):
+ expected = _targets(expected_targets)
+ logits = np.asarray([record[1] for record in records], dtype=float)
+ targets = np.asarray([record[2] for record in records], dtype=float)
+ flat = expected.reshape(-1)
+ outcomes._require(
+ logits.shape == (2 * len(expected), 2)
+ and targets.shape == logits.shape
+ and all(record[3] == 2 for record in records)
+ and np.isfinite(logits).all()
+ and np.array_equal(targets, np.column_stack([1 - flat, flat])),
+ "Native calibration records changed request/question/option target order",
+ )
+ return (logits[:, 1] - logits[:, 0]).reshape(len(expected), 2)
+
+
+def probabilities(margins):
+ values = np.asarray(margins, dtype=float)
+ outcomes._require(
+ values.ndim == 2 and values.shape[1] == 2 and len(values) > 0 and np.isfinite(values).all(),
+ "Need finite native margins ordered cheap then strong",
+ )
+ return np.exp(-np.logaddexp(0, -values))
+
+
+def outcome_nll(margins, targets):
+ values, y = np.asarray(margins, dtype=float), _targets(targets)
+ probabilities(values)
+ outcomes._require(values.shape == y.shape, "Native NLL targets do not align")
+ return float((np.logaddexp(0, values) - y * values).sum(axis=1).mean())
+
+
+def selected_epoch(history, targets):
+ y = _targets(targets)
+ outcomes._require(
+ [row["epoch"] for row in history] == list(range(PROTOCOL["selection_epochs"] + 1)),
+ "Retain epoch0 and every declared native selection epoch",
+ )
+ losses = [row["summed_binary_nll"] for row in history]
+ recomputed = [outcome_nll(row["margins"], y) for row in history]
+ outcomes._require(
+ _matches_recorded(losses, recomputed),
+ "Preserved native margins do not reproduce the outcome NLL",
+ )
+ return min(range(len(losses)), key=lambda epoch: (losses[epoch], epoch))
+
+
+def _config_values(epochs):
+ runtime, recipe = PROTOCOL["runtime"], PROTOCOL["recipe"]
+ return {
+ "epochs": epochs,
+ "micro_batch": runtime["micro_batch"],
+ "grad_accum": runtime["grad_accum"],
+ "encoder_lr": recipe["encoder_lr"],
+ "head_lr": recipe["head_lr"],
+ "weight_decay": recipe["weight_decay"],
+ "seed": PROTOCOL["seed"],
+ "max_len": finetune.MAX_LEN,
+ "head_max_len": finetune.HEAD_LEN,
+ "loss": "soft-ce",
+ "shuffle_options": ("choice",),
+ "gradient_checkpointing": runtime["gradient_checkpointing"],
+ "label_smoothing": 0.0,
+ "log_every": 32,
+ }
+
+
+def preflight(run, head_run, baseline, training_plan_path, inputs_path, rows_path, grades_path, cost_path, price_path):
+ run = Path(run)
+ _unopened(run, head_run)
+ for name in ("training-start.json", "trained.json", "frozen.json"):
+ if (run / name).exists():
+ raise FileExistsError(run / name)
+ plan = checked_plan(run, head_run)
+ head = plan["head_plan"]
+ outcomes._baseline(head, baseline)
+ rows, grades = outcomes._training(head, training_plan_path, inputs_path, rows_path, grades_path)
+ costs, statistics, proxy = outcomes.cost_proxy(cost_path, price_path, training_plan_path, grades_path, grades)
+ targets = _targets([[int(row[question]) for question in QUESTION_ORDER] for row in grades["rows"]])
+ return {
+ "plan": plan,
+ "rows": rows,
+ "targets": targets,
+ "held": np.array(head["training_folds"]) == 0,
+ "costs": costs,
+ "statistics": statistics,
+ "cost_proxy_provenance": {key: value for key, value in proxy.items() if key != "rows"},
+ "training_grade_provenance": {key: value for key, value in grades.items() if key != "rows"},
+ "training_artifacts_sha256": {
+ "plan": file_hash(training_plan_path),
+ "inputs": file_hash(inputs_path),
+ "rows": file_hash(rows_path),
+ "grades": file_hash(grades_path),
+ "cost_proxy": file_hash(cost_path),
+ "price_snapshot": file_hash(price_path),
+ },
+ }
+
+
+def train(
+ run,
+ head_run,
+ baseline,
+ training_plan_path,
+ inputs_path,
+ rows_path,
+ grades_path,
+ cost_path,
+ price_path,
+ *,
+ device="cuda",
+ local_files_only=False,
+):
+ outcomes._require(device == "cuda", "This matched study trains on CUDA only")
+ context = preflight(
+ run, head_run, baseline, training_plan_path, inputs_path, rows_path, grades_path, cost_path, price_path
+ )
+ return _train_cuda(Path(run), Path(head_run), context, local_files_only)
+
+
+def _train_cuda(run, head_run, context, local_files_only):
+ os.environ.update(PROTOCOL["eager_flags"])
+ import laya.train as lt
+ import torch
+ from huggingface_hub import snapshot_download
+ from peft import LoraConfig, get_peft_model
+ from safetensors.torch import save_file
+
+ versions = {name: importlib.metadata.version(name) for name in PROTOCOL["versions"]}
+ outcomes._require(
+ {name: value.split("+", 1)[0] for name, value in versions.items()} == PROTOCOL["versions"]
+ and torch.cuda.is_available(),
+ "Pinned native dependencies and an available CUDA GPU are required",
+ )
+ device, runtime, recipe = torch.device("cuda"), PROTOCOL["runtime"], PROTOCOL["recipe"]
+ write(
+ run / "training-start.json",
+ {
+ "plan_sha256": file_hash(run / "plan.json"),
+ "started_at": benefit.now(),
+ "versions": versions,
+ "device": "cuda",
+ "runtime": runtime,
+ "eager_flags": PROTOCOL["eager_flags"],
+ "training_artifacts_sha256": context["training_artifacts_sha256"],
+ },
+ )
+ started = time.monotonic()
+ print("Loading pinned Laya checkpoint for two-outcome training", flush=True)
+ snapshot = snapshot_download(
+ finetune.BASE,
+ revision=finetune.REVISION,
+ allow_patterns=["rl_agent_config.json", "model.safetensors", "encoder/*", "tokenizer/*"],
+ local_files_only=local_files_only,
+ )
+ targets, held = context["targets"], context["held"]
+ fit_rows = [row for row, flag in zip(context["rows"], held, strict=True) if not flag]
+ held_rows = [row for row, flag in zip(context["rows"], held, strict=True) if flag]
+ history, refit_losses = [], []
+
+ def margins(model, tokenizer, encoded, y):
+ records = lt.calibration_records(
+ model,
+ tokenizer,
+ encoded,
+ device,
+ finetune.MAX_LEN,
+ finetune.HEAD_LEN,
+ batch_size=runtime["evaluation_batch"],
+ )
+ return records_to_margins(records, y)
+
+ with tempfile.TemporaryDirectory(prefix=".outcome-initial-", dir=run) as temporary:
+ initial = Path(temporary)
+ for name in ("rl_agent_config.json", "model.safetensors"):
+ shutil.copy2(Path(snapshot) / name, initial / name)
+ for name in ("encoder", "tokenizer"):
+ shutil.copytree(Path(snapshot) / name, initial / name, symlinks=False)
+ initial_sha = file_hash(initial / "model.safetensors")
+
+ def load(with_lora):
+ torch.manual_seed(PROTOCOL["seed"])
+ np.random.seed(PROTOCOL["seed"])
+ random.seed(PROTOCOL["seed"])
+ model, tokenizer, config = lt.load_checkpoint(str(initial))
+ config.update(max_len=finetune.MAX_LEN, head_max_len=finetune.HEAD_LEN)
+ model.encoder.config.reference_compile = False
+ if with_lora:
+ model.encoder = get_peft_model(
+ model.encoder,
+ LoraConfig(
+ r=recipe["rank"],
+ lora_alpha=2 * recipe["rank"],
+ lora_dropout=recipe["dropout"],
+ target_modules=finetune.lora_targets(recipe),
+ bias="none",
+ ),
+ )
+ return model.to(device), tokenizer, config
+
+ model, tokenizer, config = load(True)
+ fitting = encode_rows(lt, tokenizer, fit_rows, targets[~held])
+ validation = encode_rows(lt, tokenizer, held_rows, targets[held])
+ selection_config = lt.TrainConfig(**_config_values(PROTOCOL["selection_epochs"]))
+ write(run / "selection-config.json", asdict(selection_config))
+
+ def observe(epoch, loss, model=model, tokenizer=tokenizer, validation=validation):
+ values = margins(model, tokenizer, validation, targets[held])
+ entry = {
+ "epoch": epoch + 1,
+ "train_loss": loss,
+ "margins": values.tolist(),
+ "summed_binary_nll": outcome_nll(values, targets[held]),
+ }
+ history.append(entry)
+ finetune.write(run / "selection-history.json", history)
+ print(json.dumps({key: entry[key] for key in ("epoch", "train_loss", "summed_binary_nll")}), flush=True)
+ model.train()
+
+ observe(-1, None)
+ lt.train_model(
+ model,
+ tokenizer,
+ fitting,
+ selection_config,
+ device,
+ finetune.MAX_LEN,
+ finetune.HEAD_LEN,
+ on_epoch_end=observe,
+ )
+ observe = None
+ epoch = selected_epoch(history, targets[held])
+ write(
+ run / "epoch-selected.json",
+ {
+ "epoch": epoch,
+ "history_sha256": file_hash(run / "selection-history.json"),
+ "validation_ids": context["plan"]["selection_validation_ids"],
+ "targets": targets[held].astype(int).tolist(),
+ "criterion": PROTOCOL["selection"],
+ },
+ )
+ del model, tokenizer, fitting, validation
+ finetune.empty_cache(torch, device)
+ checkpoints = {}
+ for variant in ("base", "lora"):
+ print(f"Preparing {variant} checkpoint; selected refit epochs={epoch}", flush=True)
+ model, tokenizer, config = load(variant == "lora")
+ validation = encode_rows(lt, tokenizer, held_rows, targets[held])
+ if variant == "lora":
+ refit_config = lt.TrainConfig(**_config_values(epoch))
+ write(run / "refit-config.json", asdict(refit_config))
+ if epoch:
+ fitting = encode_rows(lt, tokenizer, context["rows"], targets)
+ refit_losses = lt.train_model(
+ model, tokenizer, fitting, refit_config, device, finetune.MAX_LEN, finetune.HEAD_LEN
+ )
+ del fitting
+ before = margins(model, tokenizer, validation, targets[held])
+ merged = before
+ if variant == "lora":
+ model.encoder = model.encoder.merge_and_unload()
+ merged = margins(model, tokenizer, validation, targets[held])
+ merge_error = float(np.max(np.abs(merged - before)))
+ outcomes._require(
+ merge_error <= PROTOCOL["max_merge_margin_error"], "LoRA merge exceeded the margin tolerance"
+ )
+ destination = run / variant
+ lt.save_checkpoint(model, tokenizer, config, str(destination))
+ save_file(
+ {name: tensor.detach().float().cpu().contiguous() for name, tensor in model.state_dict().items()},
+ str(destination / "model.safetensors"),
+ )
+ del model
+ finetune.empty_cache(torch, device)
+ model, tokenizer, _ = lt.load_checkpoint(str(destination))
+ model.encoder.config.reference_compile = False
+ model.to(device).eval()
+ exported = margins(model, tokenizer, validation, targets[held])
+ export_error = float(np.max(np.abs(exported - before)))
+ outcomes._require(
+ export_error <= PROTOCOL["max_export_margin_error"], "FP32 export exceeded the margin tolerance"
+ )
+ checkpoints[variant] = {
+ "files_sha256": {
+ str(path.relative_to(destination)): file_hash(path)
+ for path in sorted(destination.rglob("*"))
+ if path.is_file()
+ },
+ "parameters": sum(parameter.numel() for parameter in model.parameters()),
+ "reference_margins": before.tolist(),
+ "merged_margins": merged.tolist(),
+ "exported_margins": exported.tolist(),
+ "max_merge_margin_error": merge_error,
+ "max_export_margin_error": export_error,
+ }
+ del model, tokenizer, validation
+ finetune.empty_cache(torch, device)
+ _unopened(run, head_run)
+ finetune.synchronize(torch, device)
+ trained = benefit.seal(
+ {
+ "format": TRAINED_FORMAT,
+ "completed_at": benefit.now(),
+ "plan_sha256": file_hash(run / "plan.json"),
+ "head_plan_sha256": context["plan"]["head_plan_sha256"],
+ "training_start_sha256": file_hash(run / "training-start.json"),
+ "training_artifacts_sha256": context["training_artifacts_sha256"],
+ "training_grade_provenance": context["training_grade_provenance"],
+ "initial_checkpoint_sha256": initial_sha,
+ "training_targets": targets.astype(int).tolist(),
+ "training_cost_proxies": context["costs"],
+ "conditional_cost_statistics": context["statistics"],
+ "cost_proxy_provenance": context["cost_proxy_provenance"],
+ "selection_history": history,
+ "selected_epoch": epoch,
+ "refit_losses": refit_losses,
+ "checkpoints": checkpoints,
+ "receipts_sha256": {
+ name: file_hash(run / name)
+ for name in (
+ "selection-config.json",
+ "selection-history.json",
+ "epoch-selected.json",
+ "refit-config.json",
+ )
+ },
+ "elapsed_seconds": time.monotonic() - started,
+ "boundary": "Training and FP32 checkpoint exports complete; no calibration outcomes consumed by native training",
+ }
+ )
+ write(run / "trained.json", trained)
+ print(
+ json.dumps(
+ {
+ "trained": str(run / "trained.json"),
+ "selected_epoch": epoch,
+ "elapsed_seconds": trained["elapsed_seconds"],
+ }
+ ),
+ flush=True,
+ )
+ return trained
+
+
+def _manifest(run, variant, manifest, verify_weights):
+ outcomes._require(
+ {"model.safetensors", "rl_agent_config.json"}.issubset(manifest), "Incomplete native checkpoint manifest"
+ )
+ for relative, expected in manifest.items():
+ path = Path(relative)
+ outcomes._require(
+ not path.is_absolute() and ".." not in path.parts and outcomes._sha(expected),
+ "Native checkpoint manifest has an invalid path or hash",
+ )
+ if verify_weights:
+ outcomes._require(file_hash(Path(run) / variant / path) == expected, "Native checkpoint bytes changed")
+
+
+def checked_trained(run, head_run, *, weights=False, replay=True):
+ run = Path(run)
+ plan = checked_plan(run, head_run, replay=replay)
+ head = plan["head_plan"]
+ trained = benefit.checked(outcomes.read(run / "trained.json"), TRAINED_FORMAT)
+ start = outcomes.read(run / "training-start.json")
+ targets = _targets(trained["training_targets"], len(head["populations"]["train"]))
+ held = np.asarray(head["training_folds"]) == 0
+ epoch = selected_epoch(trained["selection_history"], targets[held])
+ outcomes._require(
+ trained["plan_sha256"] == file_hash(run / "plan.json")
+ and trained["head_plan_sha256"] == plan["head_plan_sha256"]
+ and trained["training_start_sha256"] == file_hash(run / "training-start.json")
+ and start["plan_sha256"] == trained["plan_sha256"]
+ and start["training_artifacts_sha256"] == trained["training_artifacts_sha256"]
+ and start["device"] == "cuda"
+ and start["runtime"] == PROTOCOL["runtime"]
+ and start["eager_flags"] == PROTOCOL["eager_flags"]
+ and {name: value.split("+", 1)[0] for name, value in start["versions"].items()} == PROTOCOL["versions"]
+ and trained["selected_epoch"] == epoch
+ and len(trained["refit_losses"]) == epoch
+ and np.isfinite(trained["refit_losses"]).all()
+ and set(trained["checkpoints"]) == {"base", "lora"}
+ and trained["conditional_cost_statistics"]
+ == outcomes.conditional_costs(targets, trained["training_cost_proxies"])
+ and all(
+ trained["training_grade_provenance"].get(name) == head["training_grade_provenance"][name]
+ for name in outcomes.GRADE_HASHES
+ ),
+ "Native training lifecycle, targets, costs or runtime changed",
+ )
+ outcomes._require(
+ all(
+ trained["training_artifacts_sha256"][name] == head["training_artifacts_sha256"][name]
+ for name in ("plan", "inputs", "rows")
+ )
+ and trained["training_grade_provenance"]["plan_sha256"] == trained["training_artifacts_sha256"]["plan"]
+ and trained["cost_proxy_provenance"]["training_grades_sha256"] == trained["training_artifacts_sha256"]["grades"]
+ and trained["cost_proxy_provenance"]["conditional_means"] == trained["conditional_cost_statistics"],
+ "Native training artifacts do not match the frozen source and cost contracts",
+ )
+ outcomes._require(
+ set(trained["receipts_sha256"])
+ == {"selection-config.json", "selection-history.json", "epoch-selected.json", "refit-config.json"},
+ "Native receipt set changed",
+ )
+ for name, expected in trained["receipts_sha256"].items():
+ outcomes._require(file_hash(run / name) == expected, "Native training receipt changed")
+ outcomes._require(
+ outcomes.read(run / "selection-history.json") == trained["selection_history"], "Native history differs"
+ )
+ selected = outcomes.read(run / "epoch-selected.json")
+ outcomes._require(
+ selected["epoch"] == epoch
+ and selected["targets"] == targets[held].astype(int).tolist()
+ and selected["validation_ids"] == plan["selection_validation_ids"]
+ and selected["history_sha256"] == file_hash(run / "selection-history.json")
+ and selected["criterion"] == PROTOCOL["selection"],
+ "Native epoch selection receipt changed",
+ )
+ for name, epochs in (("selection-config.json", PROTOCOL["selection_epochs"]), ("refit-config.json", epoch)):
+ config = outcomes.read(run / name)
+ for key, value in _config_values(epochs).items():
+ outcomes._require(
+ config.get(key) == (list(value) if isinstance(value, tuple) else value),
+ "Native training config changed",
+ )
+ for variant, checkpoint in trained["checkpoints"].items():
+ _manifest(run, variant, checkpoint["files_sha256"], weights)
+ reference, merged, exported = (
+ np.asarray(checkpoint[key], dtype=float)
+ for key in ("reference_margins", "merged_margins", "exported_margins")
+ )
+ outcomes._require(
+ all(
+ value.shape == (int(held.sum()), 2) and np.isfinite(value).all()
+ for value in (reference, merged, exported)
+ ),
+ "Native export parity rows changed",
+ )
+ for key, value, tolerance in (
+ ("max_merge_margin_error", merged, PROTOCOL["max_merge_margin_error"]),
+ ("max_export_margin_error", exported, PROTOCOL["max_export_margin_error"]),
+ ):
+ error = float(np.max(np.abs(value - reference)))
+ outcomes._require(checkpoint[key] == error and error <= tolerance, "Native export parity exceeds tolerance")
+ return plan, trained
+
+
+def checkpoint_margins(checkpoint, rows, device):
+ """Infer two untempered margins in input order; caller verifies weight bytes."""
+ checkpoint = Path(checkpoint)
+ outcomes._require(
+ device in ("cpu", "cuda", "mps") and bool(rows), "Need nonempty rows and an explicit inference device"
+ )
+ outcomes._require(
+ all((checkpoint / name).is_file() for name in ("rl_agent_config.json", "model.safetensors")),
+ "Native inference requires a complete local checkpoint",
+ )
+ os.environ.update(PROTOCOL["eager_flags"])
+ import laya.train as lt
+ import torch
+
+ runtime_device = torch.device(device)
+ model, tokenizer, _ = lt.load_checkpoint(str(checkpoint))
+ try:
+ model.encoder.config.reference_compile = False
+ model.to(runtime_device).eval()
+ items = encode_rows(lt, tokenizer, rows)
+ records = lt.calibration_records(
+ model,
+ tokenizer,
+ items,
+ runtime_device,
+ finetune.MAX_LEN,
+ finetune.HEAD_LEN,
+ batch_size=finetune.runtime_settings(device)["evaluation_batch"],
+ )
+ return records_to_margins(records, np.zeros((len(rows), 2)))
+ finally:
+ del model
+ finetune.empty_cache(torch, runtime_device)
+
+
+def score_checkpoint(checkpoint, rows, device, conditional_cost_statistics):
+ """Return one benefit-per-credit score per request with no threshold fitting."""
+ return outcomes.normalized_scores(
+ probabilities(checkpoint_margins(checkpoint, rows, device)), conditional_cost_statistics
+ )[0].tolist()
+
+
+def _policies(rows, scores, rate):
+ return {
+ str(target): solver_report.select_cutoff(rows, scores, target, router_credits=None, credit_usd=rate)
+ for target in outcomes.TARGETS
+ }
+
+
+def freeze(run, head_run, baseline, original_path, *, device="cuda"):
+ run = Path(run)
+ _unopened(run, head_run)
+ if (run / "frozen.json").exists() or (run / "scoring-start.json").exists():
+ raise FileExistsError(run / "frozen.json")
+ plan, trained = checked_trained(run, head_run, weights=True, replay=False)
+ head = plan["head_plan"]
+ embedding = outcomes.checked_model(head_run)
+ outcomes._require(
+ embedding["training_grades_sha256"] == trained["training_artifacts_sha256"]["grades"]
+ and embedding["cost_proxy_sha256"] == trained["training_artifacts_sha256"]["cost_proxy"]
+ and embedding["head"]["targets"] == trained["training_targets"]
+ and embedding["head"]["conditional_cost_statistics"] == trained["conditional_cost_statistics"],
+ "Native and embedding comparison must use identical training outcomes and cost means",
+ )
+ outcomes._require(file_hash(original_path) == head["original_frozen_sha256"], "Original calibration study changed")
+ dataset, original, _, artifacts = benefit._inputs(baseline, original_path)
+ outcomes._require(
+ digest(dataset) == head["dataset_sha256"] and artifacts == head["baseline_artifacts_sha256"],
+ "Native held inputs differ from the prepared population",
+ )
+ write(
+ run / "scoring-start.json",
+ {
+ "trained_sha256": file_hash(run / "trained.json"),
+ "selected_embedding_sha256": file_hash(Path(head_run) / "selected-model.json"),
+ "device": device,
+ "started_at": benefit.now(),
+ "eager_flags": PROTOCOL["eager_flags"],
+ },
+ )
+ models = {}
+ for variant in ("base", "lora"):
+ margins, scores, expected = {}, {}, {}
+ for split in ("calibration", "test"):
+ print(f"Scoring frozen {variant} native outcomes: {split} inputs", flush=True)
+ rows = outcomes.held_rows(head, dataset, split)
+ values = checkpoint_margins(run / variant, rows, device)
+ ratio, costs = outcomes.normalized_scores(probabilities(values), trained["conditional_cost_statistics"])
+ margins[split], scores[split], expected[split] = values.tolist(), ratio.tolist(), costs.tolist()
+ models[variant] = {
+ "margins": margins,
+ "scores": scores,
+ "expected_costs_proxy": expected,
+ "policies": _policies(original["calibration_rows"], scores["calibration"], original["credit_usd"]),
+ }
+ _unopened(run, head_run)
+ return benefit.seal(
+ {
+ "format": FROZEN_FORMAT,
+ "frozen_at": benefit.now(),
+ "trained_sha256": file_hash(run / "trained.json"),
+ "plan_sha256": file_hash(run / "plan.json"),
+ "head_plan_sha256": plan["head_plan_sha256"],
+ "selected_embedding_sha256": file_hash(Path(head_run) / "selected-model.json"),
+ "scoring_start_sha256": file_hash(run / "scoring-start.json"),
+ "models": models,
+ "grading_provenance": outcomes.grading_provenance(original),
+ "calibration_rows": original["calibration_rows"],
+ "calibration_grades_sha256": original["calibration_grades_sha256"],
+ "credit_usd": original["credit_usd"],
+ }
+ )
+
+
+def checked_frozen(run, head_run, *, weights=False):
+ run = Path(run)
+ plan, trained = checked_trained(run, head_run, weights=weights)
+ frozen = benefit.checked(outcomes.read(run / "frozen.json"), FROZEN_FORMAT)
+ embedding = outcomes.checked_model(head_run)
+ start = outcomes.read(run / "scoring-start.json")
+ outcomes._require(
+ frozen["trained_sha256"] == file_hash(run / "trained.json")
+ and frozen["plan_sha256"] == file_hash(run / "plan.json")
+ and frozen["head_plan_sha256"] == plan["head_plan_sha256"]
+ and frozen["selected_embedding_sha256"] == file_hash(Path(head_run) / "selected-model.json")
+ and frozen["scoring_start_sha256"] == file_hash(run / "scoring-start.json")
+ and start["trained_sha256"] == frozen["trained_sha256"]
+ and start["selected_embedding_sha256"] == frozen["selected_embedding_sha256"]
+ and start["eager_flags"] == PROTOCOL["eager_flags"]
+ and trained["training_targets"] == embedding["head"]["targets"]
+ and trained["training_artifacts_sha256"]["grades"] == embedding["training_grades_sha256"]
+ and trained["training_artifacts_sha256"]["cost_proxy"] == embedding["cost_proxy_sha256"]
+ and trained["conditional_cost_statistics"] == embedding["head"]["conditional_cost_statistics"]
+ and frozen["grading_provenance"] == plan["head_plan"]["grading_provenance"]
+ and set(frozen["models"]) == {"base", "lora"},
+ "Native scoring/freeze lifecycle differs",
+ )
+ outcomes._require(
+ [row["id"] for row in frozen["calibration_rows"]]
+ == [row["id"] for row in plan["head_plan"]["populations"]["calibration"]],
+ "Native calibration IDs differ",
+ )
+ for model in frozen["models"].values():
+ for split in ("calibration", "test"):
+ values = np.asarray(model["margins"][split])
+ outcomes._require(
+ values.shape == (len(plan["head_plan"]["populations"][split]), 2), "Native margins misaligned"
+ )
+ scores, costs = outcomes.normalized_scores(probabilities(values), trained["conditional_cost_statistics"])
+ outcomes._require(
+ _matches_recorded(model["scores"][split], scores)
+ and _matches_recorded(model["expected_costs_proxy"][split], costs),
+ "Native ratio scores differ from canonical margins and training costs",
+ )
+ outcomes._require(
+ model["policies"]
+ == _policies(frozen["calibration_rows"], model["scores"]["calibration"], frozen["credit_usd"]),
+ "Native cutoffs differ from calibration-only selection",
+ )
+ return plan, trained, frozen
+
+
+def authorize_test(run, head_run, *, verify_weights=False):
+ outcomes._unopened(run)
+ plan, trained, _frozen = checked_frozen(run, head_run, weights=verify_weights)
+ outcomes.checked_frozen(head_run)
+ return benefit.seal(
+ {
+ "format": ACCESS_FORMAT,
+ "authorized_at": benefit.now(),
+ "native_frozen_sha256": file_hash(Path(run) / "frozen.json"),
+ "head_frozen_sha256": file_hash(Path(head_run) / "frozen.json"),
+ "head_plan_sha256": plan["head_plan_sha256"],
+ "checkpoint_manifest_sha256": digest(trained["checkpoints"]),
+ "checkpoint_verification": "local-bytes" if verify_weights else "manifest-only",
+ "source_code_sha256": code_hashes(),
+ "replay_validation": replay_validation(plan),
+ }
+ )
+
+
+def report(run, head_run, grades_path):
+ run, head_run = Path(run), Path(head_run)
+ plan, trained, frozen = checked_frozen(run, head_run)
+ outcomes.checked_frozen(head_run)
+ access = benefit.checked(outcomes.read(run / "test-access.json"), ACCESS_FORMAT)
+ shared = benefit.checked(outcomes.read(head_run / "test-access.json"), outcomes.ACCESS_FORMAT)
+ outcomes._require(
+ access["native_frozen_sha256"] == file_hash(run / "frozen.json")
+ and access["head_frozen_sha256"] == file_hash(head_run / "frozen.json")
+ and access["head_plan_sha256"] == plan["head_plan_sha256"]
+ and access["checkpoint_manifest_sha256"] == digest(trained["checkpoints"])
+ and access["source_code_sha256"] == code_hashes()
+ and access["replay_validation"] == replay_validation(plan)
+ and shared["frozen_sha256"] == access["head_frozen_sha256"]
+ and shared["plan_sha256"] == plan["head_plan_sha256"]
+ and shared["selected_model_sha256"] == frozen["selected_embedding_sha256"]
+ and shared["source_code_sha256"] == outcomes.code_hashes(),
+ "Native/shared test-access receipts differ",
+ )
+ head = plan["head_plan"]
+ ids = [row["id"] for row in head["populations"]["test"]]
+ grades = solver_policies._grade_rows(grades_path, "test", head["original_solver_plan_sha256"], ids)
+ outcomes._require(
+ grades["policies_frozen_sha256"] == head["original_frozen_sha256"]
+ and outcomes.grading_provenance(grades) == frozen["grading_provenance"],
+ "Native test answers differ from the original solver/grading study",
+ )
+ measured = {}
+ for variant, model in frozen["models"].items():
+ measured[variant] = {}
+ for target in outcomes.TARGETS:
+ policy = model["policies"][str(target)]
+ mode = policy["mode"]
+ routes = (np.asarray(model["scores"]["test"]) >= policy["cutoff"]).tolist() if mode == "threshold" else mode
+ measured[variant][str(target)] = {
+ "mode": mode,
+ "cutoff": policy["cutoff"],
+ "metrics": solver_report.summarize(
+ grades["rows"],
+ routes,
+ router_credits=None if mode == "threshold" else 0,
+ credit_usd=frozen["credit_usd"],
+ ),
+ }
+ return {
+ "format": "sie-solver-outcome-native-report-v1",
+ "n": len(ids),
+ "ids": ids,
+ "training_pool_rows": len(head["training_pool"]),
+ "training_rows": len(head["populations"]["train"]),
+ "selected_epoch": trained["selected_epoch"],
+ "frozen_sha256": file_hash(run / "frozen.json"),
+ "native_access_sha256": file_hash(run / "test-access.json"),
+ "shared_access_sha256": file_hash(head_run / "test-access.json"),
+ "test_grades_sha256": file_hash(grades_path),
+ "primary_max_quality_loss_pp": 1,
+ "models": measured,
+ "interpretation": "One shared two-question checkpoint, epoch selected on training papers only and freshly refit. Cost-normalization means match embedding training. Native hosting unpriced; report gross solver savings. Calibration loss targets are empirical, not test guarantees.",
+ }
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ commands = parser.add_subparsers(dest="command", required=True)
+ for command in ("prepare", "train", "freeze", "authorize-test", "report"):
+ child = commands.add_parser(command)
+ child.add_argument("--run", type=Path, required=True)
+ child.add_argument("--head-run", type=Path, required=True)
+ if command in ("train", "freeze"):
+ child.add_argument("--baseline", type=Path, required=True)
+ child.add_argument("--device", choices=("cuda",), default="cuda")
+ if command == "train":
+ for name in (
+ "training-plan",
+ "training-inputs",
+ "training-rows",
+ "training-grades",
+ "cost-proxy",
+ "price-snapshot",
+ ):
+ child.add_argument(f"--{name}", type=Path, required=True)
+ child.add_argument("--local-files-only", action="store_true")
+ if command == "freeze":
+ child.add_argument("--original-frozen", type=Path, required=True)
+ if command == "authorize-test":
+ child.add_argument("--verify-weights", action="store_true")
+ if command == "report":
+ child.add_argument("--test-grades", type=Path, required=True)
+ args = parser.parse_args()
+ destination = (
+ args.run
+ / {
+ "prepare": "plan.json",
+ "train": "trained.json",
+ "freeze": "frozen.json",
+ "authorize-test": "test-access.json",
+ "report": "report.json",
+ }[args.command]
+ )
+ if destination.exists() or (args.command == "prepare" and args.run.exists()):
+ raise FileExistsError(destination)
+ if args.command == "prepare":
+ result = prepare(args.head_run)
+ elif args.command == "train":
+ train(
+ args.run,
+ args.head_run,
+ args.baseline,
+ args.training_plan,
+ args.training_inputs,
+ args.training_rows,
+ args.training_grades,
+ args.cost_proxy,
+ args.price_snapshot,
+ device=args.device,
+ local_files_only=args.local_files_only,
+ )
+ return
+ elif args.command == "freeze":
+ result = freeze(args.run, args.head_run, args.baseline, args.original_frozen, device=args.device)
+ elif args.command == "authorize-test":
+ result = authorize_test(args.run, args.head_run, verify_weights=args.verify_weights)
+ else:
+ result = report(args.run, args.head_run, args.test_grades)
+ write(destination, result)
+ print(json.dumps({"output": str(destination), "sha256": file_hash(destination)}))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/benefit_route.py b/examples/train-a-router/benefit_route.py
new file mode 100644
index 000000000..e2d70730b
--- /dev/null
+++ b/examples/train-a-router/benefit_route.py
@@ -0,0 +1,258 @@
+"""Use a frozen benefit router to answer one new question through SIE.
+
+The selected calibration rule is fixed. A request needs at most one embedding,
+one answer-model call and one conditional final-answer call. There are no
+automatic retries. Cloud and self-hosted servers use the same public SDK.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import time
+from pathlib import Path
+
+import numpy as np
+from sie_sdk.types import Item
+
+import benefit
+import benefit_native
+import benefit_outcomes
+import benefit_outcomes_native
+import solver
+from data import digest, file_hash, render_request, write
+from serving import INSTRUCTION, MODEL, REVISION, SingleSendClient, error_receipt, normalize, usage
+
+CONTESTS = ("AMC8", "AMC10", "AMC12", "AIME", "HMMT-Feb", "HMMT-Nov")
+
+
+def embed(request, receipt_path=None):
+ """Save the billing receipt before validating a successful response body."""
+ with SingleSendClient(timeout_s=120, read_timeout_s=120) as client:
+ metadata = client.get_model(MODEL)
+ if metadata.get("revision") != REVISION or metadata.get("name", metadata.get("id")) != MODEL:
+ raise ValueError("Live encoder identity or revision differs")
+ reply = client.encode(
+ MODEL,
+ Item(text=request),
+ output_types=["dense"],
+ instruction=INSTRUCTION,
+ is_query=True,
+ wait_for_capacity=False,
+ max_oom_retries=0,
+ provision_timeout_s=60,
+ )
+ receipt = usage(reply)
+ if receipt_path is not None:
+ write(receipt_path, receipt)
+ vector = normalize(reply["dense"])
+ if vector.shape != (2560,):
+ raise ValueError("Wrong vector shape")
+ return vector, receipt
+
+
+def choose(frozen, request, target, *, self_hosted=False, receipt_path=None, outcome_head=None):
+ """Select one model; bypass embedding calls for an explicit static policy."""
+ policy = frozen["selections"][str(target)]
+ mode, score, receipt = policy["mode"], None, None
+ if mode == "threshold":
+ vector, receipt = embed(request, receipt_path)
+ if not self_hosted and not solver.valid_credit(receipt.get("credits")):
+ raise ValueError("Cloud embedding has no usable charge receipt")
+ vector = np.asarray(vector, dtype=float)
+ if outcome_head is not None:
+ score = float(benefit_outcomes.score(vector[None, :], outcome_head)[0])
+ else:
+ weights = np.asarray(frozen["head"]["weights"], dtype=float)
+ if vector.shape != weights.shape or not np.isfinite(vector).all():
+ raise ValueError("Embedding does not match the trained head")
+ score = float(vector @ weights + frozen["head"]["bias"])
+ if not np.isfinite(score):
+ raise ValueError("The trained head returned an invalid score")
+ arm = "strong" if score >= policy["cutoff"] else "cheap"
+ elif mode in ("always_strong", "always_cheap"):
+ arm = mode.removeprefix("always_")
+ else:
+ raise ValueError("Unknown frozen routing rule")
+ return {
+ "router": "qwen_embedding_outcomes" if outcome_head is not None else "qwen_embedding_ridge",
+ "arm": arm,
+ "model": solver.SETTINGS["arms"][arm]["model"],
+ "mode": mode,
+ "score": score,
+ "cutoff": policy["cutoff"],
+ "calibration_quality_loss_target_pp": target,
+ "embedding_receipt": receipt,
+ "note": "The target describes calibration results, not a guarantee for this answer.",
+ }
+
+
+def choose_native(native_run, head_run, request, target, variant, device, *, outcome_mode=False):
+ """Apply a selected native model with its original calibration cutoff."""
+ if variant not in ("base", "lora") or device not in ("cpu", "mps", "cuda"):
+ raise ValueError("Choose a native variant and supported local device")
+ routing_started = time.perf_counter()
+ trained = None
+ if outcome_mode:
+ _, trained, frozen = benefit_outcomes_native.checked_frozen(native_run, head_run)
+ else:
+ _, frozen = benefit_native.checked_frozen(native_run, head_run)
+ model = frozen["models"][variant]
+ policy = model["policies"][str(target)]
+ score, elapsed, verification_elapsed = None, None, None
+ if policy["mode"] == "threshold":
+ checkpoint = Path(native_run) / variant
+ verification_started = time.perf_counter()
+ # Metadata validation above checks all manifest paths/hashes. Only the
+ # selected optional checkpoint needs to be downloaded for live routing.
+ manifest = (
+ trained["checkpoints"][variant]["files_sha256"] if trained else model["checkpoint_files_sha256"]
+ )
+ for relative, expected in manifest.items():
+ if file_hash(checkpoint / relative) != expected:
+ raise ValueError("Selected native checkpoint bytes differ from the frozen manifest")
+ verification_elapsed = time.perf_counter() - verification_started
+ started = time.perf_counter()
+ score = (
+ benefit_outcomes_native.score_checkpoint(
+ checkpoint, [{"request": request}], device, trained["conditional_cost_statistics"]
+ )[0]
+ if trained
+ else benefit_native.score_checkpoint(checkpoint, [{"request": request}], device)[0]
+ )
+ elapsed = time.perf_counter() - started
+ arm = "strong" if score >= policy["cutoff"] else "cheap"
+ elif policy["mode"] in ("always_strong", "always_cheap"):
+ arm = policy["mode"].removeprefix("always_")
+ else:
+ raise ValueError("Unknown native routing rule")
+ return {
+ "router": "laya_outcomes_" + variant if outcome_mode else "laya_" + variant,
+ "arm": arm,
+ "model": solver.SETTINGS["arms"][arm]["model"],
+ "mode": policy["mode"],
+ "score": score,
+ "cutoff": policy["cutoff"],
+ "calibration_quality_loss_target_pp": target,
+ "embedding_receipt": None,
+ "native_model_calls": int(elapsed is not None),
+ "native_questions_per_request": 2 if outcome_mode else 1,
+ "local_checkpoint_verification_seconds": verification_elapsed,
+ "local_model_load_and_score_seconds": elapsed,
+ "local_router_end_to_end_seconds": time.perf_counter() - routing_started,
+ "latency_note": "Fresh CLI timing includes metadata checks, selected-checkpoint hashing and cold model loading; it is not persistent-server inference latency.",
+ "cost_note": "Local native compute is unpriced; reported credits cover the answer API only.",
+ "note": "The target describes calibration results, not a guarantee for this answer.",
+ }
+
+
+def answer(
+ run, output, contest, question, target=1, *, self_hosted=False, native_run=None, variant="lora", device="cpu"
+):
+ if output.exists():
+ raise FileExistsError(output)
+ if contest not in CONTESTS or not question.strip() or len(question) > 20000 or target not in benefit.TARGETS:
+ raise ValueError(
+ "Provide a supported competition, a nonempty question of at most 20,000 characters, and a declared target"
+ )
+ outcome_mode = benefit.read(run / "frozen.json").get("format") == benefit_outcomes.FROZEN_FORMAT
+ outcome_head = None
+ if outcome_mode:
+ frozen, selected = benefit_outcomes.checked_frozen(run)
+ plan, outcome_head = selected["plan"], selected["head"]
+ else:
+ frozen = benefit._checked_frozen(run)
+ plan = frozen["plan"]
+ if plan["solver_settings_sha256"] != digest(solver.SETTINGS):
+ raise ValueError("Answer-model settings differ from the training and evaluation protocol")
+ request = render_request(contest, question)
+ row = {"id": "new-" + digest(request)[:16], "request": request}
+ write(
+ output / "intent.json",
+ {
+ "id": row["id"],
+ "request_sha256": digest(request),
+ "frozen_sha256": file_hash(run / "frozen.json"),
+ "target_pp": target,
+ "self_hosted": self_hosted,
+ "router": "laya_" + variant if native_run else "qwen_embedding_outcomes" if outcome_mode else "qwen_embedding_ridge",
+ "outcome_model": outcome_mode,
+ "automatic_retries": False,
+ },
+ )
+ try:
+ routing = (
+ choose_native(native_run, run, request, target, variant, device, outcome_mode=outcome_mode)
+ if native_run
+ else choose(
+ frozen,
+ request,
+ target,
+ self_hosted=self_hosted,
+ receipt_path=output / "embedding-receipt.json",
+ outcome_head=outcome_head,
+ )
+ )
+ write(output / "routing.json", routing)
+ with SingleSendClient(timeout_s=60) as client:
+ metadata = client.get_model(routing["model"])
+ if (
+ metadata.get("revision") != solver.SETTINGS["arms"][routing["arm"]]["revision"]
+ or metadata.get("name", metadata.get("id")) != routing["model"]
+ ):
+ raise ValueError("Answer-model identity or revision differs from the evaluated model")
+ record = solver.solve(row, routing["arm"], solver.SETTINGS, self_hosted=self_hosted)
+ solver.validate_record(record, row, routing["arm"], solver.SETTINGS)
+ write(output / "answer.json", record)
+ if record["status"] != "complete":
+ raise RuntimeError("Answer collection failed; inspect the preserved receipt before another attempt")
+ except Exception as error:
+ write(output / "incomplete.json", error_receipt(error))
+ raise
+ embedding_charge = (routing["embedding_receipt"] or {}).get("credits", 0)
+ summary = {
+ "model": routing["model"],
+ "answer": record["answer"],
+ "solver_calls": len(record["calls"]),
+ "embedding_calls": int(routing["embedding_receipt"] is not None),
+ "native_model_calls": routing.get("native_model_calls", 0),
+ "total_api_credits": None if self_hosted else record["credits"] + embedding_charge,
+ "local_compute_cost": "unpriced",
+ "answer_quality": "Not graded; no reference answer was supplied for this new question",
+ }
+ write(output / "result.json", summary)
+ return summary
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--run", required=True, type=Path, help="Frozen embedding router directory")
+ parser.add_argument("--output", required=True, type=Path, help="New directory for this request and its receipts")
+ parser.add_argument("--contest", required=True, choices=CONTESTS)
+ parser.add_argument("--question", required=True)
+ parser.add_argument("--target", type=int, choices=benefit.TARGETS, default=1)
+ parser.add_argument("--self-hosted", action="store_true")
+ parser.add_argument("--native-run", type=Path, help="Use local frozen Laya instead of the embedding router")
+ parser.add_argument("--variant", choices=("base", "lora"), default="lora")
+ parser.add_argument("--device", choices=("cpu", "mps", "cuda"), default="cpu")
+ args = parser.parse_args()
+ print(
+ json.dumps(
+ answer(
+ args.run,
+ args.output,
+ args.contest,
+ args.question,
+ args.target,
+ self_hosted=args.self_hosted,
+ native_run=args.native_run,
+ variant=args.variant,
+ device=args.device,
+ ),
+ indent=2,
+ )
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/benefit_training_review.py b/examples/train-a-router/benefit_training_review.py
new file mode 100644
index 000000000..05a31e317
--- /dev/null
+++ b/examples/train-a-router/benefit_training_review.py
@@ -0,0 +1,132 @@
+"""Retain two expanded training rows after explicit symbolic-answer review.
+
+This amendment was made after training answers were collected, before fitting
+the expanded models. It does not change the training population, learner,
+calibration, or test grader. Original references and automatic grades remain
+in each adjudication so the source correction is visible and reproducible.
+"""
+
+from __future__ import annotations
+
+import argparse
+import re
+from pathlib import Path
+
+import benefit_expanded
+import solver
+from benefit_expanded_gold_v2 import grade_expanded
+from data import digest, file_hash, write
+from solver_answers import extract_final_answer
+
+REVIEW = {
+ "e2h-amc-eval-1621": {
+ "original": "2001^{2002}",
+ "canonical": "2001^{2002}",
+ "answers": {"2001^{2002}": True, "2001^{2002^{2003^{2004}}}": False},
+ "proof": "The four nested logarithms are defined iff the third logarithm is positive, the second is greater than1, the first is greater than2002, and x>2001^2002. Since2001>1 and2002^(2003^2004)>2002, the predicted power tower is strictly larger than the boundary. No huge integer or floating-point approximation is needed.",
+ "source_correction": False,
+ },
+ "e2h-amc-eval-1775": {
+ "original": r"\sqrt{\sqrt{4^{2012}+1}}",
+ "canonical": r"\sqrt{4^{2012}+1}",
+ "answers": {r"\sqrt{2^{4024}+1}": True, "2^{2012}": False},
+ "proof": "The nth tent-map iterate has2^n segments of width2^-n and vertical displacement1. Total length is2^n*sqrt(2^-2n+1)=sqrt(4^n+1). The source solution derives this expression but its final boxed answer adds an erroneous extra square root. At n=2012,4^2012=2^4024 exactly, so the strong answer is equal;2^2012 has square4^2012, strictly less than4^2012+1, so the cheap answer is unequal.",
+ "source_correction": True,
+ },
+}
+
+
+def reviewed_grade(identity, original, response):
+ automatic = grade_expanded(identity, original, response)
+ if identity not in REVIEW:
+ if automatic["status"] in ("gold_unparseable", "grading_error"):
+ raise ValueError(f"Unreviewed grading failure: {identity}")
+ return automatic
+ rule = REVIEW[identity]
+ final = re.sub(r"\s+", "", extract_final_answer(response))
+ if original != rule["original"] or final not in rule["answers"]:
+ raise ValueError(f"Symbolic training adjudication does not cover this answer: {identity}")
+ return automatic | {
+ "correct": rule["answers"][final],
+ "status": "graded",
+ "training_adjudication": {
+ "identity": identity,
+ "original_reference": original,
+ "canonical_reference": rule["canonical"],
+ "proof": rule["proof"],
+ "source_correction": rule["source_correction"],
+ "automatic_grade": automatic,
+ "reviewed_after_training_collection": True,
+ },
+ }
+
+
+def grade(directory, output=None):
+ directory = Path(directory)
+ output = Path(output) if output else directory / "grades-train.json"
+ if output.exists():
+ raise FileExistsError(output)
+ plan, inputs, settings = benefit_expanded.checked_training(directory)
+ gold = solver.read(directory / "gold-for-scoring-only.json")
+ if digest(gold) != plan["gold_sha256"] or plan["grader"] != benefit_expanded.grader_metadata():
+ raise ValueError("Expanded references or frozen grading dependency changed")
+ collection = directory / "answers-train"
+ summary = solver.read(collection / "collection.json")
+ binding = solver.read(collection / "collection-plan.json")
+ solver.collection_binding(binding, directory, "train", inputs["train"], settings, None, True)
+ if summary["status"] != "complete" or summary["completed"] != 2 * len(inputs["train"]) or summary["failed"]:
+ raise ValueError("Expanded paired collection is incomplete")
+ rows, reviewed = [], []
+ for row in inputs["train"]:
+ outcome = {"id": row["id"]}
+ for arm in settings["arms"]:
+ path = collection / f"{row['id']}-{arm}.json"
+ record = solver.read(path)
+ solver.validate_record(record, row, arm, settings)
+ if record["status"] != "complete":
+ raise ValueError("Incomplete expanded training answer")
+ measured = reviewed_grade(row["id"], gold[row["id"]], record["answer"])
+ outcome.update(
+ {
+ f"{arm}_correct": measured["correct"],
+ f"{arm}_credits": record["credits"],
+ f"{arm}_grade": measured,
+ f"{arm}_record_sha256": file_hash(path),
+ }
+ )
+ if "training_adjudication" in measured:
+ reviewed.append({"id": row["id"], "arm": arm, **measured["training_adjudication"]})
+ rows.append(outcome)
+ if {row["id"] for row in reviewed} != set(REVIEW) or len(reviewed) != 2 * len(REVIEW):
+ raise ValueError("The fixed expanded-training review population differs")
+ result = {
+ "split": "train",
+ "plan_sha256": file_hash(directory / "plan.json"),
+ "collection_plan_sha256": file_hash(collection / "collection-plan.json"),
+ "collection_summary_sha256": file_hash(collection / "collection.json"),
+ "policies_frozen_sha256": None,
+ **benefit_expanded.source_hashes(),
+ "gold_review_sha256": plan["gold_review_sha256"],
+ "training_gold_rules_sha256": plan["training_gold_rules_sha256"],
+ "post_collection_training_adjudication": {
+ "source_sha256": file_hash(Path(__file__)),
+ "review_sha256": digest(REVIEW),
+ "scope": "Two expanded-only training rows; no exclusions, no calibration/test changes, and no learner changes. Parent grader hashes identify the retained automatic protocol, augmented explicitly by this review.",
+ "records": reviewed,
+ },
+ "rows": rows,
+ }
+ write(output, result)
+ return result
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--directory", type=Path, required=True)
+ parser.add_argument("--output", type=Path)
+ args = parser.parse_args()
+ grade(args.directory, args.output)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/data.py b/examples/train-a-router/data.py
new file mode 100644
index 000000000..c5b5f7579
--- /dev/null
+++ b/examples/train-a-router/data.py
@@ -0,0 +1,312 @@
+"""Pinned public inputs, duplicate-family isolation and chronological splits."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import re
+import urllib.request
+from collections import Counter, defaultdict
+from pathlib import Path
+
+import numpy as np
+
+SOURCE_REVISION = "55bc0d2fb10954151e669d2026b87fa896f2fa26"
+SOURCE_URL = (
+ "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/"
+ "002087323da76e925235e0c6acb12f402bf8fd9e/evaluations/2026-10-17/"
+ "request-routing-difficulty-policy-v1/source/"
+)
+SOURCE_HASHES = {
+ "train.parquet": "1686c7df11b60848f05142e5b2215a96a9ab6952b3867827542f2c4d92621baa",
+ "eval.parquet": "2a04f50a80a16bfc5c1ff5667f9403e238184bbe7c325eff99f684d89f8f97fe",
+}
+LABELS = ("routine", "advanced")
+PROTOCOL = {
+ "name": "contest-relative-grouped-v2",
+ "target": "Above versus below the training-population median human difficulty for the same contest family",
+ "input": "Contest family plus problem text; no year, problem index, ratings, answer or solution",
+ "training_years": "through 2017",
+ "validation_years": [2018],
+ "calibration_years": [2019],
+ "evaluation_years": "2020 onward, excluding every previously used duplicate family",
+ "duplicates": "Connected components of canonical text or five-token Jaccard>=0.70; masked-number Jaccard>=0.85 with >=12 tokens. Drop components crossing chronological partitions.",
+ "label_thresholds": "Median rating per contest tag from eligible training population only; fixed across every learning budget",
+ "training_order": "Round-robin tag and label strata, each hash ordered; nested 64/128/256/512 prefixes",
+ "embedding_candidates": {
+ "methods": ["knn", "ridge"],
+ "sizes": [64, 128, 256, 512],
+ "penalties": [0.01, 0.1, 1.0, 10.0],
+ "neighbors": 5,
+ },
+ "selection": "Ridge penalty on validation; all cutoffs on separate calibration; no evaluation-driven choices",
+ "native_training": {
+ "seeds": [89, 137, 211],
+ "epochs": 8,
+ "selection": "minimum validation cross entropy; calibration fits cutoff after checkpoint selection",
+ },
+ "reporting": "All declared families/seeds, paired case intervals and per-seed spread; no automatic winner or cost claim",
+ "primary": "ridge512 versus knn64 on frozen temporal evaluation; report ridge64 and knn512 controls",
+}
+QUESTION = {
+ "difficulty": {
+ "type": "choice",
+ "instructions": "Classify the problem relative to other problems in the named competition. Routine is the easier half and advanced the harder half of that competition's historical human-rated difficulty. Judge reasoning and insight needed; do not solve it.",
+ "criteria": {
+ "routine": "Relatively routine for this competition: familiar techniques and fewer demanding reasoning steps.",
+ "advanced": "Relatively advanced for this competition: more demanding insight or multi-step reasoning.",
+ },
+ }
+}
+
+
+def digest(value):
+ return hashlib.sha256(
+ json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False).encode()
+ ).hexdigest()
+
+
+def file_hash(path):
+ h = hashlib.sha256()
+ with Path(path).open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ h.update(chunk)
+ return h.hexdigest()
+
+
+def write(path, value):
+ path = Path(path)
+ path.parent.mkdir(parents=True, exist_ok=True)
+ with path.open("x") as handle:
+ json.dump(value, handle, indent=2, allow_nan=False)
+ handle.write("\n")
+
+
+def download_source(output):
+ output.mkdir(parents=True, exist_ok=True)
+ for name, expected in SOURCE_HASHES.items():
+ path = output / name
+ if not path.exists():
+ with urllib.request.urlopen(SOURCE_URL + name, timeout=60) as response:
+ body = response.read(10_000_001)
+ if len(body) > 10_000_000 or hashlib.sha256(body).hexdigest() != expected:
+ raise ValueError("Source download failed its size/hash check")
+ with path.open("xb") as handle:
+ handle.write(body)
+ if file_hash(path) != expected:
+ raise ValueError(f"Pinned source differs: {name}")
+
+
+def canonical(text):
+ text = text.casefold().replace("\\dfrac", "\\frac").replace("\\tfrac", "\\frac")
+ return "".join(re.findall(r"[a-z0-9]+", text))
+
+
+def fingerprint(text, masked=False):
+ tokens = re.findall(r"[a-z]+|\d+", text.casefold())
+ if masked:
+ if len(tokens) < 12:
+ return set()
+ tokens = ["#" if token.isdigit() else token for token in tokens]
+ return {tuple(tokens[i : i + 5]) for i in range(len(tokens) - 4)}
+
+
+def duplicate_components(rows):
+ parent = list(range(len(rows)))
+
+ def find(i):
+ while parent[i] != i:
+ parent[i] = parent[parent[i]]
+ i = parent[i]
+ return i
+
+ def union(i, j):
+ a, b = find(i), find(j)
+ parent[max(a, b)] = min(a, b)
+
+ seen = {}
+ for i, row in enumerate(rows):
+ key = canonical(row["problem"])
+ if key in seen:
+ union(i, seen[key])
+ seen[key] = i
+ # An inverted shingle index avoids comparing all pairs without a common token run.
+ for masked, threshold in ((False, 0.70), (True, 0.85)):
+ prints = [fingerprint(row["problem"], masked) for row in rows]
+ index = defaultdict(list)
+ for i, current in enumerate(prints):
+ candidates = Counter(j for token in current for j in index[token])
+ for j, common in candidates.items():
+ if common / (len(current) + len(prints[j]) - common) >= threshold:
+ union(i, j)
+ for token in current:
+ index[token].append(i)
+ groups = defaultdict(list)
+ for i in range(len(rows)):
+ groups[find(i)].append(i)
+ return list(groups.values())
+
+
+def partition(year):
+ return "train" if year <= 2017 else "validation" if year == 2018 else "calibration" if year == 2019 else "test"
+
+
+def render_request(tag, text):
+ return f"Competition: {tag}\nProblem: {text.strip()}"
+
+
+def interleave(rows):
+ groups = defaultdict(list)
+ for row in rows:
+ groups[(row["contest"], row["label"])].append(row)
+ for group in groups.values():
+ group.sort(key=lambda r: digest(["training-order-v2", r["id"]]))
+ result = []
+ for i in range(max(map(len, groups.values()))):
+ result.extend(groups[key][i] for key in sorted(groups) if i < len(groups[key]))
+ return result
+
+
+def prepare(source, output, previously_used):
+ import pyarrow.parquet as pq
+
+ if output.exists():
+ raise FileExistsError(output)
+ rows = []
+ for name, expected in SOURCE_HASHES.items():
+ if file_hash(source / name) != expected:
+ raise ValueError("Source hash differs")
+ split = name.split(".")[0]
+ rows.extend(dict(r, id=f"e2h-amc-{split}-{i}") for i, r in enumerate(pq.read_table(source / name).to_pylist()))
+ used = set(json.loads(previously_used.read_text()))
+ known_ids = {r["id"] for r in rows}
+ if not used or not used <= known_ids:
+ raise ValueError("Previously used IDs must be a nonempty subset of the pinned source")
+ components = duplicate_components(rows)
+ selected, removed, families = [], Counter(), {}
+ for component in components:
+ members = [rows[i] for i in component]
+ ids = sorted(r["id"] for r in members)
+ family = digest(ids)
+ splits = {partition(r["year"]) for r in members}
+ if len(splits) != 1:
+ removed["cross_partition_family_rows"] += len(members)
+ continue
+ split = next(iter(splits))
+ if split == "test" and any(identity in used for identity in ids):
+ removed["previously_used_test_family_rows"] += len(members)
+ continue
+ # Retain one representative, chosen without ratings or model outputs.
+ row = min(members, key=lambda r: digest(["representative-v2", r["id"]]))
+ selected.append(dict(row, split=split, family=family))
+ families[family] = ids
+ removed["within_family_duplicates"] += len(members) - 1
+ training_pool = [r for r in selected if r["split"] == "train"]
+ medians = {
+ tag: float(np.median([r["rating"] for r in training_pool if r["tag"] == tag]))
+ for tag in sorted({r["tag"] for r in training_pool})
+ }
+ splits = {s: [] for s in ("train", "validation", "calibration", "test")}
+ for row in selected:
+ tag = row["tag"]
+ splits[row["split"]].append(
+ {
+ "id": row["id"],
+ "request": render_request(tag, row["problem"]),
+ "label": int(row["rating"] >= medians[tag]),
+ "contest": tag,
+ "year": row["year"],
+ "family": row["family"],
+ "paper_group": f"{row['contest']}:{row['year']}",
+ }
+ )
+ splits["train"] = interleave(splits["train"])[:512]
+ for split in ("validation", "calibration", "test"):
+ splits[split].sort(key=lambda r: digest(["input-order-v2", r["id"]]))
+ if len(splits["train"]) != 512 or any(len(r) < 50 for r in splits.values()):
+ raise ValueError("Insufficient examples for the declared protocol")
+ dataset = {"protocol": PROTOCOL, "source_revision": SOURCE_REVISION, "label_thresholds": medians, "splits": splits}
+ audit(dataset)
+ output.mkdir(parents=True)
+ write(output / "data.json", dataset)
+ write(output / "questions.json", QUESTION)
+ write(output / "protocol.json", PROTOCOL)
+ write(output / "prior-used-ids.json", sorted(used))
+ write(
+ output / "split-audit.json",
+ {
+ "data_sha256": digest(dataset),
+ "protocol_sha256": digest(PROTOCOL),
+ "source_hashes": SOURCE_HASHES,
+ "previously_used_ids_sha256": file_hash(previously_used),
+ "counts": {
+ s: {
+ "n": len(r),
+ "labels": dict(Counter(x["label"] for x in r)),
+ "contests": dict(Counter(x["contest"] for x in r)),
+ }
+ for s, r in splits.items()
+ },
+ "removed": dict(removed),
+ "duplicate_families": {k: v for k, v in families.items() if len(v) > 1},
+ "label_threshold_fit_rows": len(training_pool),
+ "label_budget_note": "64/128/256/512 counts classifier feedback; the target definition uses the eligible pre-2018 source population",
+ },
+ )
+ for split, values in splits.items():
+ write(output / f"{split}.json", values)
+ with (output / f"{split}.jsonl").open("x") as handle:
+ for r in values:
+ handle.write(
+ json.dumps(
+ {"state": r["request"], "questions": QUESTION, "expected": {"difficulty": LABELS[r["label"]]}}
+ )
+ + "\n"
+ )
+ print(
+ json.dumps(
+ {
+ "output": str(output),
+ "counts": {s: len(r) for s, r in splits.items()},
+ "removed": dict(removed),
+ "data_sha256": digest(dataset),
+ },
+ indent=2,
+ )
+ )
+
+
+def audit(dataset):
+ seen_ids, seen_text, seen_families, seen_groups = set(), set(), set(), set()
+ if set(dataset["splits"]) != {"train", "validation", "calibration", "test"}:
+ raise ValueError("Expected four disjoint splits")
+ for split, rows in dataset["splits"].items():
+ ids = {r["id"] for r in rows}
+ texts = {canonical(r["request"]) for r in rows}
+ families = {r["family"] for r in rows}
+ groups = {r["paper_group"] for r in rows}
+ if len(ids) != len(rows) or len(texts) != len(rows) or len(families) != len(rows):
+ raise ValueError("Duplicate row/family within a split")
+ if seen_ids & ids or seen_text & texts or seen_families & families or seen_groups & groups:
+ raise ValueError("Cross-split identity, text, family or paper leakage")
+ if any(partition(r["year"]) != split for r in rows) or {r["label"] for r in rows} != {0, 1}:
+ raise ValueError("Invalid year or label distribution")
+ seen_ids |= ids
+ seen_text |= texts
+ seen_families |= families
+ seen_groups |= groups
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--source", type=Path, default=Path("evidence/source"))
+ parser.add_argument("--output", type=Path, required=True)
+ parser.add_argument("--previously-used", type=Path, required=True)
+ args = parser.parse_args()
+ download_source(args.source)
+ prepare(args.source, args.output, args.previously_used)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/docs/laya-adaptation.md b/examples/train-a-router/docs/laya-adaptation.md
new file mode 100644
index 000000000..15f8bb14a
--- /dev/null
+++ b/examples/train-a-router/docs/laya-adaptation.md
@@ -0,0 +1,58 @@
+# What other Laya adaptations tell us
+
+Laya can learn task-specific decisions through post-training. The useful
+question for this example is whether that adaptation improves the money and
+answer-quality tradeoff on new requests. Results on a different benchmark
+provide ideas to test; they do not predict our router's improvement.
+
+The following are the authors' reported results, checked on 2026-10-19.
+We did not reproduce these outside experiments.
+
+| Example | What was adapted | Reported result |
+| --- | --- | --- |
+| [Official typed-decisions walkthrough](https://github.com/NandhaKishorM/laya/blob/main/docs/finetune.md) | Full tuning on 1,200 cases containing 6,000 teacher-labeled decisions, with a separate calibration slice. | Base accuracy 36.2%; published adapted checkpoint 76.6% on 2,000 test decisions. This measures agreement with the benchmark's teacher labels. |
+| [Official three-seed loss comparison](https://github.com/NandhaKishorM/laya/blob/main/research/scripts/step4_benchmark/GATE_D_RESULTS.md) | Same data and full-tuning recipe; compare RLCD, Laya's reinforcement-learning-style objective, with supervised cross-entropy. | Mean accuracy 77.62% versus 77.63%. The difference changes sign across seeds; the report retains RLCD as the default. |
+| [Community duplicate-question LoRA](https://github.com/vishalmysore/layaMOE/blob/main/results/qqp_lora.json) | Rank-16 adapters in the last four encoder layers, including attention and feed-forward layers, plus decision-head training. Starts from the typed-decisions checkpoint. | Accuracy 77.2% to 78.0%; F1 0.767 to 0.791. Its paired significance test reports p = 0.299, so this small experiment does not establish a reliable accuracy gain. |
+
+The official loss comparison is useful because it qualifies an earlier
+single-seed cross-entropy win. Across three seeds, the mean difference is
+negligible and the supervised runs vary more. It does not show that one
+objective will consistently improve this binary LoRA router. Both losses in
+that panel use the same teacher probability targets; it is not a comparison
+of hard labels with soft labels.
+
+## What we use in this example
+
+We use one fixed recipe so that the stages remain understandable:
+
+- Start from the English Laya checkpoint and keep the input questions fixed.
+- Train rank-16 attention adapters and the complete native decision head.
+- Learn two actual answer-correctness targets, one for each answer model.
+- Choose the number of epochs on held-out training families, then refit all
+ training examples and choose the routing cutoff on separate calibration data.
+- Repeat the same recipe with 500 and 2,000 training questions.
+- Check that merging and saving the adapters preserves the decision scores.
+
+Our labels say whether each recorded answer matches the dataset's reference.
+The teacher-label stage substitutes a teacher-generated reference and compares
+it with a matched reference-label control. It does not treat the teacher's
+self-reported confidence as a verified probability of correctness.
+
+The official training interface supports both hard expected answers and teacher
+probability targets. That makes either kind of supervision technically possible;
+it does not establish that purchasing probability labels would improve this
+task. The recorded example uses the pinned training implementation and settings
+described in [methodology](methodology.md).
+
+## What the comparison can establish
+
+LoRA limits how many weights are updated, which can help with small datasets.
+Full fine-tuning has more capacity but is not automatically better: learning
+rates, regularization, stopping, and data size also matter. Our current study
+compares unchanged Laya, its LoRA adaptation, and embedding classifiers. It does
+not establish that LoRA generally outperforms full fine-tuning.
+
+Model accuracy on another benchmark and teacher self-agreement are not ceilings
+on this router's independently measured answer quality. The final comparison
+must show the actual quality gap and dollar savings, using the same recorded
+answers for every routing method.
diff --git a/examples/train-a-router/docs/methodology.md b/examples/train-a-router/docs/methodology.md
new file mode 100644
index 000000000..8134dad8d
--- /dev/null
+++ b/examples/train-a-router/docs/methodology.md
@@ -0,0 +1,229 @@
+# How the comparison is measured
+
+The router chooses between two answer models. We measure whether its choices
+save money while keeping final-answer accuracy close to always using the
+stronger model. Predicting a difficulty label is not the goal.
+
+## Questions, labels, and independent evaluation
+
+The questions come from [GSM8K](https://github.com/openai/grade-school-math),
+pinned to revision `3101c7d5072418e28b9008a6636bde82a006892c`.
+The download contains the source license and attribution.
+
+| Role | Questions | What can use their answers |
+| --- | ---: | --- |
+| Small training set | 500 | Fit models and choose training settings. This is a subset of the expanded set. |
+| Expanded training set | 2,000 | Fit models and choose training settings. |
+| Calibration | 1,000 | Choose the routing cutoff after the models are trained. |
+| Test | 3,000 | Measure the saved models and cutoffs once. |
+
+The 2,000 training questions had already been used for development. They are
+not counted as fresh test evidence: they contain all 1,319 official GSM8K test
+questions and 681 official training questions. The calibration and test questions were
+selected from unused families in the original GSM8K training split. Thus our
+study roles differ from the original benchmark's train/test split; the results
+are a custom routing evaluation, not standard GSM8K benchmark scores.
+
+A **question family** groups exact duplicates and similar question templates.
+Before collecting fresh answers, we grouped normalized text and word shingles,
+then checked character four-gram similarity both with the original numbers and
+with numbers replaced by a placeholder. Similarity thresholds are 0.80 and
+0.85 respectively. The latter helps catch the same problem with different
+numbers. Every detected family stays within one study role; each fresh role
+uses only one representative per family. The 500-question subset also keeps
+whole families together.
+
+These checks found 6,751 unused families, enough for the 4,000 fresh questions.
+The source audit preserves the detected pairs and exclusions. Lexical checks
+cannot establish that all semantic relatives have been found, or that a model
+never saw GSM8K during its original pretraining.
+
+## Record both possible answers
+
+For each question, we record one answer from each model using fixed prompts,
+token limits, model revisions, and runtimes:
+
+| Answer model | Initial output limit | Runtime used for fresh collection |
+| --- | ---: | --- |
+| Qwen3.5-4B | 1,536 tokens | SGLang 0.5.10.post1, CUDA 12 |
+| Qwen3.8-27B-FP8 | 3,072 tokens | SGLang 0.5.20, CUDA 13 |
+
+An initial answer that is truncated or lacks a final answer receives at most
+one additional 256-token call to extract its final answer. The corresponding
+charges are included. There is no retry based on an answer being incorrect.
+The recorded settings and model revisions are in [solver.py](../solver.py).
+
+Both answers are checked against the dataset's reference answer. Equivalent
+numeric forms count as equal. A small arithmetic adapter handles clock-hour
+answers; durations remain ordinary quantities. Unresolved parsing or unit
+comparisons are retained for review instead of silently dropping a question
+or treating the answer as wrong. The one calibration-format adjudication
+preserves the original verdict and binds the reviewed answer to its file hash.
+It was resolved before choosing the cutoffs.
+
+Every router is evaluated against these same recorded answer pairs. We do not
+regenerate answers separately for each router, which would mix routing changes
+with variation in answer generation.
+
+## Learn correctness and the value of an upgrade
+
+Each question supplies two labels: whether the cheaper model was correct and
+whether the stronger model was correct. Both can succeed; both can fail; the
+cheaper model can occasionally be the only one to succeed.
+
+The routers predict both correctness probabilities. Their routing score is:
+
+```text
+predicted improvement in correctness
+-----------------------------------
+predicted extra cost of the stronger answer
+```
+
+The denominator has a floor of one credit. Expected costs use four means from
+training only: each model's cost when correct and when incorrect. This matters
+because long, unsuccessful answers can cost more than short, successful ones.
+The stronger model is selected when this score is at or above the saved cutoff.
+
+The embedding approach keeps **Qwen3-Embedding-4B** fixed. Its 2,560-number
+representation feeds two logistic classifiers: each is a weighted sum plus a
+bias, followed by a conversion to a probability. Only those small classifiers
+are trained. Five folds of training families choose one shared regularization
+strength from `0.01, 0.1, 1, 10`, using prediction loss. Stronger regularization
+wins an exact tie. The final classifiers are refit on all training rows.
+
+The decision-model approach uses one shared **Laya** checkpoint, based on
+ModernBERT-large, and asks it the same two correctness questions. The unchanged
+checkpoint is the baseline. Adaptation trains rank-16 attention LoRA updates
+and the complete decision head. The recipe uses seed 89, adapter learning rate
+0.0002, head learning rate 0.0001, dropout 0.05, and weight decay 0.01.
+
+We check epochs 0 through 8 on one reserved training-family fold. **Epoch 0**
+means no adaptation. The epoch with the lowest combined prediction loss is
+selected, with earlier epochs winning ties. We then restart from the original
+checkpoint and train on all rows for that many epochs. Calibration and test
+outcomes never select the training duration. Both arithmetic training sizes
+selected one epoch. Adapter merging and checkpoint export are checked for
+score drift before use.
+
+For each training size, a matched unchanged-Laya control uses the same
+training-derived cost estimates as its adapted counterpart. Its weights stay
+fixed, and calibration chooses its own cutoff. This separates changing the
+weights from updating the expected answer costs. The controls were saved in a
+separate, hash-checked supplement before opening test answers; they do not
+replace or modify the original frozen comparison.
+
+This comparison does not establish that LoRA generally beats full fine-tuning,
+or that a model's parameter count alone explains its performance.
+
+## Choose a cutoff, then open the test results
+
+The primary target is an empirical **one-percentage-point quality gap on
+calibration**. We choose the lowest-cost qualifying cutoff, also considering
+always-cheap and always-strong policies. Secondary targets of 0, 2, and 5
+points are declared in advance and retained in the report.
+
+The source plan aimed for at least 30% savings near a one-point quality gap.
+The primary 2,000-example embedding rule saves 20.87% with a 0.47-point test
+gap, so it does not reach that savings aim. The declared two-point calibration
+setting saves 33.77% with a 0.83-point test gap; its gap interval extends to
+1.30 points. We report that as a secondary operating point, without changing
+the primary comparison or claiming a guaranteed one-point bound.
+
+All model files, input-only test scores, and calibration cutoffs are saved
+before test answers are graded. A test result can exceed its calibration
+target: the target is not a guarantee. The public download contains the frozen
+decisions and test-access record. Replaying these published results is an
+**exposed reproduction**, not another blind experiment.
+
+The report keeps additional controls: models trained on competition problems
+with their original cutoffs, and those same weights with arithmetic-only
+cutoff calibration. These distinguish adjusting the cutoff from training
+weights on the new workload. Their original training-derived cost means stay
+fixed. They are diagnostic comparisons, not extra stages learners must run.
+
+Separate experiments used only the 2,000 exposed development questions. An
+answer-aware cascade, which inspects the cheaper model's answer before deciding
+whether to upgrade, was screened with training-fold validation. Two fixed
+changes to the cheaper solver's generation settings were also tested on all
+2,000 questions. Neither solver change met the improvement rule specified
+before collecting those answers. These diagnostics did not change the final
+model pair, trained routers, cutoffs, or test questions; their executed sources
+and results are retained as optional evidence, outside the walkthrough.
+
+## Teacher labels when answer keys are unavailable
+
+An independent **GPT-6 Astra** teacher receives the training questions
+using high reasoning effort. It sees only each question, never the benchmark
+reference or either answer model's output. The answer contract requests one
+number in the requested units, without unit text or currency symbols.
+
+The numeric-format collection completed **1,539 of the planned 2,000
+references** before the provider account exhausted its credits. We closed
+that acquisition and preserved its incomplete collection record. The remaining
+461 questions are missing because of acquisition availability, not an answer
+quality filter. All 1,539 completed references satisfied the numeric contract;
+both teacher and reference-label classifiers use exactly those same questions.
+
+A valid numeric reference is used to grade both answer models. Incorrect
+teacher answers remain in the training labels. We do not replace them with
+benchmark answers. A matched control trains the same classifiers with benchmark
+labels on exactly the same eligible questions and family folds. Comparing the
+two isolates the effect of the labeling pipeline, including the conditional
+cost means derived from those labels. A separate benchmark audit measures
+the teacher's reliability; it does not decide which teacher labels to retain.
+The teacher's references agree with the benchmark on **98.25%** of these
+training questions. The derived correctness labels agree on **98.70%** of
+cheaper-model answers and **98.51%** of stronger-model answers. These are
+training-label checks, separate from the held-out routing comparison.
+
+An earlier teacher run used a general mathematics answer format. Its unit text
+exposed limitations in the generic parser, so we preserved that attempt and
+collected the numeric-format run before opening the test results. The label
+cost report keeps these acquisition expenses separate from inference savings.
+Known numeric-run usage is **$6.40**; the earlier general-format acquisition
+used **$8.75**. Attempts without verified usage remain recorded separately,
+so these figures are usage-based estimates rather than a complete invoice.
+
+A numerically stable, algebraically equivalent logistic-loss calculation is
+used for both matched heads. It avoids floating-point cancellation; the
+objective, regularization grid, folds, and selection rule are unchanged.
+
+The teacher stage replaces **training** answer keys. Calibration and test still
+use trusted benchmark answers. In a deployment without any labels, reserve a
+small independently reviewed sample for quality measurement; agreement with
+the training teacher alone cannot establish real answer quality.
+
+## Money, uncertainty, and serving costs
+
+Fresh answer generation runs on Modal. We estimate its equivalent Cloud cost
+from recorded physical-call token counts and a saved Cloud price table. These
+are **Cloud-price estimates**, not observed Cloud bills or Modal invoices.
+Embedding charges are observed Cloud credits, including the original collection
+batch size. One credit corresponds to $0.00001 in this report.
+
+For each router, the report sums the selected answer charges and, where
+applicable, the embedding charges. Training, data collection, teacher labeling,
+and idle hosting are separate expenses. Laya hosting is unpriced, so its
+reported answer-model savings are the maximum allowance for router hosting
+before breaking even. Tiny local classifier CPU work is also unpriced. If the
+selected policy always uses one model, it skips router inference entirely.
+
+**Quality gap** is always-strong accuracy minus routed accuracy, in percentage
+points. It includes both answers lost by choosing the cheaper model and answers
+recovered when only that model succeeds. A one-point gap means 100 fewer
+correct answers per 10,000 requests, on average.
+
+The training recipe uses one seed. More examples require more training work;
+this is not an experiment with a fixed compute budget across training sizes.
+
+The 95% intervals resample questions while keeping each pair of answers and
+its routing decision together. They describe sampling uncertainty on this
+workload, conditional on the saved models and rules. They do not include
+training-seed variation or adjust jointly for all comparisons, and do not
+protect against future domain changes. With 3,000 test
+questions, one changed answer is about 0.033 points; the actual uncertainty
+also depends on how often the two policies disagree in correctness.
+
+Warm latency is measured separately from checkpoint loading and answer
+generation. The [serving guide](serving.md) records hardware, network inclusion,
+and the limits of the small sequential timing sample.
diff --git a/examples/train-a-router/docs/serving.md b/examples/train-a-router/docs/serving.md
new file mode 100644
index 000000000..a6f98c98d
--- /dev/null
+++ b/examples/train-a-router/docs/serving.md
@@ -0,0 +1,183 @@
+# Run your router on new requests
+
+A router adds work before answer generation. Whether it is worthwhile depends
+on the money it saves, the quality it preserves, and the time and infrastructure
+needed to make each decision. Follow the [README](../README.md) to train it;
+this page covers live use and serving trade-offs.
+
+## What runs for each request
+
+| | Laya decision model | Qwen embeddings + two classifiers |
+| --- | --- | --- |
+| Router size | About 421 million parameters | About 4 billion encoder parameters, plus two tiny classifiers |
+| Decision work | Two correctness questions through one shared checkpoint | One embedding call, then two weighted sums on CPU |
+| What training changes | Rank-16 attention LoRA updates and the decision head | Two sets of 2,560 weights and a bias; the encoder stays fixed |
+| Where inference runs | Your CPU or NVIDIA CUDA machine | Encoder in SIE Cloud or your SIE server; classifiers on your CPU |
+| Training requirement | NVIDIA CUDA GPU | CPU with the downloaded embeddings |
+
+The LoRA updates are merged into the saved checkpoint. More training examples
+do not increase either router's model size or number of inference calls. Qwen
+is roughly ten times larger than Laya, with different architecture and
+pretraining. This comparison does not isolate parameter count as the cause
+of a quality difference.
+
+The saved cutoff chooses the answer model. An always-strong or always-cheap
+policy skips router inference entirely.
+
+## Use SIE Cloud
+
+Run these commands from `examples/train-a-router` after preparing and training
+`run-output/my-router`. Create an account and API key in the
+[SIE Console](https://console.superlinked.com), then use the credits available
+in your account. The embedding route runs from a laptop without a local model.
+
+```sh
+# Install the CPU dependencies and connect the SDK to your SIE Cloud account.
+uv sync --frozen --group outcomes
+export SIE_BASE_URL=https://api.superlinked.com
+export SIE_API_KEY='paste-your-key'
+```
+
+```sh
+# Embed one new arithmetic question, select an answer model, and request its answer.
+uv run --frozen --group outcomes python router.py answer \
+ --study evidence --run run-output/my-router --method embedding \
+ --question 'A rectangle has perimeter 30 and length 9. What is its area?' \
+ --target 1 --output run-output/cloud-answer
+```
+
+The output directory contains the decision, answer, and API receipts.
+`result.json` reports the selected model and observed credits. The new answer
+is not graded because you did not supply an answer key. Use a fresh output
+folder for each request. `--target 1` selects the rule calibrated for a
+one-point quality gap; it does not guarantee this particular answer is correct.
+
+```sh
+# Run your adapted Laya checkpoint on CPU and use SIE Cloud for the selected answer model.
+uv run --frozen --group outcomes --group training python router.py answer \
+ --study evidence --run run-output/my-router --method lora --device cpu \
+ --question 'A rectangle has perimeter 30 and length 9. What is its area?' \
+ --target 1 --output run-output/lora-cloud-answer
+```
+
+Use `--device cuda` for NVIDIA inference. `--method teacher` uses the classifiers
+trained from teacher labels; `--method laya` uses the unchanged decision model.
+Omit `--run` to use the recorded routers from the download. Recorded Laya
+inference also needs the optional model files:
+
+```sh
+# Download the optional verified decision-model checkpoints for local inference.
+uv run --frozen --group outcomes python router.py fetch --study evidence --weights
+```
+
+Each answer uses one initial call and, only when needed, one final-answer
+formatting call. Inference is not automatically retried. A failed request
+leaves a receipt to inspect before trying again. Both answer models and their
+formatting calls passed separate live production checks.
+
+## Run the same models on your own SIE server
+
+The [self-hosted setup](../self-host/README.md) supplies this example's pinned
+model configurations and a Docker Compose launcher. It requires Linux and
+three NVIDIA GPUs; the validation hardware is H100 80 GB. The encoder, smaller
+answer model, and stronger answer model run in separate environments because
+their CUDA and SGLang requirements differ. This is the full inference setup;
+LoRA training alone needs one CUDA GPU.
+
+Once your endpoint is running, the same public `SIEClient` calls work:
+
+```sh
+# Send a routed question to the local three-model SIE deployment.
+SIE_BASE_URL=http://127.0.0.1:8080 SIE_API_KEY='' \
+uv run --frozen --group outcomes python router.py answer \
+ --study evidence --run run-output/my-router --method embedding \
+ --question 'A rectangle has perimeter 30 and length 9. What is its area?' \
+ --target 1 --self-hosted --output run-output/self-hosted-answer
+```
+
+The empty key is for the supplied trusted, local deployment. Use your own
+endpoint and key when connecting to an authenticated deployment.
+`--self-hosted` accepts responses without Cloud billing receipts; it keeps
+model-revision checks and does not start the server. Your GPU hosting costs
+remain separate from the report's equivalent Cloud-price estimates.
+
+| Purpose | Model | Pinned revision |
+| --- | --- | --- |
+| Decision model before adaptation | `convaiinnovations/laya` | `aa8c91ca088ec597df95a0d1c76b3063cb2ae5e8` |
+| Embedding model | `Qwen/Qwen3-Embedding-4B` | `5cf2132abc99cad020ac570b19d031efec650f2b` |
+| Cheaper answer model | `Qwen/Qwen3.5-4B` | `851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a` |
+| Stronger answer model | `Qwen/Qwen3.8-27B-FP8` | `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a` |
+
+Embedding results also depend on the fixed instruction, dimensions, and
+normalization in [serving.py](../serving.py). Changing the encoder or instruction
+requires new embeddings, classifier training, and evaluation. Matching model
+weights alone does not guarantee identical outputs across runtimes or hardware.
+
+## Latency: compare warm decisions separately from startup
+
+We measured the same 20 training questions, one at a time, after one separate
+warmup question. No answer generation is included.
+
+| Router and serving setup | Median | 95th percentile |
+| --- | ---: | ---: |
+| Unchanged Laya, resident on a Modal H100 | 14.65 ms | 18.74 ms |
+| LoRA with 2,000 examples, resident on the same H100 | 15.23 ms | 16.00 ms |
+| The same LoRA checkpoint, resident on a Modal CPU worker | 436.77 ms | 550.16 ms |
+| SIE Cloud embedding + local classifiers, client on Modal CPU | 107.67 ms | 128.58 ms |
+
+The local classifier work in the embedding route took **0.36 ms median**.
+Most of its measured time is the embedding request and network round trip.
+The 21 Cloud calls used 21 credits, or **$0.00021**. The main quality report
+uses its own measured embedding charges; this small timing sample does not
+replace them.
+
+Laya's measurement includes tokenization, both correctness predictions, and
+score calculation, with the model already loaded. It excludes HTTP. Cloud
+measurements include network time from a Modal CPU worker. These are different
+serving setups, not a comparison on equal hardware, and 20 sequential requests
+do not measure concurrent throughput. The native CPU probe used eight Torch
+threads and FP32 weights on a Modal worker; it is not a laptop measurement.
+
+```sh
+# Time Cloud embedding decisions on the same saved 20 questions plus one warmup.
+uv run --frozen --group outcomes python benchmark_arithmetic.py \
+ --inputs evidence/arithmetic/timing/inputs.json \
+ --model run-output/my-router/embedding-2000.json \
+ --output run-output/warm-embedding.json
+```
+
+This command makes 21 embedding calls. Add `--self-hosted` and set your endpoint
+variables to measure your own server.
+
+```sh
+# On your CUDA machine, time your resident Laya router on the same questions.
+uv run --frozen --group outcomes --group training python benchmark_arithmetic_native.py \
+ --inputs evidence/arithmetic/timing/inputs.json \
+ --native run-output/my-router/native-2000 --device cuda --variant lora \
+ --output run-output/warm-laya.json
+```
+
+Use `--device cpu` to measure your CPU. The timing comparison uses a fixed zero
+score threshold; it does not select or change a routing policy.
+
+The single-question answer CLI verifies and loads a Laya checkpoint each time.
+Its cold startup time is therefore larger than the warm numbers above. A
+persistent application keeps the verified model loaded and reuses it. Report
+checkpoint download, loading, router inference, and answer generation separately.
+
+## Turn savings into a hosting budget
+
+An embedding call is cheap, but the router's **choice of answer model** can
+change the bill substantially. Sending more questions to 27B can increase the
+total cost even when embedding costs stay almost unchanged.
+
+The report's embedding savings include measured encoder credits. Laya's
+answer-model savings leave an allowance for hosting the router: if the answer
+bill falls by $20 per 10,000 requests, you can spend at most $20 on router
+hosting before those savings disappear. That is an example calculation;
+use the measured allowance for your chosen policy. Idle GPU time and utilization
+matter. Tiny local classifier CPU work is also unpriced.
+
+Training, teacher labeling, and collecting evaluation answers are separate
+one-time costs. The [methodology](methodology.md) explains which charges were
+observed and which are estimates from recorded token counts.
diff --git a/examples/train-a-router/evidence.py b/examples/train-a-router/evidence.py
new file mode 100644
index 000000000..5e1b85b40
--- /dev/null
+++ b/examples/train-a-router/evidence.py
@@ -0,0 +1,120 @@
+"""Download and verify the immutable public experiment; weights are opt-in."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import tempfile
+import urllib.request
+from pathlib import Path, PurePosixPath
+
+from data import file_hash
+
+REVISION = "4136e68feaa90b5f50dce9f30df212982d393356"
+MANIFEST_SHA256 = "6fa7b8e94d5894177cf702f00df9daacabcdc8bbab237eac3f0a15eb28c3e709"
+FOLDER = "evaluations/2026-10-18/train-a-router-grouped-v2"
+LORA_REVISION = "d884d137386ccc4c89eb6d78449197a4e4be9892"
+LORA_MANIFEST_SHA256 = "f9f2f521de0dc6bc1711e7695378702efc03a55f017d29b7e52dc6552ae5872f"
+LORA_FOLDER = "evaluations/2026-10-18/train-a-router-lora-followup"
+SOLVER_REVISION = "7810c9631d8fca236ea8a074bab86b25f2b3fa0e"
+SOLVER_MANIFEST_SHA256 = "9df32300204ac4f24ffcd032c824619ddf9d0db100179d8d947397d191db6a59"
+SOLVER_FOLDER = "evaluations/2026-10-19/train-a-router-answer-quality"
+ARITHMETIC_REVISION = "71e0d5109e46f8b9c698806e7d3e47b2aa0da87d"
+ARITHMETIC_MANIFEST_SHA256 = "45e0d829c043cb9aaec2ab77859a93a34fda0a3e8765ca89ca81d6e0461d725d"
+ARITHMETIC_FOLDER = "evaluations/2026-10-19/train-a-router-arithmetic"
+BASE_URL = "https://huggingface.co/datasets/superlinked/sie-task-evidence/resolve/"
+
+
+def entries(body, *, manifest_sha256=None):
+ if hashlib.sha256(body).hexdigest() != (manifest_sha256 or MANIFEST_SHA256):
+ raise ValueError("Manifest differs from the pinned public experiment")
+ manifest = json.loads(body)
+ for name, item in manifest["files"].items():
+ path = PurePosixPath(name)
+ if path.is_absolute() or ".." in path.parts or "\\" in name or str(path) != name:
+ raise ValueError("Unsafe artifact path")
+ if not isinstance(item["bytes"], int) or not 0 <= item["bytes"] <= 2_000_000_000:
+ raise ValueError("Invalid artifact size")
+ return manifest["files"]
+
+
+def verify_file(path, expected):
+ if path.stat().st_size != expected["bytes"] or file_hash(path) != expected["sha256"]:
+ raise ValueError(f"Artifact hash/size differs: {path.name}")
+
+
+def verify(directory, *, manifest_sha256=None):
+ files = entries((directory / "manifest.json").read_bytes(), manifest_sha256=manifest_sha256)
+ for name, expected in files.items():
+ if expected["group"] == "core":
+ verify_file(directory / name, expected)
+ return files
+
+
+def fetch(
+ directory, weights=None, *, lora_study=False, solver_study=False, arithmetic_study=False,
+ solver_records=False, solver_weights=False,
+):
+ if sum((lora_study, solver_study, arithmetic_study)) > 1:
+ raise ValueError("Choose one study")
+ answer_study = solver_study or arithmetic_study
+ if answer_study and weights:
+ raise ValueError("Choose one study; solver evidence contains no router weights")
+ if solver_records and not answer_study:
+ raise ValueError("--solver-records requires --solver-study")
+ if solver_weights and not answer_study:
+ raise ValueError("--solver-weights requires --solver-study")
+ if lora_study and weights not in (None, "lora"):
+ raise ValueError("The LoRA follow-up supports only --weights lora")
+ revision = LORA_REVISION if lora_study else REVISION
+ folder = LORA_FOLDER if lora_study else FOLDER
+ manifest_sha256 = LORA_MANIFEST_SHA256 if lora_study else MANIFEST_SHA256
+ if solver_study:
+ revision, folder, manifest_sha256 = SOLVER_REVISION, SOLVER_FOLDER, SOLVER_MANIFEST_SHA256
+ if arithmetic_study:
+ revision, folder, manifest_sha256 = ARITHMETIC_REVISION, ARITHMETIC_FOLDER, ARITHMETIC_MANIFEST_SHA256
+ if "PENDING_PUBLICATION" in (revision, manifest_sha256):
+ raise RuntimeError("Public evidence has not been pinned yet")
+ destination = directory / "manifest.json"
+ if destination.exists() and file_hash(destination) != manifest_sha256:
+ raise ValueError("Destination contains a different experiment")
+ base = f"{BASE_URL}{revision}/{folder}/"
+ with urllib.request.urlopen(base + "manifest.json", timeout=60) as response:
+ body = response.read(2_000_001 if answer_study else 200_001)
+ files = entries(body, manifest_sha256=manifest_sha256)
+ directory.mkdir(parents=True, exist_ok=True)
+ groups = {"core"}
+ if solver_records:
+ groups.add("answers")
+ if solver_weights:
+ groups.add("native-weights")
+ if weights:
+ groups.update(f"seed89-{v}" for v in (("full", "lora") if weights == "both" else (weights,)))
+ for name, expected in files.items():
+ if expected["group"] not in groups:
+ continue
+ path = directory / name
+ if path.exists():
+ verify_file(path, expected)
+ continue
+ path.parent.mkdir(parents=True, exist_ok=True)
+ with tempfile.NamedTemporaryFile(dir=path.parent, prefix=".download-", delete=False) as handle:
+ temporary = Path(handle.name)
+ try:
+ with urllib.request.urlopen(base + name, timeout=300) as response:
+ size = 0
+ for chunk in iter(lambda: response.read(1024 * 1024), b""):
+ size += len(chunk)
+ if size > expected["bytes"]:
+ raise ValueError("Artifact exceeds its declared size")
+ handle.write(chunk)
+ handle.flush()
+ verify_file(temporary, expected)
+ temporary.replace(path)
+ finally:
+ temporary.unlink(missing_ok=True)
+ print(f"Verified {name}", flush=True)
+ if destination.exists() and destination.read_bytes() != body:
+ raise ValueError("Destination contains a different experiment")
+ destination.write_bytes(body)
+ verify(directory, manifest_sha256=manifest_sha256)
diff --git a/examples/train-a-router/finetune.py b/examples/train-a-router/finetune.py
new file mode 100644
index 000000000..12f778dfc
--- /dev/null
+++ b/examples/train-a-router/finetune.py
@@ -0,0 +1,489 @@
+"""Train native Laya decisions on a CUDA or Apple Silicon GPU.
+
+Training reads train.jsonl, validation.jsonl and calibration.jsonl. Evaluation is a separate
+command, after checkpoint/threshold selection has been written and hashed.
+Run with ``uv run --frozen --group training python finetune.py --help``.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import importlib.metadata
+import json
+import random
+import shutil
+import time
+from dataclasses import asdict
+from pathlib import Path
+
+import numpy as np
+
+BASE = "convaiinnovations/laya"
+REVISION = "aa8c91ca088ec597df95a0d1c76b3063cb2ae5e8"
+MAX_LEN = 512
+HEAD_LEN = 256
+
+LORA_RECIPES = {
+ "original": {
+ "rank": 16,
+ "encoder_lr": 2e-4,
+ "head_lr": 1e-4,
+ "dropout": 0.05,
+ "weight_decay": 0.01,
+ "smoothing": 0.0,
+ "head": "all",
+ },
+ "slow": {
+ "rank": 16,
+ "encoder_lr": 5e-5,
+ "head_lr": 1e-5,
+ "dropout": 0.05,
+ "weight_decay": 0.01,
+ "smoothing": 0.0,
+ "head": "all",
+ },
+ "regularized": {
+ "rank": 8,
+ "encoder_lr": 5e-5,
+ "head_lr": 1e-5,
+ "dropout": 0.1,
+ "weight_decay": 0.1,
+ "smoothing": 0.1,
+ "head": "all",
+ },
+ "frozen-head": {
+ "rank": 8,
+ "encoder_lr": 1e-4,
+ "head_lr": 1e-5,
+ "dropout": 0.1,
+ "weight_decay": 0.1,
+ "smoothing": 0.1,
+ "head": "none",
+ },
+ "scorer-only": {
+ "rank": 8,
+ "encoder_lr": 1e-4,
+ "head_lr": 1e-5,
+ "dropout": 0.1,
+ "weight_decay": 0.1,
+ "smoothing": 0.1,
+ "head": "scorer",
+ },
+}
+LORA_RECIPES["top-layers"] = {**LORA_RECIPES["slow"], "last_layers": 4}
+
+
+def lora_targets(recipe):
+ if recipe.get("last_layers"):
+ # The pinned ModernBERT-large encoder has 28 layers.
+ indices = "|".join(str(i) for i in range(28 - recipe["last_layers"], 28))
+ return rf".*layers\.({indices})\.attn\.(Wqkv|Wo)"
+ return r".*attn\.(Wqkv|Wo)"
+
+
+def write(path, value):
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(json.dumps(value, indent=2, allow_nan=False) + "\n")
+
+
+def sha(path):
+ h = hashlib.sha256()
+ with path.open("rb") as handle:
+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+ h.update(chunk)
+ return h.hexdigest()
+
+
+def cutoff(scores, labels):
+ """Calibration balanced accuracy; ties prefer the native zero-margin boundary."""
+ scores, labels = np.asarray(scores), np.asarray(labels)
+ if len(scores) != len(labels) or not np.isfinite(scores).all() or set(labels.tolist()) != {0, 1}:
+ raise ValueError("Calibration needs finite aligned scores and both classes")
+ values = np.unique(scores)
+ candidates = np.r_[values[0] - 1, (values[:-1] + values[1:]) / 2, values[-1] + 1, 0.0]
+
+ def quality(t):
+ prediction = scores >= t
+ return np.mean([np.mean(prediction[labels == label] == label) for label in (0, 1)])
+
+ return float(max(candidates, key=lambda t: (quality(t), -abs(t), -t)))
+
+
+def items(lt, tok, path, smoothing=0.0):
+ rows = lt.read_jsonl(str(path))
+ result, skipped = lt.items_from_rows(tok, rows, MAX_LEN, HEAD_LEN, label_smoothing=smoothing)
+ if skipped or len(result) != len(rows):
+ raise ValueError(f"Training data must not silently drop rows: {skipped}")
+ return result
+
+
+def runtime_settings(device_type):
+ if device_type == "mps":
+ # Torch 2.9.1 checkpoint recomputation does not restore MPS dropout RNG.
+ # Small batches keep memory bounded without that optimization.
+ return {"micro_batch": 1, "grad_accum": 16, "gradient_checkpointing": False, "evaluation_batch": 4}
+ return {"micro_batch": 8, "grad_accum": 2, "gradient_checkpointing": True, "evaluation_batch": 16}
+
+
+def empty_cache(torch, device):
+ if device.type == "cuda":
+ torch.cuda.empty_cache()
+ elif device.type == "mps":
+ torch.mps.empty_cache()
+
+
+def synchronize(torch, device):
+ if device.type == "cuda":
+ torch.cuda.synchronize()
+ elif device.type == "mps":
+ torch.mps.synchronize()
+
+
+def scores(lt, model, tok, data, device):
+ records = lt.calibration_records(
+ model, tok, data, device, MAX_LEN, HEAD_LEN, batch_size=runtime_settings(device.type)["evaluation_batch"]
+ )
+ logits = np.array([r[1] for r in records], dtype=np.float64)
+ labels = np.array([int(np.argmax(r[2])) for r in records])
+ if logits.shape != (len(data), 2) or not np.isfinite(logits).all():
+ raise ValueError("Expected finite two-option logits")
+ return logits[:, 1] - logits[:, 0], labels
+
+
+def auc(values, labels):
+ """Pairwise ROC AUC, with half credit for equal scores."""
+ values, labels = np.asarray(values), np.asarray(labels)
+ if len(values) != len(labels) or not np.isfinite(values).all() or set(labels.tolist()) != {0, 1}:
+ raise ValueError("AUC needs finite aligned scores and both classes")
+ difference = values[labels == 1, None] - values[labels == 0][None, :]
+ return float(np.mean((difference > 0) + 0.5 * (difference == 0)))
+
+
+def better(candidate, best, metric):
+ if best is None:
+ return True
+ if metric == "auc":
+ return candidate["auc"] > best["auc"]
+ return candidate["cross_entropy"] < best["cross_entropy"]
+
+
+def metrics(values, labels, threshold):
+ predictions = (values >= threshold).astype(int)
+ return {
+ "n": len(labels),
+ "accuracy": float(np.mean(predictions == labels)),
+ "balanced_accuracy": float(np.mean([np.mean(predictions[labels == v] == v) for v in (0, 1)])),
+ "errors": int(np.sum(predictions != labels)),
+ "false_routine": int(np.sum((predictions == 0) & (labels == 1))),
+ "false_advanced": int(np.sum((predictions == 1) & (labels == 0))),
+ "scores": values.tolist(),
+ "labels": labels.tolist(),
+ "predictions": predictions.tolist(),
+ }
+
+
+def train(args):
+ import laya.train as lt
+ import torch
+ from huggingface_hub import snapshot_download
+ from peft import LoraConfig, get_peft_model
+ from safetensors.torch import save_file
+
+ output = args.output
+ output.mkdir(parents=True, exist_ok=False)
+ device = torch.device(args.device)
+ runtime = runtime_settings(device.type)
+ torch.manual_seed(args.seed)
+ random.seed(args.seed)
+ np.random.seed(args.seed)
+ config = lt.TrainConfig(
+ epochs=8,
+ micro_batch=runtime["micro_batch"],
+ grad_accum=runtime["grad_accum"],
+ encoder_lr=2.5e-5,
+ head_lr=1e-4,
+ seed=args.seed,
+ max_len=MAX_LEN,
+ head_max_len=HEAD_LEN,
+ loss="soft-ce",
+ shuffle_options=("choice",),
+ gradient_checkpointing=runtime["gradient_checkpointing"],
+ log_every=32,
+ )
+ recipe = LORA_RECIPES[args.recipe]
+ plan = {
+ "base": BASE,
+ "revision": REVISION,
+ "config": asdict(config),
+ "runtime": runtime,
+ "variants": args.variants,
+ "selection": (
+ "Maximum validation ROC AUC, including epoch zero; ties retain earlier epoch. "
+ if args.validation_metric == "auc"
+ else "Minimum validation cross entropy; ties retain earlier epoch. "
+ )
+ + "Cutoff fitted once on separate calibration after selection.",
+ "validation_metric": args.validation_metric,
+ "recipe": args.recipe,
+ "export_dtype": args.export_dtype,
+ "lora": {
+ **recipe,
+ "alpha": 2 * recipe["rank"],
+ "targets": lora_targets(recipe),
+ "frozen_head_dropout": "Retained as training-time regularization",
+ "export": "Merge encoder adapters into the unchanged native architecture",
+ },
+ "data_sha256": {
+ n: sha(args.data / n) for n in ("train.jsonl", "validation.jsonl", "calibration.jsonl", "questions.json")
+ },
+ "test_access": "None. Separate evaluate command only after frozen.json exists.",
+ "versions": {n: importlib.metadata.version(n) for n in ("laya", "torch", "transformers", "peft")},
+ "device": torch.cuda.get_device_name() if device.type == "cuda" else str(device),
+ "reporting": "All declared seeds and model families; no test-based family or seed selection",
+ }
+ write(output / "plan.json", plan)
+ shutil.copy2(args.data / "questions.json", output / "questions.json")
+ snapshot = snapshot_download(
+ BASE,
+ revision=REVISION,
+ allow_patterns=[
+ "rl_agent_config.json",
+ "model.safetensors",
+ "encoder/*",
+ "tokenizer/*",
+ ],
+ )
+ base = output / "base"
+ # A shared Hub snapshot may already contain unrelated sub-models. Copy only
+ # the native root checkpoint, even when those other files are cached.
+ base.mkdir()
+ for name in ("rl_agent_config.json", "model.safetensors"):
+ shutil.copy2(Path(snapshot) / name, base / name)
+ for name in ("encoder", "tokenizer"):
+ shutil.copytree(Path(snapshot) / name, base / name, symlinks=False)
+ summary = {}
+ for variant in plan["variants"]:
+ torch.manual_seed(args.seed)
+ model, tok, cfg = lt.load_checkpoint(str(base))
+ cfg.update(max_len=MAX_LEN, head_max_len=HEAD_LEN)
+ model.to(device)
+ training = items(lt, tok, args.data / "train.jsonl", recipe["smoothing"] if variant == "lora" else 0.0)
+ validation = items(lt, tok, args.data / "validation.jsonl")
+ destination = base if variant == "base" else output / variant
+ if variant == "lora":
+ model.encoder = get_peft_model(
+ model.encoder,
+ LoraConfig(
+ r=recipe["rank"],
+ lora_alpha=2 * recipe["rank"],
+ lora_dropout=recipe["dropout"],
+ target_modules=lora_targets(recipe),
+ bias="none",
+ ),
+ )
+ if args.recipe != "original":
+ for name, parameter in model.named_parameters():
+ if not name.startswith("encoder."):
+ parameter.requires_grad_(
+ not name.startswith("act_head.")
+ and (recipe["head"] == "all" or (recipe["head"] == "scorer" and name.startswith("scorer.")))
+ )
+ count = {
+ "total": sum(p.numel() for p in model.parameters()),
+ "trainable": sum(p.numel() for p in model.parameters() if p.requires_grad),
+ }
+ history = []
+ best = None
+ started = time.perf_counter()
+
+ def select(epoch, loss, model=model, tok=tok, validation=validation, history=history, variant=variant):
+ nonlocal best
+ values, labels = scores(lt, model, tok, validation, device)
+ threshold = 0.0
+ result = metrics(values, labels, threshold)
+ result.update(
+ epoch=epoch + 1,
+ train_loss=loss,
+ threshold=threshold,
+ cross_entropy=float(np.mean(np.logaddexp(0, values) - labels * values)),
+ auc=auc(values, labels),
+ )
+ history.append(result)
+ if better(result, best, args.validation_metric):
+ best = dict(result)
+ if variant != "base":
+ # Keep the unmerged PEFT weights private to this training run.
+ torch.save(model.state_dict(), output / f"{variant}-best.pt")
+ write(output / f"{variant}-history.json", history)
+ print(
+ json.dumps(
+ {
+ "variant": variant,
+ "epoch": epoch + 1,
+ "validation_accuracy": result["accuracy"],
+ "validation_ce": result["cross_entropy"],
+ "validation_auc": result["auc"],
+ "loss": loss,
+ }
+ ),
+ flush=True,
+ )
+ model.train()
+
+ if variant == "base":
+ select(-1, None)
+ calibration = items(lt, tok, args.data / "calibration.jsonl")
+ cal_values, cal_labels = scores(lt, model, tok, calibration, device)
+ best["threshold"] = cutoff(cal_values, cal_labels)
+ else:
+ config.encoder_lr = recipe["encoder_lr"] if variant == "lora" else 2.5e-5
+ config.head_lr = recipe["head_lr"] if variant == "lora" else 1e-4
+ config.weight_decay = recipe["weight_decay"] if variant == "lora" else 0.01
+ config.label_smoothing = recipe["smoothing"] if variant == "lora" else 0.0
+ write(output / f"{variant}-train-config.json", asdict(config))
+ if args.validation_metric == "auc":
+ select(-1, None)
+ lt.train_model(model, tok, training, config, device, MAX_LEN, HEAD_LEN, on_epoch_end=select)
+ select = None
+ model.load_state_dict(torch.load(output / f"{variant}-best.pt", weights_only=True, map_location=device))
+ model.eval()
+ before, _ = scores(lt, model, tok, validation, device)
+ if variant == "lora":
+ model.encoder = model.encoder.merge_and_unload()
+ after, _ = scores(lt, model, tok, validation, device)
+ merge_error = float(np.max(np.abs(before - after)))
+ if merge_error > 1e-3:
+ raise ValueError(f"LoRA merge changed margins: {merge_error}")
+ write(output / "lora-merge-parity.json", {"max_margin_error": merge_error})
+ lt.save_checkpoint(model, tok, cfg, str(destination))
+ if args.export_dtype == "float32":
+ save_file(
+ {name: tensor.detach().float().cpu().contiguous() for name, tensor in model.state_dict().items()},
+ str(destination / "model.safetensors"),
+ )
+ del model
+ empty_cache(torch, device)
+ model, tok, _ = lt.load_checkpoint(str(destination))
+ model.to(device).eval()
+ exported, _ = scores(lt, model, tok, validation, device)
+ error = float(np.max(np.abs(exported - before)))
+ if error > 0.03:
+ raise ValueError(f"Native export margin drift exceeds tolerance: {error}")
+ # Native serialization uses half-precision weights. Fit the cutoff on
+ # the checkpoint users actually load, after merging and serialization.
+ calibration = items(lt, tok, args.data / "calibration.jsonl")
+ cal_values, cal_labels = scores(lt, model, tok, calibration, device)
+ best["threshold"] = cutoff(cal_values, cal_labels)
+ changes = int(np.sum((exported >= best["threshold"]) != (before >= best["threshold"])))
+ write(
+ output / f"{variant}-export-parity.json",
+ {
+ "max_margin_error": error,
+ "validation_decision_changes_at_final_cutoff": changes,
+ "calibration": "Cutoff fitted on serialized checkpoint",
+ },
+ )
+ summary[variant] = {
+ "threshold": best["threshold"],
+ "threshold_source": "separate calibration split, serialized checkpoint",
+ "validation_cross_entropy": best["cross_entropy"],
+ "validation_auc": best["auc"],
+ "seed": args.seed,
+ "epoch": best["epoch"],
+ "validation_accuracy_at_zero": best["accuracy"],
+ "parameters": count,
+ "elapsed_seconds": time.perf_counter() - started,
+ "checkpoint_sha256": sha(destination / "model.safetensors"),
+ }
+ write(output / f"{variant}-calibration.json", metrics(cal_values, cal_labels, best["threshold"]))
+ write(output / "selection.json", summary)
+ select = None
+ del model
+ empty_cache(torch, device)
+ write(output / "frozen.json", {"plan_sha256": sha(output / "plan.json"), "selected": summary})
+ print("FROZEN", json.dumps(summary), flush=True)
+
+
+def evaluate(args):
+ import laya.train as lt
+ import torch
+
+ frozen = json.loads((args.output / "frozen.json").read_text())
+ plan = json.loads((args.output / "plan.json").read_text())
+ if (
+ sha(args.output / "plan.json") != frozen["plan_sha256"]
+ or sha(args.questions) != plan["data_sha256"]["questions.json"]
+ ):
+ raise ValueError("Plan or question changed after freeze")
+ rows = json.loads(args.evaluation.read_text())
+ questions = json.loads(args.questions.read_text())
+ data = [
+ {
+ "state": row["request"],
+ "questions": questions,
+ "expected": {"difficulty": ("routine", "advanced")[row["label"]]},
+ }
+ for row in rows
+ ]
+ result = {
+ "frozen_sha256": sha(args.output / "frozen.json"),
+ "data_sha256": sha(args.evaluation),
+ "ids": [r["id"] for r in rows],
+ "models": {},
+ }
+ device = torch.device(args.device)
+ for name, selected in frozen["selected"].items():
+ path = args.output / name
+ if sha(path / "model.safetensors") != selected["checkpoint_sha256"]:
+ raise ValueError("Checkpoint changed after freeze")
+ model, tok, _ = lt.load_checkpoint(str(path))
+ model.to(device).eval()
+ encoded, skipped = lt.items_from_rows(tok, data, MAX_LEN, HEAD_LEN)
+ if skipped or len(encoded) != len(data):
+ raise ValueError(f"Evaluation silently dropped rows: {skipped}")
+ values, labels = scores(lt, model, tok, encoded, device)
+ measured = metrics(values, labels, selected["threshold"])
+ # Same warm model, text, device and batch size across all three variants.
+ latencies = []
+ for index in range(min(35, len(encoded))):
+ synchronize(torch, device)
+ start = time.perf_counter()
+ scores(lt, model, tok, encoded[index : index + 1], device)
+ synchronize(torch, device)
+ if index >= 5:
+ latencies.append(time.perf_counter() - start)
+ measured["batch1_seconds"] = {
+ "median": float(np.median(latencies)),
+ "p95": float(np.quantile(latencies, 0.95)),
+ "samples": len(latencies),
+ }
+ result["models"][name] = measured
+ write(args.output / f"evaluation-{args.evaluation.stem}.json", result)
+ print(name, measured["accuracy"], measured["errors"], measured["batch1_seconds"], flush=True)
+ del model
+ empty_cache(torch, device)
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ sub = parser.add_subparsers(dest="command", required=True)
+ training = sub.add_parser("train")
+ training.add_argument("--data", type=Path, required=True)
+ training.add_argument("--seed", type=int, choices=(89, 137, 211), default=89)
+ training.add_argument("--recipe", choices=tuple(LORA_RECIPES), default="original")
+ training.add_argument("--export-dtype", choices=("float16", "float32"), default="float16")
+ training.add_argument("--validation-metric", choices=("cross-entropy", "auc"), default="cross-entropy")
+ training.add_argument("--variants", nargs="+", choices=("base", "full", "lora"), default=["base", "full", "lora"])
+ testing = sub.add_parser("evaluate")
+ testing.add_argument("--evaluation", type=Path, required=True)
+ testing.add_argument("--questions", type=Path, required=True)
+ for child in (training, testing):
+ child.add_argument("--output", type=Path, required=True)
+ child.add_argument("--device", default="cuda")
+ args = parser.parse_args()
+ (train if args.command == "train" else evaluate)(args)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/generalization_data.py b/examples/train-a-router/generalization_data.py
new file mode 100644
index 000000000..0e7fbe09c
--- /dev/null
+++ b/examples/train-a-router/generalization_data.py
@@ -0,0 +1,164 @@
+"""Prepare an arithmetic transfer test without using any router outcomes.
+
+The competition test contains only 479 clean questions. This separately
+declared population measures transfer to arithmetic, not a larger sample of
+that same competition distribution. No arithmetic examples train the routers.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+from collections import Counter
+from fractions import Fraction
+from pathlib import Path
+
+import solver
+from benefit_expanded_gold_v2 import outcome_grader_metadata
+from data import SOURCE_HASHES, digest, duplicate_components, file_hash, render_request, write
+
+REVISION = "3101c7d5072418e28b9008a6636bde82a006892c"
+HASHES = {
+ "train.jsonl": "17f347dc51477c50d4efb83959dbb7c56297aba886e5544ee2aaed3024813465",
+ "test.jsonl": "3730d312f6e3440559ace48831e51066acaca737f6eabec99bccb9e4b3c39d14",
+}
+SOURCE_URL = f"https://raw.githubusercontent.com/openai/grade-school-math/{REVISION}/grade_school_math/data/"
+COUNT = 2000
+
+
+def final_answer(solution):
+ if solution.count("####") != 1:
+ raise ValueError("GSM8K source must have exactly one final-answer marker")
+ answer = solution.split("####", 1)[1].strip()
+ if re.fullmatch(r"[+-]?(?:\d+|\d{1,3}(?:,\d{3})+)(?:\.\d+)?", answer) is None:
+ raise ValueError("Unexpected arithmetic source answer")
+ Fraction(answer.replace(",", ""))
+ return answer
+
+
+def prepare(source, math_source, output):
+ import pyarrow.parquet as pq
+
+ source, math_source, output = map(Path, (source, math_source, output))
+ for name in ("inputs.json", "settings.json", "source-plan.json", "gold-for-scoring-only.json", "source-audit.json"):
+ if (output / name).exists():
+ raise FileExistsError(output / name)
+ rows = []
+ for name, expected in SOURCE_HASHES.items():
+ if file_hash(math_source / name) != expected:
+ raise ValueError("Competition source changed")
+ rows.extend(
+ {"id": f"e2h-amc-{name.split('.')[0]}-{index}", "problem": row["problem"], "dataset": "competition"}
+ for index, row in enumerate(pq.read_table(math_source / name).to_pylist())
+ )
+ for name, expected in HASHES.items():
+ if file_hash(source / name) != expected:
+ raise ValueError("Pinned arithmetic source changed")
+ for index, line in enumerate((source / name).read_text().splitlines()):
+ row = json.loads(line)
+ rows.append(
+ {
+ "id": f"gsm8k-{name.split('.')[0]}-{index}",
+ "problem": row["question"],
+ "gold": final_answer(row["answer"]),
+ "dataset": "arithmetic",
+ "original_split": name.split(".")[0],
+ }
+ )
+ retained, cross_source, duplicates = [], [], []
+ for component in duplicate_components(rows):
+ members = [rows[index] for index in component]
+ arithmetic = [row for row in members if row["dataset"] == "arithmetic"]
+ if not arithmetic:
+ continue
+ identities = sorted(row["id"] for row in members)
+ if len(arithmetic) != len(members):
+ cross_source.append(identities)
+ continue
+ preferred = [row for row in arithmetic if row["original_split"] == "test"] or arithmetic
+ representative = min(preferred, key=lambda row: digest(["arithmetic-transfer-representative-v1", row["id"]]))
+ retained.append(representative | {"family": digest(identities)})
+ if len(members) > 1:
+ duplicates.append({"ids": identities, "representative": representative["id"]})
+ official = sorted(
+ [row for row in retained if row["original_split"] == "test"],
+ key=lambda row: digest(["arithmetic-transfer-order-v1", row["id"]]),
+ )
+ supplement = sorted(
+ [row for row in retained if row["original_split"] == "train"],
+ key=lambda row: digest(["arithmetic-transfer-order-v1", row["id"]]),
+ )
+ if not 0 < len(official) <= COUNT <= len(retained):
+ raise ValueError("Source cannot satisfy the predeclared transfer population")
+ selected = official + supplement[: COUNT - len(official)]
+ inputs = [{"id": row["id"], "request": render_request("GSM8K", row["problem"])} for row in selected]
+ gold = {row["id"]: row["gold"] for row in selected}
+ metadata = [{key: row[key] for key in ("id", "family", "original_split")} for row in selected]
+ audit = {
+ "source_only": True,
+ "solver_outcomes_read": False,
+ "router_training_arithmetic_examples": 0,
+ "arithmetic_source_rows": sum(row["dataset"] == "arithmetic" for row in rows),
+ "eligible_families": len(retained),
+ "selected": len(selected),
+ "selected_by_original_split": dict(Counter(row["original_split"] for row in selected)),
+ "cross_source_families_removed": cross_source,
+ "within_arithmetic_duplicate_families": duplicates,
+ "duplicate_rule": "Same pinned canonical-text and five-token/numeric-masked Jaccard connected-components rule as data.py. One arithmetic representative per family; exclude any component touching any competition source question.",
+ "limitation": "These lexical duplicate checks do not prove the absence of every semantic relationship or pretrained-model contamination.",
+ }
+ write(output / "inputs.json", inputs)
+ write(output / "settings.json", solver.SETTINGS)
+ write(output / "gold-for-scoring-only.json", gold)
+ write(output / "rows.json", metadata)
+ write(output / "source-audit.json", audit)
+ plan = {
+ "format": "sie-arithmetic-transfer-source-v1",
+ "n": COUNT,
+ "source_revision": REVISION,
+ "source_url": SOURCE_URL,
+ "source_hashes": HASHES,
+ "competition_source_hashes": SOURCE_HASHES,
+ "files_sha256": {
+ name: file_hash(output / name)
+ for name in ("inputs.json", "settings.json", "gold-for-scoring-only.json", "rows.json", "source-audit.json")
+ },
+ "inputs_sha256": digest(inputs),
+ "settings_sha256": digest(solver.SETTINGS),
+ "request_format": "Competition: GSM8K followed by the public question; no answer, solution or original-split marker is supplied",
+ "selection": "Keep every eligible official-test family, then identity-hash-selected official-train families to reach2000. None are used in router fitting or cutoff selection.",
+ "study": "Separate arithmetic transfer evaluation; not a replacement for the competition test, a standard GSM8K leaderboard score, or an assumed production workload mix.",
+ "models": "Apply all already-declared original/expanded Laya and embedding routers plus the teacher-label control unchanged. No transfer training or cutoff retuning.",
+ "targets_pp": [1, 0, 2, 5],
+ "primary_target_pp": 1,
+ "quality": "Paired final-answer correctness relative to the pinned numeric reference; use frozen question-only arithmetic normalization/equivalence rules.",
+ "cost": "Self-hosted generation token usage priced with the same explicit uncachedCloud token-price snapshot; not observedCloud bills. Embedding charges, teacher costs and GPU acquisition costs are recorded separately.",
+ "precision_planning": "2000 independent questions gives approximately +/-0.98pp two-sided95% precision if routed-vs-strong correctness differs on5%; this is an assumption, not a promised realized interval.",
+ "source_code_sha256": {
+ name: file_hash(Path(__file__).with_name(name))
+ for name in ("generalization_data.py", "data.py", "solver.py", "benefit_expanded_gold_v2.py")
+ },
+ **outcome_grader_metadata(),
+ }
+ write(output / "source-plan.json", plan)
+ return {
+ "n": COUNT,
+ "by_original_split": audit["selected_by_original_split"],
+ "cross_source_families_removed": len(cross_source),
+ "duplicate_families": len(duplicates),
+ "plan_sha256": file_hash(output / "source-plan.json"),
+ }
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--source", type=Path, required=True)
+ parser.add_argument("--math-source", type=Path, required=True)
+ parser.add_argument("--output", type=Path, required=True)
+ args = parser.parse_args()
+ print(json.dumps(prepare(args.source, args.math_source, args.output), indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/plot_arithmetic.py b/examples/train-a-router/plot_arithmetic.py
new file mode 100644
index 000000000..123576546
--- /dev/null
+++ b/examples/train-a-router/plot_arithmetic.py
@@ -0,0 +1,143 @@
+"""Draw the saved routing comparison in dollars and answer-quality loss."""
+
+from __future__ import annotations
+
+import argparse
+import copy
+import json
+from pathlib import Path
+
+ORDER = (
+ "base",
+ "base_arithmetic_500",
+ "base_arithmetic_2000",
+ "lora_calibration_only",
+ "lora_500",
+ "lora_2000",
+ "embedding_calibration_only",
+ "embedding_500",
+ "embedding_2000",
+ "embedding_teacher_2000",
+ "embedding_teacher_matched_reference",
+)
+LABELS = {
+ "always_strong": "Always Qwen 27B",
+ "always_cheap": "Always Qwen 4B",
+ "base": "Laya, unchanged; original cost estimates",
+ "base_arithmetic_500": "Laya, unchanged; costs from 500 examples",
+ "base_arithmetic_2000": "Laya, unchanged; costs from 2,000 examples",
+ "lora_calibration_only": "Existing LoRA, new cutoff",
+ "lora_500": "Laya + LoRA, 500 examples",
+ "lora_2000": "Laya + LoRA, 2,000 examples",
+ "embedding_calibration_only": "Existing embedding router, new cutoff",
+ "embedding_500": "Embeddings + classifiers, 500 examples",
+ "embedding_2000": "Embeddings + classifiers, 2,000 examples",
+ "embedding_teacher_2000": "Embeddings + teacher labels",
+ "embedding_teacher_matched_reference": "Same questions, reference labels",
+}
+
+
+def label_for(report, name):
+ label = LABELS.get(name, name)
+ if name in ("embedding_teacher_2000", "embedding_teacher_matched_reference"):
+ count = report["methods"][name]["training_rows"]
+ return f"{label}, {count:,} examples"
+ return label
+
+
+def plot(report, output, *, target="1", cost_controls=None):
+ import matplotlib
+
+ matplotlib.use("Agg")
+ import matplotlib.pyplot as plt
+ from matplotlib.ticker import FuncFormatter
+
+ if report.get("format") != "sie-arithmetic-adaptation-report-v1":
+ raise ValueError("Expected the saved arithmetic adaptation report")
+ if cost_controls is not None:
+ if (
+ cost_controls.get("format") != "sie-arithmetic-cost-controls-report-v1"
+ or cost_controls.get("core_frozen_sha256") != report["frozen_sha256"]
+ or cost_controls.get("test_grades_sha256") != report["test_grades_sha256"]
+ or cost_controls.get("n") != report["n"]
+ or set(cost_controls.get("methods", {})) != {"base_arithmetic_500", "base_arithmetic_2000"}
+ ):
+ raise ValueError("Cost controls must use the same frozen study and test answers")
+ report = copy.deepcopy(report)
+ report["methods"].update(cost_controls["methods"])
+ rows = [(name, report["baselines"][name]) for name in ("always_strong", "always_cheap")]
+ for name in ORDER:
+ if name in report["methods"]:
+ rows.append((name, report["methods"][name]["policies"][str(target)]["metrics"]))
+ if len(rows) == 2:
+ raise ValueError("No evaluated router methods found")
+ output = Path(output)
+ output.mkdir(parents=True, exist_ok=False)
+ plt.rcParams.update({"font.size": 10, "axes.spines.top": False, "axes.spines.right": False})
+ fig, (costs, gaps) = plt.subplots(
+ 1, 2, figsize=(12.5, max(6.5, len(rows) * 0.48 + 2.4)), sharey=True,
+ gridspec_kw={"width_ratios": [1.1, 1]},
+ )
+ for index, (name, metric) in enumerate(rows):
+ known = metric.get("net_total_cost_usd_per1000") is not None
+ cost = 10 * metric["net_total_cost_usd_per1000" if known else "total_solver_cost_usd_per1000"]
+ color = "#325b81" if "embedding" in name or "teacher" in name else "#a76236"
+ if name.startswith("always_"):
+ color = "#68737d"
+ costs.barh(index, cost, height=0.58, color=color, alpha=0.85, hatch=None if known else "//")
+ costs.annotate(
+ f"${cost:.2f}" + (" + hosting" if not known else ""),
+ (cost, index), xytext=(5, 0), textcoords="offset points", va="center", fontsize=9,
+ )
+ gap = metric["gap_vs_strong_pp"]
+ interval = metric["paired_bootstrap_95ci"]["gap_vs_strong_pp"]
+ gaps.errorbar(
+ gap, index, xerr=[[max(0, gap - interval[0])], [max(0, interval[1] - gap)]],
+ fmt="o", color=color, markersize=6, capsize=3,
+ )
+ gaps.annotate(f"{gap:.2f}", (interval[1], index), xytext=(7, 0), textcoords="offset points", va="center", fontsize=9)
+ costs.set_yticks(range(len(rows)), [label_for(report, name) for name, _ in rows])
+ costs.invert_yaxis()
+ costs.xaxis.set_major_formatter(FuncFormatter(lambda value, _: f"${value:g}"))
+ costs.set_xlim(0, costs.get_xlim()[1] * 1.37)
+ costs.set_title("Cost per 10,000 requests", loc="left", fontweight="bold", pad=15)
+ costs.set_xlabel("Lower costs less")
+ gaps.axvline(0, color="#68737d", linestyle="--", linewidth=1)
+ gaps.axvline(float(target), color="#a1a8ae", linestyle=":", linewidth=1)
+ gaps.set_xlim(min(-0.5, gaps.get_xlim()[0]), gaps.get_xlim()[1] + 0.8)
+ gaps.set_title("Answer accuracy lost (percentage points)", loc="left", fontweight="bold", pad=15)
+ gaps.set_xlabel("Lower preserves more quality; bars show 95% intervals")
+ for axis in (costs, gaps):
+ axis.grid(axis="x", alpha=0.15)
+ axis.set_axisbelow(True)
+ axis.tick_params(axis="y", length=0)
+ fig.suptitle(f"Routing on {report['n']:,} held-out arithmetic questions", fontsize=16, y=0.99)
+ fig.text(
+ 0.02, 0.055,
+ f"Cutoffs selected for a {target}-point calibration target; measured test gaps can differ. "
+ "Costs use the saved Cloud token prices.\n"
+ "Embedding bars include the embedding charge. Hatched bars exclude unpriced Laya hosting. "
+ "Training is a separate one-time cost.",
+ fontsize=9, color="#495560", va="bottom",
+ )
+ fig.tight_layout(rect=(0, 0.12, 1, 0.95))
+ for suffix in ("svg", "png"):
+ fig.savefig(output / f"cost-and-quality-{target}pp.{suffix}", dpi=180, bbox_inches="tight")
+ plt.close(fig)
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--report", type=Path, required=True)
+ parser.add_argument("--output", type=Path, required=True)
+ parser.add_argument("--target", choices=("0", "1", "2", "5"), default="1")
+ parser.add_argument("--cost-controls", type=Path, help="Include the separate unchanged-Laya cost controls")
+ args = parser.parse_args()
+ plot(
+ json.loads(args.report.read_text()), args.output, target=args.target,
+ cost_controls=json.loads(args.cost_controls.read_text()) if args.cost_controls else None,
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/policies.py b/examples/train-a-router/policies.py
new file mode 100644
index 000000000..d4c8690d1
--- /dev/null
+++ b/examples/train-a-router/policies.py
@@ -0,0 +1,162 @@
+"""Frozen-encoder classifiers with separate selection and calibration data."""
+
+from __future__ import annotations
+
+import json
+
+import numpy as np
+
+from data import PROTOCOL, audit, digest, file_hash, write
+from serving import INSTRUCTION, MODEL, REVISION, vectors
+
+PENALTIES = (0.01, 0.1, 1.0, 10.0)
+
+
+def ridge(x, y, penalty):
+ if penalty <= 0 or x.ndim != 2 or len(x) != len(y) or len(y) < 2:
+ raise ValueError("Need positive regularization and aligned training data")
+ xmean, ymean = x.mean(axis=0), y.mean()
+ centered = x - xmean
+ weights = centered.T @ np.linalg.solve(centered @ centered.T + penalty * np.eye(len(x)), y - ymean)
+ return weights, float(ymean - xmean @ weights)
+
+
+def knn(bank, labels, queries, k=5):
+ if not 1 <= k <= len(bank):
+ raise ValueError("Invalid neighbor count")
+ # Stable sorting makes ties follow the declared training order.
+ indices = np.argsort(-(queries @ bank.T), axis=1, kind="stable")[:, :k]
+ return labels[indices].mean(axis=1)
+
+
+def cutoff(scores, labels):
+ scores, labels = np.asarray(scores), np.asarray(labels)
+ if len(scores) != len(labels) or not len(scores) or not np.isfinite(scores).all():
+ raise ValueError("Invalid calibration observations")
+ values = np.unique(scores)
+ candidates = np.r_[
+ np.nextafter(values[0], -np.inf), (values[:-1] + values[1:]) / 2, np.nextafter(values[-1], np.inf)
+ ]
+
+ # Balanced accuracy; ties prefer the default 0.5 score boundary.
+ def quality(t):
+ prediction = scores >= t
+ return np.mean([np.mean(prediction[labels == label] == label) for label in (0, 1)])
+
+ if set(labels.tolist()) != {0, 1}:
+ raise ValueError("Calibration needs both classes")
+ return float(max(candidates, key=lambda t: (quality(t), -abs(t - 0.5), -t)))
+
+
+def metrics(labels, predictions):
+ y, p = np.asarray(labels), np.asarray(predictions)
+ if y.shape != p.shape or not len(y):
+ raise ValueError("Unaligned predictions")
+ return {
+ "n": len(y),
+ "accuracy": float(np.mean(p == y)),
+ "errors": int(np.sum(p != y)),
+ "balanced_accuracy": float(np.mean([np.mean(p[y == v] == v) for v in (0, 1)])),
+ "false_routine": int(np.sum((p == 0) & (y == 1))),
+ "false_advanced": int(np.sum((p == 1) & (y == 0))),
+ }
+
+
+def paired(y, before, after):
+ delta = (np.asarray(after) == y).astype(int) - (np.asarray(before) == y).astype(int)
+ counts = np.random.default_rng(187).multinomial(
+ len(y), [(delta == -1).mean(), (delta == 0).mean(), (delta == 1).mean()], 10000
+ )
+ samples = 100 * (counts[:, 2] - counts[:, 0]) / len(y)
+ return {
+ "gain_pp": float(100 * delta.mean()),
+ "gain_pp_95ci": np.quantile(samples, [0.025, 0.975]).tolist(),
+ "method": "paired case bootstrap, conditional on trained models",
+ }
+
+
+def train(dataset, recordings, count, method):
+ audit(dataset)
+ if count not in PROTOCOL["embedding_candidates"]["sizes"] or method not in ("ridge", "knn"):
+ raise ValueError("Undeclared training budget or method")
+ train_rows = dataset["splits"]["train"][:count]
+ validation, calibration = (dataset["splits"][s] for s in ("validation", "calibration"))
+ x, vx, cx = (vectors(rows, recordings) for rows in (train_rows, validation, calibration))
+ y, vy, cy = (np.array([r["label"] for r in rows]) for rows in (train_rows, validation, calibration))
+ policy = {
+ "format": "sie-contest-relative-router-v2",
+ "dataset_sha256": digest(dataset),
+ "method": method,
+ "train_ids": [r["id"] for r in train_rows],
+ "labels": count,
+ "encoder": MODEL,
+ "revision": REVISION,
+ "instruction": INSTRUCTION,
+ "development_vectors_sha256": file_hash(recordings / "vectors.jsonl.gz"),
+ "selection": "validation chooses ridge penalty; separate calibration chooses cutoff",
+ }
+ if method == "ridge":
+ candidates = [ridge(x, y, p) for p in PENALTIES]
+ losses = [float(np.mean((vx @ w + b - vy) ** 2)) for w, b in candidates]
+ selected = int(np.argmin(losses))
+ weights, bias = candidates[selected]
+ policy.update(weights=weights.tolist(), bias=bias, penalty=PENALTIES[selected], validation_mse=losses)
+ calibration_scores = cx @ weights + bias
+ else:
+ policy.update(k=5, bank=x.tolist(), targets=y.tolist())
+ calibration_scores = knn(x, y, cx)
+ policy["threshold"] = cutoff(calibration_scores, cy)
+ policy["sha256"] = digest(policy)
+ return policy
+
+
+def scores(policy, x):
+ if (
+ policy["format"] != "sie-contest-relative-router-v2"
+ or (policy["encoder"], policy["revision"], policy["instruction"]) != (MODEL, REVISION, INSTRUCTION)
+ or policy["method"] not in ("ridge", "knn")
+ ):
+ raise ValueError("Unsupported policy configuration")
+ expected = policy["sha256"]
+ if digest({k: v for k, v in policy.items() if k != "sha256"}) != expected:
+ raise ValueError("Frozen policy changed")
+ if policy["method"] == "ridge":
+ return x @ np.array(policy["weights"]) + policy["bias"]
+ return knn(np.array(policy["bank"]), np.array(policy["targets"]), x, policy["k"])
+
+
+def train_suite(dataset, recordings, output):
+ output.mkdir(parents=True, exist_ok=False)
+ chosen = {}
+ for count in (64, 128, 256, 512):
+ for method in ("knn", "ridge"):
+ name = f"{method}{count}"
+ policy = train(dataset, recordings, count, method)
+ write(output / f"{name}.json", policy)
+ chosen[name] = policy["sha256"]
+ print(f"Frozen {name}", flush=True)
+ write(
+ output / "frozen.json", {"dataset_sha256": digest(dataset), "policies": chosen, "primary": PROTOCOL["primary"]}
+ )
+
+
+def evaluate_suite(dataset, recordings, policies):
+ frozen = json.loads((policies / "frozen.json").read_text())
+ if frozen["dataset_sha256"] != digest(dataset):
+ raise ValueError("Dataset changed after freeze")
+ rows = dataset["splits"]["test"]
+ x, y = vectors(rows, recordings), np.array([r["label"] for r in rows])
+ result, predictions = {}, {}
+ for name, expected in frozen["policies"].items():
+ policy = json.loads((policies / f"{name}.json").read_text())
+ if policy["sha256"] != expected:
+ raise ValueError("Selected policy changed")
+ predictions[name] = (scores(policy, x) >= policy["threshold"]).astype(int)
+ result[name] = metrics(y, predictions[name])
+ return {
+ "dataset_sha256": digest(dataset),
+ "metrics": result,
+ "primary_comparison": paired(y, predictions["knn64"], predictions["ridge512"]),
+ "predictions": {k: p.tolist() for k, p in predictions.items()},
+ "ids": [r["id"] for r in rows],
+ }
diff --git a/examples/train-a-router/pyproject.toml b/examples/train-a-router/pyproject.toml
new file mode 100644
index 000000000..f42e4f146
--- /dev/null
+++ b/examples/train-a-router/pyproject.toml
@@ -0,0 +1,26 @@
+[project]
+name = "sie-train-a-router-example"
+version = "0.1.0"
+description = "Learn, validate and serve a request router with frozen embeddings or a fine-tuned decision model"
+requires-python = ">=3.12,<3.13"
+dependencies = ["numpy==2.4.6", "sie-sdk==0.10.0", "httpx>=0.28,<1"]
+
+[tool.uv]
+package = false
+
+[tool.uv.sources]
+sie-sdk = { git = "https://github.com/superlinked/sie", rev = "d680237f02c2b67fd9afd91f30b73a29cb18fae0", subdirectory = "packages/sie_sdk" }
+
+[dependency-groups]
+data = ["pyarrow>=18,<24"]
+dev = ["ruff>=0.14"]
+evaluation = ["pyarrow>=18,<24", "math-verify[antlr4_13_2]==0.9.0"]
+outcomes = ["scipy==1.17.0"]
+training = ["laya==0.3.29", "peft==0.18.1", "torch==2.9.1", "transformers==4.57.6"]
+plots = [
+ "matplotlib>=3.10,<4",
+]
+
+[tool.ruff]
+line-length = 120
+target-version = "py312"
diff --git a/examples/train-a-router/router.py b/examples/train-a-router/router.py
new file mode 100644
index 000000000..932204106
--- /dev/null
+++ b/examples/train-a-router/router.py
@@ -0,0 +1,69 @@
+"""Build and use a router that trades answer-model cost against answer quality.
+
+fetch: download the public data and recorded results
+replay: reproduce the recorded cost and quality tables on CPU
+prepare: create a separate folder for your training runs
+fit-native: adapt the Laya decision model with LoRA on a CUDA GPU
+fit-embedding: fit two small classifiers on saved Qwen embeddings on CPU
+fit-teacher: compare teacher and benchmark labels using the same saved inputs
+evaluate: measure your trained router on the public test recordings
+answer: route a new question and request its answer through SIE
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+import arithmetic_replay
+import arithmetic_route
+import evidence
+
+COMMANDS = ("fetch", "replay", "prepare", "fit-native", "fit-embedding", "fit-teacher", "evaluate", "answer")
+
+
+def main(argv=None):
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ parser.add_argument("command", choices=COMMANDS)
+ arguments = sys.argv[1:] if argv is None else argv
+ args, remaining = parser.parse_args(arguments[:1]), arguments[1:]
+ if args.command not in ("fetch", "answer"):
+ arithmetic_replay.main([args.command, *remaining])
+ return
+ command = argparse.ArgumentParser(prog=f"router.py {args.command}")
+ command.add_argument("--study", type=Path, default=Path("evidence"))
+ if args.command == "fetch":
+ command.add_argument("--records", action="store_true", help="Include raw answer recordings for auditing")
+ command.add_argument(
+ "--weights", action="store_true", help="Include recorded native weights for local inference"
+ )
+ selected = command.parse_args(remaining)
+ evidence.fetch(
+ selected.study, arithmetic_study=True,
+ solver_records=selected.records, solver_weights=selected.weights,
+ )
+ return
+ command.add_argument("--run", type=Path, help="Your prepared/trained folder; omit to use the recorded router")
+ command.add_argument("--population", choices=arithmetic_replay.POPULATIONS, default="expanded")
+ command.add_argument("--method", choices=("embedding", "lora", "laya", "teacher"), default="embedding")
+ command.add_argument("--question", required=True)
+ command.add_argument("--target", type=int, choices=(0, 1, 2, 5), default=1)
+ command.add_argument("--output", type=Path, required=True)
+ command.add_argument("--self-hosted", action="store_true")
+ command.add_argument(
+ "--device", choices=("cpu", "cuda"),
+ default="cpu", help="Local native inference device"
+ )
+ selected = command.parse_args(remaining)
+ result = arithmetic_route.answer(
+ selected.study, selected.output, selected.question, run=selected.run,
+ population=selected.population, method=selected.method, target=selected.target,
+ self_hosted=selected.self_hosted, device=selected.device,
+ )
+ print(json.dumps(result, indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/self-host/Dockerfile.proxy b/examples/train-a-router/self-host/Dockerfile.proxy
new file mode 100644
index 000000000..df6c1b817
--- /dev/null
+++ b/examples/train-a-router/self-host/Dockerfile.proxy
@@ -0,0 +1,6 @@
+FROM haproxy:3.2-bookworm@sha256:baecd003ae5c2376e58d04a5b2ffc7d82d30e9bebe2869f76222b819f719dee3
+USER root
+RUN apt-get update && apt-get install -y --no-install-recommends lua-cjson \
+ && rm -rf /var/lib/apt/lists/*
+USER haproxy
+COPY haproxy.cfg chat-model.lua /usr/local/etc/haproxy/
diff --git a/examples/train-a-router/self-host/README.md b/examples/train-a-router/self-host/README.md
new file mode 100644
index 000000000..51247b50d
--- /dev/null
+++ b/examples/train-a-router/self-host/README.md
@@ -0,0 +1,98 @@
+# Serve the three models on your own GPUs
+
+This recipe exposes one local SIE endpoint for the encoder and both answer
+models. The routing commands use the same prompts, model revisions and
+conditional final-answer calls as SIE Cloud.
+
+This is an optional inference deployment. The main example can use SIE Cloud
+for these calls and needs only one GPU for LoRA training.
+
+You need **Linux, Docker Compose, NVIDIA Container Toolkit and three separate
+NVIDIA GPUs**. The validation hardware is three H100 80 GB GPUs. The strong
+model's pinned image requires a CUDA 13 compatible driver. This is a larger
+deployment than the single GPU used to train Laya's LoRA adapter. Sharing one
+GPU between all three workers has not been validated.
+
+The first start downloads about 48 GB of model weights, in addition to the
+container images. The named `model-cache` volume retains those weights.
+Each worker gets a separate runtime because the answer models need different
+SGLang/CUDA versions. The answer workers use released images pinned by digest.
+The encoder builds the `sglang-embedding` bundle from an exact public SIE
+commit: the released generation bundle does not accept embedding adapters.
+This initial CPU image build also takes time. The image digests, build commit
+and model revisions are pinned in [compose.yaml](compose.yaml) and [models/](models/).
+
+Run these commands from `examples/train-a-router`:
+
+```sh
+# Check the Compose configuration and start all workers, waiting for model readiness.
+docker compose -f self-host/compose.yaml config --quiet
+docker compose -f self-host/compose.yaml up --build --detach --wait --wait-timeout 1200
+```
+
+The default device IDs are 0, 1 and 2. Set `CHEAP_GPU`, `STRONG_GPU` and
+`ENCODER_GPU` before startup to choose different, distinct devices. Cold model
+loading takes several minutes. Inspect startup with
+`docker compose -f self-host/compose.yaml logs --tail 100` if readiness fails.
+The health checks require each pinned model profile to be loaded; a running
+HTTP process alone does not count as ready. The SDK smoke below additionally
+checks real inference and the structured final-answer path.
+
+```sh
+# Install the client dependencies and check metadata, embeddings and both answer protocols.
+uv sync --frozen --group outcomes
+export SIE_BASE_URL=http://127.0.0.1:8080
+export SIE_API_KEY=''
+uv run --frozen --group outcomes python self-host/smoke.py \
+ --output run-output/self-host-smoke
+```
+
+The smoke check makes one embedding request and an initial answer plus a
+forced structured finalizer for each answer model. The cheap worker preloads
+both its speculative and grammar-safe engines, reserving 40% of GPU memory
+for each. This avoids loading a second engine during an answer. Token admission
+is capped at 8,192 per profile; this recipe is for sequential requests.
+It validates immutable model revisions and saves an incomplete receipt
+if any step fails. Use a new output directory for another attempt.
+
+After following the main example's training instructions:
+
+```sh
+# Route a new request with your saved classifiers, then call only the selected answer model.
+uv run --frozen --group outcomes python router.py answer \
+ --study evidence --run run-output/my-router --population expanded \
+ --method embedding \
+ --question 'A rectangle has perimeter 30 and length 9. What is its area?' \
+ --target 1 --self-hosted --output run-output/self-hosted-answer
+```
+
+`--self-hosted` allows missing Cloud billing receipts; GPU hosting is still
+unpriced. It preserves model identity checks. The saved calibration target is
+not a guarantee for an individual answer or a new workload.
+
+```sh
+# Stop the workers and proxy while preserving the downloaded model cache.
+docker compose -f self-host/compose.yaml down
+```
+
+The frontend listens only on `127.0.0.1:8080`; worker ports 18081–18083 are also
+published only on loopback. SGLang child ports stay on the private Compose
+network. This is a trusted local deployment with no API-key authentication.
+Docker administrators and other local users are privileged peers. Do not
+publish these ports on a network without designing and reviewing authentication.
+
+The small proxy accepts only the three pinned model metadata paths, the
+encoder path and chat completion requests for the two answer models. It bounds
+request bodies and headers, rejects ambiguous JSON model fields and strips
+client credentials before forwarding. There are no automatic request retries.
+
+The GPU smoke passed with both exact released answer images and the pinned
+encoder build: all three model revisions, embeddings, both initial/finalizer
+pairs, and one public routed request. Application processes ran as UID 1000
+with writable caches and the original image dependencies. Modal connected the
+isolated workers through authenticated tunnels and required materializing the
+encoder Dockerfile's final `FROM` build argument to its intended stage; the
+public Compose build uses the original Dockerfile and ordinary build arguments.
+A complete Docker Compose GPU stack has not been executed as one deployment.
+These checks do not establish throughput or bit-for-bit equivalence with
+SIE Cloud.
diff --git a/examples/train-a-router/self-host/chat-model.lua b/examples/train-a-router/self-host/chat-model.lua
new file mode 100644
index 000000000..deab88b0c
--- /dev/null
+++ b/examples/train-a-router/self-host/chat-model.lua
@@ -0,0 +1,67 @@
+local json = require("cjson.safe")
+json.decode_invalid_numbers(false)
+json.decode_max_depth(32)
+
+local allowed = {
+ ["Qwen/Qwen3.5-4B"] = true,
+ ["Qwen/Qwen3.8-27B-FP8"] = true,
+}
+
+local function unique_top_level_keys(body)
+ local seen, depth, previous, i = {}, 0, "", 1
+ while i <= #body do
+ local c = body:sub(i, i)
+ if c == '"' then
+ local start = i
+ i = i + 1
+ while i <= #body do
+ local q = body:sub(i, i)
+ if q == "\\" then
+ i = i + 2
+ elseif q == '"' then
+ break
+ else
+ i = i + 1
+ end
+ end
+ if depth == 1 and (previous == "{" or previous == ",") then
+ local key = json.decode(body:sub(start, i))
+ if type(key) ~= "string" or seen[key] then
+ return false
+ end
+ seen[key] = true
+ end
+ previous = '"'
+ elseif c == "{" or c == "[" then
+ depth = depth + 1
+ previous = c
+ elseif c == "}" or c == "]" then
+ depth = depth - 1
+ previous = c
+ elseif not c:match("%s") then
+ previous = c
+ end
+ i = i + 1
+ end
+ return true
+end
+
+core.register_action("validate-body", {"http-req"}, function(txn, kind)
+ txn:set_var("txn.valid_body", false)
+ local body = txn.sf:req_body()
+ local length = txn.sf:req_hdr("content-length")
+ if type(body) ~= "string" or not length:match("^%d+$")
+ or tonumber(length) ~= #body or #body == 0 or #body > 131072 then
+ return
+ end
+ if kind == "chat" then
+ local value = json.decode(body)
+ if body:match("^%s*(.)") ~= "{" or type(value) ~= "table"
+ or type(value.model) ~= "string" or not allowed[value.model]
+ or not unique_top_level_keys(body) then
+ return
+ end
+ txn:set_var("txn.model", value.model)
+ end
+ txn:set_var("txn.valid_body", true)
+end, 1)
diff --git a/examples/train-a-router/self-host/compose.yaml b/examples/train-a-router/self-host/compose.yaml
new file mode 100644
index 000000000..d56880f45
--- /dev/null
+++ b/examples/train-a-router/self-host/compose.yaml
@@ -0,0 +1,87 @@
+name: sie-train-a-router
+
+x-worker: &worker
+ restart: "no"
+ shm_size: 8gb
+ volumes:
+ - ./models:/app/models:ro
+ - ./healthcheck.py:/opt/router-healthcheck.py:ro
+ - model-cache:/app/.cache/huggingface
+ environment:
+ SIE_AUTH_MODE: none
+ HF_HOME: /app/.cache/huggingface
+ healthcheck:
+ test: [CMD, python, /opt/router-healthcheck.py, cheap]
+ interval: 10s
+ timeout: 10s
+ start_period: 10m
+ retries: 3
+
+services:
+ cheap:
+ <<: *worker
+ image: ghcr.io/superlinked/sie-server:v0.10.1-cuda12-sglang@sha256:ff397c6df05721261e519986f1a0922bfe746ad180114cd93941a3ebf04bf463
+ command: [serve, --host, 0.0.0.0, --port, "8080", --models-dir, /app/models, --models, "Qwen/Qwen3.5-4B,Qwen/Qwen3.5-4B:no-spec", --preload, "Qwen/Qwen3.5-4B,Qwen/Qwen3.5-4B:no-spec", --device, cuda]
+ ports: ["127.0.0.1:18081:8080"]
+ deploy:
+ resources:
+ reservations:
+ devices:
+ - driver: nvidia
+ device_ids: ["${CHEAP_GPU:-0}"]
+ capabilities: [gpu]
+ strong:
+ <<: *worker
+ healthcheck:
+ test: [CMD, python, /opt/router-healthcheck.py, strong]
+ interval: 10s
+ timeout: 10s
+ start_period: 10m
+ retries: 3
+ image: ghcr.io/superlinked/sie-server:v0.10.1-cuda13-sglang-cu130@sha256:b9262c56066e97d6d1892c57edf96140d951316aa1fd7929bf6eb1424d9d6c81
+ command: [serve, --host, 0.0.0.0, --port, "8080", --models-dir, /app/models, --models, Qwen/Qwen3.8-27B-FP8, --preload, Qwen/Qwen3.8-27B-FP8, --device, cuda]
+ ports: ["127.0.0.1:18082:8080"]
+ deploy:
+ resources:
+ reservations:
+ devices:
+ - driver: nvidia
+ device_ids: ["${STRONG_GPU:-1}"]
+ capabilities: [gpu]
+ encoder:
+ <<: *worker
+ healthcheck:
+ test: [CMD, python, /opt/router-healthcheck.py, encoder]
+ interval: 10s
+ timeout: 10s
+ start_period: 10m
+ retries: 3
+ image: sie-train-a-router-encoder:975d5309
+ build:
+ context: https://github.com/superlinked/sie.git#975d5309ba3af9eaeb13725b20dcf10d906c3e2a
+ dockerfile: packages/sie_server/Dockerfile.cuda12
+ args:
+ BUNDLE: sglang-embedding
+ SIE_SRC_REV: 975d5309ba3af9eaeb13725b20dcf10d906c3e2a
+ command: [serve, --host, 0.0.0.0, --port, "8080", --models-dir, /app/models, --models, Qwen/Qwen3-Embedding-4B, --preload, Qwen/Qwen3-Embedding-4B, --device, cuda]
+ ports: ["127.0.0.1:18083:8080"]
+ deploy:
+ resources:
+ reservations:
+ devices:
+ - driver: nvidia
+ device_ids: ["${ENCODER_GPU:-2}"]
+ capabilities: [gpu]
+ proxy:
+ build:
+ context: .
+ dockerfile: Dockerfile.proxy
+ network_mode: host
+ restart: "no"
+ read_only: true
+ cap_drop: [ALL]
+ security_opt: [no-new-privileges:true]
+ depends_on: [cheap, strong, encoder]
+
+volumes:
+ model-cache:
diff --git a/examples/train-a-router/self-host/haproxy.cfg b/examples/train-a-router/self-host/haproxy.cfg
new file mode 100644
index 000000000..32321f7f4
--- /dev/null
+++ b/examples/train-a-router/self-host/haproxy.cfg
@@ -0,0 +1,65 @@
+global
+ maxconn 16
+ tune.bufsize 147456
+ tune.http.maxhdr 32
+ tune.lua.bool-sample-conversion normal
+ lua-load /usr/local/etc/haproxy/chat-model.lua
+
+defaults
+ mode http
+ timeout connect 5s
+ timeout client 360s
+ timeout server 360s
+ timeout http-request 5s
+ timeout http-keep-alive 5s
+ retries 0
+
+frontend local_sie
+ bind 127.0.0.1:8080
+ acl get method GET
+ acl post method POST
+ acl chat path -m str /v1/chat/completions
+ acl encode path -m str /v1/encode/Qwen/Qwen3-Embedding-4B
+ acl cheap_metadata path -m str /v1/models/Qwen/Qwen3.5-4B
+ acl strong_metadata path -m str /v1/models/Qwen/Qwen3.8-27B-FP8
+ acl encoder_metadata path -m str /v1/models/Qwen/Qwen3-Embedding-4B
+ http-request deny deny_status 400 if { query -m found }
+ http-request deny deny_status 431 if { req.hdrs -m len gt 8192 }
+ http-request deny deny_status 400 if { req.hdr(transfer-encoding) -m found }
+ http-request deny deny_status 400 if { req.hdr(content-encoding) -m found }
+ http-request deny deny_status 400 if { req.hdr(expect) -m found }
+ http-request deny deny_status 404 unless get or post
+ http-request deny deny_status 404 if get !cheap_metadata !strong_metadata !encoder_metadata
+ http-request deny deny_status 404 if post !chat !encode
+ http-request deny deny_status 400 if get { req.hdr(content-length) -m found }
+ http-request deny deny_status 400 if post !{ req.hdr_cnt(content-length) eq 1 }
+ http-request deny deny_status 413 if post { req.hdr(content-length) -m int gt 131072 }
+ http-request deny deny_status 400 if chat !{ req.hdr(content-type) -m str application/json }
+ http-request deny deny_status 400 if encode !{ req.hdr(content-type) -m str application/json application/msgpack application/x-msgpack }
+ http-request wait-for-body time 2s if post
+ http-request lua.validate-body chat if post chat
+ http-request lua.validate-body encode if post encode
+ http-request deny deny_status 400 if post !{ var(txn.valid_body) -m bool }
+ http-request del-header Authorization
+ http-request del-header Proxy-Authorization
+ http-request del-header X-API-Key
+ http-request del-header Cookie
+ http-request del-header Forwarded
+ http-request del-header X-Forwarded-For
+ http-request del-header X-Forwarded-Host
+ http-request del-header X-Forwarded-Proto
+ use_backend cheap if cheap_metadata
+ use_backend strong if strong_metadata
+ use_backend encoder if encoder_metadata
+ use_backend encoder if encode
+ use_backend cheap if { var(txn.model) -m str Qwen/Qwen3.5-4B }
+ use_backend strong if { var(txn.model) -m str Qwen/Qwen3.8-27B-FP8 }
+
+backend cheap
+ server worker 127.0.0.1:18081
+
+backend strong
+ server worker 127.0.0.1:18082
+
+backend encoder
+ server worker 127.0.0.1:18083
diff --git a/examples/train-a-router/self-host/healthcheck.py b/examples/train-a-router/self-host/healthcheck.py
new file mode 100644
index 000000000..94a3bf1fd
--- /dev/null
+++ b/examples/train-a-router/self-host/healthcheck.py
@@ -0,0 +1,42 @@
+"""Require every configured profile to be loaded at its pinned revision."""
+
+from __future__ import annotations
+
+import argparse
+
+from sie_sdk import SIEClient
+
+MODELS = {
+ "cheap": (
+ ("Qwen/Qwen3.5-4B", "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a"),
+ ("Qwen/Qwen3.5-4B:no-spec", "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a"),
+ ),
+ "strong": (("Qwen/Qwen3.8-27B-FP8", "017b9c7af6b5689d5dd426a76e0bc077eb5ca20a"),),
+ "encoder": (("Qwen/Qwen3-Embedding-4B", "5cf2132abc99cad020ac570b19d031efec650f2b"),),
+}
+
+
+def check(client, arm):
+ for model, revision in MODELS[arm]:
+ metadata = client.get_model(model)
+ if (
+ metadata.get("name") != model
+ or metadata.get("loaded") is not True
+ or metadata.get("revision") != revision
+ ):
+ raise RuntimeError(f"Pinned model is not loaded: {model}")
+ if arm == "encoder" and metadata.get("dims", {}).get("dense") != 2560:
+ raise RuntimeError("Encoder dimensions differ")
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("arm", choices=MODELS)
+ args = parser.parse_args()
+ # Metadata reads never trigger a model load or inference request.
+ with SIEClient("http://127.0.0.1:8080", api_key="", timeout_s=3, read_timeout_s=3) as client:
+ check(client, args.arm)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/self-host/models/Qwen__Qwen3-Embedding-4B.yaml b/examples/train-a-router/self-host/models/Qwen__Qwen3-Embedding-4B.yaml
new file mode 100644
index 000000000..1596888df
--- /dev/null
+++ b/examples/train-a-router/self-host/models/Qwen__Qwen3-Embedding-4B.yaml
@@ -0,0 +1,36 @@
+sie_id: Qwen/Qwen3-Embedding-4B
+hf_id: Qwen/Qwen3-Embedding-4B
+hf_revision: 5cf2132abc99cad020ac570b19d031efec650f2b
+inputs:
+ text: true
+ image: false
+ audio: false
+ video: false
+tasks:
+ encode:
+ dense:
+ dim: 2560
+ sparse: null
+ multivector: null
+ score: null
+ extract: null
+max_sequence_length: 32768
+profiles:
+ default:
+ max_batch_tokens: 16384
+ compute_precision: bfloat16
+ adapter_path: sie_server.adapters.sglang.embedding:SGLangEmbeddingAdapter
+ adapter_options:
+ loadtime:
+ mem_fraction_static: 0.85
+ # Up to four batches in flight to SGLang at once, so the GPU keeps working
+ # while a batch waits on its HTTP round trips: about 1.5x the documents per
+ # second at 8 and 32 in flight on an H100.
+ max_concurrent_dispatch: 4
+ runtime:
+ pooling: last_token
+ normalize: true
+ query_template: |-
+ Instruct: {instruction}
+ Query: {text}
+ default_instruction: Given a query, retrieve relevant passages that answer the query
diff --git a/examples/train-a-router/self-host/models/Qwen__Qwen3.5-4B.yaml b/examples/train-a-router/self-host/models/Qwen__Qwen3.5-4B.yaml
new file mode 100644
index 000000000..e79343006
--- /dev/null
+++ b/examples/train-a-router/self-host/models/Qwen__Qwen3.5-4B.yaml
@@ -0,0 +1,346 @@
+sie_id: Qwen/Qwen3.5-4B
+hf_id: Qwen/Qwen3.5-4B
+hf_revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
+inputs:
+ # Qwen3.5-4B is a unified vision-language model (Gated DeltaNet + sparse MoE
+ # with an integrated vision encoder). The wire surface accepts text+image
+ # via SIE-native generate (inline ``images``) and the OpenAI chat-completions
+ # schema (``image_url`` content parts); video is documented by Qwen but not
+ # yet wired through the SIE gateway.
+ text: true
+ image: true
+ audio: false
+ video: false
+tasks:
+ generate:
+ # Native context length is 262,144 tokens (YaRN extends to ~1M).
+ # Keep the bare request context at 8192. The explicit ``:long-context``
+ # profile below exposes the already-measured 32K A100 launch shape without
+ # widening latency/capacity expectations for every caller.
+ context_length: 8192
+ max_output_tokens: 4096
+ # Route grammar-constrained requests (response_format / grammar) to the
+ # non-speculative variant. The free-text profiles run NEXTN
+ # speculative decoding for free-text throughput, but NEXTN bypasses SGLang's
+ # Outlines grammar FSM (leaks out-of-schema keys, truncates mid-JSON —
+ # verified on L4). The `no-spec` profile (speculative off, same Outlines
+ # backend) enforces the grammar correctly; the gateway rewrites grammar
+ # requests to `Qwen/Qwen3.5-4B:no-spec` so free-text keeps NEXTN.
+ grammar_profile: no-spec
+ capabilities:
+ # NOTE: scoped to what the production backend (``outlines``, set
+ # per-profile under ``adapter_options.loadtime``) actually supports
+ # on SGLang. SGLang's outlines_backend explicitly skips ebnf with
+ # ``Skip unsupported key_type='ebnf'`` — confirmed by a compatibility smoke.
+ # If an operator switches a Qwen3.5 profile to ``grammar_backend:
+ # xgrammar`` they should re-add ``"ebnf"`` here — xgrammar smoke on
+ # the same hardware passed all three modes (regex/json_schema/ebnf).
+ grammar: ["json_schema", "regex"]
+ streaming: true
+ tools: true
+ # NOTE: not advertising code:true here yet — Qwen3.5-4B is the strongest
+ # model on paper but its NEXTN/hybrid serving path does not come up
+ # reliably on the current eval image, so it has no measured HumanEval/MBPP
+ # baseline. The ``model="code"`` alias points to the measured
+ # Qwen3-4B-Instruct-2507; promote this once 3.5-4B is measured.
+ # Grammar backend: ``outlines`` (set per-profile under
+ # ``adapter_options.loadtime``). Earlier revisions forced ``xgrammar``
+ # here because the worker-side Outlines preflight (``compile_outlines``)
+ # crashed on Qwen3.5 with ``'TokenizersBackend' object has no attribute
+ # 'vocabulary'`` — it passed the raw transformers>=5 tokenizer straight
+ # to Outlines' processor factories. That preflight now wraps the
+ # tokenizer in Outlines' ``TransformerTokenizer`` adapter (the same wrap
+ # SGLang does internally), so ``outlines`` works for json_schema/regex.
+ # EBNF is *not* in capabilities because SGLang's outlines_backend does
+ # not implement it (the smoke surfaces this with a 400). If you need
+ # EBNF on Qwen3.5, switch the profile to ``grammar_backend: xgrammar``
+ # and re-add ``"ebnf"`` to the capability list above.
+ # Qwen3.5 emits ``...`` reasoning by default
+ # ("Qwen3.5 will think by default before response", per model card).
+ # We disable it for the benchmark / OpenAI-compat path so the
+ # visible output is the answer only. Operators wanting CoT can
+ # flip this profile-side.
+ chat_template_kwargs:
+ enable_thinking: false
+ prewarm_grammars:
+ # NOTE: Outlines regexes are IMPLICITLY ANCHORED and its FSM engine
+ # (interegular) rejects ``^``/``$`` anchors with
+ # ``interegular.patterns.Unsupported`` — which, unhandled inside
+ # SGLang's scheduler thread, SIGQUITs the whole server. XGrammar
+ # tolerated anchors; Outlines does not. So write the bare pattern
+ # (``(yes|no)``, not ``^(yes|no)$``). The worker-side preflight
+ # (``compile_outlines``) catches a stray anchored regex as
+ # ``grammar_compile_failed`` before it can reach — and crash — SGLang.
+ - name: yes_no
+ kind: regex
+ value: "(yes|no)"
+ - name: short_answer
+ kind: json_schema
+ value:
+ type: object
+ properties:
+ answer:
+ type: string
+ required: [answer]
+max_sequence_length: 8192
+# ── KV-cache math (empirical, 2026-05-18) ──
+#
+# Qwen3.5-4B uses a hybrid Gated DeltaNet + Gated Attention architecture
+# where 24 of 32 layers are recurrent (DeltaNet) and only 8 are KV-cache
+# bearing (Gated Attention with 4 KV heads × head_dim=256). The mamba
+# scheduler reserves additional state buffers for the recurrent layers
+# (``--mamba-scheduler-strategy extra_buffer`` is required for NEXTN spec
+# decoding to coexist with the radix cache). Effective KV memory ends up
+# much smaller than a comparable dense 4B model because the recurrent
+# buffer eats roughly ``mamba_full_memory_ratio=0.9`` of the static
+# allocation.
+#
+# Empirical SGLang ``/get_server_info`` dump (sglang 0.5.10, NEXTN on,
+# ``--mamba-scheduler-strategy extra_buffer``):
+#
+# L4 (22 GB), mem_fraction_static=0.90, context_length=2048 OR 8192:
+# weight=8.62 GB, kvcache=0.96 GB, token_capacity=31,610
+# effective_max_running_requests_per_dp = 3
+# (context_length above 8192 OOMs on L4)
+#
+# A100-40GB, mem_fraction_static=0.85, context_length=32768:
+# max_total_num_tokens=847,848, max_req_input_len=32,762
+# effective_max_running_requests_per_dp = 48
+#
+# ``kv_budget_tokens`` is sized at ``token_capacity / 4``, matching the
+# pattern set by Qwen3-4B-Instruct-2507. Re-calibrate with an SGLang capacity
+# probe if its version or ``mem_fraction_static`` changes.
+profiles:
+ # Local 80 GB deployment: reserve 40% for each of the default and no-spec
+ # engines. Preload both so final-answer formatting needs no inference retry.
+ default:
+ max_batch_tokens: 8192
+ compute_precision: bfloat16
+ adapter_path: sie_server.adapters.sglang.generation:SGLangGenerationAdapter
+ # Bound admission for sequential requests while both profiles remain resident.
+ kv_budget_tokens: 8192
+ adapter_options:
+ loadtime:
+ # Apple-Silicon (MLX) repo served by the MLX subprocess backend. On
+ # non-CUDA devices SGLangGenerationAdapter.create_for_device swaps to
+ # MLXGenerationAdapter, which reads this. Ignored on CUDA (SGLang
+ # swallows it via **kwargs).
+ mlx_repo: mlx-community/Qwen3.5-4B-4bit
+ mem_fraction_static: 0.40
+ served_model_name: Qwen/Qwen3.5-4B
+ disable_cuda_graph: true
+ attention_backend: triton
+ grammar_backend: outlines
+ reasoning_parser: qwen3
+ tool_call_parser: qwen3_coder
+ # NEXTN/MTP per the Qwen3.5-4B model card recipe. SGLang 0.5.10
+ # implements NEXTN under the EAGLE codepath internally; the
+ # ``/server_info`` dump reports ``speculative_algorithm: EAGLE``
+ # — that is expected and not a misconfiguration.
+ speculative:
+ enabled: true
+ algorithm: nextn
+ num_steps: 3
+ eagle_topk: 1
+ num_draft_tokens: 4
+ # Required for NEXTN spec decoding to coexist with radix cache
+ # on Qwen3.5's hybrid Gated DeltaNet architecture. The adapter
+ # also sets the SGLANG_ENABLE_SPEC_V2=1 env var when
+ # speculative.enabled — both are required as a pair.
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--mamba-scheduler-strategy"
+ - "extra_buffer"
+ runtime:
+ first_chunk_timeout_s: 60
+ # The Stage-5 A100-80GB c=4 OSS queue run at 37771022d observed one
+ # governed 10s inter-chunk abort in its 300-row CaseHOLD tranche. 15s
+ # matches sibling large-generation profiles while preserving the 300s
+ # overall bound; the single full rerun remains required for promotion.
+ inter_chunk_timeout_s: 15
+ overall_timeout_s: 300
+ default_sampling:
+ temperature: 0.7
+ top_p: 0.8
+ presence_penalty: 1.5
+ stop_tokens:
+ - "<|im_end|>"
+ # Historical acceptance-benchmark identity. Keep it as a zero-delta alias so
+ # those results remain reproducible while production uses the bare model id.
+ a100-40gb:
+ extends: default
+ # Explicit reasoning mode. The tokenizer switch changes output semantics,
+ # so it is a named profile rather than the bare model default. Thinking uses
+ # Qwen's recommended sampling recipe; the launch shape remains the measured
+ # A100 default.
+ thinking:
+ extends: default
+ chat_template_kwargs:
+ enable_thinking: true
+ adapter_options:
+ runtime:
+ first_chunk_timeout_s: 60
+ inter_chunk_timeout_s: 15
+ overall_timeout_s: 300
+ default_sampling:
+ temperature: 1.0
+ top_p: 0.95
+ top_k: 20
+ presence_penalty: 1.5
+ stop_tokens:
+ - "<|im_end|>"
+ # The checkpoint is native to 262K and the checked-in A100 calibration
+ # already boots SGLang at 32K. Expose that conservative measured window as
+ # an opt-in variant while the bare model remains 8K.
+ long-context:
+ extends: default
+ adapter_options:
+ loadtime:
+ max_seq_length: 32768
+ mlx_repo: mlx-community/Qwen3.5-4B-4bit
+ mem_fraction_static: 0.85
+ served_model_name: Qwen/Qwen3.5-4B
+ disable_cuda_graph: true
+ attention_backend: triton
+ grammar_backend: outlines
+ reasoning_parser: qwen3
+ tool_call_parser: qwen3_coder
+ speculative:
+ enabled: true
+ algorithm: nextn
+ num_steps: 3
+ eagle_topk: 1
+ num_draft_tokens: 4
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--mamba-scheduler-strategy"
+ - "extra_buffer"
+ # L4 is severely constrained for Qwen3.5-4B + NEXTN — fits only ~3
+ # concurrent requests, context cap ≈ 8192. Use this explicit profile for
+ # smoke / c=1 only; it is not the production default.
+ l4-smoke:
+ max_batch_tokens: 8192
+ compute_precision: bfloat16
+ adapter_path: sie_server.adapters.sglang.generation:SGLangGenerationAdapter
+ # Empirical: 31,610 / 4 = 7,902 → round to 7,168 (= 7 × 1024).
+ kv_budget_tokens: 7168
+ adapter_options:
+ loadtime:
+ mlx_repo: mlx-community/Qwen3.5-4B-4bit
+ # Empirical floor on L4 — anything lower OOMs at model load.
+ mem_fraction_static: 0.90
+ served_model_name: Qwen/Qwen3.5-4B
+ disable_cuda_graph: true
+ attention_backend: triton
+ grammar_backend: outlines
+ reasoning_parser: qwen3
+ tool_call_parser: qwen3_coder
+ speculative:
+ enabled: true
+ algorithm: nextn
+ num_steps: 3
+ eagle_topk: 1
+ num_draft_tokens: 4
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--mamba-scheduler-strategy"
+ - "extra_buffer"
+ - "--disable-overlap-schedule"
+ runtime:
+ first_chunk_timeout_s: 60
+ inter_chunk_timeout_s: 10
+ overall_timeout_s: 300
+ default_sampling:
+ temperature: 0.7
+ top_p: 0.8
+ presence_penalty: 1.5
+ stop_tokens:
+ - "<|im_end|>"
+ h100:
+ max_batch_tokens: 65536
+ compute_precision: bfloat16
+ adapter_path: sie_server.adapters.sglang.generation:SGLangGenerationAdapter
+ # Analytical estimate — re-calibrate with an SGLang capacity probe once
+ # H100 access is set up. Scales ~2× off the A100-40GB number
+ # (80 GB / 40 GB), so ~420k tokens at context=32768.
+ kv_budget_tokens: 421888
+ adapter_options:
+ loadtime:
+ mem_fraction_static: 0.85
+ served_model_name: Qwen/Qwen3.5-4B
+ disable_cuda_graph: true
+ attention_backend: triton
+ grammar_backend: outlines
+ reasoning_parser: qwen3
+ tool_call_parser: qwen3_coder
+ speculative:
+ enabled: true
+ algorithm: nextn
+ num_steps: 3
+ eagle_topk: 1
+ num_draft_tokens: 4
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--mamba-scheduler-strategy"
+ - "extra_buffer"
+ runtime:
+ first_chunk_timeout_s: 60
+ inter_chunk_timeout_s: 10
+ overall_timeout_s: 300
+ default_sampling:
+ temperature: 0.7
+ top_p: 0.8
+ presence_penalty: 1.5
+ stop_tokens:
+ - "<|im_end|>"
+ # No-speculative companion for grammar-constrained requests on A100. It
+ # keeps the measured A100 runtime shape while disabling NEXTN, whose decode
+ # path bypasses the Outlines grammar FSM.
+ no-spec:
+ max_batch_tokens: 8192
+ compute_precision: bfloat16
+ adapter_path: sie_server.adapters.sglang.generation:SGLangGenerationAdapter
+ # Bound admission for sequential requests while both profiles remain resident.
+ kv_budget_tokens: 8192
+ adapter_options:
+ loadtime:
+ mlx_repo: mlx-community/Qwen3.5-4B-4bit
+ mem_fraction_static: 0.40
+ served_model_name: Qwen/Qwen3.5-4B
+ disable_cuda_graph: true
+ attention_backend: triton
+ grammar_backend: outlines
+ reasoning_parser: qwen3
+ tool_call_parser: qwen3_coder
+ speculative:
+ enabled: false
+ # mamba_scheduler_strategy is harmless when speculative is off;
+ # the radix-cache compatibility error only fires with spec on.
+ # Leaving the flag present keeps both profiles syntactically
+ # identical so a config diff between them shows only the
+ # speculative block (the intent of the ablation).
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--mamba-scheduler-strategy"
+ - "extra_buffer"
+ runtime:
+ # Outlines can spend more than 60s compiling/decoding larger
+ # json_schema requests before the first visible token. Keep this below
+ # overall_timeout_s so genuinely stuck requests still fail within 5 min.
+ first_chunk_timeout_s: 180
+ inter_chunk_timeout_s: 10
+ overall_timeout_s: 300
+ default_sampling:
+ temperature: 0.7
+ top_p: 0.8
+ presence_penalty: 1.5
+ stop_tokens:
+ - "<|im_end|>"
+ default-no-admission:
+ extends: default
+ admission_enabled: false
diff --git a/examples/train-a-router/self-host/models/Qwen__Qwen3.8-27B-FP8.yaml b/examples/train-a-router/self-host/models/Qwen__Qwen3.8-27B-FP8.yaml
new file mode 100644
index 000000000..6179e670c
--- /dev/null
+++ b/examples/train-a-router/self-host/models/Qwen__Qwen3.8-27B-FP8.yaml
@@ -0,0 +1,547 @@
+sie_id: Qwen/Qwen3.8-27B-FP8
+hf_id: Qwen/Qwen3.8-27B-FP8
+hf_revision: 017b9c7af6b5689d5dd426a76e0bc077eb5ca20a
+# Official first-party blockwise FP8 checkpoint. The pinned config declares
+# Qwen3_5ForConditionalGeneration, 64 layers in the same 48 GDN + 16 full-
+# attention layout as Qwen3.6-27B, and a native 262,144-token window. Its
+# quantization_config is dynamic FP8 with 128x128 weight blocks, so SGLang
+# detects the checkpoint quantization directly; no online --quantization flag
+# belongs in these profiles.
+inputs:
+ text: true
+ image: true
+ audio: false
+ # Qwen documents native video understanding, but video is not yet wired
+ # through the SIE generation request surface.
+ video: false
+# A request with ONE image reads it at up to 3,211,264 pixels (3,136 visual
+# tokens, one per merged 32x32 patch), the resolution a dense form or statement
+# needs: at the family's 1,003,520 a full page came back with misread names and
+# digits. SIE_SGLANG_SINGLE_IMAGE_MAX_PIXELS raises the bound for single-image
+# requests only; a request with two or more images keeps --mm-process-config's
+# 1,003,520 per image, so multi-image requests fit the 8K window exactly as
+# before. The worker reserves the matching bound per request. Every profile
+# also compiles JSON-schema grammars without free whitespace, so greedy
+# decoding cannot pad a reply with indentation or run to the output cap in
+# spaces.
+# Every profile also bounds each JSON number to 19 integer and 19 fractional
+# digits (``json_number_max_digits``). XGrammar leaves digit runs unbounded, so
+# a greedy decode that starts repeating a digit (``8214263.0000...``) ran to
+# the output cap on 2 of 30 page-image extractions on the batch profile.
+tasks:
+ generate:
+ # Keep the bare route conservative. Native context is exposed only through
+ # exact-hardware profiles whose admission and launch shapes are smoke-tested.
+ context_length: 8192
+ max_output_tokens: 4096
+ grammar_profile: no-spec
+ capabilities:
+ grammar: ["json_schema", "regex"]
+ streaming: true
+ tools: true
+ # Qwen3.8 defaults to thinking. The bare model and every non-thinking
+ # profile are answer-only; reasoning is exposed only through the explicit
+ # thinking profiles below.
+ chat_template_kwargs:
+ enable_thinking: false
+ prewarm_grammars:
+ - name: yes_no
+ kind: regex
+ value: "(yes|no)"
+ - name: short_answer
+ kind: json_schema
+ value:
+ type: object
+ properties:
+ answer:
+ type: string
+ required: [answer]
+max_sequence_length: 8192
+profiles:
+ # Correctness-first bare route: official FP8 weights, BF16 KV, FP32 GDN
+ # state, no speculation, and an 8K admission window. Decode runs on CUDA
+ # graphs (SGLang's default; prefill graphs stay off on this engine). At one
+ # request in flight the graphs replay the same kernels, so greedy replies
+ # are byte-identical to the eager launch: 60 of 60 page extractions and
+ # photo questions matched, while GPU-seconds per extraction fell 6.4x.
+ default:
+ max_batch_tokens: 16384
+ compute_precision: bfloat16
+ adapter_path: sie_server.adapters.sglang.cuda13:SGLangCuda13Adapter
+ kv_budget_tokens: 8192
+ adapter_options:
+ loadtime:
+ extra_env:
+ SIE_SGLANG_SINGLE_IMAGE_MAX_PIXELS: "3211264"
+ mem_fraction_static: 0.85
+ served_model_name: Qwen/Qwen3.8-27B-FP8
+ attention_backend: flashinfer
+ grammar_backend: xgrammar
+ json_number_max_digits: 19
+ tool_call_parser: qwen3_coder
+ speculative:
+ enabled: false
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--constrained-json-disable-any-whitespace"
+ - "--kv-cache-dtype"
+ - "bfloat16"
+ - "--mamba-ssm-dtype"
+ - "float32"
+ runtime:
+ first_chunk_timeout_s: 90
+ inter_chunk_timeout_s: 15
+ overall_timeout_s: 600
+ # Official Qwen3.8 non-thinking sampling recipe. Grammar-constrained
+ # requests default to temperature 0 and no presence penalty instead
+ # (see GRAMMAR_SAMPLING_DEFAULTS in core/runtime_options.py).
+ default_sampling:
+ temperature: 0.7
+ top_p: 0.8
+ top_k: 20
+ presence_penalty: 1.5
+ stop_tokens:
+ - "<|im_end|>"
+
+ no-spec:
+ extends: default
+
+ # H100-80GB: one native-window admission with in-checkpoint MTP. FA3 plus
+ # BF16 GDN state is the measured SM90 batch-one winner. Profile-scoped
+ # DeepGEMM fast warmup covers every decode M through 1024 and samples the
+ # prefill range through the explicit 32K boundary. The matching no-spec
+ # profile is the grammar-safe fallback. Every native-window pair uses the
+ # synchronous scheduler: pinned SGLang speculative V2 has a request-dependent
+ # overlap-path crash on hybrid Mamba, while these profiles admit only one
+ # running request and have no useful batch to overlap.
+ h100-256k:
+ max_output_tokens: 32768
+ max_batch_tokens: 262144
+ compute_precision: bfloat16
+ adapter_path: sie_server.adapters.sglang.cuda13:SGLangCuda13Adapter
+ grammar_profile: h100-256k-no-spec
+ kv_budget_tokens: 262144
+ adapter_options:
+ loadtime:
+ max_seq_length: 262144
+ mem_fraction_static: 0.85
+ served_model_name: Qwen/Qwen3.8-27B-FP8
+ attention_backend: fa3
+ grammar_backend: xgrammar
+ json_number_max_digits: 19
+ tool_call_parser: qwen3_coder
+ extra_env:
+ SIE_SGLANG_SINGLE_IMAGE_MAX_PIXELS: "3211264"
+ SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
+ speculative:
+ enabled: true
+ algorithm: eagle
+ num_steps: 3
+ eagle_topk: 1
+ num_draft_tokens: 4
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--constrained-json-disable-any-whitespace"
+ - "--kv-cache-dtype"
+ - "bfloat16"
+ - "--mamba-ssm-dtype"
+ - "bfloat16"
+ - "--mamba-radix-cache-strategy"
+ - "extra_buffer"
+ - "--disable-overlap-schedule"
+ - "--page-size"
+ - "64"
+ - "--chunked-prefill-size"
+ - "32768"
+ - "--max-prefill-tokens"
+ - "32768"
+ - "--max-running-requests"
+ - "1"
+ - "--cuda-graph-max-bs-decode"
+ - "1"
+ runtime:
+ first_chunk_timeout_s: 900
+ inter_chunk_timeout_s: 30
+ overall_timeout_s: 1800
+ default_sampling:
+ temperature: 0.7
+ top_p: 0.8
+ top_k: 20
+ presence_penalty: 1.5
+ stop_tokens:
+ - "<|im_end|>"
+ h100-256k-no-spec:
+ extends: h100-256k
+ adapter_options:
+ loadtime:
+ max_seq_length: 262144
+ mem_fraction_static: 0.85
+ served_model_name: Qwen/Qwen3.8-27B-FP8
+ attention_backend: fa3
+ grammar_backend: xgrammar
+ json_number_max_digits: 19
+ tool_call_parser: qwen3_coder
+ extra_env:
+ SIE_SGLANG_SINGLE_IMAGE_MAX_PIXELS: "3211264"
+ SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
+ speculative:
+ enabled: false
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--constrained-json-disable-any-whitespace"
+ - "--kv-cache-dtype"
+ - "bfloat16"
+ - "--mamba-ssm-dtype"
+ - "bfloat16"
+ - "--mamba-radix-cache-strategy"
+ - "extra_buffer"
+ - "--disable-overlap-schedule"
+ - "--page-size"
+ - "64"
+ - "--chunked-prefill-size"
+ - "32768"
+ - "--max-prefill-tokens"
+ - "32768"
+ - "--max-running-requests"
+ - "1"
+ - "--cuda-graph-max-bs-decode"
+ - "1"
+
+ # H100-80GB batch lane: the h100-256k launch shape without speculation,
+ # admitting up to 32 running requests with CUDA graphs captured for every
+ # decode batch size through 32. Speculation stays off so xgrammar enforces
+ # every constrained token, which makes this the concurrent grammar-safe
+ # native-window lane. Measured on one H100 80GB with a ~670-token json_schema
+ # request: 72 tok/s decode at batch 1 against 11.6 on the bare route, and
+ # 148 requests/min at 8 in flight against 44. Standalone so its grammar twin
+ # inherits only this shape.
+ # Unlike the single-admission native-window pairs it keeps SGLang's overlap
+ # scheduler: those pairs disable it for the speculative-V2 overlap crash on
+ # hybrid Mamba, which does not apply to a non-speculative launch. On 222
+ # page-image json_schema extractions, overlap plus 32 admissions cut
+ # GPU-seconds per request by 10% at 1 in flight, 12% at 8 and 25% at 32,
+ # with the extraction score unchanged.
+ h100-256k-batch:
+ grammar_profile: h100-256k-batch-no-spec
+ max_output_tokens: 32768
+ max_batch_tokens: 262144
+ compute_precision: bfloat16
+ adapter_path: sie_server.adapters.sglang.cuda13:SGLangCuda13Adapter
+ kv_budget_tokens: 262144
+ adapter_options:
+ loadtime:
+ max_seq_length: 262144
+ mem_fraction_static: 0.85
+ served_model_name: Qwen/Qwen3.8-27B-FP8
+ attention_backend: fa3
+ grammar_backend: xgrammar
+ json_number_max_digits: 19
+ tool_call_parser: qwen3_coder
+ extra_env:
+ SIE_SGLANG_SINGLE_IMAGE_MAX_PIXELS: "3211264"
+ SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
+ speculative:
+ enabled: false
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--constrained-json-disable-any-whitespace"
+ - "--kv-cache-dtype"
+ - "bfloat16"
+ - "--mamba-ssm-dtype"
+ - "bfloat16"
+ - "--mamba-radix-cache-strategy"
+ - "extra_buffer"
+ - "--page-size"
+ - "64"
+ - "--chunked-prefill-size"
+ - "32768"
+ - "--max-prefill-tokens"
+ - "32768"
+ - "--max-running-requests"
+ - "32"
+ - "--cuda-graph-max-bs-decode"
+ - "32"
+ runtime:
+ first_chunk_timeout_s: 900
+ inter_chunk_timeout_s: 30
+ overall_timeout_s: 1800
+ default_sampling:
+ temperature: 0.7
+ top_p: 0.8
+ top_k: 20
+ presence_penalty: 1.5
+ stop_tokens:
+ - "<|im_end|>"
+ # Constrained-decoding identity for the batch lane; identical because the
+ # parent is already non-speculative and grammar-safe.
+ h100-256k-batch-no-spec:
+ extends: h100-256k-batch
+
+ # H100-80GB thinking: the h100-256k launch shape without speculation, plus
+ # the qwen3 reasoning parser and the strict-thinking lane so a premature EOS
+ # cannot leave the answer inside the private reasoning channel. The chat
+ # template's own default effort (xhigh) applies because the gateway does not
+ # forward request-level reasoning_effort. Sampling is the model card's
+ # thinking recipe. Standalone so its children inherit only this shape.
+ h100-256k-thinking:
+ grammar_profile: h100-256k-thinking-no-spec
+ max_output_tokens: 32768
+ max_batch_tokens: 262144
+ compute_precision: bfloat16
+ adapter_path: sie_server.adapters.sglang.cuda13:SGLangStrictThinkingAdapter
+ kv_budget_tokens: 262144
+ chat_template_kwargs:
+ enable_thinking: true
+ adapter_options:
+ loadtime:
+ max_seq_length: 262144
+ mem_fraction_static: 0.85
+ served_model_name: Qwen/Qwen3.8-27B-FP8
+ attention_backend: fa3
+ grammar_backend: xgrammar
+ json_number_max_digits: 19
+ reasoning_parser: qwen3
+ tool_call_parser: qwen3_coder
+ extra_env:
+ SIE_SGLANG_SINGLE_IMAGE_MAX_PIXELS: "3211264"
+ SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
+ speculative:
+ enabled: false
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--constrained-json-disable-any-whitespace"
+ - "--kv-cache-dtype"
+ - "bfloat16"
+ - "--mamba-ssm-dtype"
+ - "bfloat16"
+ - "--mamba-radix-cache-strategy"
+ - "extra_buffer"
+ - "--disable-overlap-schedule"
+ - "--page-size"
+ - "64"
+ - "--chunked-prefill-size"
+ - "32768"
+ - "--max-prefill-tokens"
+ - "32768"
+ - "--max-running-requests"
+ - "1"
+ - "--cuda-graph-max-bs-decode"
+ - "1"
+ runtime:
+ first_chunk_timeout_s: 900
+ inter_chunk_timeout_s: 30
+ overall_timeout_s: 1800
+ default_sampling:
+ temperature: 1.0
+ top_p: 0.95
+ top_k: 20
+ presence_penalty: 0.0
+ stop_tokens:
+ - "<|im_end|>"
+ # Constrained-decoding identity for the thinking launch; identical because
+ # the parent is already non-speculative and grammar-safe.
+ h100-256k-thinking-no-spec:
+ extends: h100-256k-thinking
+ # Hardware-neutral alias. Thinking needs the native window, so it resolves
+ # to the H100 256K lane rather than the 8K bare route.
+ thinking:
+ extends: h100-256k-thinking
+
+ # H200-141GB: deliberately retain the measured H100 SM90 recipe and
+ # one-admission launch shape; the larger card provides more memory headroom
+ # without a second tuning surface.
+ h200-256k:
+ max_output_tokens: 32768
+ max_batch_tokens: 262144
+ compute_precision: bfloat16
+ adapter_path: sie_server.adapters.sglang.cuda13:SGLangCuda13Adapter
+ grammar_profile: h200-256k-no-spec
+ kv_budget_tokens: 262144
+ adapter_options:
+ loadtime:
+ max_seq_length: 262144
+ mem_fraction_static: 0.85
+ served_model_name: Qwen/Qwen3.8-27B-FP8
+ attention_backend: fa3
+ grammar_backend: xgrammar
+ json_number_max_digits: 19
+ tool_call_parser: qwen3_coder
+ extra_env:
+ SIE_SGLANG_SINGLE_IMAGE_MAX_PIXELS: "3211264"
+ SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
+ speculative:
+ enabled: true
+ algorithm: eagle
+ num_steps: 3
+ eagle_topk: 1
+ num_draft_tokens: 4
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--constrained-json-disable-any-whitespace"
+ - "--kv-cache-dtype"
+ - "bfloat16"
+ - "--mamba-ssm-dtype"
+ - "bfloat16"
+ - "--mamba-radix-cache-strategy"
+ - "extra_buffer"
+ - "--disable-overlap-schedule"
+ - "--page-size"
+ - "64"
+ - "--chunked-prefill-size"
+ - "32768"
+ - "--max-prefill-tokens"
+ - "32768"
+ - "--max-running-requests"
+ - "1"
+ - "--cuda-graph-max-bs-decode"
+ - "1"
+ runtime:
+ first_chunk_timeout_s: 900
+ inter_chunk_timeout_s: 30
+ overall_timeout_s: 1800
+ default_sampling:
+ temperature: 0.7
+ top_p: 0.8
+ top_k: 20
+ presence_penalty: 1.5
+ stop_tokens:
+ - "<|im_end|>"
+ h200-256k-no-spec:
+ extends: h200-256k
+ adapter_options:
+ loadtime:
+ max_seq_length: 262144
+ mem_fraction_static: 0.85
+ served_model_name: Qwen/Qwen3.8-27B-FP8
+ attention_backend: fa3
+ grammar_backend: xgrammar
+ json_number_max_digits: 19
+ tool_call_parser: qwen3_coder
+ extra_env:
+ SIE_SGLANG_SINGLE_IMAGE_MAX_PIXELS: "3211264"
+ SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
+ speculative:
+ enabled: false
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--constrained-json-disable-any-whitespace"
+ - "--kv-cache-dtype"
+ - "bfloat16"
+ - "--mamba-ssm-dtype"
+ - "bfloat16"
+ - "--mamba-radix-cache-strategy"
+ - "extra_buffer"
+ - "--disable-overlap-schedule"
+ - "--page-size"
+ - "64"
+ - "--chunked-prefill-size"
+ - "32768"
+ - "--max-prefill-tokens"
+ - "32768"
+ - "--max-running-requests"
+ - "1"
+ - "--cuda-graph-max-bs-decode"
+ - "1"
+
+ # RTX PRO 6000 96GB (SM120): retain FlashInfer and 2K prefill chunks from the
+ # official Blackwell recipe while sharing the measured BF16 GDN state and
+ # profile-scoped DeepGEMM fast warmup. Both RTX profiles deliberately leave
+ # --max-prefill-tokens at SGLang's pinned default; only the chunk granularity
+ # is overridden here, and the Stage-5 full-window smoke guards that pairing.
+ rtx-pro-6000-256k:
+ max_output_tokens: 32768
+ max_batch_tokens: 262144
+ compute_precision: bfloat16
+ adapter_path: sie_server.adapters.sglang.cuda13:SGLangCuda13Adapter
+ grammar_profile: rtx-pro-6000-256k-no-spec
+ kv_budget_tokens: 262144
+ adapter_options:
+ loadtime:
+ max_seq_length: 262144
+ mem_fraction_static: 0.85
+ served_model_name: Qwen/Qwen3.8-27B-FP8
+ attention_backend: flashinfer
+ grammar_backend: xgrammar
+ json_number_max_digits: 19
+ tool_call_parser: qwen3_coder
+ extra_env:
+ SIE_SGLANG_SINGLE_IMAGE_MAX_PIXELS: "3211264"
+ SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
+ speculative:
+ enabled: true
+ algorithm: eagle
+ num_steps: 3
+ eagle_topk: 1
+ num_draft_tokens: 4
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--constrained-json-disable-any-whitespace"
+ - "--kv-cache-dtype"
+ - "bfloat16"
+ - "--mamba-ssm-dtype"
+ - "bfloat16"
+ - "--mamba-radix-cache-strategy"
+ - "extra_buffer"
+ - "--disable-overlap-schedule"
+ - "--page-size"
+ - "64"
+ - "--chunked-prefill-size"
+ - "2048"
+ - "--max-running-requests"
+ - "1"
+ - "--cuda-graph-max-bs-decode"
+ - "1"
+ runtime:
+ first_chunk_timeout_s: 900
+ inter_chunk_timeout_s: 30
+ overall_timeout_s: 1800
+ default_sampling:
+ temperature: 0.7
+ top_p: 0.8
+ top_k: 20
+ presence_penalty: 1.5
+ stop_tokens:
+ - "<|im_end|>"
+ rtx-pro-6000-256k-no-spec:
+ extends: rtx-pro-6000-256k
+ adapter_options:
+ loadtime:
+ max_seq_length: 262144
+ mem_fraction_static: 0.85
+ served_model_name: Qwen/Qwen3.8-27B-FP8
+ attention_backend: flashinfer
+ grammar_backend: xgrammar
+ json_number_max_digits: 19
+ tool_call_parser: qwen3_coder
+ extra_env:
+ SIE_SGLANG_SINGLE_IMAGE_MAX_PIXELS: "3211264"
+ SGLANG_JIT_DEEPGEMM_FAST_WARMUP: "1"
+ speculative:
+ enabled: false
+ extra_launch_args:
+ - "--mm-process-config"
+ - '{"image":{"min_pixels":65536,"max_pixels":1003520}}'
+ - "--constrained-json-disable-any-whitespace"
+ - "--kv-cache-dtype"
+ - "bfloat16"
+ - "--mamba-ssm-dtype"
+ - "bfloat16"
+ - "--mamba-radix-cache-strategy"
+ - "extra_buffer"
+ - "--disable-overlap-schedule"
+ - "--page-size"
+ - "64"
+ - "--chunked-prefill-size"
+ - "2048"
+ - "--max-running-requests"
+ - "1"
+ - "--cuda-graph-max-bs-decode"
+ - "1"
diff --git a/examples/train-a-router/self-host/smoke.py b/examples/train-a-router/self-host/smoke.py
new file mode 100644
index 000000000..8921c469e
--- /dev/null
+++ b/examples/train-a-router/self-host/smoke.py
@@ -0,0 +1,76 @@
+"""Check the three pinned workers through one local SIE endpoint."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import time
+from pathlib import Path
+from urllib.parse import urlparse
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+import serving
+import solver
+from data import file_hash, write
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--output", type=Path, required=True)
+ args = parser.parse_args()
+ if urlparse(os.environ.get("SIE_BASE_URL", "")).hostname not in ("127.0.0.1", "localhost", "::1"):
+ raise ValueError("This smoke test only calls an explicitly configured loopback endpoint")
+ args.output.mkdir(parents=True, exist_ok=False)
+ receipt = {"format": "sie-router-self-host-smoke-v1", "status": "incomplete", "calls": [],
+ "source_sha256": {p.name: file_hash(p) for p in (Path(__file__), Path(solver.__file__), Path(serving.__file__))},
+ "observed_cloud_credits": None, "hosting_cost": "unpriced"}
+ started = time.monotonic()
+ try:
+ with serving.SingleSendClient(timeout_s=300, read_timeout_s=300) as client:
+ metadata = {}
+ for model, revision in [(serving.MODEL, serving.REVISION),
+ *[(arm["model"], arm["revision"]) for arm in solver.SETTINGS["arms"].values()]]:
+ value = client.get_model(model)
+ if value["revision"] != revision:
+ raise ValueError("Pinned worker model revision differs")
+ metadata[model] = value
+ if metadata[serving.MODEL]["dims"]["dense"] != 2560:
+ raise ValueError("Encoder dimensions differ")
+ receipt["metadata"] = metadata
+ write(args.output / "metadata.json", metadata)
+ request = "Competition: AMC10\nProblem: A rectangle has perimeter 30 and length 9. What is its area?"
+ vector, usage = serving.embed(request)
+ if vector.shape != (2560,):
+ raise ValueError("Embedding shape differs")
+ receipt["embedding"] = {"dimensions": len(vector), "squared_norm": float(vector @ vector), "receipt": usage}
+ for name, arm in solver.SETTINGS["arms"].items():
+ print(f"Checking {name} initial response and forced structured finalizer", flush=True)
+ messages = [{"role": "system", "content": arm["system"]}, {"role": "user", "content": request}]
+ first = solver.call(client, arm, messages, self_hosted=True)
+ entry = {"arm": name, "model": arm["model"], "initial": first}
+ receipt["calls"].append(entry)
+ write(args.output / f"{name}-initial.json", entry)
+ if not first["response_valid"] or not first["answer"].strip():
+ raise ValueError("Worker returned no initial answer")
+ final_messages = [{"role": "system", "content": solver.FINAL_SYSTEM},
+ {"role": "user", "content": json.dumps({"problem": request, "partial_work": first["answer"]})}]
+ final = solver.call(client, arm, final_messages, final=True, self_hosted=True)
+ structured = json.loads(final["answer"])
+ if not isinstance(structured, dict) or set(structured) != {"answer"} or not isinstance(structured["answer"], str):
+ raise ValueError("Forced finalizer violated the recorded JSON schema")
+ entry["forced_finalizer"] = final
+ write(args.output / f"{name}.json", entry)
+ receipt["status"] = "complete"
+ except Exception as error:
+ receipt["error"] = serving.error_receipt(error)
+ raise
+ finally:
+ receipt["elapsed_seconds"] = time.monotonic() - started
+ write(args.output / "smoke.json", receipt)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/self-host/test_healthcheck.py b/examples/train-a-router/self-host/test_healthcheck.py
new file mode 100644
index 000000000..f1874dd0c
--- /dev/null
+++ b/examples/train-a-router/self-host/test_healthcheck.py
@@ -0,0 +1,50 @@
+"""The worker must not become healthy after a failed or partial preload."""
+
+import unittest
+
+import healthcheck
+
+
+class Client:
+ def __init__(self, missing=None, wrong_revision=False):
+ self.missing = missing
+ self.wrong_revision = wrong_revision
+ self.calls = []
+
+ def get_model(self, model):
+ self.calls.append(model)
+ revision = next(revision for rows in healthcheck.MODELS.values() for name, revision in rows if name == model)
+ return {"name": model, "loaded": model != self.missing, "revision": "changed" if self.wrong_revision else revision,
+ "dims": {"dense": 2560}}
+
+
+class HealthcheckTests(unittest.TestCase):
+ def test_cheap_requires_both_loaded_profiles(self):
+ client = Client(missing="Qwen/Qwen3.5-4B:no-spec")
+ with self.assertRaises(RuntimeError):
+ healthcheck.check(client, "cheap")
+ self.assertEqual(client.calls, [model for model, _ in healthcheck.MODELS["cheap"]])
+
+ def test_failed_encoder_preload_does_not_pass(self):
+ with self.assertRaises(RuntimeError):
+ healthcheck.check(Client(missing="Qwen/Qwen3-Embedding-4B"), "encoder")
+
+ def test_wrong_revision_does_not_pass(self):
+ with self.assertRaises(RuntimeError):
+ healthcheck.check(Client(wrong_revision=True), "strong")
+
+ def test_wrong_profile_with_same_revision_does_not_pass(self):
+ class WrongProfileClient(Client):
+ def get_model(self, model):
+ return {**super().get_model(model), "name": "Qwen/Qwen3.5-4B"}
+
+ with self.assertRaises(RuntimeError):
+ healthcheck.check(WrongProfileClient(), "cheap")
+
+ def test_pinned_loaded_profiles_pass(self):
+ for arm in healthcheck.MODELS:
+ healthcheck.check(Client(), arm)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/self-host/test_proxy.py b/examples/train-a-router/self-host/test_proxy.py
new file mode 100644
index 000000000..c009c8ca3
--- /dev/null
+++ b/examples/train-a-router/self-host/test_proxy.py
@@ -0,0 +1,202 @@
+"""Exercise the real HAProxy boundary against local HTTP fixtures, without models."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import http.client
+import json
+import socket
+import subprocess
+import threading
+import time
+import unittest
+from contextlib import closing
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+
+CHEAP = "Qwen/Qwen3.5-4B"
+STRONG = "Qwen/Qwen3.8-27B-FP8"
+ENCODER = "Qwen/Qwen3-Embedding-4B"
+CALLS = []
+
+
+class Fixture(BaseHTTPRequestHandler):
+ def do_GET(self):
+ self.reply()
+
+ def do_POST(self):
+ self.reply()
+
+ def reply(self):
+ body = self.rfile.read(int(self.headers.get("Content-Length", "0")))
+ record = {
+ "worker": self.server.server_port,
+ "path": self.path,
+ "sha256": hashlib.sha256(body).hexdigest(),
+ "headers": dict(self.headers),
+ }
+ CALLS.append(record)
+ payload = json.dumps(record).encode()
+ self.send_response(429 if self.headers.get("X-Fixture-Error") else 200)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(payload)))
+ self.send_header("X-SIE-Model-Revision", "fixture-revision")
+ self.end_headers()
+ self.wfile.write(payload)
+
+ def log_message(self, *_):
+ pass
+
+
+def request(method, path, body=None, headers=None):
+ with closing(http.client.HTTPConnection("127.0.0.1", 8080, timeout=5)) as client:
+ client.request(method, path, body, headers or {})
+ reply = client.getresponse()
+ return reply.status, dict(reply.getheaders()), reply.read()
+
+
+def raw_request(data):
+ with socket.create_connection(("127.0.0.1", 8080), timeout=5) as client:
+ client.sendall(data)
+ client.shutdown(socket.SHUT_WR)
+ reply = client.recv(65536)
+ return int(reply.split(b" ", 2)[1]) if reply else 0
+
+
+class ProxyBoundary(unittest.TestCase):
+ def chat(self, body):
+ return request("POST", "/v1/chat/completions", body, {"Content-Type": "application/json"})
+
+ def rejected(self, method, path, body=None, headers=None):
+ before = len(CALLS)
+ status, _, _ = request(method, path, body, headers)
+ self.assertGreaterEqual(status, 400)
+ self.assertEqual(len(CALLS), before)
+
+ def test_exact_routes_and_untouched_body_and_revision(self):
+ for model, port in ((CHEAP, 18081), (STRONG, 18082)):
+ body = json.dumps({"model": model, "messages": [{"role": "user", "content": "2+2?"}]}).encode()
+ status, headers, payload = self.chat(body)
+ self.assertEqual(status, 200)
+ self.assertEqual(json.loads(payload)["worker"], port)
+ self.assertEqual(json.loads(payload)["sha256"], hashlib.sha256(body).hexdigest())
+ self.assertEqual({k.lower(): v for k, v in headers.items()}["x-sie-model-revision"], "fixture-revision")
+
+ def test_escaped_model_and_model_like_message(self):
+ body = ('{"m\\u006fdel":"Qwen\\/Qwen3.5-4B",'
+ '"messages":[{"role":"user","content":"\\\"model\\\":\\\"Qwen/Qwen3.8-27B-FP8\\\""}]}')
+ status, _, payload = self.chat(body)
+ self.assertEqual(status, 200)
+ self.assertEqual(json.loads(payload)["worker"], 18081)
+
+ def test_duplicate_top_level_model_and_escaped_duplicate(self):
+ for key in ('"model"', '"m\\u006fdel"'):
+ for value in (CHEAP, STRONG):
+ body = '{"model":"' + CHEAP + '",' + key + ':"' + value + '","messages":[]}'
+ self.rejected("POST", "/v1/chat/completions", body, {"Content-Type": "application/json"})
+
+ def test_missing_wrong_model_and_malformed_json(self):
+ for body in ('{}', '[]', '{"model":null}', '{"model":123}', '{"model":"unknown"}',
+ '{"model":"' + ENCODER + '"}', '{"model":"' + CHEAP + ':no-spec"}',
+ '{"model":"' + CHEAP + '","temperature":NaN}', '{"model":"' + CHEAP + '",}'):
+ self.rejected("POST", "/v1/chat/completions", body, {"Content-Type": "application/json"})
+
+ def test_metadata_and_msgpack_encode(self):
+ for model, port in ((CHEAP, 18081), (STRONG, 18082), (ENCODER, 18083)):
+ status, _, payload = request("GET", "/v1/models/" + model)
+ self.assertEqual(status, 200)
+ self.assertEqual(json.loads(payload)["worker"], port)
+ payload = b"\x81\xa5items\x90"
+ status, _, result = request("POST", "/v1/encode/" + ENCODER, payload,
+ {"Content-Type": "application/msgpack"})
+ self.assertEqual(status, 200)
+ self.assertEqual(json.loads(result)["sha256"], hashlib.sha256(payload).hexdigest())
+
+ def test_wrong_endpoint_method_and_query(self):
+ for method, path in (("GET", "/v1/models"), ("GET", "/healthz"),
+ ("POST", "/v1/encode/" + CHEAP), ("POST", "/v1/models/" + CHEAP),
+ ("GET", "/v1/chat/completions"), ("DELETE", "/v1/models/" + CHEAP),
+ ("GET", "/v1/models/Qwen%2FQwen3.5-4B"),
+ ("GET", "/v1/models/" + CHEAP + "?upstream=evil.invalid")):
+ self.rejected(method, path)
+
+ def test_credentials_and_forwarding_headers_removed(self):
+ supplied = {"Authorization": "Bearer fake-fixture", "X-API-Key": "fake-fixture",
+ "Proxy-Authorization": "fake-fixture", "Cookie": "fixture=1",
+ "Forwarded": "host=evil.invalid", "X-Forwarded-Host": "evil.invalid"}
+ status, _, payload = request("GET", "/v1/models/" + CHEAP, headers=supplied)
+ self.assertEqual(status, 200)
+ forwarded = {k.lower() for k in json.loads(payload)["headers"]}
+ self.assertFalse(forwarded.intersection(k.lower() for k in supplied))
+
+ def test_oversized_body_and_headers(self):
+ self.rejected("POST", "/v1/chat/completions", b" " * 131073, {"Content-Type": "application/json"})
+ self.rejected("GET", "/v1/models/" + CHEAP, headers={"X-Large": "x" * 9000})
+
+ def test_upstream_error_status_body_and_revision_are_preserved(self):
+ before = len(CALLS)
+ status, headers, payload = request("GET", "/v1/models/" + CHEAP, headers={"X-Fixture-Error": "1"})
+ self.assertEqual(status, 429)
+ self.assertEqual(len(CALLS), before + 1)
+ self.assertEqual(json.loads(payload)["worker"], 18081)
+ self.assertEqual({k.lower(): v for k, v in headers.items()}["x-sie-model-revision"], "fixture-revision")
+
+ def test_incomplete_chunked_conflicting_and_missing_lengths(self):
+ start = b"POST /v1/chat/completions HTTP/1.1\r\nHost: localhost\r\nContent-Type: application/json\r\n"
+ for suffix in (b"Content-Length: 100\r\n\r\n{}",
+ b"Transfer-Encoding: chunked\r\n\r\n2\r\n{}\r\n0\r\n\r\n",
+ b"Content-Length: 2\r\nContent-Length: 3\r\n\r\n{}",
+ b"\r\n{}"):
+ before = len(CALLS)
+ status = raw_request(start + suffix)
+ self.assertTrue(status == 0 or status >= 400)
+ self.assertEqual(len(CALLS), before)
+
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--config", type=Path, default=Path(__file__).with_name("haproxy.cfg"))
+ parser.add_argument("--output", type=Path, required=True)
+ args = parser.parse_args()
+ if args.output.exists():
+ raise ValueError("Preserve existing test receipts; choose a new output")
+ subprocess.run(["haproxy", "-c", "-f", str(args.config)], check=True)
+ servers = [ThreadingHTTPServer(("127.0.0.1", port), Fixture) for port in (18081, 18082, 18083)]
+ for server in servers:
+ threading.Thread(target=server.serve_forever, daemon=True).start()
+ started = time.monotonic()
+ process = subprocess.Popen(["haproxy", "-db", "-f", str(args.config)])
+ try:
+ for _ in range(100):
+ if process.poll() is not None:
+ raise RuntimeError("HAProxy exited before validation")
+ try:
+ with socket.create_connection(("127.0.0.1", 8080), timeout=0.1):
+ break
+ except OSError:
+ time.sleep(0.05)
+ result = unittest.TextTestRunner(verbosity=2).run(unittest.defaultTestLoader.loadTestsFromTestCase(ProxyBoundary))
+ receipt = {"format": "sie-self-host-proxy-fixtures-v1", "tests": result.testsRun,
+ "failures": len(result.failures), "errors": len(result.errors),
+ "elapsed_seconds": time.monotonic() - started, "model_inference": False,
+ "haproxy": subprocess.check_output(["haproxy", "-v"], text=True).strip(),
+ "source_sha256": {p.name: hashlib.sha256(p.read_bytes()).hexdigest()
+ for p in (Path(__file__), args.config, args.config.with_name("chat-model.lua"))}}
+ args.output.write_text(json.dumps(receipt, indent=2) + "\n")
+ if not result.wasSuccessful():
+ raise SystemExit(1)
+ finally:
+ process.terminate()
+ try:
+ process.wait(timeout=5)
+ except subprocess.TimeoutExpired:
+ process.kill()
+ process.wait(timeout=5)
+ for server in servers:
+ server.shutdown()
+ server.server_close()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/serving.py b/examples/train-a-router/serving.py
new file mode 100644
index 000000000..ebfc19076
--- /dev/null
+++ b/examples/train-a-router/serving.py
@@ -0,0 +1,154 @@
+"""Bounded SIE SDK collection and deployment for the embedding policy."""
+
+from __future__ import annotations
+
+import gzip
+import json
+import math
+import time
+
+import numpy as np
+from sie_sdk import SIEClient
+from sie_sdk.types import Item
+
+from data import digest, write
+
+MODEL = "Qwen/Qwen3-Embedding-4B"
+REVISION = "5cf2132abc99cad020ac570b19d031efec650f2b"
+INSTRUCTION = "Represent the relative difficulty of the mathematics problem within its named competition for classification as routine or advanced."
+
+
+class SingleSendClient(SIEClient):
+ def _record_retry(self):
+ raise RuntimeError("Automatic retries are disabled in this example")
+
+
+def usage(reply):
+ request = reply.get("request") or {}
+ native = reply.get("usage") or {}
+ billed = request.get("usage") or {}
+ credits = request.get("credits_debited", billed.get("credits_charged", native.get("credits_charged")))
+ return {"request_id": request.get("id"), "credits": credits, "rate_book_version": billed.get("rate_book_version")}
+
+
+def error_receipt(error):
+ result = {"error_type": type(error).__name__, "status": "failed_or_unknown", "credits": None}
+ status = getattr(error, "status_code", None)
+ if isinstance(status, int):
+ result["http_status"] = status
+ request = getattr(error, "request", None)
+ if isinstance(request, dict):
+ result.update(usage({"request": request}))
+ return result
+
+
+def normalize(values):
+ x = np.asarray(values, dtype=np.float64)
+ norms = np.linalg.norm(x, axis=-1, keepdims=True)
+ if not np.isfinite(x).all() or np.any(norms == 0):
+ raise ValueError("Invalid vector")
+ return x / norms
+
+
+def collect(rows, output, *, self_hosted=False):
+ output.mkdir(parents=True, exist_ok=False)
+ started, credits, calls = time.monotonic(), 0.0, []
+ with SingleSendClient(timeout_s=120, read_timeout_s=120) as client:
+ metadata = client.get_model(MODEL)
+ if metadata["revision"] != REVISION or metadata["dims"]["dense"] != 2560:
+ raise ValueError("Serving model differs from the pinned encoder")
+ write(
+ output / "plan.json",
+ {
+ "model": MODEL,
+ "revision": REVISION,
+ "instruction": INSTRUCTION,
+ "rows_sha256": digest(rows),
+ "ids": [r["id"] for r in rows],
+ "input_sha256": {r["id"]: digest(r["request"]) for r in rows},
+ "is_query": True,
+ "dimensions": 2560,
+ "batch_size": 16,
+ "max_seconds": 900,
+ "max_observed_credits": 5000,
+ },
+ )
+ try:
+ with gzip.open(output / "vectors.jsonl.gz", "xt") as handle:
+ for offset in range(0, len(rows), 16):
+ if time.monotonic() - started > 900 or credits >= 5000:
+ raise RuntimeError("Collection budget reached")
+ batch = rows[offset : offset + 16]
+ reply = client.encode(
+ MODEL,
+ [Item(id=r["id"], text=r["request"]) for r in batch],
+ output_types=["dense"],
+ instruction=INSTRUCTION,
+ is_query=True,
+ wait_for_capacity=False,
+ max_oom_retries=0,
+ provision_timeout_s=60,
+ )
+ if len(reply) != len(batch) or {r["id"] for r in reply} != {r["id"] for r in batch}:
+ raise ValueError("Response ID mismatch")
+ receipt = usage(reply[0])
+ if any(usage(r) != receipt for r in reply):
+ raise ValueError("Batch receipts differ")
+ charge = receipt["credits"]
+ if not self_hosted and (
+ not isinstance(charge, (int, float)) or not math.isfinite(charge) or charge < 0
+ ):
+ raise ValueError("Cloud response has no valid credit receipt")
+ credits += charge or 0
+ calls.append(receipt | {"ids": [r["id"] for r in batch]})
+ for row in reply:
+ vector = normalize(row["dense"])
+ if vector.shape != (2560,):
+ raise ValueError("Wrong vector shape")
+ handle.write(json.dumps({"id": row["id"], "vector": vector.tolist()}) + "\n")
+ handle.flush()
+ print(f"Encoded {offset + len(batch)}/{len(rows)}; observed credits {credits:g}", flush=True)
+ except Exception as error:
+ write(output / "incomplete.json", error_receipt(error) | {"completed_calls": calls})
+ raise
+ write(
+ output / "collection.json",
+ {"calls": calls, "credits": None if self_hosted else credits, "elapsed_seconds": time.monotonic() - started},
+ )
+
+
+def vectors(rows, directory):
+ if (directory / "incomplete.json").exists():
+ raise ValueError("Incomplete collection")
+ plan = json.loads((directory / "plan.json").read_text())
+ if (plan["model"], plan["revision"], plan["instruction"]) != (MODEL, REVISION, INSTRUCTION):
+ raise ValueError("Vector configuration differs")
+ if any(plan["input_sha256"].get(r["id"]) != digest(r["request"]) for r in rows):
+ raise ValueError("Recorded vector belongs to a different input")
+ with gzip.open(directory / "vectors.jsonl.gz", "rt") as handle:
+ records = [json.loads(line) for line in handle]
+ by_id = {r["id"]: r["vector"] for r in records}
+ if len(records) != len(by_id) or set(by_id) != set(plan["ids"]):
+ raise ValueError("Duplicate or missing vector IDs")
+ return normalize([by_id[r["id"]] for r in rows])
+
+
+def embed(request):
+ with SingleSendClient(timeout_s=120, read_timeout_s=120) as client:
+ metadata = client.get_model(MODEL)
+ if metadata["revision"] != REVISION:
+ raise ValueError("Live encoder revision differs")
+ reply = client.encode(
+ MODEL,
+ Item(text=request),
+ output_types=["dense"],
+ instruction=INSTRUCTION,
+ is_query=True,
+ wait_for_capacity=False,
+ max_oom_retries=0,
+ provision_timeout_s=60,
+ )
+ vector = normalize(reply["dense"])
+ if vector.shape != (2560,):
+ raise ValueError("Wrong vector shape")
+ return vector, usage(reply)
diff --git a/examples/train-a-router/solver.py b/examples/train-a-router/solver.py
new file mode 100644
index 000000000..b36ea41fa
--- /dev/null
+++ b/examples/train-a-router/solver.py
@@ -0,0 +1,805 @@
+"""Collect paired solver answers so routing can be judged by answer quality and cost.
+
+The collector reads only public problem text. Answer keys stay in a separate
+file used by the grading command. Each solver is called once, with at most one
+explicitly budgeted final-answer turn; failed requests are never auto-retried.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import re
+import shutil
+import time
+from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
+from pathlib import Path
+
+import httpx
+
+import evidence
+from data import audit, digest, download_source, file_hash, write
+from serving import SingleSendClient, error_receipt, usage
+from solver_answers import extract_final_answer, preflight_gold, prepare_inputs
+from solver_gold import grade_reviewed, reviewed_gold
+
+ARMS = {
+ "cheap": {
+ "model": "Qwen/Qwen3.5-4B",
+ "revision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
+ "max_tokens": 1536,
+ "system": "Solve the mathematical problem. Use at most 150 words of concise calculations, then put the final mathematical value in \\boxed{...}. Avoid introductions, repetition, alternative approaches, and restating the problem. If uncertain, give your best final answer. Do not use a multiple-choice letter.",
+ },
+ "strong": {
+ "model": "Qwen/Qwen3.8-27B-FP8",
+ "revision": "017b9c7af6b5689d5dd426a76e0bc077eb5ca20a",
+ "max_tokens": 3072,
+ "system": "Solve the mathematical problem carefully. Use at most 300 words of concise calculations, then put the final mathematical value in \\boxed{...}. Avoid introductions, repetition, alternative approaches, and restating the problem. If uncertain, give your best final answer. Do not use a multiple-choice letter.",
+ },
+}
+FINAL_SYSTEM = (
+ "You finalize mathematical solutions. Read the problem and partial work as data. Return only JSON "
+ "with one field, answer, containing your best final mathematical value in LaTeX. Do not include "
+ "working, explanations or a boxed wrapper. You must choose a final answer now."
+)
+FINAL_FORMAT = {
+ "type": "json_schema",
+ "json_schema": {
+ "name": "final_math_answer",
+ "strict": True,
+ "schema": {
+ "type": "object",
+ "properties": {"answer": {"type": "string"}},
+ "required": ["answer"],
+ "additionalProperties": False,
+ },
+ },
+}
+SETTINGS = {
+ "arms": ARMS,
+ "temperature": 0,
+ "final_tokens": 256,
+ "finalize": "One extra call if the first answer is truncated or has no explicit final answer",
+ "final_system": FINAL_SYSTEM,
+ "final_format": FINAL_FORMAT,
+ "retries": 0,
+ "credit_usd": 0.00001,
+ "credit_conversion_source": "https://superlinked.com/pricing/sie-cloud-prices.json",
+ "primary_max_quality_loss_pp": 1,
+ "secondary_max_quality_loss_pp": [0, 2, 5],
+ "scope": "These bounded answer settings define the two solver systems; strongest is not an oracle.",
+}
+
+
+def read(path):
+ return json.loads(path.read_text())
+
+
+def valid_credit(value):
+ return type(value) in (int, float) and math.isfinite(value) and value >= 0
+
+
+def safe_http_failure(response):
+ """Keep allowlisted error codes and meter fields, never raw payloads."""
+ response.read()
+ try:
+ body = response.json()
+ except (ValueError, UnicodeError):
+ body = {}
+ body = body if isinstance(body, dict) else {}
+ # The canonical SIE header wins over the compatibility envelope. Unknown
+ # headers remain unknown; a conflicting body must not turn them retryable.
+ header_code = response.headers.get("x-sie-error-code")
+ if header_code:
+ raw_code, code_source = header_code, "header"
+ elif "error" in body:
+ error = body["error"]
+ raw_code, code_source = (error.get("code") if isinstance(error, dict) else None), "error"
+ elif "detail" in body:
+ detail = body["detail"]
+ raw_code, code_source = (detail.get("code") if isinstance(detail, dict) else None), "detail"
+ else:
+ raw_code, code_source = body.get("code"), "body"
+ aliases = {"rate_limit_exceeded": "RATE_LIMIT", "provisioning": "PROVISIONING"}
+ code = aliases.get(raw_code, raw_code) if isinstance(raw_code, str) else None
+ known_codes = {
+ "RATE_LIMIT",
+ "BILLING_CAPACITY_UNAVAILABLE",
+ "QUEUE_FULL",
+ "QUEUE_UNAVAILABLE",
+ "transport_failure",
+ "PROVISIONING",
+ "MODEL_LOADING",
+ "RESOURCE_EXHAUSTED",
+ "MODEL_LOAD_FAILED",
+ "INPUT_TOO_LONG",
+ "INSUFFICIENT_CREDITS",
+ "ACCOUNT_DISABLED",
+ "INSUFFICIENT_SCOPE",
+ }
+ code = code if isinstance(code, str) and code in known_codes else "unrecognized"
+ retry_after = response.headers.get("retry-after")
+ try:
+ retry_after = float(retry_after)
+ except (ValueError, TypeError):
+ retry_after = None
+ if not valid_credit(retry_after):
+ retry_after = None
+ known_rejection = (
+ # The header/native discriminator identifies managed pre-dispatch
+ # admission. A bare compatibility rate_limit_exceeded has also been
+ # used for queue failures after publication, so normalization alone is
+ # insufficient evidence that execution never started.
+ (response.status_code == 429 and code == "RATE_LIMIT" and raw_code == "RATE_LIMIT")
+ or (
+ response.status_code == 503
+ and code
+ in {
+ "BILLING_CAPACITY_UNAVAILABLE",
+ "QUEUE_FULL",
+ "PROVISIONING",
+ "MODEL_LOADING",
+ }
+ )
+ or (
+ response.status_code == 503
+ and code in {"QUEUE_UNAVAILABLE", "transport_failure"}
+ and retry_after is not None
+ )
+ )
+ request_id = response.headers.get("x-sie-request-id")
+ if not isinstance(request_id, str) or not re.fullmatch(r"[A-Za-z0-9_.:-]{1,128}", request_id):
+ request_id = None
+ native_usage = body.get("usage")
+ native_usage = native_usage if isinstance(native_usage, dict) else {}
+ credits = native_usage.get("credits_charged")
+ if not valid_credit(credits):
+ header = response.headers.get("x-sie-credits-debited", "")
+ credits = int(header) if re.fullmatch(r"0|[1-9][0-9]{0,18}", header) else None
+ return {
+ "http_status": response.status_code,
+ "error_code": code,
+ "error_code_source": code_source,
+ "request_id": request_id,
+ "credits": credits,
+ "retry_after_seconds": retry_after,
+ "known_pre_execution_rejection": known_rejection,
+ }
+
+
+class RecordingClient(SingleSendClient):
+ """Use the SDK's public transport injection to retain errors before it retries."""
+
+ def __init__(self, *, http_transport=None, **kwargs):
+ self.http_failures = []
+ # Let the SDK resolve its public environment contract. Construction makes
+ # no HTTP requests, and the temporary transport is closed immediately.
+ with SingleSendClient(**kwargs) as configured:
+ base_url = configured.base_url
+ kwargs.pop("base_url", None)
+ transport = httpx.Client(
+ base_url=base_url, transport=http_transport, event_hooks={"response": [self.capture_failure]}
+ )
+ super().__init__(base_url, http_client=transport, **kwargs)
+
+ def capture_failure(self, response):
+ if response.status_code >= 400:
+ self.http_failures.append(safe_http_failure(response))
+
+
+def save_latest(path, value):
+ temporary = path.with_suffix(path.suffix + ".new")
+ # Immutable attempt receipts remain authoritative after an interrupted
+ # replacement; a stale convenience copy can be rebuilt from them.
+ temporary.unlink(missing_ok=True)
+ write(temporary, value)
+ temporary.replace(path)
+
+
+def prepare(reference, source, output):
+ if output.exists():
+ raise FileExistsError(output)
+ evidence.verify(reference)
+ dataset = read(reference / "data.json")
+ audit(dataset)
+ download_source(source)
+ inputs, gold, provenance = prepare_inputs(source, dataset)
+ write(output / "inputs.json", inputs)
+ write(output / "gold-for-scoring-only.json", gold)
+ write(output / "settings.json", SETTINGS)
+ write(
+ output / "plan.json",
+ {
+ "dataset_sha256": digest(dataset),
+ "inputs_sha256": digest(inputs),
+ "gold_sha256": digest(gold),
+ "settings_sha256": digest(SETTINGS),
+ "counts": {split: len(rows) for split, rows in inputs.items()},
+ "source": provenance,
+ "inference_reads_gold": False,
+ "selection": "Use calibration answers only; freeze routing choices before collecting test answers",
+ "grading": "Exact source answers with pinned symbolic equivalence checks; no teacher agreement proxy",
+ },
+ )
+ write(output / "gold-preflight.json", preflight_gold(gold))
+ canonical, review = reviewed_gold(gold, source=source)
+ review["preflight_sha256"] = file_hash(output / "gold-preflight.json")
+ write(output / "gold-reviewed.json", canonical)
+ write(output / "gold-adjudication-details.json", review)
+ write(
+ output / "gold-review.json",
+ {
+ "preflight_sha256": review["preflight_sha256"],
+ "original_gold_sha256": review["original_gold_sha256"],
+ "canonical_gold_sha256": review["canonical_gold_sha256"],
+ "approved_ids": sorted(set(gold) - set(review["needs_review"])),
+ "adjudicated_ids": review["approved_ids"],
+ "unresolved_ids": sorted(review["needs_review"]),
+ "ready": review["ready"],
+ },
+ )
+ print(json.dumps({"prepared": str(output), "counts": {k: len(v) for k, v in inputs.items()}}))
+
+
+def checked_inputs(directory):
+ plan, inputs, settings = (read(directory / name) for name in ("plan.json", "inputs.json", "settings.json"))
+ if plan["inputs_sha256"] != digest(inputs) or plan["settings_sha256"] != digest(settings):
+ raise ValueError("Solver inputs or settings changed after preparation")
+ if settings != SETTINGS:
+ raise ValueError("Solver source settings differ from the saved protocol")
+ return plan, inputs, settings
+
+
+def check_review(directory, selected):
+ """Inference checks only reviewed IDs and hashes, never the answer key."""
+ review_path = directory / "gold-review.json"
+ if not review_path.exists():
+ raise ValueError("Resolve the gold preflight before collecting answers")
+ review = read(review_path)
+ if review["preflight_sha256"] != file_hash(directory / "gold-preflight.json"):
+ raise ValueError("Gold review belongs to a different preflight")
+ if review.get("ready") is False or set(selected) - set(review["approved_ids"]):
+ raise ValueError("Gold review leaves selected problems unresolved")
+
+
+def call(client, arm, messages, *, final=False, self_hosted=False):
+ started = time.monotonic()
+ reply = client.chat_completions(
+ model=arm["model"],
+ messages=messages,
+ temperature=0,
+ max_tokens=256 if final else arm["max_tokens"],
+ response_format=FINAL_FORMAT if final else None,
+ wait_for_capacity=False,
+ max_oom_retries=0,
+ provision_timeout_s=180,
+ read_timeout_s=180,
+ )
+ receipt = usage(reply)
+ credits = receipt["credits"]
+ choices = reply.get("choices")
+ choice = choices[0] if isinstance(choices, list) and choices and isinstance(choices[0], dict) else {}
+ message = choice.get("message")
+ message = message if isinstance(message, dict) else {}
+ answer = message.get("content")
+ return {
+ "messages_sha256": digest(messages),
+ "elapsed_seconds": time.monotonic() - started,
+ "answer": answer or "",
+ "response_valid": isinstance(answer, str),
+ "finish_reason": choice.get("finish_reason"),
+ "execution_revision": client.last_model_revision,
+ "billing": "self_hosted" if self_hosted else "cloud",
+ "receipt": receipt,
+ "billing_known": self_hosted or (type(credits) in (int, float) and math.isfinite(credits) and credits >= 0),
+ "tokens": {
+ k: v
+ for k, v in reply.get("usage", {}).items()
+ if k in ("prompt_tokens", "completion_tokens", "total_tokens")
+ },
+ }
+
+
+def solve(row, arm_name, settings, *, self_hosted=False):
+ arm = settings["arms"][arm_name]
+ record = initial_record(row, arm_name, settings)
+ client = None
+ try:
+ with RecordingClient(timeout_s=180, read_timeout_s=180) as client:
+ messages = [{"role": "system", "content": arm["system"]}, {"role": "user", "content": row["request"]}]
+ first = call(client, arm, messages, self_hosted=self_hosted)
+ record["calls"].append(first)
+ if not first["response_valid"]:
+ raise ValueError("Successful response has no text answer")
+ if not first["billing_known"]:
+ raise ValueError("Successful response has no usable billed-credit receipt")
+ answer = first["answer"]
+ if first["finish_reason"] == "length" or not extract_final_answer(answer):
+ final_messages = [
+ {"role": "system", "content": FINAL_SYSTEM},
+ {"role": "user", "content": json.dumps({"problem": row["request"], "partial_work": answer})},
+ ]
+ final = call(client, arm, final_messages, final=True, self_hosted=self_hosted)
+ record["calls"].append(final)
+ if not final["response_valid"]:
+ raise ValueError("Successful final response has no text answer")
+ if not final["billing_known"]:
+ raise ValueError("Successful final response has no usable billed-credit receipt")
+ try:
+ parsed = json.loads(final["answer"])
+ if (
+ not isinstance(parsed, dict)
+ or set(parsed) != {"answer"}
+ or not isinstance(parsed["answer"], str)
+ ):
+ raise ValueError("Malformed final answer")
+ answer = "\\boxed{" + parsed["answer"] + "}"
+ except (ValueError, TypeError):
+ answer = ""
+ record["final_format_error"] = True
+ record["answer"] = answer
+ record["status"] = "complete"
+ except Exception as error: # noqa: BLE001 -- preserve failed attempts without publishing sensitive error text
+ record["error"] = error_receipt(error)
+ failures = list(client.http_failures) if client is not None else []
+ record["http_failures"] = failures
+ safe_rejection = bool(failures) and failures[-1]["known_pre_execution_rejection"]
+ if failures and "error" in record:
+ record["error"].update(failures[-1])
+ record["observed_credits"] = sum(
+ c["receipt"]["credits"]
+ for c in record["calls"]
+ if type(c["receipt"]["credits"]) in (int, float)
+ and math.isfinite(c["receipt"]["credits"])
+ and c["receipt"]["credits"] >= 0
+ )
+ record["observed_credits"] += sum(f["credits"] for f in failures if valid_credit(f["credits"]))
+ record["has_unknown_charges"] = any(not c["billing_known"] for c in record["calls"]) or (
+ "error" in record and not safe_rejection
+ )
+ record["safe_to_resume"] = safe_rejection and not record["has_unknown_charges"]
+ record["credits"] = record["observed_credits"] if record["status"] == "complete" and not self_hosted else None
+ return record
+
+
+def initial_record(row, arm_name, settings):
+ arm = settings["arms"][arm_name]
+ return {
+ "id": row["id"],
+ "arm": arm_name,
+ "request_sha256": digest(row["request"]),
+ "settings_sha256": digest(settings),
+ "model": arm["model"],
+ "catalog_revision": arm["revision"],
+ "status": "incomplete",
+ "calls": [],
+ }
+
+
+def validate_record(record, row, arm, settings):
+ if (
+ record.get("id") != row["id"]
+ or record.get("arm") != arm
+ or record.get("settings_sha256") != digest(settings)
+ or record.get("request_sha256") != digest(row["request"])
+ or record.get("model") != settings["arms"][arm]["model"]
+ or record.get("catalog_revision") != settings["arms"][arm]["revision"]
+ or record.get("status") not in ("complete", "incomplete")
+ or not valid_credit(record.get("observed_credits"))
+ ):
+ raise ValueError("Answer does not match the frozen request, settings and identity")
+ if record["status"] == "complete":
+ if record.get("has_unknown_charges") or not isinstance(record.get("answer"), str):
+ raise ValueError("Complete answer has unresolved execution or billing")
+ calls = record.get("calls")
+ if not isinstance(calls, list) or not 1 <= len(calls) <= 2:
+ raise ValueError("Complete answer has an invalid call count")
+ hosted = {call.get("billing") for call in calls}
+ if hosted == {"cloud"}:
+ charges = [call.get("receipt", {}).get("credits") for call in calls]
+ if not all(valid_credit(charge) for charge in charges) or (
+ sum(charges) != record["credits"] or record["credits"] != record["observed_credits"]
+ ):
+ raise ValueError("Complete answer credits do not match its call receipts")
+ elif hosted != {"self_hosted"} or record.get("credits") is not None:
+ raise ValueError("Complete answer has inconsistent hosting or cost evidence")
+
+
+def collection_binding(plan, directory, split, rows, settings, frozen_hash, self_hosted):
+ if (
+ plan["ids"] != [r["id"] for r in rows]
+ or len(plan["ids"]) != len(set(plan["ids"]))
+ or plan["split"] != split
+ or plan["plan_sha256"] != file_hash(directory / "plan.json")
+ or plan["settings_sha256"] != digest(settings)
+ or plan.get("policies_frozen_sha256") != frozen_hash
+ or plan["self_hosted"] != self_hosted
+ ):
+ raise ValueError("Collection does not match the prepared split, inputs and frozen policies")
+
+
+def load_attempts(output, rows, settings):
+ """Archive old-format records once, and validate immutable attempt receipts."""
+ records, attempts, interrupted = {}, {}, set()
+ for row in rows:
+ for arm in settings["arms"]:
+ key = (row["id"], arm)
+ name = f"{row['id']}-{arm}"
+ canonical = output / f"{name}.json"
+ history = output / "attempts" / name
+ if canonical.exists() and not list(history.glob("attempt-*.json")):
+ history.mkdir(parents=True, exist_ok=True)
+ shutil.copyfile(canonical, history / "attempt-0001.json")
+ paths = sorted(path for path in history.glob("attempt-*.json") if not path.name.endswith(".intent.json"))
+ history_records = []
+ for path in paths:
+ record = read(path)
+ validate_record(record, row, arm, settings)
+ history_records.append(record)
+ if canonical.exists():
+ record = read(canonical)
+ validate_record(record, row, arm, settings)
+ if not history_records or record != history_records[-1]:
+ raise ValueError("Latest answer differs from its immutable attempt history")
+ records[key] = record
+ elif history_records:
+ # A process may stop after writing the immutable receipt but
+ # before replacing the convenience copy. No generation repeats.
+ records[key] = history_records[-1]
+ save_latest(canonical, history_records[-1])
+ intents = sorted(history.glob("attempt-*.intent.json"))
+ dangling = [path for path in intents if not path.with_name(path.name.replace(".intent", "")).exists()]
+ if dangling:
+ interrupted.add(key)
+ attempts[key] = history_records
+ return records, attempts, interrupted
+
+
+def collect(
+ directory,
+ split,
+ limit,
+ workers,
+ max_credits,
+ max_seconds,
+ *,
+ self_hosted=False,
+ resume=False,
+ retry_unknown=False,
+ max_attempts=3,
+):
+ _plan, inputs, settings = checked_inputs(directory)
+ frozen_hash = None
+ if split == "test":
+ frozen = read(directory / "policies-frozen.json")
+ if frozen["plan_sha256"] != file_hash(directory / "plan.json"):
+ raise ValueError("Test collection requires routing choices frozen against this plan")
+ frozen_hash = file_hash(directory / "policies-frozen.json")
+ if limit is not None and limit <= 0:
+ raise ValueError("A row limit must be positive")
+ rows = inputs[split][:limit] if limit else inputs[split]
+ if (
+ not rows
+ or not 1 <= workers <= 16
+ or not math.isfinite(max_credits)
+ or not math.isfinite(max_seconds)
+ or max_credits <= 0
+ or max_seconds <= 0
+ or type(max_attempts) is not int
+ or max_attempts <= 0
+ ):
+ raise ValueError("Use nonempty inputs, 1..16 workers and positive collection limits")
+ if retry_unknown and not resume:
+ raise ValueError("Unknown execution may only be retried with an explicit --resume")
+ check_review(directory, [r["id"] for r in rows])
+ output = directory / f"answers-{split}"
+ if resume:
+ plan = read(output / "collection-plan.json")
+ collection_binding(plan, directory, split, rows, settings, frozen_hash, self_hosted)
+ else:
+ output.mkdir(exist_ok=False)
+ write(
+ output / "collection-plan.json",
+ {
+ "ids": [r["id"] for r in rows],
+ "split": split,
+ "plan_sha256": file_hash(directory / "plan.json"),
+ "settings_sha256": digest(settings),
+ "policies_frozen_sha256": frozen_hash,
+ "max_credits": max_credits,
+ "credit_limit_scope": "Observed spend; each in-flight task may finish its initial and one finalization call",
+ "max_seconds": max_seconds,
+ "workers": workers,
+ "self_hosted": self_hosted,
+ },
+ )
+ records, history, interrupted = load_attempts(output, rows, settings)
+ todo = []
+ for row in rows:
+ for arm in settings["arms"]:
+ key = (row["id"], arm)
+ previous = records.get(key)
+ if previous and previous["status"] == "complete":
+ continue
+ attempt_folder = output / "attempts" / f"{row['id']}-{arm}"
+ attempt_numbers = {path.name.split(".")[0] for path in attempt_folder.glob("attempt-*.json")}
+ if len(attempt_numbers) >= max_attempts:
+ raise ValueError("Attempt limit reached; inspect receipts before explicitly raising --max-attempts")
+ unknown = key in interrupted or (previous is not None and not previous.get("safe_to_resume", False))
+ if unknown and not retry_unknown:
+ raise ValueError(
+ "Prior execution or charge is unknown; inspect receipts before --resume --retry-unknown"
+ )
+ todo.append((row, arm))
+ if not todo:
+ print(f"{split}: every answer is already complete; rebuilding the summary without requests", flush=True)
+ sessions = output / "sessions"
+ previous_sessions = sorted(sessions.glob("session-*"))
+ if not previous_sessions and (output / "collection.json").exists():
+ initial = sessions / "session-0000"
+ initial.mkdir(parents=True)
+ shutil.copyfile(output / "collection.json", initial / "collection.json")
+ shutil.copyfile(output / "collection-plan.json", initial / "plan.json")
+ previous_sessions = [initial]
+ session = sessions / f"session-{len(previous_sessions) + 1:04d}"
+ write(
+ session / "plan.json",
+ {
+ "collection_plan_sha256": file_hash(output / "collection-plan.json"),
+ "resume": resume,
+ "retry_unknown": retry_unknown,
+ "max_attempts": max_attempts,
+ "workers": workers,
+ "max_credits": max_credits,
+ "max_seconds": max_seconds,
+ "pending": [{"id": row["id"], "arm": arm} for row, arm in todo],
+ "budget_scope": "New observed acquisition charges in this invocation; in-flight requests are drained",
+ },
+ )
+ tasks = iter(todo)
+ started, credits, failures_this_session, sent = time.monotonic(), 0.0, 0, 0
+ stopped = False
+ fatal = None
+ try:
+ # Check public model identity once before inference; no answer key is read here.
+ if todo:
+ with SingleSendClient(timeout_s=60) as client:
+ for arm in settings["arms"].values():
+ metadata = client.get_model(arm["model"])
+ if metadata["revision"] != arm["revision"]:
+ raise ValueError("Solver revision differs from the prepared protocol")
+ with ThreadPoolExecutor(max_workers=workers) as pool:
+ pending = {}
+
+ def schedule():
+ nonlocal stopped, sent
+ if failures_this_session or credits >= max_credits or time.monotonic() - started >= max_seconds:
+ stopped = True
+ return
+ task = next(tasks, None)
+ if task is not None:
+ row, arm = task
+ identity = row["id"]
+ folder = output / "attempts" / f"{identity}-{arm}"
+ numbers = [int(path.name.split(".")[0].split("-")[1]) for path in folder.glob("attempt-*.json")]
+ number = max(numbers, default=0) + 1
+ target = folder / f"attempt-{number:04d}.json"
+ write(
+ target.with_name(target.stem + ".intent.json"),
+ {
+ "id": identity,
+ "arm": arm,
+ "session": session.name,
+ "request_sha256": digest(row["request"]),
+ "settings_sha256": digest(settings),
+ },
+ )
+ pending[pool.submit(solve, row, arm, settings, self_hosted=self_hosted)] = (row, arm, target)
+ sent += 1
+
+ for _ in range(workers):
+ schedule()
+ while pending:
+ done, _ = wait(pending, timeout=10, return_when=FIRST_COMPLETED)
+ for future in done:
+ row, arm, target = pending.pop(future)
+ identity = row["id"]
+ try:
+ record = future.result()
+ except Exception as error: # noqa: BLE001 -- retain unknown worker attempts and drain others
+ record = initial_record(row, arm, settings) | {
+ "error": error_receipt(error),
+ "observed_credits": 0,
+ "credits": None,
+ "has_unknown_charges": True,
+ "safe_to_resume": False,
+ }
+ validate_record(record, row, arm, settings)
+ write(target, record)
+ save_latest(output / f"{identity}-{arm}.json", record)
+ records[(identity, arm)] = record
+ history[(identity, arm)].append(record)
+ credits += record["observed_credits"]
+ failures_this_session += int(record["status"] != "complete")
+ completed = sum(r["status"] == "complete" for r in records.values())
+ print(
+ f"{split}: {completed}/{len(settings['arms']) * len(rows)} complete, "
+ f"{failures_this_session} failed this session, {credits:g} new observed credits",
+ flush=True,
+ )
+ for _ in done:
+ schedule()
+ except Exception as error: # noqa: BLE001 -- preserve a content-free incomplete-run receipt
+ fatal = error_receipt(error)
+ completed = sum(r["status"] == "complete" for r in records.values())
+ failed = sum(r["status"] != "complete" for r in records.values())
+ status = (
+ "complete" if completed == len(settings["arms"]) * len(rows) and not failed and fatal is None else "incomplete"
+ )
+ all_attempts = [record for attempts in history.values() for record in attempts]
+ summary = {
+ "status": status,
+ "completed": completed,
+ "failed": failed,
+ "stopped": stopped,
+ "observed_credits": sum(record["observed_credits"] for record in all_attempts),
+ "selected_answer_credits": sum(
+ record["observed_credits"] for record in records.values() if record["status"] == "complete"
+ ),
+ "failed_attempts": sum(record["status"] != "complete" for record in all_attempts),
+ "unknown_charge_attempts": sum(record.get("has_unknown_charges", True) for record in all_attempts),
+ "prior_interrupted_attempts": len(interrupted),
+ "acquisition_cost_complete": not interrupted
+ and not any(record.get("has_unknown_charges", True) for record in all_attempts),
+ "session": session.name,
+ "session_observed_credits": credits,
+ "session_attempts": sent,
+ "elapsed_seconds": time.monotonic() - started,
+ "fatal_error": fatal,
+ }
+ write(session / "collection.json", summary)
+ save_latest(output / "collection.json", summary)
+ if status != "complete":
+ raise RuntimeError(
+ "Collection incomplete; preserve receipts and inspect the recorded failure before continuing"
+ )
+
+
+def grade(directory, split, output=None):
+ output = output or directory / f"grades-{split}.json"
+ if output.exists():
+ raise FileExistsError(output)
+ plan, inputs, settings = checked_inputs(directory)
+ gold = read(directory / "gold-for-scoring-only.json")
+ if digest(gold) != plan["gold_sha256"]:
+ raise ValueError("Answer key changed after preparation")
+ collection = directory / f"answers-{split}"
+ summary = read(collection / "collection.json")
+ if summary["status"] != "complete":
+ raise ValueError("Refuse an incomplete collection")
+ collection_plan = read(collection / "collection-plan.json")
+ selected = collection_plan["ids"]
+ if (
+ collection_plan["plan_sha256"] != file_hash(directory / "plan.json")
+ or collection_plan["settings_sha256"] != digest(settings)
+ or collection_plan["split"] != split
+ or summary["completed"] != len(selected) * len(settings["arms"])
+ or summary["failed"] != 0
+ ):
+ raise ValueError("Collection does not match the prepared plan and count")
+ if not selected or selected != [r["id"] for r in inputs[split]][: len(selected)]:
+ raise ValueError("Collection IDs differ from the prepared split order")
+ if split == "test" and collection_plan.get("policies_frozen_sha256") != file_hash(
+ directory / "policies-frozen.json"
+ ):
+ raise ValueError("Frozen policies changed after test collection")
+ check_review(directory, selected)
+ by_id = {r["id"]: r for r in inputs[split]}
+ result = []
+ for identity in selected:
+ row = by_id[identity]
+ outcome = {"id": identity}
+ for arm in settings["arms"]:
+ path = collection / f"{identity}-{arm}.json"
+ answer = read(path)
+ if (
+ answer["status"] != "complete"
+ or answer["settings_sha256"] != digest(settings)
+ or answer["request_sha256"] != digest(row["request"])
+ or answer["id"] != identity
+ or answer["arm"] != arm
+ ):
+ raise ValueError("Answer does not match the frozen request and settings")
+ validate_record(answer, row, arm, settings)
+ measured = grade_reviewed(identity, gold[identity], answer["answer"])
+ if measured["status"] in ("gold_unparseable", "grading_error"):
+ raise ValueError(f"Grader could not safely score {identity}; inspect the source answer")
+ outcome.update(
+ {
+ f"{arm}_correct": measured["correct"],
+ f"{arm}_credits": answer["credits"],
+ f"{arm}_grade": measured,
+ f"{arm}_record_sha256": file_hash(path),
+ }
+ )
+ result.append(outcome)
+ write(
+ output,
+ {
+ "split": split,
+ "plan_sha256": file_hash(directory / "plan.json"),
+ "collection_plan_sha256": file_hash(collection / "collection-plan.json"),
+ "collection_summary_sha256": file_hash(collection / "collection.json"),
+ "policies_frozen_sha256": collection_plan.get("policies_frozen_sha256"),
+ "grader_source_sha256": file_hash(Path(__file__).with_name("solver_answers.py")),
+ "gold_review_source_sha256": file_hash(Path(__file__).with_name("solver_gold.py")),
+ "gold_review_sha256": file_hash(directory / "gold-review.json"),
+ "rows": result,
+ },
+ )
+ print(
+ json.dumps(
+ {
+ "n": len(result),
+ **{arm: sum(r[f"{arm}_correct"] for r in result) / len(result) for arm in settings["arms"]},
+ }
+ )
+ )
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ sub = parser.add_subparsers(dest="command", required=True)
+ child = sub.add_parser("prepare")
+ child.add_argument("--evidence", type=Path, default=Path("evidence"))
+ child.add_argument("--source", type=Path, default=Path("run-output/solver-source"))
+ child.add_argument("--output", type=Path, required=True)
+ for command in ("collect", "grade"):
+ child = sub.add_parser(command)
+ child.add_argument("--run", type=Path, required=True)
+ child.add_argument("--split", choices=("validation", "calibration", "test"), required=True)
+ if command == "collect":
+ child.add_argument("--limit", type=int)
+ child.add_argument("--workers", type=int, default=4)
+ child.add_argument("--max-credits", type=int, default=1000000)
+ child.add_argument("--max-seconds", type=int, default=7200)
+ child.add_argument("--self-hosted", action="store_true")
+ child.add_argument(
+ "--resume", action="store_true", help="Continue missing answers; preserve every earlier attempt"
+ )
+ child.add_argument(
+ "--retry-unknown",
+ action="store_true",
+ help="Explicitly permit resubmission when prior execution or charge is unknown",
+ )
+ child.add_argument(
+ "--max-attempts",
+ type=int,
+ default=3,
+ help="Maximum recorded attempts per problem and solver across resumptions",
+ )
+ else:
+ child.add_argument("--output", type=Path, help="Write grades to a new file, preserving archived grades")
+ args = parser.parse_args()
+ if args.command == "prepare":
+ prepare(args.evidence, args.source, args.output)
+ elif args.command == "collect":
+ collect(
+ args.run,
+ args.split,
+ args.limit,
+ args.workers,
+ args.max_credits,
+ args.max_seconds,
+ self_hosted=args.self_hosted,
+ resume=args.resume,
+ retry_unknown=args.retry_unknown,
+ max_attempts=args.max_attempts,
+ )
+ else:
+ grade(args.run, args.split, args.output)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/solver_answers.py b/examples/train-a-router/solver_answers.py
new file mode 100644
index 000000000..d560a45a8
--- /dev/null
+++ b/examples/train-a-router/solver_answers.py
@@ -0,0 +1,357 @@
+"""Join pinned solver golds and grade explicit final answers without model judging.
+
+Run ``preflight_gold`` before collecting solver outputs. Its review list is an
+audit gate, not permission to exclude difficult-to-grade problems. Math-Verify
+uses signal timeouts: grade on the main thread, or in a separate process.
+"""
+
+from __future__ import annotations
+
+import importlib.metadata
+import re
+from collections import Counter
+from fractions import Fraction
+from pathlib import Path
+
+from data import SOURCE_HASHES, SOURCE_REVISION, audit, digest, file_hash, render_request
+
+MAX_ANSWER_CHARS = 4096
+MAX_RESPONSE_CHARS = 100_000
+PARSING_SECONDS = 2
+GRADER_VERSION = "explicit-final-math-verify-v3"
+_BOX = re.compile(r"\\(?:boxed|fbox)\s*\{")
+_FINAL = re.compile(
+ r"(?im)(?=(?:^|(?<=[.!?,;])\s+)(?:(?:therefore|thus|hence|so)[,:]?\s+)?"
+ r"(?:\*\*)?(?:the\s+)?(?:final\s+answer|answer)"
+ r"(?:\*\*)?\s*(?::|=|\bis\b)\s*(.+)$)"
+)
+
+
+def prepare_inputs(source: Path, dataset: dict) -> tuple[dict, dict, dict]:
+ """Return solver-only inputs, separate answers and checked source provenance."""
+ import pyarrow.parquet as pq
+
+ audit(dataset)
+ if dataset.get("source_revision") != SOURCE_REVISION:
+ raise ValueError("Dataset source revision differs from the pinned source")
+ originals = {}
+ for name, expected in SOURCE_HASHES.items():
+ path = source / name
+ if file_hash(path) != expected:
+ raise ValueError(f"Source hash differs: {name}")
+ rows = pq.read_table(path, columns=["problem", "answer", "tag", "year"]).to_pylist()
+ originals[name.split(".")[0]] = rows
+
+ inputs, gold = {}, {}
+ for split in ("validation", "calibration", "test"):
+ inputs[split] = []
+ for row in dataset["splits"][split]:
+ match = re.fullmatch(r"e2h-amc-(train|eval)-(0|[1-9]\d*)", row["id"])
+ if match is None or int(match[2]) >= len(originals[match[1]]):
+ raise ValueError("Unknown pinned-source row ID")
+ original = originals[match[1]][int(match[2])]
+ if (
+ row["request"] != render_request(original["tag"], original["problem"])
+ or row["contest"] != original["tag"]
+ or row["year"] != original["year"]
+ ):
+ raise ValueError(f"Source/request join differs: {row['id']}")
+ answer = original["answer"]
+ if not isinstance(answer, str) or not answer.strip():
+ raise ValueError(f"Missing source answer: {row['id']}")
+ inputs[split].append({"id": row["id"], "request": row["request"]})
+ gold[row["id"]] = answer
+ return (
+ inputs,
+ gold,
+ {
+ "source_revision": SOURCE_REVISION,
+ "source_hashes": dict(SOURCE_HASHES),
+ "dataset_sha256": digest(dataset),
+ "inputs_sha256": digest(inputs),
+ "gold_sha256": digest(gold),
+ "counts": {split: len(rows) for split, rows in inputs.items()},
+ },
+ )
+
+
+def _boxed_body(text: str, opening: int) -> str | None:
+ depth = 1
+ for index in range(opening + 1, len(text)):
+ if index and text[index - 1] == "\\":
+ continue
+ if text[index] == "{":
+ depth += 1
+ elif text[index] == "}":
+ depth -= 1
+ if not depth:
+ return text[opening + 1 : index]
+ return None
+
+
+def _boxes(text: str) -> list[tuple[int, str | None]]:
+ result, end = [], -1
+ for match in _BOX.finditer(text):
+ if match.start() < end:
+ continue
+ body = _boxed_body(text, match.end() - 1)
+ result.append((match.start(), body))
+ if body is None:
+ break
+ end = match.end() + len(body) + 1
+ return result
+
+
+def _clean(text: str) -> str:
+ text = text.strip().removeprefix("**").removesuffix("**").strip()
+ if text.endswith("."):
+ text = text[:-1].rstrip()
+ for left, right in (("$$", "$$"), ("$", "$"), (r"\[", r"\]"), (r"\(", r"\)")):
+ if text.startswith(left) and text.endswith(right):
+ text = text[len(left) : -len(right)].strip()
+ break
+ return text.replace(r"\left", "").replace(r"\right", "").strip()
+
+
+def _answer_only(text: str) -> bool:
+ if "\n" in text or len(text) > MAX_ANSWER_CHARS:
+ return False
+ # A single name/Yes/No is an answer; a sentence containing a number is not.
+ if re.fullmatch(r"[A-Za-z]+", text):
+ return True
+ words = re.findall(r"(? str:
+ """Extract the last explicit answer, or an answer-only response.
+
+ Balanced boxes preserve nested fractions. An unfinished final box does not
+ fall back to an earlier intermediate answer. Unmarked derivations return "".
+ """
+ if not isinstance(response, str) or len(response) > MAX_RESPONSE_CHARS:
+ return ""
+ candidates = _boxes(response)
+ for match in _FINAL.finditer(response):
+ value = match[1].strip()
+ box = _boxes(value)
+ if box:
+ value = box[-1][1]
+ candidates.append((match.start(), value))
+ if candidates:
+ value = max(candidates, key=lambda item: item[0])[1]
+ answer = _clean(value) if value is not None else ""
+ else:
+ answer = _clean(response)
+ if not _answer_only(answer):
+ return ""
+ return answer if 0 < len(answer) <= MAX_ANSWER_CHARS else ""
+
+
+def _parts(text: str) -> list[str]:
+ depth, start, parts = 0, 0, []
+ for index, character in enumerate(text):
+ if character in "({[":
+ depth += 1
+ elif character in ")}]":
+ depth -= 1
+ if depth < 0:
+ raise ValueError("Unbalanced answer delimiters")
+ elif character == "," and depth == 0:
+ parts.append(text[start:index].strip())
+ start = index + 1
+ if depth:
+ raise ValueError("Unbalanced answer delimiters")
+ return parts + [text[start:].strip()]
+
+
+def _text_answer(text: str) -> str | None:
+ match = re.fullmatch(r"\\(?:text|mathrm)\{([A-Za-z][A-Za-z -]*)\}", text)
+ if match:
+ return " ".join(match[1].casefold().split())
+ if re.fullmatch(r"[A-Z][a-z]+", text) or text.casefold() in ("yes", "no"):
+ return text.casefold()
+ return None
+
+
+def _parse(text: str, depth: int = 0) -> dict:
+ from math_verify import LatexExtractionConfig, parse
+
+ text = _clean(text)
+ if not text or len(text) > MAX_ANSWER_CHARS or depth > 4 or "$" in text:
+ raise ValueError("Invalid answer length or nested math environment")
+ named = _text_answer(text)
+ if named is not None:
+ return {"kind": "text", "value": named}
+ tuple_container = text
+ if text.startswith(r"\{") and text.endswith(r"\}"):
+ tuple_container = text[2:-2].strip()
+ parts = _parts(tuple_container)
+ if len(parts) > 1 and all(part.startswith("(") and part.endswith(")") for part in parts):
+ if len(parts) > 32:
+ raise ValueError("Too many answer tuples")
+ return {"kind": "set_of_tuples", "value": [_parse(part, depth + 1) for part in parts]}
+ if text.startswith("(") and text.endswith(")"):
+ parts = _parts(text[1:-1])
+ if len(parts) > 1:
+ if len(parts) > 32 or any(not part for part in parts):
+ raise ValueError("Invalid tuple")
+ return {"kind": "tuple", "value": [_parse(part, depth + 1) for part in parts]}
+ ratio = re.fullmatch(r"([+-]?\d+(?:\.\d+)?)\s*:\s*([+-]?\d+(?:\.\d+)?)", text)
+ if ratio:
+ # A score such as 6:0 must not become an infinity or an equivalent 3:0.
+ return {"kind": "ratio", "value": (Fraction(ratio[1]), Fraction(ratio[2]))}
+ # Math-Verify deliberately accepts 9% == 9. Keep mathematical scaling
+ # explicit here; source questions asking for a percentage have a separate
+ # reviewed rule for an omitted percent sign.
+ text = text.replace(r"\%", r"\cdot\frac{1}{100}").replace("%", r"\cdot\frac{1}{100}")
+ values = parse(
+ "$" + text + "$",
+ extraction_config=[LatexExtractionConfig(boxed_match_priority=0)],
+ fallback_mode="no_fallback",
+ extraction_mode="first_match",
+ parsing_timeout=PARSING_SECONDS,
+ raise_on_error=True,
+ )
+ if not values:
+ raise ValueError("No mathematical expression parsed")
+ return {"kind": "math", "value": values}
+
+
+def _summary(parsed: dict) -> dict:
+ if parsed["kind"] in ("tuple", "set_of_tuples"):
+ return {"kind": parsed["kind"], "items": [_summary(part) for part in parsed["value"]]}
+ if parsed["kind"] != "math":
+ return {"kind": parsed["kind"], "value": str(parsed["value"])[:160]}
+ # Avoid formatting enormous symbolic objects outside the parser's timeout.
+ return {"kind": "math", "types": [type(value).__name__ for value in parsed["value"]]}
+
+
+def _equivalent(gold: dict, prediction: dict) -> bool:
+ from math_verify import verify
+ from sympy import Equality
+
+ if gold["kind"] != prediction["kind"]:
+ return False
+ if gold["kind"] == "set_of_tuples":
+ remaining = list(prediction["value"])
+ for item in gold["value"]:
+ matching = next((i for i, value in enumerate(remaining) if _equivalent(item, value)), None)
+ if matching is None:
+ return False
+ remaining.pop(matching)
+ return not remaining
+ if gold["kind"] == "tuple":
+ return len(gold["value"]) == len(prediction["value"]) and all(
+ _equivalent(a, b) for a, b in zip(gold["value"], prediction["value"], strict=True)
+ )
+ if gold["kind"] != "math":
+ return gold["value"] == prediction["value"]
+ for value in prediction["value"]:
+ relations = value.atoms(Equality)
+ if len(relations) > 32:
+ raise ValueError("Too many answer equalities")
+ for relation in relations:
+ left, right = relation.lhs, relation.rhs
+ if left.is_Symbol and left not in right.free_symbols:
+ continue # A final assignment such as x=7 may name the value.
+ if right.is_Symbol and right not in left.free_symbols:
+ continue
+ # Math-Verify otherwise unwraps any equation's RHS, even 8=7.
+ # Only a true equality may supply a final value this way.
+ if not verify(left, right, strict=True, timeout_seconds=PARSING_SECONDS, raise_on_error=True):
+ return False
+ return bool(
+ verify(
+ gold["value"],
+ prediction["value"],
+ strict=True,
+ timeout_seconds=PARSING_SECONDS,
+ raise_on_error=True,
+ )
+ )
+
+
+def _review_reasons(answer: str) -> list[str]:
+ text, reasons = _clean(answer), []
+ if re.search(r"\\(?:l|c)dots|\.\.\.", text):
+ reasons.append("ellipsis_requires_explicit_answer_set")
+ if ":" in text:
+ reasons.append("ratio_or_score_semantics")
+ commas = text.replace("{,}", ",")
+ top_level = _parts(commas)
+ if len(top_level) > 1 and not re.fullmatch(r"[+-]?\d{1,3}(?:,\d{3})+", commas):
+ reasons.append("bare_list_or_multiple_answers")
+ if len(top_level) == 1 and text.startswith("(") and text.endswith(")") and len(_parts(text[1:-1])) == 2:
+ reasons.append("ordered_pair_or_interval")
+ return reasons
+
+
+def grader_metadata() -> dict:
+ return {
+ "protocol": GRADER_VERSION,
+ "math_verify": importlib.metadata.version("math-verify"),
+ "strict": True,
+ "parsing_timeout_seconds": PARSING_SECONDS,
+ "comparison_timeout_seconds": PARSING_SECONDS,
+ "extraction": "last explicit final answer or answer-only response; no intermediate-expression search",
+ "percentage_semantics": "percent is multiplication by 1/100; omitted signs require a reviewed source rule",
+ "equations": "named assignments or verified equalities; false equations cannot contribute only their RHS",
+ }
+
+
+def preflight_gold(goldmap: dict[str, str]) -> dict:
+ """Audit every answer; keep unresolved IDs visible instead of dropping rows."""
+ from math_verify.errors import TimeoutException
+
+ review, counts = {}, Counter()
+ for identity, answer in goldmap.items():
+ reasons, parsed = [], None
+ try:
+ reasons = _review_reasons(answer)
+ value = _parse(answer)
+ parsed = _summary(value)
+ if not _equivalent(value, value):
+ reasons.append("gold_does_not_verify_against_itself")
+ except (Exception, TimeoutException) as error: # noqa: BLE001 — preserve parser failures in the audit.
+ reasons.append("gold_parse_error:" + type(error).__name__)
+ if reasons:
+ review[identity] = {"answer": answer, "reasons": reasons, "parsed": parsed}
+ counts["needs_review" if reasons else "ready"] += 1
+ return {
+ "grader": grader_metadata(),
+ "gold_sha256": digest(goldmap),
+ "n": len(goldmap),
+ "ready": not review,
+ "counts": dict(counts),
+ "needs_review": review,
+ }
+
+
+def grade_answer(gold: str, response: str) -> dict:
+ """Grade a final answer; gold/parser errors remain distinct from wrong answers."""
+ from math_verify.errors import TimeoutException
+
+ final = extract_final_answer(response)
+ result = {"correct": False, "status": "no_final_answer", "final_answer": final, "gold": None, "prediction": None}
+ try:
+ expected = _parse(gold)
+ result["gold"] = _summary(expected)
+ result["gold_review_reasons"] = _review_reasons(gold)
+ except (Exception, TimeoutException) as error: # noqa: BLE001 — gold failures must not disappear.
+ return result | {"status": "gold_unparseable", "error_type": type(error).__name__}
+ if not final:
+ return result
+ try:
+ if expected["kind"] == "text":
+ literal = _text_answer(final) or " ".join(final.casefold().split())
+ predicted = {"kind": "text", "value": literal}
+ else:
+ predicted = _parse(final)
+ result["prediction"] = _summary(predicted)
+ except (Exception, TimeoutException) as error: # noqa: BLE001 — distinguish invalid output from a wrong answer.
+ return result | {"status": "prediction_unparseable", "error_type": type(error).__name__}
+ try:
+ return result | {"status": "graded", "correct": _equivalent(expected, predicted)}
+ except (Exception, TimeoutException) as error: # noqa: BLE001 — expose symbolic-comparison failures.
+ return result | {"status": "grading_error", "error_type": type(error).__name__}
diff --git a/examples/train-a-router/solver_gold.py b/examples/train-a-router/solver_gold.py
new file mode 100644
index 000000000..5007ca12b
--- /dev/null
+++ b/examples/train-a-router/solver_gold.py
@@ -0,0 +1,242 @@
+"""Source-only adjudications fixed before inspecting any solver answers.
+
+Rules bind to both row ID and original answer. Two numeric transcription errors
+are corrected; the other rules declare notation/semantics already established
+by the source problem and solution. No evaluation rows are removed.
+"""
+
+from __future__ import annotations
+
+import re
+from fractions import Fraction
+from itertools import pairwise
+
+from data import SOURCE_HASHES, digest, file_hash
+from solver_answers import extract_final_answer, grade_answer, preflight_gold
+
+PROTOCOL = "source-only-gold-adjudication-v2"
+# Original answer, canonical answer, semantic type. Full source reasoning and
+# problem/solution hashes accompany the published gold-adjudication artifact.
+RULES = {
+ "e2h-amc-train-977": ("6:0", "6:00", "clock_time"),
+ "e2h-amc-eval-1694": (r"1{,}10{,}0", "1010000", "source_correction"),
+ "e2h-amc-train-945": ("(7,9)", "(7,9)", "ordered_tuple"),
+ "e2h-amc-eval-2468": ("(8,2)", "(8,2)", "ordered_tuple"),
+ "e2h-amc-eval-1731": ("-6,0,2,8", r"\{-6,0,2,8\}", "finite_set"),
+ "e2h-amc-eval-718": ("100,110,11", r"\{11,100,110\}", "finite_set"),
+ "e2h-amc-train-711": ("1346,2016,2019", r"\{1346,2016,2019\}", "finite_set"),
+ "e2h-amc-eval-2077": ("(-1,2)", "(-1,2)", "ordered_tuple"),
+ "e2h-amc-train-285": ("1,2,3,4", r"\{1,2,3,4\}", "finite_set"),
+ "e2h-amc-train-684": ("1:2", r"\frac{1}{2}", "ratio"),
+ "e2h-amc-eval-111": ("(2,2),(12,17),(17,12)", r"\{(2,2),(12,17),(17,12)\}", "set_of_ordered_tuples"),
+ "e2h-amc-eval-1815": ("2:3", r"\frac{2}{3}", "ratio"),
+ "e2h-amc-eval-2927": (r"2 \sqrt{2}, 4+2 \sqrt{2}", r"\{2\sqrt{2},4+2\sqrt{2}\}", "finite_set"),
+ "e2h-amc-eval-1773": (r"0, \pm i \sqrt{6}", r"\{-i\sqrt{6},0,i\sqrt{6}\}", "finite_set"),
+ "e2h-amc-eval-1728": (
+ "(12,3,4),(6,5.4,7.6)",
+ r"\{(12,3,4),(6,\frac{27}{5},\frac{38}{5})\}",
+ "set_of_ordered_tuples",
+ ),
+ "e2h-amc-train-645": (r"0, \frac{\sqrt{5}-1}{2}", r"\{0,\frac{\sqrt{5}-1}{2}\}", "finite_set"),
+ "e2h-amc-eval-769": ("7,15", r"\{7,15\}", "finite_set"),
+ "e2h-amc-eval-2738": (r"0,3,6,9, \ldots, 6048", r"\{3k\mid k\in\mathbb{Z},0\leq k\leq2016\}", "progression"),
+ "e2h-amc-eval-2152": ("(-1030, -990)", "(-1030,-990)", "ordered_tuple"),
+ "e2h-amc-eval-620": (r"21{,}0", "21000", "source_correction"),
+ "e2h-amc-eval-2906": ("4:1", "4", "ratio"),
+ # These seven questions explicitly request a percentage (or the integer N
+ # in N percent), unlike other questions merely containing percentage data.
+ "e2h-amc-train-56": (r"80 \%", r"80\%", "percentage"),
+ "e2h-amc-eval-2677": ("40", r"40\%", "percentage"),
+ "e2h-amc-train-146": ("78", r"78\%", "percentage"),
+ "e2h-amc-train-140": ("54", r"54\%", "percentage"),
+ "e2h-amc-eval-55": ("200", r"200\%", "percentage"),
+ "e2h-amc-eval-812": ("75", r"75\%", "percentage"),
+ "e2h-amc-eval-1803": ("19", r"19\%", "percentage"),
+}
+CORRECTION_REASONS = {
+ "e2h-amc-eval-1694": "The double sum is 100*5050 + 100*5050 = 1010000; the source lost zero digits.",
+ "e2h-amc-eval-620": "The logarithmic product simplifies to (20*21/2)*100 = 21000; the source lost zero digits.",
+}
+
+
+def _clock_minutes(value: str) -> int | None:
+ value = re.sub(r"\\(?:text|mathrm)\{([^{}]*)\}", r"\1", value)
+ value = re.sub(r"\\[,;! ]|~", " ", value).strip().casefold()
+ value = re.sub(r"\b([ap])\s*\.\s*m\.?", r"\1m", value)
+ value = re.sub(r"\s+in the (?:afternoon|evening)$", " pm", value)
+ value = re.sub(r"\s+in the morning$", " am", value)
+ match = re.fullmatch(r"(\d{1,2})(?::(\d{1,2}))?\s*(am|pm)?", value)
+ if match is None:
+ return None
+ hour, minute, meridiem = int(match[1]), int(match[2] or 0), match[3]
+ if hour > 23 or minute > 59 or (meridiem and not 1 <= hour <= 12):
+ return None
+ if meridiem:
+ hour = hour % 12 + (12 if meridiem == "pm" else 0)
+ elif hour < 12:
+ # This source question explicitly starts at noon and asks about later
+ # the same day. A bare hour therefore uses its afternoon interpretation.
+ hour += 12
+ return 60 * hour + minute
+
+
+def _compact(value: str) -> str:
+ value = value.replace(r"\left", "").replace(r"\right", "")
+ value = re.sub(r"\\(?:ldots|cdots|dots)|…", "...", value)
+ value = re.sub(r"\\(?:text|mathrm)\{([^{}]*)\}", r"\1", value)
+ value = re.sub(r"\\mathbb\s*\{?Z\}?", "Z", value)
+ for old, new in ((r"\leq", "≤"), (r"\le", "≤"), (r"\in", "∈"), (r"\mid", ":"), (r"\cdot", "*")):
+ value = value.replace(old, new)
+ value = re.sub(r"\\[,;! ]|\s+", "", value)
+ if value.startswith(r"\{") and value.endswith(r"\}"):
+ value = value[2:-2]
+ elif value.startswith("{") and value.endswith("}"):
+ value = value[1:-1]
+ return value
+
+
+def _progression(value: str) -> tuple[int, int, int] | None:
+ """Recognize bounded integer progressions, without evaluating expressions."""
+ compact = _compact(value)
+ listed = re.fullmatch(r"((?:-?\d+,){2,32})\.\.\.,(-?\d+)", compact)
+ if listed:
+ prefix = [int(part) for part in listed[1].rstrip(",").split(",")]
+ start, step, last = prefix[0], prefix[1] - prefix[0], int(listed[2])
+ if (
+ step > 0
+ and all(b - a == step for a, b in pairwise(prefix))
+ and last >= prefix[-1]
+ and (last - start) % step == 0
+ ):
+ return start, step, last
+ return None
+ builder = re.fullmatch(r"(-?\d+)\*?([a-z])[:|](.+)", compact)
+ if builder:
+ coefficient, variable, domain = int(builder[1]), builder[2], builder[3]
+ if coefficient <= 0:
+ return None
+ # Integer domain is mandatory: a real interval is not a discrete set.
+ member = variable + "∈Z"
+ if domain.startswith(member + ","):
+ bounds = domain[len(member) + 1 :]
+ elif domain.endswith("," + member):
+ bounds = domain[: -len(member) - 1]
+ else:
+ bounds = ""
+ bound = re.fullmatch(r"(-?\d+)(≤|<)" + variable + r"(≤|<)(-?\d+)", bounds)
+ if bound:
+ low = int(bound[1]) + int(bound[2] == "<")
+ high = int(bound[4]) - int(bound[3] == "<")
+ return (coefficient * low, coefficient, coefficient * high) if low <= high else None
+ indices = re.fullmatch(variable + r"=(0,1,(?:2,)?\.\.\.,\d+)", domain)
+ if indices:
+ return 0, coefficient, coefficient * int(domain.rsplit(",", 1)[1])
+ divisible = re.fullmatch(r"([a-z])∈Z[:|](-?\d+)(≤|<)\1(≤|<)(-?\d+),(\d+)[:|]\1", compact)
+ if divisible:
+ low = int(divisible[2]) + int(divisible[3] == "<")
+ high = int(divisible[5]) - int(divisible[4] == "<")
+ step = int(divisible[6])
+ if step > 0:
+ first, last = -(-low // step) * step, (high // step) * step
+ return (first, step, last) if first <= last else None
+ words = re.sub(r"\s+", " ", value.casefold().strip())
+ natural = re.fullmatch(
+ r"(?:all )?multiples of (\d+) (?:from (-?\d+) to|between (-?\d+) and) (-?\d+)( inclusive)?", words
+ )
+ if natural and (natural[2] is not None or natural[5]):
+ step, start, last = int(natural[1]), int(natural[2] or natural[3]), int(natural[4])
+ if step > 0 and start <= last and start % step == 0 and last % step == 0:
+ return start, step, last
+ return None
+
+
+def _special(correct: bool, final: str, kind: str, expected, observed) -> dict:
+ return {
+ "correct": correct,
+ "status": "graded" if final else "no_final_answer",
+ "final_answer": final,
+ "gold": {"kind": kind, "value": expected},
+ "prediction": {"kind": kind, "value": observed},
+ }
+
+
+def grade_reviewed(identity: str, originalgold: str, response: str) -> dict:
+ """Use the fixed semantic rule for this source row, preserving every row."""
+ if identity not in RULES:
+ return grade_answer(originalgold, response)
+ original, canonical, kind = RULES[identity]
+ if originalgold != original:
+ raise ValueError("Adjudication does not match the original source answer")
+ final = extract_final_answer(response)
+ if kind == "clock_time":
+ observed = _clock_minutes(final) if final else None
+ result = _special(observed == 18 * 60, final, kind, 18 * 60, observed)
+ elif kind == "progression":
+ observed = _progression(final) if final else None
+ result = _special(observed == (0, 3, 6048), final, kind, (0, 3, 6048), observed)
+ elif kind == "percentage":
+ # In "what percentage?", 40 and 40% both mean forty percent. Explicit
+ # 0.4% must still be rejected. Do not apply this convention to counts.
+ expected = canonical if "%" in final else canonical.removesuffix(r"\%")
+ result = grade_answer(expected, response)
+ else:
+ if kind == "ratio":
+ ratio = re.fullmatch(r"([+-]?\d+(?:\.\d+)?)\s*:\s*([+-]?\d+(?:\.\d+)?)", final)
+ if ratio:
+ if not Fraction(ratio[2]):
+ result = _special(False, final, kind, canonical, "zero denominator")
+ return result | {"adjudication": {"protocol": PROTOCOL, "id": identity, "kind": kind}}
+ normalized = Fraction(ratio[1]) / Fraction(ratio[2])
+ response = "\\boxed{" + str(normalized) + "}"
+ result = grade_answer(canonical, response)
+ return result | {"adjudication": {"protocol": PROTOCOL, "id": identity, "kind": kind}}
+
+
+def reviewed_gold(gold: dict[str, str], source=None) -> tuple[dict, dict]:
+ """Apply only source-bound rules and return explicit approval/coverage metadata."""
+ if source is not None:
+ for name, expected in SOURCE_HASHES.items():
+ if file_hash(source / name) != expected:
+ raise ValueError("Gold adjudications require the pinned source")
+ corrected = dict(gold)
+ approved, corrections = [], []
+ for identity, original in gold.items():
+ if identity not in RULES:
+ continue
+ expected, canonical, kind = RULES[identity]
+ if original != expected:
+ raise ValueError(f"Source gold changed for adjudicated row: {identity}")
+ measured = grade_reviewed(identity, original, "\\boxed{" + canonical + "}")
+ if measured["status"] != "graded" or not measured["correct"]:
+ raise ValueError(f"Canonical reviewed answer cannot be graded: {identity}")
+ corrected[identity] = canonical
+ approved.append(identity)
+ corrections.append(
+ {
+ "id": identity,
+ "original": original,
+ "canonical": canonical,
+ "kind": kind,
+ "source_answer_correction": kind == "source_correction",
+ "reason": CORRECTION_REASONS.get(
+ identity,
+ "Source question explicitly asks for a percentage; an omitted percent sign is allowed."
+ if kind == "percentage"
+ else "Source problem and solution establish this answer type.",
+ ),
+ }
+ )
+ preflight = preflight_gold(gold)
+ unresolved = {key: value for key, value in preflight["needs_review"].items() if key not in approved}
+ return corrected, {
+ "protocol": PROTOCOL,
+ "source_hashes": dict(SOURCE_HASHES),
+ "original_gold_sha256": digest(gold),
+ "canonical_gold_sha256": digest(corrected),
+ "approved_ids": sorted(approved),
+ "correction_records": corrections,
+ "n": len(gold),
+ "excluded": [],
+ "ready": not unresolved,
+ "needs_review": unresolved,
+ }
diff --git a/examples/train-a-router/solver_policies.py b/examples/train-a-router/solver_policies.py
new file mode 100644
index 000000000..8a6fbe535
--- /dev/null
+++ b/examples/train-a-router/solver_policies.py
@@ -0,0 +1,584 @@
+"""Freeze downstream routing on calibration answers, then report fixed test policies.
+
+This CPU-only orchestration reads recorded router scores and solver grades. It
+does not call models, collect answers, or select a policy using test answers.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+import numpy as np
+
+import evidence
+import policies
+import solver_report
+from data import LABELS, digest, file_hash, write
+from serving import MODEL, REVISION, vectors
+
+STAGES = ("base", "lora512", "lora2581", "ridge64", "ridge512")
+TARGETS = (1, 0, 2, 5)
+SCORE_FORMAT = "sie-solver-router-scores-v1"
+FROZEN_FORMAT = "sie-solver-policies-v1"
+GRADING_HASHES = ("grader_source_sha256", "gold_review_source_sha256", "gold_review_sha256")
+
+
+def read(path):
+ return json.loads(Path(path).read_text())
+
+
+def seal(document):
+ return document | {"sha256": digest(document)}
+
+
+def check_seal(document, expected_format):
+ if document.get("format") != expected_format or document.get("sha256") != digest(
+ {key: value for key, value in document.items() if key != "sha256"}
+ ):
+ raise ValueError("Frozen document format or content hash differs")
+
+
+def finite_scores(values, count):
+ result = np.asarray(values, dtype=float)
+ if result.shape != (count,) or not np.isfinite(result).all():
+ raise ValueError("Expected one finite router score per input")
+ return result
+
+
+def _native_stage(directory, variant, dataset, calibration_sha, test_sha):
+ plan, frozen, calibration, test = (
+ read(directory / name)
+ for name in ("plan.json", "frozen.json", f"{variant}-calibration.json", "evaluation-test.json")
+ )
+ selected = frozen["selected"][variant]
+ if (
+ frozen["plan_sha256"] != file_hash(directory / "plan.json")
+ or plan["data_sha256"]["calibration.jsonl"] != calibration_sha
+ or plan["config"]["seed"] != 89
+ or selected["seed"] != 89
+ or test["frozen_sha256"] != file_hash(directory / "frozen.json")
+ or test["data_sha256"] != test_sha
+ or test["ids"] != [row["id"] for row in dataset["splits"]["test"]]
+ ):
+ raise ValueError("Native scores differ from their frozen model, seed or data")
+ threshold = float(selected["threshold"])
+ if not np.isfinite(threshold):
+ raise ValueError("Nonfinite original threshold")
+ scores = {}
+ for split, measured in (("calibration", calibration), ("test", test["models"][variant])):
+ rows = dataset["splits"][split]
+ values = finite_scores(measured["scores"], len(rows))
+ if (
+ measured["labels"] != [row["label"] for row in rows]
+ or measured["predictions"] != (values >= threshold).astype(int).tolist()
+ ):
+ raise ValueError("Native scores, labels or original decisions are not aligned")
+ scores[split] = values.tolist()
+ return {
+ "family": "native",
+ "seed": 89,
+ "variant": variant,
+ "original_threshold": threshold,
+ "checkpoint_sha256": selected["checkpoint_sha256"],
+ "scores": scores,
+ }
+
+
+def load_stage_scores(baseline, study, policy_directory):
+ """Load five public frozen stages, verifying metadata only; weights are optional."""
+ baseline, study, policy_directory = map(Path, (baseline, study, policy_directory))
+ evidence.verify(baseline)
+ evidence.verify(study, manifest_sha256=evidence.LORA_MANIFEST_SHA256)
+ dataset = read(baseline / "data.json")
+ rows = {split: dataset["splits"][split] for split in ("calibration", "test")}
+ if read(baseline / "test.json") != rows["test"]:
+ raise ValueError("Native test data differs from the router dataset")
+ calibration = [json.loads(line) for line in (baseline / "calibration.jsonl").read_text().splitlines()]
+ if [row["state"] for row in calibration] != [row["request"] for row in rows["calibration"]] or [
+ row["expected"]["difficulty"] for row in calibration
+ ] != [LABELS[row["label"]] for row in rows["calibration"]]:
+ raise ValueError("Native calibration order differs from the router dataset")
+ artifacts = {}
+
+ def bind(root_name, root, names):
+ for name in names:
+ artifacts[f"{root_name}/{name}"] = file_hash(root / name)
+
+ bind("baseline", baseline, ("manifest.json", "data.json", "calibration.jsonl", "test.json"))
+ bind("study", study, ("manifest.json", "all-frozen.json", "evaluation-plan.json", "evaluation-complete.json"))
+ stages = {}
+ for name, root_name, root, relative, variant in (
+ ("base", "baseline", baseline, "native/seed-89", "base"),
+ ("lora512", "study", study, "screening/original/seed-89", "lora"),
+ ("lora2581", "study", study, "expanded/seed-89", "lora"),
+ ):
+ stages[name] = _native_stage(
+ root / relative,
+ variant,
+ dataset,
+ artifacts["baseline/calibration.jsonl"],
+ artifacts["baseline/test.json"],
+ )
+ names = ("plan.json", "frozen.json", f"{variant}-calibration.json", "evaluation-test.json")
+ bind(root_name, root, [f"{relative}/{name}" for name in names])
+ frozen = read(policy_directory / "frozen.json")
+ if frozen["dataset_sha256"] != digest(dataset):
+ raise ValueError("Embedding policy suite uses a different dataset")
+ bind("policies", policy_directory, ("frozen.json", "ridge64.json", "ridge512.json"))
+ matrices = {}
+ for split, directory in (("calibration", "development-embeddings"), ("test", "test-embeddings")):
+ matrices[split] = vectors(rows[split], baseline / directory)
+ bind("baseline", baseline, [f"{directory}/{name}" for name in ("plan.json", "vectors.jsonl.gz")])
+ reference = read(baseline / "embedding-evaluation.json")
+ if reference["dataset_sha256"] != digest(dataset) or reference["ids"] != [row["id"] for row in rows["test"]]:
+ raise ValueError("Embedding reference uses different test inputs")
+ bind("baseline", baseline, ("embedding-evaluation.json",))
+ for size in (64, 512):
+ name = f"ridge{size}"
+ policy = read(policy_directory / f"{name}.json")
+ if (
+ policy["sha256"] != frozen["policies"][name]
+ or policy["dataset_sha256"] != digest(dataset)
+ or policy["development_vectors_sha256"] != artifacts["baseline/development-embeddings/vectors.jsonl.gz"]
+ or policy["labels"] != size
+ or policy["train_ids"] != [row["id"] for row in dataset["splits"]["train"][:size]]
+ ):
+ raise ValueError("Embedding policy differs from its frozen training inputs")
+ scores = {
+ split: finite_scores(policies.scores(policy, matrix), len(rows[split]))
+ for split, matrix in matrices.items()
+ }
+ threshold = float(policy["threshold"])
+ if (
+ not np.isfinite(threshold)
+ or (scores["test"] >= threshold).astype(int).tolist() != reference["predictions"][name]
+ ):
+ raise ValueError("Embedding decisions differ from the frozen reference")
+ stages[name] = {
+ "family": "embedding",
+ "encoder": MODEL,
+ "revision": REVISION,
+ "original_threshold": threshold,
+ "policy_sha256": policy["sha256"],
+ "scores": {split: values.tolist() for split, values in scores.items()},
+ }
+ return seal(
+ {
+ "format": SCORE_FORMAT,
+ "dataset_sha256": digest(dataset),
+ "ids": {split: [row["id"] for row in values] for split, values in rows.items()},
+ "input_sha256": {
+ split: {row["id"]: digest(row["request"]) for row in values} for split, values in rows.items()
+ },
+ "stages": stages,
+ "source_artifacts_sha256": artifacts,
+ "source_code_sha256": {
+ name: file_hash(Path(__file__).with_name(name)) for name in ("policies.py", "serving.py")
+ },
+ "interpretation": "Pre-existing difficulty-router scores on an exposed benchmark; no solver test answers used",
+ }
+ )
+
+
+def _checked_scores(scores):
+ check_seal(scores, SCORE_FORMAT)
+ if set(scores["stages"]) != set(STAGES) or set(scores["ids"]) != {"calibration", "test"}:
+ raise ValueError("Expected all five declared stages and both populations")
+ seen = set()
+ for split, ids in scores["ids"].items():
+ if not ids or len(ids) != len(set(ids)) or seen.intersection(ids):
+ raise ValueError("Router populations must have unique, disjoint IDs")
+ if set(scores["input_sha256"][split]) != set(ids):
+ raise ValueError("Router input hashes do not match IDs")
+ seen.update(ids)
+ for name in STAGES:
+ stage = scores["stages"][name]
+ if stage["family"] != ("embedding" if name.startswith("ridge") else "native"):
+ raise ValueError("Router family differs from the declared stage")
+ finite_scores(stage["scores"][split], len(ids))
+ if not np.isfinite(stage["original_threshold"]):
+ raise ValueError("Nonfinite original threshold")
+
+
+def _grading_provenance(document):
+ hashes = {name: document.get(name) for name in GRADING_HASHES}
+ if any(
+ not isinstance(value, str) or len(value) != 64 or any(char not in "0123456789abcdef" for char in value)
+ for value in hashes.values()
+ ):
+ raise ValueError("Grading provenance requires SHA256 hashes for the grader, gold-review source and gold review")
+ return hashes
+
+
+def _grade_rows(path, split, plan_sha, ids, *, quality_only=False):
+ grades = read(path)
+ _grading_provenance(grades)
+ if grades.get("split") != split or grades.get("plan_sha256") != plan_sha:
+ raise ValueError("Grades use a different split or solver plan")
+ if [row["id"] for row in grades["rows"]] != ids:
+ raise ValueError("Grades must contain every expected ID in the frozen order")
+ # The pure reporter validates both boolean outcomes and complete measured costs.
+ if quality_only:
+ solver_report.summarize_quality(grades["rows"], "always_strong")
+ else:
+ solver_report.summarize(grades["rows"], "always_strong")
+ return grades
+
+
+def _credits(path, split, plan_sha, ids, input_hashes):
+ if path is None:
+ return None, None
+ record = read(path)
+ if (
+ record.get("plan_sha256") != plan_sha
+ or (record.get("model"), record.get("revision")) != (MODEL, REVISION)
+ or record.get("batch_size") != 1
+ ):
+ raise ValueError("Router credits require the same plan and pinned encoder measured one request at a time")
+ rows = record[split]
+ if [row["id"] for row in rows] != ids:
+ raise ValueError("Router credit IDs do not match the frozen order")
+ values = []
+ for row in rows:
+ value = row["credits"]
+ if isinstance(value, bool) or not isinstance(value, (int, float)) or not np.isfinite(value) or value < 0:
+ raise ValueError("Router credits must be measured finite nonnegative numbers")
+ if "request_sha256" in row and row["request_sha256"] != input_hashes[row["id"]]:
+ raise ValueError("Router credit receipt belongs to another request")
+ values.append(value)
+ return values, file_hash(path)
+
+
+def _selections(rows, scores, embedding_credits, credit_usd, *, quality_only=False):
+ if [row["id"] for row in rows] != scores["ids"]["calibration"]:
+ raise ValueError("Calibration outcomes differ from the frozen score order")
+ if quality_only:
+ return {}
+ return {
+ name: {
+ str(target): solver_report.select_cutoff(
+ rows,
+ scores["stages"][name]["scores"]["calibration"],
+ target,
+ router_credits=embedding_credits if name.startswith("ridge") else None,
+ credit_usd=credit_usd,
+ )
+ for target in TARGETS
+ }
+ for name in STAGES
+ }
+
+
+def freeze_calibrations(
+ plan_path,
+ calibration_grades_path,
+ scores,
+ router_credits_path=None,
+ credit_usd=1e-5,
+ *,
+ quality_only=False,
+):
+ """Return a sealed policy document without reading any solver test answer."""
+ plan_path = Path(plan_path)
+ if quality_only and router_credits_path is not None:
+ raise ValueError("Quality-only mode does not accept priced router credits")
+ if (plan_path.parent / "answers-test").exists() or (plan_path.parent / "grades-test.json").exists():
+ raise ValueError("Refuse a new policy freeze after test-answer collection has started")
+ _checked_scores(scores)
+ plan, inputs = read(plan_path), read(plan_path.with_name("inputs.json"))
+ if plan["dataset_sha256"] != scores["dataset_sha256"] or plan["inputs_sha256"] != digest(inputs):
+ raise ValueError("Solver plan inputs differ from the frozen router population")
+ for split, ids in scores["ids"].items():
+ if (
+ [row["id"] for row in inputs[split]] != ids
+ or plan["counts"][split] != len(ids)
+ or {row["id"]: digest(row["request"]) for row in inputs[split]} != scores["input_sha256"][split]
+ ):
+ raise ValueError("Solver and router input IDs, order or requests differ")
+ plan_sha = file_hash(plan_path)
+ grades = _grade_rows(
+ calibration_grades_path,
+ "calibration",
+ plan_sha,
+ scores["ids"]["calibration"],
+ quality_only=quality_only,
+ )
+ credits, credit_sha = _credits(
+ router_credits_path,
+ "calibration",
+ plan_sha,
+ scores["ids"]["calibration"],
+ scores["input_sha256"]["calibration"],
+ )
+ keys = ("id", "cheap_correct", "strong_correct")
+ if not quality_only:
+ keys += ("cheap_credits", "strong_credits")
+ rows = [{key: row[key] for key in keys} for row in grades["rows"]]
+ selections = _selections(rows, scores, credits, credit_usd, quality_only=quality_only)
+ return seal(
+ {
+ "format": FROZEN_FORMAT,
+ "analysis_mode": "quality_only" if quality_only else "cost",
+ "plan_sha256": plan_sha,
+ "calibration_grades_sha256": file_hash(calibration_grades_path),
+ **_grading_provenance(grades),
+ "source_scores": scores,
+ "calibration_rows": rows,
+ "embedding_calibration_credits": credits,
+ "embedding_calibration_credits_sha256": credit_sha,
+ "credit_usd": None if quality_only else credit_usd,
+ "primary_max_quality_loss_pp": None if quality_only else 1,
+ "secondary_max_quality_loss_pp": [] if quality_only else [0, 2, 5],
+ "selections": selections,
+ "selection_note": (
+ "Quality-only: retain original difficulty cutoffs; unknown costs cannot define a cheapest policy"
+ if quality_only
+ else "Calibrate cost policies using measured solver credits"
+ ),
+ "source_code_sha256": {
+ name: file_hash(Path(__file__).with_name(name)) for name in ("solver_policies.py", "solver_report.py")
+ },
+ "test_access": "Router scores only; solver test answers are not opened during policy selection",
+ "interpretation": "Targets constrain observed calibration gaps; prospective test quality is not guaranteed",
+ }
+ )
+
+
+def report(frozen_path, test_grades_path, router_credits_path=None):
+ """Apply every predeclared policy; never choose a winner from test results."""
+ frozen = read(frozen_path)
+ check_seal(frozen, FROZEN_FORMAT)
+ grading = _grading_provenance(frozen)
+ if frozen.get("analysis_mode") not in ("cost", "quality_only"):
+ raise ValueError("Unknown frozen analysis mode")
+ quality_only = frozen["analysis_mode"] == "quality_only"
+ if quality_only and router_credits_path is not None:
+ raise ValueError("Quality-only mode does not accept priced router credits")
+ scores = frozen["source_scores"]
+ _checked_scores(scores)
+ if (
+ frozen["primary_max_quality_loss_pp"] != (None if quality_only else 1)
+ or frozen["secondary_max_quality_loss_pp"] != ([] if quality_only else [0, 2, 5])
+ or (quality_only and (frozen["credit_usd"] is not None or frozen["embedding_calibration_credits"] is not None))
+ or frozen["source_code_sha256"]
+ != {name: file_hash(Path(__file__).with_name(name)) for name in ("solver_policies.py", "solver_report.py")}
+ or frozen["selections"]
+ != _selections(
+ frozen["calibration_rows"],
+ scores,
+ frozen["embedding_calibration_credits"],
+ frozen["credit_usd"],
+ quality_only=quality_only,
+ )
+ ):
+ raise ValueError("Frozen calibration policy, declared targets or reporting code differs")
+ ids = scores["ids"]["test"]
+ grades = _grade_rows(test_grades_path, "test", frozen["plan_sha256"], ids, quality_only=quality_only)
+ if grades.get("policies_frozen_sha256") != file_hash(frozen_path) or _grading_provenance(grades) != grading:
+ raise ValueError(
+ "Test grades are not bound to the pre-collection policy freeze, same grader and same gold review"
+ )
+ credits, credits_sha = _credits(
+ router_credits_path,
+ "test",
+ frozen["plan_sha256"],
+ ids,
+ scores["input_sha256"]["test"],
+ )
+ rows, rate = grades["rows"], frozen["credit_usd"]
+
+ def summarize(routes, overhead=None):
+ if quality_only:
+ return solver_report.summarize_quality(rows, routes)
+ return solver_report.summarize(rows, routes, router_credits=overhead, credit_usd=rate)
+
+ stages = {}
+ for name in STAGES:
+ stage = scores["stages"][name]
+ values = np.array(stage["scores"]["test"])
+ overhead = credits if name.startswith("ridge") else None
+ calibrated = {}
+ for target in () if quality_only else TARGETS:
+ selection = frozen["selections"][name][str(target)]
+ mode = selection["mode"]
+ routes = (values >= selection["cutoff"]).tolist() if mode == "threshold" else mode
+ calibrated[str(target)] = {
+ "mode": mode,
+ "cutoff": selection["cutoff"],
+ "metrics": summarize(routes, overhead if mode == "threshold" else 0),
+ }
+ stages[name] = {
+ "original_threshold": stage["original_threshold"],
+ "original_difficulty_policy": summarize((values >= stage["original_threshold"]).tolist(), overhead),
+ "calibrated_cost_policies": calibrated,
+ }
+ return {
+ "format": "sie-solver-policy-report-v1",
+ "analysis_mode": frozen["analysis_mode"],
+ "plan_sha256": frozen["plan_sha256"],
+ "policies_frozen_sha256": file_hash(frozen_path),
+ "test_grades_sha256": file_hash(test_grades_path),
+ "source_scores_sha256": scores["sha256"],
+ "embedding_test_credits_sha256": credits_sha,
+ "ids": ids,
+ "n": len(ids),
+ "credit_usd": rate,
+ "primary_max_quality_loss_pp": frozen["primary_max_quality_loss_pp"],
+ "secondary_max_quality_loss_pp": frozen["secondary_max_quality_loss_pp"],
+ "selection_note": frozen["selection_note"],
+ "baselines": {name: summarize(name, 0) for name in ("always_strong", "always_cheap")},
+ "stages": stages,
+ "interpretation": (
+ "Quality-only replay of original frozen difficulty cutoffs. Solver and infrastructure prices are unknown; "
+ "no costs, savings, break-even prices or cost-calibrated policies are reported. "
+ "All policies share the same paired solver outcomes on an exposed benchmark."
+ )
+ if quality_only
+ else (
+ "Every frozen policy shares the same paired solver outcomes. Positive gap means answer-accuracy loss; "
+ "negative gap preserves cheap-solver rescues. Unknown native overhead is excluded from gross savings. "
+ "Calibration targets are not test guarantees; this population is an exposed benchmark."
+ ),
+ }
+
+
+def _display_table(headers, rows):
+ widths = [max(len(str(row[i])) for row in [headers, *rows]) for i in range(len(headers))]
+ for index, row in enumerate([headers, *rows]):
+ print(
+ " | ".join(
+ str(cell).ljust(width) if i == 0 else str(cell).rjust(width)
+ for i, (cell, width) in enumerate(zip(row, widths, strict=True))
+ )
+ )
+ if index == 0:
+ print("-+-".join("-" * width for width in widths))
+
+
+def display(result):
+ """Print quality and explicitly separated gross/net costs without changing the report."""
+ names = {
+ "base": "Base Laya",
+ "lora512": "LoRA 512",
+ "lora2581": "LoRA 2,581",
+ "ridge64": "Qwen ridge 64",
+ "ridge512": "Qwen ridge 512",
+ }
+ groups = [
+ (
+ "Original frozen difficulty cutoffs",
+ [
+ ("Always strong", result["baselines"]["always_strong"]),
+ ("Always cheap", result["baselines"]["always_cheap"]),
+ ]
+ + [(names[name], result["stages"][name]["original_difficulty_policy"]) for name in STAGES],
+ )
+ ]
+ quality_only = result["analysis_mode"] == "quality_only"
+ if not quality_only:
+ target = result["primary_max_quality_loss_pp"]
+ groups.append(
+ (
+ f"Cost policies selected for a {target}pp calibration loss target (test results)",
+ [
+ (names[name], result["stages"][name]["calibrated_cost_policies"][str(target)]["metrics"])
+ for name in STAGES
+ ],
+ )
+ )
+ print(f"Answer quality on {result['n']} paired test requests (exposed benchmark)")
+ print("Gap = always-strong accuracy minus policy accuracy; negative gaps mean more correct answers.")
+ print("Regressions / rescues count answers lost / gained versus always strong.")
+ if quality_only:
+ print("Costs unknown: quality-only comparison; no cost-based policy selection.")
+
+ def number(metrics, key, *, missing="unpriced", digits=4):
+ value = metrics.get(key)
+ return missing if value is None else f"{value:.{digits}f}"
+
+ for title, entries in groups:
+ print(f"\n{title}")
+ _display_table(
+ ["Policy", "Answer %", "Gap pp", "Cheap %", "Regressions", "Rescues"],
+ [
+ [
+ name,
+ f"{100 * metrics['answer_accuracy']:.2f}",
+ f"{metrics['gap_vs_strong_pp']:+.2f}",
+ f"{100 * metrics['cheap_share']:.2f}",
+ metrics["regressions"],
+ metrics["rescues"],
+ ]
+ for name, metrics in entries
+ ],
+ )
+ if not quality_only:
+ print("\nCosts per 1,000 requests; savings versus always strong")
+ _display_table(
+ ["Policy", "Solver $", "Gross save %", "Router $", "Net total $", "Net save %"],
+ [
+ [
+ name,
+ number(metrics, "total_solver_cost_usd_per1000"),
+ number(metrics, "gross_solver_savings_pct", missing="n/a", digits=2),
+ number(metrics, "router_cost_usd_per1000"),
+ number(metrics, "net_total_cost_usd_per1000"),
+ number(
+ metrics,
+ "net_savings_pct",
+ missing="n/a" if metrics["router_overhead_known"] else "unpriced",
+ digits=2,
+ ),
+ ]
+ for name, metrics in entries
+ ],
+ )
+ if not quality_only:
+ print("\nGross savings exclude router costs; net totals and savings include them.")
+ print("unpriced = unknown router overhead; n/a = undefined percentage at a zero baseline cost.")
+ print("Calibration loss targets are not test guarantees. The 0, 2 and 5pp targets remain in JSON.")
+ print("Paired bootstrap intervals and full provenance are retained in JSON.")
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ commands = parser.add_subparsers(dest="command", required=True)
+ freeze = commands.add_parser("freeze", help="Select cutoffs from complete calibration answers only")
+ for name in ("plan", "calibration-grades", "baseline", "study", "policies", "output"):
+ freeze.add_argument(f"--{name}", type=Path, required=True)
+ freeze.add_argument("--router-credits", type=Path)
+ freeze.add_argument("--credit-usd", type=float, default=1e-5)
+ freeze.add_argument(
+ "--quality-only",
+ action="store_true",
+ help="Keep original cutoffs and report quality without costs when self-hosted prices are unknown",
+ )
+ evaluate = commands.add_parser("report", help="Report all frozen policies on complete test answers")
+ for name in ("frozen", "test-grades", "output"):
+ evaluate.add_argument(f"--{name}", type=Path, required=True)
+ evaluate.add_argument("--router-credits", type=Path)
+ args = parser.parse_args()
+ if args.output.exists():
+ raise FileExistsError(args.output)
+ if args.command == "freeze":
+ scores = load_stage_scores(args.baseline, args.study, args.policies)
+ result = freeze_calibrations(
+ args.plan,
+ args.calibration_grades,
+ scores,
+ args.router_credits,
+ args.credit_usd,
+ quality_only=args.quality_only,
+ )
+ else:
+ result = report(args.frozen, args.test_grades, args.router_credits)
+ write(args.output, result)
+ if args.command == "report":
+ display(result)
+ print(json.dumps({"output": str(args.output), "sha256": file_hash(args.output)}))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/solver_report.py b/examples/train-a-router/solver_report.py
new file mode 100644
index 000000000..a646aa4fb
--- /dev/null
+++ b/examples/train-a-router/solver_report.py
@@ -0,0 +1,326 @@
+"""Paired answer quality and measured solver costs for a fixed routing policy.
+
+These functions do no inference. Each row must contain both solver outcomes for
+the same problem, collected with frozen solver settings. A positive quality gap
+means lost accuracy relative to always using the strong solver; rescues can make
+the gap negative. Calibration constraints are empirical, not risk guarantees.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Mapping, Sequence
+from numbers import Real
+
+import numpy as np
+
+BOOTSTRAP_SAMPLES = 2000
+BOOTSTRAP_SEED = 89
+BASELINES = ("always_strong", "always_cheap")
+
+
+def _number(value, name):
+ if isinstance(value, (bool, np.bool_)) or not isinstance(value, Real):
+ raise TypeError(f"{name} must be a finite number")
+ result = float(value)
+ if not np.isfinite(result) or result < 0:
+ raise ValueError(f"{name} must be finite and nonnegative")
+ return result
+
+
+def _aligned(values, ids, name):
+ if isinstance(values, Mapping):
+ if set(values) != set(ids):
+ raise ValueError(f"{name} IDs do not match the rows")
+ return [values[key] for key in ids]
+ if isinstance(values, (str, bytes)) or not isinstance(values, (Sequence, np.ndarray)):
+ raise TypeError(f"{name} must be an aligned sequence or an ID mapping")
+ if isinstance(values, np.ndarray) and values.ndim != 1:
+ raise ValueError(f"{name} must be one-dimensional")
+ if len(values) != len(ids):
+ raise ValueError(f"{name} length does not match the rows")
+ return list(values)
+
+
+def _inputs(rows, router_credits, credit_usd):
+ if not isinstance(rows, Sequence) or isinstance(rows, (str, bytes)) or not rows:
+ raise ValueError("rows must be a nonempty sequence")
+ required = {"id", "cheap_correct", "strong_correct", "cheap_credits", "strong_credits"}
+ ids, cheap, strong, cheap_cost, strong_cost = [], [], [], [], []
+ for row in rows:
+ if not isinstance(row, Mapping) or not required <= row.keys():
+ raise ValueError("Each row requires an ID, both correctness labels, and both measured credit costs")
+ if not isinstance(row["id"], str) or not row["id"].strip():
+ raise ValueError("Each row requires a nonempty string ID")
+ ids.append(row["id"])
+ for key in ("cheap_correct", "strong_correct"):
+ if not isinstance(row[key], (bool, np.bool_)):
+ raise TypeError(f"{key} must be boolean")
+ cheap.append(row["cheap_correct"])
+ strong.append(row["strong_correct"])
+ cheap_cost.append(_number(row["cheap_credits"], "cheap_credits"))
+ strong_cost.append(_number(row["strong_credits"], "strong_credits"))
+ if len(set(ids)) != len(ids):
+ raise ValueError("Row IDs must be unique")
+ rate = _number(credit_usd, "credit_usd")
+ overhead = None
+ if router_credits is not None:
+ if isinstance(router_credits, Real):
+ overhead = np.full(len(ids), _number(router_credits, "router_credits"))
+ else:
+ overhead = np.array(
+ [_number(value, "router_credits") for value in _aligned(router_credits, ids, "router_credits")]
+ )
+ return (
+ ids,
+ np.array(cheap, dtype=bool),
+ np.array(strong, dtype=bool),
+ np.array(cheap_cost),
+ np.array(strong_cost),
+ overhead,
+ rate,
+ )
+
+
+def _routes(routes, ids):
+ if isinstance(routes, str):
+ if routes not in BASELINES:
+ raise ValueError("Unknown baseline; use always_strong or always_cheap")
+ return np.full(len(ids), routes == "always_strong"), routes
+ values = _aligned(routes, ids, "routes")
+ if any(not isinstance(value, (bool, np.bool_)) for value in values):
+ raise ValueError("routes must be boolean: true selects the strong solver")
+ return np.array(values, dtype=bool), "router"
+
+
+def _means(correct, strong, cost, strong_cost, cheap_share, overhead, rate):
+ """Accept scalars or vectors of bootstrap means, keeping pairs together."""
+ unit = 1000 * rate
+ gross = (strong_cost - cost) * unit
+ with np.errstate(divide="ignore", invalid="ignore"):
+ gross_pct = np.where(strong_cost > 0, 100 * (strong_cost - cost) / strong_cost, np.nan)
+ result = {
+ "answer_accuracy": correct,
+ "always_strong_answer_accuracy": strong,
+ "gap_vs_strong_pp": 100 * (strong - correct),
+ "total_solver_cost_usd_per1000": cost * unit,
+ "always_strong_solver_cost_usd_per1000": strong_cost * unit,
+ "gross_solver_savings_usd_per1000": gross,
+ "gross_solver_savings_pct": gross_pct,
+ "cheap_share": cheap_share,
+ "break_even_router_usd_per1000": gross,
+ }
+ if overhead is not None:
+ with np.errstate(divide="ignore", invalid="ignore"):
+ net_pct = np.where(strong_cost > 0, 100 * (strong_cost - cost - overhead) / strong_cost, np.nan)
+ result.update(
+ router_cost_usd_per1000=overhead * unit,
+ net_total_cost_usd_per1000=(cost + overhead) * unit,
+ net_savings_usd_per1000=(strong_cost - cost - overhead) * unit,
+ net_savings_pct=net_pct,
+ )
+ return result
+
+
+def _summary(inputs, decisions, mode):
+ ids, cheap, strong, cheap_cost, strong_cost, overhead, rate = inputs
+ if overhead is not None and mode in BASELINES:
+ overhead = np.zeros(len(ids))
+ correct = np.where(decisions, strong, cheap)
+ losses = strong.astype(int) - correct.astype(int)
+ cost = np.where(decisions, strong_cost, cheap_cost)
+ means = _means(
+ correct.mean(),
+ strong.mean(),
+ cost.mean(),
+ strong_cost.mean(),
+ (~decisions).mean(),
+ None if overhead is None else overhead.mean(),
+ rate,
+ )
+ means["gap_vs_strong_pp"] = 100 * int(losses.sum()) / len(ids)
+ result = {key: float(value) if np.isfinite(value) else None for key, value in means.items()}
+ result.update(
+ n=len(ids),
+ routing_mode=mode,
+ regressions=int((strong & ~correct).sum()),
+ rescues=int((~strong & correct).sum()),
+ router_overhead_known=overhead is not None,
+ router_overhead_applied=overhead is not None and mode not in BASELINES,
+ )
+ rng = np.random.default_rng(BOOTSTRAP_SEED)
+ samples = {key: [] for key in means}
+ for start in range(0, BOOTSTRAP_SAMPLES, 128):
+ indices = rng.integers(0, len(ids), (min(128, BOOTSTRAP_SAMPLES - start), len(ids)))
+ values = _means(
+ correct[indices].mean(axis=1),
+ strong[indices].mean(axis=1),
+ cost[indices].mean(axis=1),
+ strong_cost[indices].mean(axis=1),
+ (~decisions)[indices].mean(axis=1),
+ None if overhead is None else overhead[indices].mean(axis=1),
+ rate,
+ )
+ values["gap_vs_strong_pp"] = 100 * losses[indices].mean(axis=1)
+ for key, value in values.items():
+ samples[key].append(value)
+ result["paired_bootstrap_95ci"] = {
+ key: np.quantile(values, [0.025, 0.975]).tolist() if np.isfinite(values).all() else None
+ for key, chunks in samples.items()
+ for values in [np.concatenate(chunks)]
+ }
+ result["bootstrap"] = {
+ "samples": BOOTSTRAP_SAMPLES,
+ "seed": BOOTSTRAP_SEED,
+ "method": "paired problem-level percentile bootstrap, conditional on the fixed policy and recorded outcomes",
+ "limitations": (
+ "Descriptive intervals assume independent problems; they do not account for shared competition papers, "
+ "policy selection, solver randomness, or future price changes. Zero-event intervals are not risk bounds. "
+ "A percentage interval is null if any resampled baseline has zero cost."
+ ),
+ }
+ return result
+
+
+def summarize(rows, routes, router_credits=None, credit_usd=1e-5):
+ """Return paired quality/cost metrics and fixed-seed descriptive 95% intervals.
+
+ ``routes`` is an aligned boolean sequence (true means strong), an exact-ID
+ mapping, or the explicit baseline ``always_strong`` / ``always_cheap``.
+ Only explicit baselines bypass routing overhead; uniform learned decisions
+ still incur it. Credit costs include all solver/finalizer calls. Router
+ credits may be a nonnegative scalar or one value per row. Unknown overhead
+ omits all net-cost fields. Accuracy and cheap share are fractions; the gap
+ is percentage points; savings percentages use the always-strong cost.
+ """
+ inputs = _inputs(rows, router_credits, credit_usd)
+ decisions, mode = _routes(routes, inputs[0])
+ return _summary(inputs, decisions, mode)
+
+
+def summarize_quality(rows, routes):
+ """Report paired answer quality when solver prices are unknown.
+
+ Correctness and IDs are required; credit fields are neither required nor
+ used. This explicit mode reports no costs, savings, or break-even price.
+ ``summarize`` and ``select_cutoff`` retain strict measured-cost validation.
+ """
+ if not isinstance(rows, Sequence) or isinstance(rows, (str, bytes)) or not rows:
+ raise ValueError("rows must be a nonempty sequence")
+ ids, cheap, strong = [], [], []
+ for row in rows:
+ if not isinstance(row, Mapping) or not {"id", "cheap_correct", "strong_correct"} <= row.keys():
+ raise ValueError("Each row requires an ID and both correctness labels")
+ if not isinstance(row["id"], str) or not row["id"].strip():
+ raise ValueError("Each row requires a nonempty string ID")
+ for key in ("cheap_correct", "strong_correct"):
+ if not isinstance(row[key], (bool, np.bool_)):
+ raise TypeError(f"{key} must be boolean")
+ ids.append(row["id"])
+ cheap.append(row["cheap_correct"])
+ strong.append(row["strong_correct"])
+ if len(set(ids)) != len(ids):
+ raise ValueError("Row IDs must be unique")
+ decisions, mode = _routes(routes, ids)
+ strong = np.asarray(strong, dtype=bool)
+ correct = np.where(decisions, strong, np.asarray(cheap, dtype=bool))
+ losses = strong.astype(int) - correct.astype(int)
+ observations = {
+ "answer_accuracy": correct,
+ "always_strong_answer_accuracy": strong,
+ "gap_vs_strong_pp": 100 * losses,
+ "cheap_share": ~decisions,
+ }
+ result = {key: float(values.mean()) for key, values in observations.items()}
+ result.update(
+ n=len(ids),
+ routing_mode=mode,
+ regressions=int((strong & ~correct).sum()),
+ rescues=int((~strong & correct).sum()),
+ cost_status="unknown",
+ cost_note="Quality-only analysis: solver and infrastructure costs are unpriced; no savings estimate is made",
+ )
+ rng = np.random.default_rng(BOOTSTRAP_SEED)
+ samples = {key: [] for key in observations}
+ for start in range(0, BOOTSTRAP_SAMPLES, 128):
+ indices = rng.integers(0, len(ids), (min(128, BOOTSTRAP_SAMPLES - start), len(ids)))
+ for key, values in observations.items():
+ samples[key].append(values[indices].mean(axis=1))
+ result["paired_bootstrap_95ci"] = {
+ key: np.quantile(np.concatenate(chunks), [0.025, 0.975]).tolist() for key, chunks in samples.items()
+ }
+ result["bootstrap"] = {
+ "samples": BOOTSTRAP_SAMPLES,
+ "seed": BOOTSTRAP_SEED,
+ "method": "paired problem-level percentile bootstrap, conditional on the fixed policy and recorded outcomes",
+ "limitations": (
+ "Descriptive intervals assume independent problems; they do not account for shared competition papers, "
+ "policy selection or solver randomness. Zero-event intervals are not risk bounds."
+ ),
+ }
+ return result
+
+
+def select_cutoff(calibration_rows, scores, max_quality_loss_pp, router_credits=None, credit_usd=1e-5):
+ """Select the cheapest empirically eligible policy using calibration only.
+
+ A threshold chooses strong when ``score >= cutoff``; tied scores stay
+ together. Always-strong and always-cheap are explicit router-free options.
+ A policy must strictly beat always-strong's cost to replace that fallback.
+ Remaining ties prefer less quality loss, then bypassing the router, then a
+ lower cutoff. This target is the observed *net* accuracy gap: rescues offset
+ regressions. It is neither a bound on harmful switches nor a confidence
+ bound on future quality. Freeze this selection before reading test answers.
+ """
+ target = _number(max_quality_loss_pp, "max_quality_loss_pp")
+ if target > 100:
+ raise ValueError("max_quality_loss_pp cannot exceed 100")
+ inputs = _inputs(calibration_rows, router_credits, credit_usd)
+ ids, cheap, strong, cheap_cost, strong_cost, overhead, _ = inputs
+ values = _aligned(scores, ids, "scores")
+ if any(isinstance(value, (bool, np.bool_)) or not isinstance(value, Real) for value in values):
+ raise ValueError("scores must be finite numbers")
+ values = np.array(values, dtype=float)
+ if not np.isfinite(values).all():
+ raise ValueError("scores must be finite numbers")
+ baseline_cost = float(strong_cost.mean())
+ best = {
+ "mode": "always_strong",
+ "cutoff": None,
+ "cost": baseline_cost,
+ "gap": 0.0,
+ "decisions": np.ones(len(ids), dtype=bool),
+ }
+ candidates = [("always_cheap", None, np.zeros(len(ids), dtype=bool))]
+ candidates.extend(("threshold", float(value), values >= value) for value in np.unique(values))
+ eligible = 1 # The always-strong bypass has zero loss and zero router overhead.
+ best_key = None
+ for mode, cutoff, decisions in candidates:
+ correct = np.where(decisions, strong, cheap)
+ gap = 100 * int((strong.astype(int) - correct.astype(int)).sum()) / len(ids)
+ if gap > target:
+ continue
+ eligible += 1
+ cost = float(np.where(decisions, strong_cost, cheap_cost).mean())
+ if mode == "threshold" and overhead is not None:
+ cost += float(overhead.mean())
+ if cost >= baseline_cost:
+ continue
+ key = (cost, gap, mode == "threshold", float("-inf") if cutoff is None else cutoff)
+ if best_key is None or key < best_key:
+ best_key = key
+ best = {"mode": mode, "cutoff": cutoff, "cost": cost, "gap": gap, "decisions": decisions}
+ mode = best["mode"]
+ return {
+ "mode": mode,
+ "cutoff": best["cutoff"],
+ "score_rule": "score >= cutoff selects strong" if mode == "threshold" else None,
+ "max_quality_loss_pp": target,
+ "selection_cost_basis": "solver plus router credits" if overhead is not None else "solver credits only",
+ "selection_rule": "minimum empirical cost below always-strong; then loss, router bypass, lower cutoff",
+ "candidates": len(candidates) + 1,
+ "empirically_eligible_candidates": eligible,
+ "calibration_ids": ids,
+ "calibration_routes": best["decisions"].tolist(),
+ "calibration": _summary(inputs, best["decisions"], "router" if mode == "threshold" else mode),
+ "interpretation": "Calibration-only empirical constraint, not a selection-adjusted or prospective quality guarantee",
+ }
diff --git a/examples/train-a-router/source_review.py b/examples/train-a-router/source_review.py
new file mode 100644
index 000000000..a755bad4b
--- /dev/null
+++ b/examples/train-a-router/source_review.py
@@ -0,0 +1,356 @@
+"""Apply a source-reference audit without changing saved model responses.
+
+The review is data, not a list of model-specific exceptions. It binds the
+original question, reference, independent proof and any exclusion. Training
+answers were already available when this amendment was made; solver test
+answers remain closed until all models and cutoffs have been saved.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import shutil
+from pathlib import Path
+
+import benefit
+import benefit_data
+import benefit_expanded_gold
+import benefit_expanded_gold_v2
+import benefit_training_review
+import solver
+import solver_gold
+import solver_policies
+import teacher_format
+from data import SOURCE_HASHES, SOURCE_REVISION, digest, file_hash, write
+
+FORMAT = "sie-router-source-review-v3"
+PROTOCOL = {
+ "references": "Pinned source answers, prior source corrections, and independently proved amendments; teacher disagreements nominate reviews but are not accepted as proof",
+ "eligibility": "Exclude missing inputs, contradictory or ambiguous prompts, and partial-credit estimation or proof-only tasks outside the final-answer comparison contract; apply source-only rules to every split",
+ "timing": "After training outcome collection and exploratory v2b fits; before corrected fits and held-out solver outcome access",
+ "formatting": "Normalize both finals by question-only rules; preserve original text and transformations",
+ "old_results": "Original acquisition plans, responses, graders and exploratory results remain unchanged",
+}
+METADATA_KEYS = ("source_review_sha256", "source_review_grader_sha256", "format_normalizer_sha256")
+SPECIAL_SEMANTICS = {
+ "clock_time",
+ "progression",
+ "rounded_percentage",
+ "prime_approximation",
+ "division_operator",
+ "ordered_labels",
+ "weekday",
+ "official_alternatives",
+ "exact_rational",
+}
+
+
+def read(path):
+ return json.loads(Path(path).read_text())
+
+
+def canonical_reference(identity, original):
+ """Preserve previously reviewed source semantics before applying new data."""
+ if identity in benefit_training_review.REVIEW:
+ rule = benefit_training_review.REVIEW[identity]
+ if original != rule["original"]:
+ raise ValueError("Symbolic source reference changed")
+ return rule["canonical"], "reviewed_symbolic_training"
+ for rules in (
+ benefit_expanded_gold_v2.RULES,
+ benefit_expanded_gold.RULES,
+ benefit_data.RULES,
+ solver_gold.RULES,
+ ):
+ if identity in rules:
+ expected, canonical, kind = rules[identity]
+ if original != expected:
+ raise ValueError("Prior source reference changed")
+ return canonical, kind
+ return original, "scalar"
+
+
+def checked_review(review):
+ if not (
+ review.get("format") == FORMAT
+ and review.get("finalized") is True
+ and review.get("protocol") == PROTOCOL
+ and review.get("source_revision") == SOURCE_REVISION
+ and review.get("source_hashes") == SOURCE_HASHES
+ and review.get("solver_test_outcomes_accessed") is False
+ and review.get("training_outcomes_already_available") is True
+ ):
+ raise ValueError("Source review must declare the pinned data and truthful outcome-access boundary")
+ records = review.get("records")
+ if not isinstance(records, list) or not records or any(not isinstance(row, dict) for row in records):
+ raise ValueError("Source review records must be a nonempty list")
+ for row in records:
+ if not (
+ isinstance(row.get("id"), str)
+ and row["id"]
+ and Path(row["id"]).name == row["id"]
+ and row["id"] not in (".", "..")
+ and "\\" not in row["id"]
+ and row.get("split") in ("train", "calibration", "test")
+ and isinstance(row.get("question"), str)
+ and row["question"].strip()
+ and row.get("question_sha256") == digest(row["question"])
+ and isinstance(row.get("request_sha256"), str)
+ and re.fullmatch(r"[0-9a-f]{64}", row["request_sha256"])
+ and row.get("decision") in ("retain", "exclude")
+ and isinstance(row.get("reason"), str)
+ and row["reason"].strip()
+ and isinstance(row.get("original_reference"), str)
+ and isinstance(row.get("reference"), str)
+ and row["reference"]
+ and isinstance(row.get("semantic_kind"), str)
+ and row["semantic_kind"]
+ ):
+ raise ValueError("Invalid source-only review row")
+ if row["decision"] == "retain" and row["semantic_kind"] in SPECIAL_SEMANTICS | {"reviewed_symbolic_training"}:
+ canonical, kind = canonical_reference(row["id"], row["original_reference"])
+ if row["reference"] != canonical or row["semantic_kind"] != kind:
+ raise ValueError("A changed reference must use source_correction, not prior special semantics")
+ if len({row["id"] for row in records}) != len(records):
+ raise ValueError("Source review IDs must be nonempty and unique")
+ return review
+
+
+def metadata(path):
+ checked_review(read(path))
+ return {
+ "source_review_sha256": file_hash(path),
+ "source_review_grader_sha256": file_hash(Path(__file__)),
+ "format_normalizer_sha256": file_hash(Path(teacher_format.__file__)),
+ }
+
+
+def eligible_rows(rows, split, review):
+ """Filter a collected population by source content, preserving its order."""
+ mapping = {row["id"]: row for row in checked_review(review)["records"]}
+ if split not in ("train", "calibration", "test") or len({row["id"] for row in rows}) != len(rows):
+ raise ValueError("Invalid split or duplicate collected question")
+ selected = []
+ for row in rows:
+ if row["id"] not in mapping:
+ raise ValueError("Collected question is missing from the source review")
+ item = mapping[row["id"]]
+ request_sha = row.get("request_sha256")
+ if "request" in row:
+ actual = digest(row["request"])
+ if request_sha is not None and request_sha != actual:
+ raise ValueError("Collected request text differs from its digest")
+ request_sha = actual
+ if row["request"].partition("\nProblem: ")[2].strip() != item["question"].strip():
+ raise ValueError("Review question differs from the collected request text")
+ if item["split"] != split or item["request_sha256"] != request_sha:
+ raise ValueError("Review belongs to a different split or question")
+ if item["decision"] == "retain":
+ selected.append(row)
+ if not selected:
+ raise ValueError("Source review removed an entire split")
+ return selected
+
+
+def grade_reference_row(identity, question, original_reference, response, review):
+ """Compare a solver final with the reviewed source, retaining its provenance."""
+ mapping = {row["id"]: row for row in checked_review(review)["records"]}
+ item = mapping[identity]
+ if item["decision"] != "retain" or item["question"] != question or item["original_reference"] != original_reference:
+ raise ValueError("Cannot grade an excluded or mismatched source question")
+ return _grade_reviewed_row(item, response)
+
+
+def _grade_reviewed_row(item, response):
+ """Use a row from an already checked review; avoid rechecking all rows per answer."""
+ identity, question, original_reference = item["id"], item["question"], item["original_reference"]
+ kind = item["semantic_kind"]
+ if kind == "reviewed_symbolic_training":
+ # These two training-only huge-power answers were proved symbolically
+ # before this amendment; do not invoke an unbounded integer expansion.
+ result = benefit_training_review.reviewed_grade(identity, original_reference, response)
+ elif kind in SPECIAL_SEMANTICS:
+ result = benefit_expanded_gold_v2.grade_expanded(identity, original_reference, response)
+ else:
+ result = teacher_format.grade_reference(question, item["reference"], response)
+ return result | {
+ "source_reference": {
+ "original": original_reference,
+ "reviewed": item["reference"],
+ "question_sha256": item["question_sha256"],
+ "semantic_kind": kind,
+ "review_reason": item["reason"],
+ },
+ }
+
+
+def grade_collection(directory, split, review_path, *, test_access_path=None):
+ """Grade eligible saved answers; test requires a prior all-model access receipt."""
+ directory, review_path = Path(directory), Path(review_path)
+ if split not in ("train", "calibration", "test"):
+ raise ValueError("Unknown collection split")
+ if split == "test":
+ if test_access_path is None or not Path(test_access_path).is_file():
+ raise ValueError("Save the all-model test-access receipt before opening test answers")
+ try:
+ access = read(test_access_path)
+ except (ValueError, OSError) as error:
+ raise ValueError("Invalid all-model test-access receipt") from error
+ if not (
+ isinstance(access, dict)
+ and access.get("format") == "sie-reviewed-outcome-all-model-access-v1"
+ and access.get("split") == "test"
+ and access.get("source_review_sha256") == file_hash(review_path)
+ and access.get("original_solver_plan_sha256") == file_hash(directory / "plan.json")
+ and isinstance(access.get("calibration_sha256"), str)
+ and re.fullmatch(r"[0-9a-f]{64}", access["calibration_sha256"])
+ and isinstance(access.get("studies"), list)
+ and access["studies"]
+ and all(isinstance(study, dict) and study for study in access["studies"])
+ ):
+ raise ValueError("All-model test-access receipt does not bind this review, plan and calibration")
+ review = checked_review(read(review_path))
+ review_metadata = metadata(review_path)
+ plan, inputs, settings = (read(directory / name) for name in ("plan.json", "inputs.json", "settings.json"))
+ if plan["inputs_sha256"] != digest(inputs) or plan["settings_sha256"] != digest(settings):
+ raise ValueError("Saved solver inputs or settings changed")
+ collection = directory / f"answers-{split}"
+ summary, binding = read(collection / "collection.json"), read(collection / "collection-plan.json")
+ frozen_hash = file_hash(directory / "policies-frozen.json") if split == "test" else None
+ solver.collection_binding(binding, directory, split, inputs[split], settings, frozen_hash, binding["self_hosted"])
+ if (
+ summary["status"] != "complete"
+ or summary["completed"] != 2 * len(inputs[split])
+ or summary["failed"]
+ or summary.get("missing", 0) != 0
+ or summary.get("unknown_execution_attempts", 0) != 0
+ ):
+ raise ValueError("Need every paired response in the original collected population")
+ gold = read(directory / "gold-for-scoring-only.json")
+ if digest(gold) != plan["gold_sha256"]:
+ raise ValueError("Acquisition reference bytes changed")
+ selected = eligible_rows(inputs[split], split, review)
+ reference = {row["id"]: row for row in review["records"]}
+ rows = []
+ for row in selected:
+ identity = row["id"]
+ item = reference[identity]
+ if gold[identity] != item["original_reference"]:
+ raise ValueError("Acquisition and audited source references differ")
+ outcome = {"id": identity}
+ for arm in ("cheap", "strong"):
+ path = collection / f"{identity}-{arm}.json"
+ record = read(path)
+ solver.validate_record(record, row, arm, settings)
+ if record["status"] != "complete":
+ raise ValueError("Incomplete saved answer")
+ measured = _grade_reviewed_row(item, record["answer"])
+ if measured["status"] in ("gold_unparseable", "grading_error", "unit_contract_unresolved"):
+ raise ValueError(f"Unresolved comparison for {identity}/{arm}: {measured['status']}")
+ outcome.update(
+ {
+ f"{arm}_correct": measured["correct"],
+ f"{arm}_credits": record["credits"],
+ f"{arm}_grade": measured,
+ f"{arm}_record_sha256": file_hash(path),
+ }
+ )
+ rows.append(outcome)
+ parent_keys = (
+ "grader_source_sha256",
+ "gold_review_source_sha256",
+ "gold_review_sha256",
+ "training_gold_rules_sha256",
+ "training_gold_review_source_sha256",
+ "outcome_grader_source_sha256",
+ "outcome_grader_protocol_sha256",
+ )
+ parent = {key: plan[key] for key in parent_keys if key in plan}
+ parent.setdefault("gold_review_sha256", file_hash(directory / "gold-review.json"))
+ parent |= benefit_expanded_gold_v2.outcome_grader_metadata()
+ if metadata(review_path) != review_metadata:
+ raise ValueError("Source review or grading code changed during grading")
+ return {
+ "split": split,
+ "plan_sha256": file_hash(directory / "plan.json"),
+ "collection_plan_sha256": file_hash(collection / "collection-plan.json"),
+ "collection_summary_sha256": file_hash(collection / "collection.json"),
+ "source_collection_policies_frozen_sha256": frozen_hash,
+ "policies_frozen_sha256": None,
+ "collected_ids": [row["id"] for row in inputs[split]],
+ "excluded_ids": [row["id"] for row in inputs[split] if reference[row["id"]]["decision"] == "exclude"],
+ "test_access_sha256": None if test_access_path is None else file_hash(test_access_path),
+ **parent,
+ **review_metadata,
+ "rows": rows,
+ }
+
+
+def calibration_snapshot(directory, output, review_path):
+ """Keep historical input provenance, replace only the eligible scored rows."""
+ directory, output = Path(directory), Path(output)
+ if output.exists():
+ raise FileExistsError(output)
+ old_path = directory / "policies-frozen.json"
+ old = read(old_path)
+ solver_policies.check_seal(old, solver_policies.FROZEN_FORMAT)
+ graded = grade_collection(directory, "calibration", review_path)
+ original_ids = [row["id"] for row in old["calibration_rows"]]
+ if original_ids != graded["collected_ids"] or len(original_ids) != len(set(original_ids)):
+ raise ValueError("Historical calibration cost order differs from the collected questions")
+ original_credits = dict(zip(original_ids, old["embedding_calibration_credits"], strict=True))
+ credits = [original_credits[row["id"]] for row in graded["rows"]]
+ output.mkdir(parents=True)
+ shutil.copyfile(directory / "plan.json", output / "plan.json")
+ write(output / "grades-calibration.json", graded)
+ keys = ("id", "cheap_correct", "strong_correct", "cheap_credits", "strong_credits")
+ amended = {key: value for key, value in old.items() if key != "sha256"}
+ amended.update(
+ {
+ "calibration_rows": [{key: row[key] for key in keys} for row in graded["rows"]],
+ "calibration_grades_sha256": file_hash(output / "grades-calibration.json"),
+ "embedding_calibration_credits": credits,
+ "selections": {},
+ **benefit_expanded_gold_v2.outcome_grader_metadata(),
+ **metadata(review_path),
+ "amendment": {
+ "created_at": benefit.now(),
+ "original_frozen_sha256": file_hash(old_path),
+ "boundary": "after_collection_before_corrected_fits_and_test_outcome_access",
+ "attest_test_outcomes_unopened": True,
+ "source_scores": "Historical full-population input provenance only; no historical cutoff is reused",
+ "excluded_calibration_ids": graded["excluded_ids"],
+ "source_review_sha256": file_hash(review_path),
+ },
+ "test_access": "Calibration-only source amendment; does not authorize test outcome access",
+ }
+ )
+ write(output / "policies-frozen.json", solver_policies.seal(amended))
+ return {
+ "n": len(graded["rows"]),
+ "excluded": graded["excluded_ids"],
+ "frozen_sha256": file_hash(output / "policies-frozen.json"),
+ }
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("command", choices=("grade-train", "calibration"))
+ parser.add_argument("--collection", type=Path, required=True)
+ parser.add_argument("--review", type=Path, required=True)
+ parser.add_argument("--output", type=Path, required=True)
+ args = parser.parse_args()
+ if args.output.exists():
+ raise FileExistsError(args.output)
+ if args.command == "calibration":
+ print(json.dumps(calibration_snapshot(args.collection, args.output, args.review), indent=2))
+ else:
+ result = grade_collection(args.collection, "train", args.review)
+ write(args.output, result)
+ print(json.dumps({"n": len(result["rows"]), "excluded": result["excluded_ids"]}, indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/teacher_collect.py b/examples/train-a-router/teacher_collect.py
new file mode 100644
index 000000000..b41a8d1ee
--- /dev/null
+++ b/examples/train-a-router/teacher_collect.py
@@ -0,0 +1,648 @@
+"""Stage and collect a bounded question-only extension of a saved teacher pilot.
+
+``prepare`` makes no API calls and never opens benchmark gold or solver answers.
+``collect`` is a single attempt: it refuses an existing start receipt, preserves
+every reservation/response, stops new dispatch on any error, and never retries.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import os
+import re
+import shutil
+import time
+from decimal import Decimal
+from pathlib import Path
+
+import httpx
+
+from data import digest, file_hash, write
+
+MODEL = "gpt-6-astra"
+PRICES = {"input": 10, "cached_input": 1, "input_cache_write": 12.5, "output": 50}
+SYSTEM = (
+ "Solve the mathematical problem independently. Return its final answer as a concise mathematical expression "
+ "or value, with any units or requested rounding, in the answer field. Use LaTeX if necessary; "
+ "do not include dollar delimiters or a boxed wrapper. Do not use external tools."
+)
+SETTINGS = {
+ "model": MODEL,
+ "instructions": SYSTEM,
+ "input": "",
+ "reasoning": {"effort": "high"},
+ "max_output_tokens": 8192,
+ "store": False,
+ "service_tier": "default",
+ "text": {
+ "format": {
+ "type": "json_schema",
+ "name": "math_answer",
+ "strict": True,
+ "schema": {
+ "type": "object",
+ "properties": {"answer": {"type": "string"}},
+ "required": ["answer"],
+ "additionalProperties": False,
+ },
+ }
+ },
+}
+MAX_USD = Decimal(25)
+MAX_SECONDS = 3600
+MAX_WORKERS = 4
+REQUEST_SECONDS = 300
+ENDPOINT = "https://api.openai.com/v1/responses"
+
+
+def read(path):
+ return json.loads(Path(path).read_text())
+
+
+def body_for(row):
+ return SETTINGS | {"input": row["request"]}
+
+
+def input_allowance(row):
+ # Text-only UTF-8 bytes bound tokenization; the extra allowance covers the
+ # request/schema framing. Requests stay below the long-context price tier.
+ allowance = len(json.dumps(body_for(row)).encode()) + 2048
+ if allowance >= 272_000:
+ raise ValueError("Teacher input exceeds the declared short-context budget")
+ return allowance
+
+
+def reservation(row):
+ return (Decimal(input_allowance(row)) * Decimal("12.5") + Decimal(8192 * 50)) / 1_000_000
+
+
+def usage_cost(usage, allowance):
+ """Return invoice estimate and conservative charge only for verified usage."""
+ if not isinstance(usage, dict):
+ raise TypeError("Missing usage")
+ details = usage.get("input_tokens_details") or {}
+ values = [
+ usage.get("input_tokens"),
+ usage.get("output_tokens"),
+ details.get("cached_tokens", 0),
+ details.get("cache_write_tokens", 0),
+ ]
+ if any(type(value) is not int or value < 0 for value in values):
+ raise ValueError("Invalid token usage")
+ incoming, outgoing, cached, written = values
+ if cached + written > incoming or incoming > allowance or outgoing > 8192:
+ raise ValueError("Usage exceeds the reserved request bounds")
+ invoice = (
+ Decimal(incoming - cached - written) * 10
+ + Decimal(cached)
+ + Decimal(written) * Decimal("12.5")
+ + Decimal(outgoing) * 50
+ ) / 1_000_000
+ conservative = (Decimal(incoming - cached) * Decimal("12.5") + Decimal(cached) + Decimal(outgoing) * 50) / 1_000_000
+ return invoice, conservative
+
+
+def _check_teacher_record(record, row):
+ if (
+ record.get("id") != row["id"]
+ or record.get("request_sha256") != digest(body_for(row))
+ or record.get("status") != "complete"
+ or record.get("response_status") != "completed"
+ or record.get("returned_model") != MODEL
+ or not isinstance(record.get("answer"), str)
+ or json.loads(record["output_text"]) != {"answer": record["answer"]}
+ ):
+ raise ValueError("Reused teacher record differs from the exact question-only request")
+ return usage_cost(record.get("usage"), input_allowance(row))
+
+
+def superseded_transport(path):
+ """Bind an explicitly authorized single repeat after a connection-only stop."""
+ path = Path(path)
+ plan, summary = read(path / "plan.json"), read(path / "collection.json")
+ if (
+ plan.get("superseded_attempt") is not None
+ or plan.get("settings") != SETTINGS
+ or summary.get("status") != "incomplete"
+ or summary.get("stop_reason") != "request_error_or_unknown_execution"
+ or summary.get("plan_sha256") != file_hash(path / "plan.json")
+ or summary.get("complete") != 64
+ or summary.get("dispatched_new") != 4
+ or file_hash(path / "source-executed/teacher_collect.py") != plan["operator_sha256"]
+ ):
+ raise ValueError("Supersession requires the preserved four-call connection-only stop")
+ exposure = Decimal(0)
+ repeated = plan["new_ids"][:4]
+ for identity in repeated:
+ record_path = path / "records" / f"{identity}.json"
+ intent_path = path / "intents" / f"{identity}.json"
+ record, intent = read(record_path), read(intent_path)
+ if (
+ file_hash(record_path) != summary["record_hashes"][identity]
+ or file_hash(intent_path) != summary["intent_hashes"][identity]
+ or record.get("status") != "incomplete"
+ or record.get("error") != {"kind": "ConnectError", "execution_unknown": True}
+ or record.get("answer") is not None
+ or record.get("usage_verified") is not False
+ or "raw_response_sha256" in record
+ or intent["plan_sha256"] != summary["plan_sha256"]
+ or intent["reservation_usd"] != plan["request_reservations_usd"][identity]
+ ):
+ raise ValueError("Only preserved unknown connection failures may be explicitly repeated once")
+ exposure += Decimal(intent["reservation_usd"])
+ return {
+ "files_sha256": {
+ str(item.relative_to(path)): file_hash(item) for item in sorted(path.rglob("*")) if item.is_file()
+ },
+ "plan_sha256": file_hash(path / "plan.json"),
+ "unknown_exposure_usd": str(exposure),
+ "explicitly_repeated_ids": repeated,
+ "reason": "The user authorized one network-enabled repeat after sandbox ConnectError; previous unknown reservations remain charged against the new total cap. No automatic retry.",
+ }
+
+
+def prepare(training, reuse, output, max_usd=Decimal(15), workers=MAX_WORKERS, max_seconds=1800, supersedes=None):
+ training, reuse, output = Path(training), Path(reuse), Path(output)
+ if output.exists():
+ raise FileExistsError(output)
+ max_usd = Decimal(str(max_usd))
+ if (
+ not 0 < max_usd <= MAX_USD
+ or not 1 <= workers <= MAX_WORKERS
+ or not REQUEST_SECONDS + 5 < max_seconds <= MAX_SECONDS
+ ):
+ raise ValueError("Budget, concurrency or duration exceeds the authorized limits")
+ source_plan, inputs, eligibility = (
+ read(training / name) for name in ("plan.json", "inputs.json", "eligibility.json")
+ )
+ rows = inputs.get("train", [])
+ identities = [row["id"] for row in rows]
+ if (
+ source_plan.get("population") != "original"
+ or source_plan.get("counts") != {"train": 510}
+ or source_plan.get("source_counts") != {"train": 512}
+ or set(inputs) != {"train"}
+ or len(rows) != 510
+ or len(set(identities)) != 510
+ or any(set(row) != {"id", "request"} for row in rows)
+ or any(not isinstance(row["request"], str) or not row["request"].strip() for row in rows)
+ or any(re.fullmatch(r"e2h-amc-(?:train|eval)-(?:0|[1-9]\d*)", row["id"]) is None for row in rows)
+ or source_plan["inputs_sha256"] != digest(inputs)
+ or source_plan["eligibility_sha256"] != file_hash(training / "eligibility.json")
+ or eligibility.get("source_only") is not True
+ or eligibility["eligible_train_ids"] != identities
+ or source_plan["eligible_train_ids"] != identities
+ ):
+ raise ValueError("Need the declared clean510 question-only training inputs")
+ old_plan, old_collection = read(reuse / "plan.json"), read(reuse / "collection.json")
+ reused_ids = [row["id"] for row in old_plan["inputs"]]
+ if (
+ old_plan.get("settings") != SETTINGS
+ or old_plan.get("prices_usd_per_million") != PRICES
+ or old_plan.get("teacher_receives_gold") is not False
+ or old_plan.get("teacher_receives_solver_answers") is not False
+ or old_plan.get("n") != 64
+ or len(set(reused_ids)) != 64
+ or len(reused_ids) != 64
+ or old_collection.get("status") != "complete"
+ or old_collection.get("complete") != 64
+ or old_collection.get("returned") != 64
+ or old_collection.get("n") != 64
+ or old_collection["plan_sha256"] != file_hash(reuse / "plan.json")
+ or set(old_collection["record_hashes"]) != set(reused_ids)
+ or not set(reused_ids) <= set(identities)
+ ):
+ raise ValueError("Reuse requires the complete64-answer pilot with identical settings")
+ by_id = {row["id"]: row for row in rows}
+ invoice, conservative = Decimal(0), Decimal(0)
+ for row in old_plan["inputs"]:
+ path = reuse / "records" / f"{row['id']}.json"
+ if row != by_id[row["id"]] or file_hash(path) != old_collection["record_hashes"][row["id"]]:
+ raise ValueError("Reused teacher question or record changed")
+ actual, upper = _check_teacher_record(read(path), row)
+ invoice += actual
+ conservative += upper
+ if conservative >= max_usd:
+ raise ValueError("Historical teacher cost already exhausts the total budget")
+ new_ids = sorted(
+ set(identities) - set(reused_ids), key=lambda identity: digest(["teacher-extension-order-v1", identity])
+ )
+ previous = superseded_transport(supersedes) if supersedes is not None else None
+ if previous and (
+ read(Path(supersedes) / "plan.json")["inputs"] != rows
+ or read(Path(supersedes) / "plan.json")["reuse"]["record_hashes"] != old_collection["record_hashes"]
+ or conservative + Decimal(previous["unknown_exposure_usd"]) >= max_usd
+ ):
+ raise ValueError("Superseded teacher population, reused answers or budget changed")
+ plan = {
+ "format": "sie-teacher-reference-extension-v1",
+ "created_unix": time.time(),
+ "model": MODEL,
+ "n": 510,
+ "inputs": rows,
+ "new_ids": new_ids,
+ "reused_ids": reused_ids,
+ "settings": SETTINGS,
+ "teacher_receives_gold": False,
+ "teacher_receives_solver_answers": False,
+ "purpose": "Teacher-label reliability over clean510 training rows and a later embedding-control training-equivalence audit; no new native candidate or model-settings search.",
+ "order": "All64 pilot answers reused unchanged; remaining446 use fixed identity-hash order, independent of answers/outcomes.",
+ "source": {
+ "plan_sha256": file_hash(training / "plan.json"),
+ "inputs_sha256": file_hash(training / "inputs.json"),
+ "eligibility_sha256": file_hash(training / "eligibility.json"),
+ },
+ "reuse": {
+ "plan_sha256": file_hash(reuse / "plan.json"),
+ "collection_sha256": file_hash(reuse / "collection.json"),
+ "record_hashes": old_collection["record_hashes"],
+ "list_price_estimate_usd": str(invoice),
+ "conservative_budget_usd": str(conservative),
+ },
+ "max_usd": str(max_usd),
+ "workers": workers,
+ "max_seconds": max_seconds,
+ "request_timeout_seconds": REQUEST_SECONDS,
+ "automatic_retries": 0,
+ "unknown_execution_repeat": False,
+ "superseded_attempt": previous,
+ "prices_usd_per_million": PRICES,
+ "price_source": "https://developers.openai.com/api/docs/pricing",
+ "price_checked_date": "2026-10-19",
+ "input_token_allowances": {identity: input_allowance(by_id[identity]) for identity in new_ids},
+ "request_reservations_usd": {identity: str(reservation(by_id[identity])) for identity in new_ids},
+ "budget_rule": "Historical conservative charges, explicitly superseded unknown reservations, settled conservative usage and every in-flight/unknown reservation may never exceed max_usd. Release excess only after verified usage. A budget stop can leave the population incomplete.",
+ "operator_sha256": file_hash(Path(__file__)),
+ "data_source_sha256": file_hash(Path(__file__).with_name("data.py")),
+ "limitations": "The model alias has no distinct dated snapshot. Cost is a token-price estimate, not an invoice. Input allowances are conservative text-byte/schema bounds and are checked against every returned usage receipt.",
+ }
+ output.mkdir(parents=True)
+ write(output / "plan.json", plan)
+ for directory, names, origin in (
+ ("source", ("plan.json", "inputs.json", "eligibility.json"), training),
+ ("reuse", ("plan.json", "collection.json"), reuse),
+ ):
+ (output / directory).mkdir()
+ for name in names:
+ shutil.copyfile(origin / name, output / directory / name)
+ (output / "records").mkdir()
+ for identity in reused_ids:
+ shutil.copyfile(reuse / "records" / f"{identity}.json", output / "records" / f"{identity}.json")
+ if previous:
+ shutil.copytree(supersedes, output / "superseded")
+ return plan
+
+
+class Budget:
+ def __init__(self, maximum, historical):
+ self.maximum, self.settled = Decimal(str(maximum)), Decimal(str(historical))
+ self.active = {}
+
+ @property
+ def exposure(self):
+ return self.settled + sum(self.active.values(), Decimal(0))
+
+ def reserve(self, identity, amount):
+ amount = Decimal(str(amount))
+ if identity in self.active or amount <= 0 or self.exposure + amount > self.maximum:
+ return False
+ self.active[identity] = amount
+ return True
+
+ def settle(self, identity, known_conservative=None):
+ reserved = self.active.pop(identity)
+ value = reserved if known_conservative is None else Decimal(str(known_conservative))
+ if value < 0 or value > reserved:
+ raise ValueError("Charge exceeds the reserved bound")
+ self.settled += value
+
+ def receipt(self):
+ return {
+ "maximum_usd": str(self.maximum),
+ "settled_or_unknown_usd": str(self.settled),
+ "inflight_usd": {key: str(value) for key, value in self.active.items()},
+ "total_exposure_usd": str(self.exposure),
+ }
+
+
+async def _http_request(client, body, secret, seconds):
+ async with asyncio.timeout(seconds):
+ response = await client.post(ENDPOINT, json=body, headers={"Authorization": "Bearer " + secret})
+ if response.status_code != 200:
+ return {"error": {"kind": "http_error", "status": response.status_code, "execution_unknown": True}}
+ if len(response.content) > 2_000_000:
+ return {"error": {"kind": "response_too_large", "execution_unknown": True}}
+ return {"payload": response.json()}
+
+
+async def _call(run, row, secret, client, request_fn):
+ started = time.time()
+ record = {
+ "id": row["id"],
+ "request_sha256": digest(body_for(row)),
+ "status": "incomplete",
+ "answer": None,
+ "usage_verified": False,
+ }
+ try:
+ async with asyncio.timeout(REQUEST_SECONDS):
+ result = await request_fn(client, body_for(row), secret, REQUEST_SECONDS)
+ if "error" in result:
+ record["error"] = result["error"]
+ else:
+ payload = result["payload"]
+ raw = run / "responses" / f"{row['id']}.json"
+ write(raw, payload)
+ record["raw_response_sha256"] = file_hash(raw)
+ record.update(
+ {
+ "response_id": payload.get("id"),
+ "returned_model": payload.get("model"),
+ "response_status": payload.get("status"),
+ "usage": payload.get("usage"),
+ "service_tier": payload.get("service_tier"),
+ "incomplete_details": payload.get("incomplete_details"),
+ }
+ )
+ try:
+ if payload.get("model") != MODEL or payload.get("service_tier") not in (None, "default"):
+ raise ValueError("Returned model or service tier differs from reserved pricing")
+ invoice, conservative = usage_cost(payload.get("usage"), input_allowance(row))
+ record.update(
+ {
+ "usage_verified": True,
+ "list_price_estimate_usd": float(invoice),
+ "conservative_list_price_usd": str(conservative),
+ }
+ )
+ except (TypeError, ValueError):
+ record["error"] = {"kind": "invalid_or_missing_usage", "execution_unknown": True}
+ output = "".join(
+ content.get("text", "")
+ for item in payload.get("output", [])
+ if item.get("type") == "message"
+ for content in item.get("content", [])
+ if content.get("type") == "output_text"
+ )
+ record["output_text"] = output
+ parsed = json.loads(output)
+ if not isinstance(parsed, dict) or set(parsed) != {"answer"} or not isinstance(parsed["answer"], str):
+ raise ValueError("Teacher response differs from its answer schema")
+ record["answer"] = parsed["answer"]
+ if record["usage_verified"] and payload.get("status") == "completed" and payload.get("model") == MODEL:
+ record["status"] = "complete"
+ else:
+ record.setdefault(
+ "error", {"kind": "incomplete_or_changed_model", "execution_unknown": not record["usage_verified"]}
+ )
+ except (Exception, asyncio.CancelledError) as error: # noqa: BLE001 — persist uncertain execution without retry.
+ record["error"] = {"kind": type(error).__name__, "execution_unknown": not record["usage_verified"]}
+ record["elapsed_seconds"] = time.time() - started
+ write(run / "records" / f"{row['id']}.json", record)
+ return record
+
+
+def checked_plan(run):
+ plan = read(run / "plan.json")
+ if (
+ plan.get("format") != "sie-teacher-reference-extension-v1"
+ or plan["settings"] != SETTINGS
+ or plan["operator_sha256"] != file_hash(Path(__file__))
+ or plan["data_source_sha256"] != file_hash(Path(__file__).with_name("data.py"))
+ or plan["prices_usd_per_million"] != PRICES
+ or plan.get("automatic_retries") != 0
+ or plan.get("unknown_execution_repeat") is not False
+ or plan.get("teacher_receives_gold") is not False
+ or plan.get("teacher_receives_solver_answers") is not False
+ or not 0 < Decimal(plan["max_usd"]) <= MAX_USD
+ or not 1 <= plan["workers"] <= MAX_WORKERS
+ or not REQUEST_SECONDS + 5 < plan["max_seconds"] <= MAX_SECONDS
+ ):
+ raise ValueError("Prepared teacher code/settings/limits changed")
+ for category, mapping in (
+ ("source", plan["source"]),
+ ("reuse", {key: plan["reuse"][key] for key in ("plan_sha256", "collection_sha256")}),
+ ):
+ for key, expected in mapping.items():
+ if file_hash(run / category / (key.removesuffix("_sha256") + ".json")) != expected:
+ raise ValueError("Copied source/reuse provenance changed")
+ rows = {row["id"]: row for row in plan["inputs"]}
+ copied_inputs = read(run / "source/inputs.json")
+ copied_source = read(run / "source/plan.json")
+ copied_eligibility = read(run / "source/eligibility.json")
+ copied_pilot = read(run / "reuse/plan.json")
+ copied_collection = read(run / "reuse/collection.json")
+ if (
+ len(rows) != 510
+ or len(plan["inputs"]) != 510
+ or len(set(plan["new_ids"])) != 446
+ or len(plan["new_ids"]) != 446
+ or len(set(plan["reused_ids"])) != 64
+ or len(plan["reused_ids"]) != 64
+ or copied_inputs != {"train": plan["inputs"]}
+ or digest(copied_inputs) != copied_source["inputs_sha256"]
+ or copied_eligibility["eligible_train_ids"] != [row["id"] for row in plan["inputs"]]
+ or plan["reused_ids"] != [row["id"] for row in copied_pilot["inputs"]]
+ or plan["reuse"]["record_hashes"] != copied_collection["record_hashes"]
+ or set(plan["new_ids"]) & set(plan["reused_ids"])
+ or set(rows) != set(plan["new_ids"]) | set(plan["reused_ids"])
+ or plan["new_ids"]
+ != sorted(plan["new_ids"], key=lambda identity: digest(["teacher-extension-order-v1", identity]))
+ ):
+ raise ValueError("Prepared teacher identities changed")
+ for identity in plan["new_ids"]:
+ if Decimal(plan["request_reservations_usd"][identity]) != reservation(rows[identity]) or plan[
+ "input_token_allowances"
+ ][identity] != input_allowance(rows[identity]):
+ raise ValueError("Teacher reservation changed")
+ invoice, conservative = Decimal(0), Decimal(0)
+ for identity, expected in plan["reuse"]["record_hashes"].items():
+ path = run / "records" / f"{identity}.json"
+ if file_hash(path) != expected:
+ raise ValueError("Reused raw teacher answer changed")
+ actual, upper = _check_teacher_record(read(path), rows[identity])
+ invoice += actual
+ conservative += upper
+ if invoice != Decimal(plan["reuse"]["list_price_estimate_usd"]) or conservative != Decimal(
+ plan["reuse"]["conservative_budget_usd"]
+ ):
+ raise ValueError("Historical teacher budget differs from raw usage")
+ if (
+ plan.get("superseded_attempt") is not None
+ and superseded_transport(run / "superseded") != plan["superseded_attempt"]
+ ):
+ raise ValueError("Superseded transport provenance or unknown exposure changed")
+ return plan, rows
+
+
+async def collect(run, secret=None, request_fn=_http_request):
+ run = Path(run)
+ plan, rows = checked_plan(run)
+ if (
+ (run / "started.json").exists()
+ or (run / "collection.json").exists()
+ or (run / "intents").exists()
+ or any((run / "records" / f"{identity}.json").exists() for identity in plan["new_ids"])
+ ):
+ raise FileExistsError("This phase has already started; no automatic resume or repeated calls")
+ secret = secret or os.environ.get("OPENAI_API_KEY")
+ if not secret:
+ raise ValueError("Set OPENAI_API_KEY before explicitly running collect")
+ previous = plan.get("superseded_attempt") or {}
+ budget = Budget(
+ plan["max_usd"],
+ Decimal(plan["reuse"]["conservative_budget_usd"]) + Decimal(previous.get("unknown_exposure_usd", "0")),
+ )
+ start, monotonic_start = time.time(), time.monotonic()
+ write(
+ run / "started.json",
+ {
+ "plan_sha256": file_hash(run / "plan.json"),
+ "started_unix": start,
+ "deadline_unix": start + plan["max_seconds"],
+ "budget": budget.receipt(),
+ },
+ )
+ stop, index, active, completed = None, 0, {}, {}
+ ledger_index = 0
+ transport = httpx.AsyncHTTPTransport(retries=0, trust_env=False)
+ async with httpx.AsyncClient(
+ timeout=REQUEST_SECONDS, follow_redirects=False, trust_env=False, transport=transport
+ ) as client:
+ while index < len(plan["new_ids"]) or active:
+ while stop is None and index < len(plan["new_ids"]) and len(active) < plan["workers"]:
+ if time.monotonic() - monotonic_start > plan["max_seconds"] - REQUEST_SECONDS - 5:
+ stop = "deadline"
+ break
+ identity = plan["new_ids"][index]
+ reserve = plan["request_reservations_usd"][identity]
+ if not budget.reserve(identity, reserve):
+ if not active:
+ stop = "budget"
+ break
+ write(
+ run / "intents" / f"{identity}.json",
+ {
+ "id": identity,
+ "plan_sha256": file_hash(run / "plan.json"),
+ "request_sha256": digest(body_for(rows[identity])),
+ "reservation_usd": reserve,
+ "budget_after_reservation": budget.receipt(),
+ "started_unix": time.time(),
+ },
+ )
+ task = asyncio.create_task(_call(run, rows[identity], secret, client, request_fn))
+ active[task] = identity
+ index += 1
+ if not active:
+ break
+ done, _ = await asyncio.wait(active, timeout=30, return_when=asyncio.FIRST_COMPLETED)
+ # Process the entire completion batch before any replacement launch.
+ for task in done:
+ identity = active.pop(task)
+ record = await task
+ completed[identity] = record
+ budget.settle(identity, record.get("conservative_list_price_usd") if record["usage_verified"] else None)
+ if record["status"] != "complete":
+ stop = stop or "request_error_or_unknown_execution"
+ write(
+ run / "ledger" / f"{ledger_index:05d}.json",
+ {
+ "completed": len(completed),
+ "dispatched": index,
+ "stop_reason": stop,
+ "budget": budget.receipt(),
+ "elapsed_seconds": time.monotonic() - monotonic_start,
+ },
+ )
+ ledger_index += 1
+ print(
+ json.dumps(
+ {
+ "event": "teacher_progress",
+ "completed_new": len(completed),
+ "planned_new": len(plan["new_ids"]),
+ "stop_reason": stop,
+ "budget": budget.receipt(),
+ }
+ ),
+ flush=True,
+ )
+ hashes = dict(plan["reuse"]["record_hashes"])
+ hashes.update({identity: file_hash(run / "records" / f"{identity}.json") for identity in completed})
+ known_invoice = Decimal(plan["reuse"]["list_price_estimate_usd"]) + sum(
+ (Decimal(str(record["list_price_estimate_usd"])) for record in completed.values() if record["usage_verified"]),
+ Decimal(0),
+ )
+ successful = len(plan["reused_ids"]) + sum(record["status"] == "complete" for record in completed.values())
+ summary = {
+ "status": "complete" if successful == plan["n"] else "incomplete",
+ "n": plan["n"],
+ "reused": len(plan["reused_ids"]),
+ "dispatched_new": index,
+ "returned": len(hashes),
+ "complete": successful,
+ "stop_reason": stop,
+ "plan_sha256": file_hash(run / "plan.json"),
+ "elapsed_seconds": time.monotonic() - monotonic_start,
+ "list_price_estimate_usd": float(known_invoice),
+ "invoice_estimate_complete": not previous and all(record["usage_verified"] for record in completed.values()),
+ "prior_unknown_exposure_usd": previous.get("unknown_exposure_usd", "0"),
+ "explicit_prior_transport_repeats": len(previous.get("explicitly_repeated_ids", [])),
+ "conservative_list_price_usd": str(budget.settled),
+ "budget": budget.receipt(),
+ "inference_retries": 0,
+ "record_hashes": hashes,
+ "intent_hashes": {
+ identity: file_hash(run / "intents" / f"{identity}.json") for identity in plan["new_ids"][:index]
+ },
+ }
+ write(run / "collection.json", summary)
+ return summary
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ sub = parser.add_subparsers(dest="command", required=True)
+ prepare_parser = sub.add_parser("prepare")
+ prepare_parser.add_argument("--training", type=Path, required=True)
+ prepare_parser.add_argument("--reuse", type=Path, required=True)
+ prepare_parser.add_argument("--output", type=Path, required=True)
+ prepare_parser.add_argument("--max-usd", type=Decimal, default=Decimal(15))
+ prepare_parser.add_argument("--workers", type=int, default=MAX_WORKERS)
+ prepare_parser.add_argument("--max-seconds", type=int, default=1800)
+ prepare_parser.add_argument("--supersedes", type=Path)
+ collect_parser = sub.add_parser("collect")
+ collect_parser.add_argument("--run", type=Path, required=True)
+ args = parser.parse_args()
+ if args.command == "prepare":
+ plan = prepare(
+ args.training, args.reuse, args.output, args.max_usd, args.workers, args.max_seconds, args.supersedes
+ )
+ print(
+ json.dumps(
+ {
+ "prepared": str(args.output),
+ "n": plan["n"],
+ "reuse": len(plan["reused_ids"]),
+ "new": len(plan["new_ids"]),
+ "max_total_usd": plan["max_usd"],
+ "historical_conservative_usd": plan["reuse"]["conservative_budget_usd"],
+ "plan_sha256": file_hash(args.output / "plan.json"),
+ },
+ indent=2,
+ )
+ )
+ else:
+ result = asyncio.run(collect(args.run))
+ print(
+ json.dumps(
+ {key: value for key, value in result.items() if key not in ("record_hashes", "intent_hashes")}, indent=2
+ )
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/teacher_format.py b/examples/train-a-router/teacher_format.py
new file mode 100644
index 000000000..322a26e89
--- /dev/null
+++ b/examples/train-a-router/teacher_format.py
@@ -0,0 +1,525 @@
+"""Question-only normalization of explicit final-answer formatting.
+
+No dataset identities, benchmark values or solver correctness enter these rules.
+Original answer strings and applied transformations remain in every receipt.
+Removing an approximation marker does not introduce a numeric tolerance.
+"""
+
+from __future__ import annotations
+
+import re
+
+from benefit_expanded_gold_v2 import grade_teacher_reference
+from solver_answers import _clean, _equivalent, _parse, extract_final_answer
+
+PROTOCOL = "question-only-final-format-v1"
+_NUMBER = r"[+-]?(?:\d+(?:\.\d*)?|\.\d+)"
+_SPACING = re.compile(r"\\(?:quad|qquad)\b|\\[,;:! ]")
+_MONEY = re.compile(r"\b(?:dollars?|cents?|nickels?|dimes?|coins?|price|costs?|tax|pay|paid|worth)\b", re.IGNORECASE)
+_UNITS = (
+ "degrees",
+ "radians",
+ "minutes per day",
+ "miles per gallon",
+ "miles per hour",
+ "kilometers per hour",
+ "kilometres per hour",
+ "dollars",
+ "cents",
+ "kilometers",
+ "kilometres",
+ "hours",
+ "minutes",
+ "miles",
+ "meters",
+ "metres",
+ "centimeters",
+ "centimetres",
+ "inches",
+ "feet",
+ "years",
+ "cm",
+ "km",
+)
+
+
+def _unit_name(unit: str) -> str:
+ unit = " ".join(unit.lower().split()).replace("metre", "meter")
+ unit = {"cm": "centimeter", "km": "kilometer", "inches": "inch", "feet": "foot"}.get(unit, unit)
+ return unit.removesuffix("s")
+
+
+def _math_prefix(value: str) -> bool:
+ no_commands = re.sub(r"\\[A-Za-z]+", "", value)
+ return value.count("{") == value.count("}") and re.search(r"[A-Za-z]{2,}", no_commands) is None
+
+
+def answer_unit(question: str) -> str | None:
+ """Infer the requested unit from the final request, never incidental facts."""
+ question = re.sub(r"\[asy\].*?\[/asy\]", " ", question, flags=re.IGNORECASE | re.DOTALL)
+ requests = [
+ match
+ for match in re.finditer(
+ r"\b(?:how\s+(?:many|much|old|long)|what|which|find|determine|compute|give|calculate)\b",
+ question,
+ re.IGNORECASE,
+ )
+ if re.match(r"how\b|what\b", match[0], re.IGNORECASE)
+ or re.search(r"(?:^|[.!?,;:]\s*|\band\s+)$", question[: match.start()], re.IGNORECASE)
+ ]
+ clause = question[requests[-1].start() :] if requests else question
+ if requests:
+ preceding = re.search(r"\bIn\s+([A-Za-z ]+),\s*$", question[: requests[-1].start()], re.IGNORECASE)
+ if preceding:
+ clause = preceding[0] + clause
+ requested_variable = re.search(
+ r"(?:values?\s+of|what\s+is|find|determine)"
+ r"\s+\$?([A-Za-z]|\\(?:theta|alpha|beta|gamma|delta|phi|omega))\$?\s*[?.]",
+ clause,
+ re.IGNORECASE,
+ )
+ if requested_variable:
+ variable = re.escape(requested_variable[1])
+ prefix = question[: requests[-1].start()]
+ unit_pattern = "|".join(re.escape(unit.removesuffix("s")) + "s?" for unit in _UNITS)
+ adjacent = {
+ _unit_name(match[1])
+ for match in re.finditer(
+ r"(? tuple[str, str] | None:
+ degree = re.fullmatch(r"(.+?)\s*(?:\^\s*\{?\s*\\circ\s*\}?|°)", value)
+ if degree and _math_prefix(degree[1]):
+ return degree[1].strip(), "degree"
+ prefix = re.fullmatch(r"(?:\\\$|\$|\\textdollar\s*)\s*(.+)", value, re.DOTALL)
+ if prefix and _math_prefix(prefix[1]):
+ return prefix[1].strip(), "dollar"
+ for unit in _UNITS:
+ singular = unit.removesuffix("s")
+ suffix = re.fullmatch(
+ r"(.+?)\s*(?:\\(?:text|mathrm)\{\s*)?(" + re.escape(singular) + r"s?)\s*\}?", value, re.IGNORECASE
+ )
+ if suffix and _math_prefix(suffix[1]):
+ return suffix[1].strip(), _unit_name(unit)
+ # Unknown noun units must not be silently discarded by Math-Verify either.
+ suffix = re.fullmatch(r"(.+?)(?:\\(?:text|mathrm)\{\s*([A-Za-z][A-Za-z -]+)\s*\}|\s+([A-Za-z][A-Za-z -]+))", value)
+ if suffix and _math_prefix(suffix[1]):
+ return suffix[1].strip(), "count:" + _unit_name(suffix[2] or suffix[3])
+ return None
+
+
+def normalize_final(question: str, final: str) -> dict:
+ """Return a value and an auditable list of generic formatting operations."""
+ value, steps = _clean(final), []
+ clean = _SPACING.sub(" ", value).strip()
+ if clean != value:
+ value = clean
+ steps.append("tex_spacing")
+ approximate = re.match(r"^(?:\\approx(?![A-Za-z])|≈)\s*(.+)$", value, re.DOTALL)
+ if approximate:
+ value = approximate[1].strip()
+ steps.append("approximation_marker_removed_without_tolerance")
+
+ value, real_radical = _real_odd_radicals(question, value)
+ if real_radical:
+ steps.append("explicit_real_odd_radical_negative_radicand")
+
+ binary_question = re.search(
+ r"(?:^|[.!?]\s+|\bProblem:\s*)(?:must|can|could|does|do|is|are|will|would)\b[^?]*\?\s*$",
+ question,
+ re.IGNORECASE | re.DOTALL,
+ )
+ binary_answer = re.match(r"^(?:\\(?:text|mathrm)\{\s*)?(yes|no)\b(?:[.,;:]|\s|\}|$)", value, re.IGNORECASE)
+ if binary_question and binary_answer:
+ value = binary_answer[1].capitalize()
+ steps.append("explicit_yes_no_question_leading_decision")
+
+ # Only names may precede these assignments; 8=7 is never unwrapped.
+ assignment = re.fullmatch(
+ r"(?:[A-Za-z]|\(\s*[A-Za-z](?:\s*,\s*[A-Za-z])+\s*\))\s*(?:=|\\in\b|∈)\s*(.+)",
+ value,
+ re.DOTALL,
+ )
+ if assignment and assignment[1].startswith(("(", r"\{")):
+ value = assignment[1].strip()
+ steps.append("named_tuple_or_set_assignment")
+
+ target, stated = answer_unit(question), explicit_unit(value)
+ if target == "year":
+ year = re.fullmatch(r"(?:\\(?:text|mathrm)\{\s*)?Year\s*\}?\s*(" + _NUMBER + r")", value, re.IGNORECASE)
+ if year:
+ value = year[1]
+ stated = None
+ steps.append("requested_year_prefix")
+ unit_status = "no_explicit_unit"
+ if target and target.startswith("angle_"):
+ if stated and stated[1] not in {"degree", "radian"}:
+ unit_status = "incompatible"
+ else:
+ scalar = stated[0] if stated else value
+ degree = stated is not None and stated[1] == "degree"
+ if stated is None:
+ degree = target == "angle_degree" or (
+ target == "angle_inferred_degree" and not re.search(r"\\pi\b|π", scalar)
+ )
+ value = "(" + scalar + r")*\pi/180" if degree else scalar
+ unit_status = "compatible" if stated else "no_explicit_unit"
+ if degree:
+ steps.append("angle_degrees_to_radians_exact")
+ elif stated:
+ steps.append("explicit_angle_radians")
+ elif target == "angle_inferred_degree" and re.search(r"\\pi\b|π", scalar):
+ steps.append("unspecified_angle_pi_expression_in_radians")
+ elif stated:
+ scalar, unit = stated
+ if target is None:
+ unit_status = "unknown_requested_unit"
+ elif unit == target or {unit, target} <= {"cent", "dollar"}:
+ unit_status = "compatible"
+ value = scalar
+ if unit == "cent":
+ value = "(" + value + ")/100"
+ steps.append("explicit_cents_to_dollars")
+ else:
+ steps.append("explicit_requested_unit:" + unit)
+ else:
+ unit_status = "incompatible"
+ elif target == "cent" and re.fullmatch(_NUMBER, value):
+ value = "(" + value + ")/100"
+ steps.append("requested_cents_to_dollars")
+
+ ratio = re.fullmatch(r"(" + _NUMBER + r")\s*:\s*(" + _NUMBER + r")", value)
+ if ratio and target == "ratio":
+ value = "(" + ratio[1] + ")/(" + ratio[2] + ")"
+ steps.append("explicit_ratio_context")
+ if re.fullmatch(r"[+-]?\d{1,3}(?:,\d{3})+(?:\.\d+)?", value):
+ value = value.replace(",", "")
+ steps.append("whole_numeric_thousands_separators")
+ return {
+ "original": final,
+ "normalized": value,
+ "operations": steps,
+ "protocol": PROTOCOL,
+ "requested_unit": target,
+ "explicit_unit": stated[1] if stated else None,
+ "unit_status": unit_status,
+ }
+
+
+def normalize_reference(question: str, answer: str) -> tuple[str, list[str]]:
+ result = normalize_final(question, answer)
+ return result["normalized"], result["operations"]
+
+
+def _real_odd_radicals(question: str, value: str) -> tuple[str, bool]:
+ """Move a negative sign outside explicit odd radicals in a real-answer task.
+
+ Arbitrary fractional powers retain their complex principal-value meaning.
+ Only bounded, symbol-free radicands proved real and negative are changed.
+ """
+ if not re.search(r"\breal\b", question, re.IGNORECASE) or len(value) > 2000:
+ return value, False
+ from sympy import latex
+
+ matches = list(re.finditer(r"\\sqrt\s*\[\s*(\d{1,2})\s*\]\s*\{", value))
+ changed = False
+ for match in reversed(matches[:20]):
+ index = int(match[1])
+ if index < 3 or index % 2 == 0:
+ continue
+ depth, end = 1, match.end()
+ while end < len(value) and depth:
+ depth += (value[end] == "{") - (value[end] == "}")
+ end += 1
+ if depth:
+ continue
+ radicand = value[match.end() : end - 1]
+ try:
+ parsed = _parse(radicand)
+ if parsed["kind"] != "math" or len(parsed["value"]) != 1:
+ continue
+ expression = parsed["value"][0]
+ if expression.free_symbols or expression.is_real is not True or expression.is_negative is not True:
+ continue
+ positive = latex(-expression)
+ except Exception: # noqa: BLE001, S112 — retain unsupported text for the bounded base grader.
+ continue
+ value = value[: match.start()] + r"(-\sqrt[" + str(index) + "]{" + positive + "})" + value[end:]
+ changed = True
+ return value, changed
+
+
+def _complex_value(parsed: dict) -> dict:
+ # Math-Verify assigns inconsistent real assumptions to bare i depending on
+ # fraction spelling. In a question explicitly about complex numbers, i is
+ # the imaginary unit in both answers. Preserve every other parsed symbol.
+ from sympy import I
+
+ if parsed["kind"] == "math":
+ return parsed | {
+ "value": [value.xreplace({s: I for s in value.free_symbols if s.name == "i"}) for value in parsed["value"]]
+ }
+ if parsed["kind"] in ("tuple", "set_of_tuples"):
+ return parsed | {"value": [_complex_value(value) for value in parsed["value"]]}
+ return parsed
+
+
+def _same_base_integer_power(reference: str, prediction: str) -> bool | None:
+ """Compare one identical integer base>1 using bounded integer exponents.
+
+ Positive-integer power towers are evaluated only up to a fixed saturation
+ bound. One saturated exponent and one exact smaller exponent prove unequal
+ powers; two saturated, structurally different exponents stay unsupported.
+ """
+ cap = 1_000_000
+
+ def parse(value):
+ if not value or len(value) > 512 or re.search(r"\d\s+\d", value):
+ raise ValueError("Unsupported power syntax")
+ compact = re.sub(r"\s+", "", value)
+ tokens = re.findall(r"\d+|[(){}^]", compact)
+ if "".join(tokens) != compact or len(tokens) > 256:
+ raise ValueError("Unsupported power syntax")
+ index = 0
+
+ def power(depth=0):
+ nonlocal index
+ if depth > 32 or index >= len(tokens):
+ raise ValueError("Power syntax bound")
+ token = tokens[index]
+ index += 1
+ if token.isdigit():
+ node = int(token)
+ if node < 1:
+ raise ValueError("Only positive integers")
+ elif token in ("(", "{"):
+ node = power(depth + 1)
+ close = ")" if token == "(" else "}"
+ if index >= len(tokens) or tokens[index] != close:
+ raise ValueError("Unbalanced power syntax")
+ index += 1
+ else:
+ raise ValueError("Unsupported power atom")
+ if index < len(tokens) and tokens[index] == "^":
+ index += 1
+ node = (node, power(depth + 1))
+ return node
+
+ result = power()
+ if index != len(tokens):
+ raise ValueError("Trailing power syntax")
+ return result
+
+ def capped(node):
+ if isinstance(node, int):
+ return min(node, cap)
+ base, exponent = capped(node[0]), capped(node[1])
+ result = 1
+ while exponent:
+ if exponent & 1:
+ result = min(result * base, cap)
+ if result == cap:
+ return cap
+ exponent >>= 1
+ base = min(base * base, cap)
+ return result
+
+ try:
+ left, right = parse(reference), parse(prediction)
+ except (ValueError, RecursionError):
+ return None
+ if not (
+ isinstance(left, tuple)
+ and isinstance(right, tuple)
+ and isinstance(left[0], int)
+ and left[0] > 1
+ and left[0] == right[0]
+ ):
+ return None
+ if left == right:
+ return True
+ a, b = capped(left[1]), capped(right[1])
+ if a == b == cap:
+ return None
+ return a == b
+
+
+def grade_reference(question: str, reference_final: str, solver_response: str) -> dict:
+ """Compare two independent final answers using question-only formatting."""
+ final = extract_final_answer(solver_response)
+ reference = normalize_final(question, reference_final)
+ prediction = normalize_final(question, final)
+ receipts = {"reference": reference, "prediction": prediction}
+ if (
+ reference["unit_status"] == prediction["unit_status"] == "unknown_requested_unit"
+ and reference["explicit_unit"] == prediction["explicit_unit"]
+ ):
+ # Equal explicit units can be compared without guessing an omitted
+ # unit. A bare number paired with an unknown unit remains unresolved.
+ for record in receipts.values():
+ record["normalized"] = explicit_unit(record["normalized"])[0]
+ record["unit_status"] = "same_explicit_units"
+ record["operations"].append("same_explicit_units_on_both_answers")
+ if any(record["unit_status"] in {"unknown_requested_unit", "incompatible"} for record in receipts.values()):
+ return {
+ "correct": False,
+ "status": "graded"
+ if all(record["unit_status"] != "unknown_requested_unit" for record in receipts.values())
+ else "unit_contract_unresolved",
+ "final_answer": final,
+ "format_normalization": receipts,
+ "unit_check": "Explicit answer units must match the requested quantity; no incidental-question-unit stripping",
+ }
+ powers = _same_base_integer_power(reference["normalized"], prediction["normalized"])
+ if powers is not None:
+ return {
+ "correct": powers,
+ "status": "graded",
+ "final_answer": final,
+ "format_normalization": receipts,
+ "integer_power_guard": "Identical integer base>1; exact or saturated positive-integer exponent comparison",
+ }
+ result = grade_teacher_reference(
+ question,
+ reference["normalized"],
+ r"\boxed{" + prediction["normalized"] + "}" if final else solver_response,
+ )
+ if (
+ final
+ and result.get("exact_numeric_guard") != "applied"
+ and re.search(r"\bcomplex\s+numbers?\b", question, re.IGNORECASE)
+ ):
+ try:
+ result = result | {
+ "correct": _equivalent(
+ _complex_value(_parse(reference["normalized"])), _complex_value(_parse(prediction["normalized"]))
+ ),
+ "complex_context": "Bare i is the imaginary unit in both parsed answers",
+ "status": "graded",
+ }
+ except Exception as error: # noqa: BLE001 — preserve the bounded base parser's failure result.
+ result = result | {"complex_context_error": type(error).__name__}
+ return result | {"final_answer": final, "format_normalization": {"reference": reference, "prediction": prediction}}
diff --git a/examples/train-a-router/tests/test_arithmetic_answers.py b/examples/train-a-router/tests/test_arithmetic_answers.py
new file mode 100644
index 000000000..468ece2cc
--- /dev/null
+++ b/examples/train-a-router/tests/test_arithmetic_answers.py
@@ -0,0 +1,36 @@
+import unittest
+
+from arithmetic_answers import clock_question, grade_reference
+
+
+class ArithmeticAnswerTests(unittest.TestCase):
+ def test_clock_hour_reference_accepts_cosmetic_period_and_minutes(self):
+ question = "The party starts at 5:00 pm. What is the latest time of day she can begin?"
+ for prediction in ("2:00 pm", r"2{:}00\ \mathrm{p.m.}", "14:00"):
+ with self.subTest(prediction=prediction):
+ result = grade_reference(question, "2", r"\boxed{" + prediction + "}")
+ self.assertTrue(result["correct"])
+
+ def test_clock_different_hour_or_minutes_is_wrong(self):
+ question = "What is the latest time they can leave home?"
+ for prediction in ("4:00 pm", "5:30 pm"):
+ self.assertFalse(grade_reference(question, "5", r"\boxed{" + prediction + "}")["correct"])
+
+ def test_duration_is_not_clock_time(self):
+ question = "A train leaves at 2 pm. How many hours does its trip take?"
+ self.assertFalse(clock_question(question))
+ self.assertFalse(grade_reference(question, "2", r"\boxed{2:00 pm}")["correct"])
+
+ def test_explicit_reference_period_is_preserved(self):
+ question = "What time does the train leave?"
+ self.assertFalse(grade_reference(question, "2:00 pm", r"\boxed{2:00 am}")["correct"])
+ self.assertTrue(grade_reference(question, "2:00 pm", r"\boxed{14:00}")["correct"])
+
+ def test_ordinary_count_keeps_existing_grading(self):
+ question = "How many apples are left?"
+ self.assertTrue(grade_reference(question, "6", r"\boxed{6}")["correct"])
+ self.assertFalse(grade_reference(question, "6", r"\boxed{7}")["correct"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_arithmetic_calibration_review.py b/examples/train-a-router/tests/test_arithmetic_calibration_review.py
new file mode 100644
index 000000000..56596ac44
--- /dev/null
+++ b/examples/train-a-router/tests/test_arithmetic_calibration_review.py
@@ -0,0 +1,79 @@
+import copy
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import arithmetic_calibration_review as review
+
+
+class CalibrationReviewTests(unittest.TestCase):
+ def fixture(self):
+ flag = {
+ "id": "example",
+ "arm": "strong",
+ "status": "unit_contract_unresolved",
+ "question": "How much of each?",
+ "reference": "30",
+ "final_answer": r"30 \text{ of each}",
+ "record_sha256": "a" * 64,
+ }
+ row = {
+ "id": "example",
+ "strong_correct": False,
+ "strong_record_sha256": "a" * 64,
+ "strong_grade": {"correct": False, "status": flag["status"], "final_answer": flag["final_answer"]},
+ }
+ decision = {
+ "id": "example",
+ "arm": "strong",
+ "record_sha256": "a" * 64,
+ "question_sha256": review.text_hash(flag["question"]),
+ "reference_sha256": review.text_hash("30"),
+ "final_answer_sha256": review.text_hash(flag["final_answer"]),
+ "correct": True,
+ "reason": "The equal-share arithmetic gives 30 for each requested category.",
+ }
+ return row, flag, decision
+
+ def test_exact_unresolved_review_retains_original_result(self):
+ row, flag, decision = self.fixture()
+ original = copy.deepcopy(row["strong_grade"])
+ review.apply_decision(row, flag, decision, "30")
+ self.assertTrue(row["strong_correct"])
+ self.assertEqual(row["strong_grade"]["base_grade"], original)
+
+ def test_resolved_or_changed_record_is_not_reviewable(self):
+ for key, value in (("record_sha256", "b" * 64), ("final_answer_sha256", "c" * 64), ("correct", 1)):
+ row, flag, decision = self.fixture()
+ decision[key] = value
+ with self.assertRaises(ValueError):
+ review.apply_decision(row, flag, decision, "30")
+ row, flag, decision = self.fixture()
+ row["strong_grade"]["status"] = "graded"
+ with self.assertRaises(ValueError):
+ review.apply_decision(row, flag, decision, "30")
+
+ def test_repeat_validation_preserves_first_scan_and_uses_fresh_output(self):
+ with tempfile.TemporaryDirectory() as directory:
+ run = Path(directory)
+ (run / "plan.json").write_text("{}")
+ original = run / "unresolved-calibration.json"
+ original.write_text('{"cases": ["reviewed"]}')
+ manifest = {"decisions": [{}], "original_scan_sha256": review.file_hash(original)}
+ before = original.read_bytes()
+
+ def base_grade(source, scratch, collection, prices, split, workers):
+ self.assertNotEqual(scratch, run)
+ self.assertEqual((scratch / "plan.json").read_bytes(), (run / "plan.json").read_bytes())
+ (scratch / "unresolved-calibration.json").write_bytes(before)
+ raise ValueError("1 unresolved comparisons; retain every row and review the complete list")
+
+ with patch.object(review.study, "grade", side_effect=base_grade):
+ actual = review.verified_scan(run, run, run, run, manifest, 1)
+ self.assertEqual(actual, original)
+ self.assertEqual(original.read_bytes(), before)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_arithmetic_controls.py b/examples/train-a-router/tests/test_arithmetic_controls.py
new file mode 100644
index 000000000..7da610433
--- /dev/null
+++ b/examples/train-a-router/tests/test_arithmetic_controls.py
@@ -0,0 +1,110 @@
+import json
+import tempfile
+import unittest
+from contextlib import ExitStack
+from pathlib import Path
+from unittest.mock import patch
+
+import arithmetic_controls as controls
+
+
+class ArithmeticControlTests(unittest.TestCase):
+ def test_probabilities_must_be_finite_and_unmodified(self):
+ statistics = {
+ "cheap": {"incorrect": 10, "correct": 2},
+ "strong": {"incorrect": 20, "correct": 8},
+ }
+ for values in ([[float("nan"), 0.9]], [[0.5, 1.01]], [[0.5]], [[0.5, 0.9], [0.2, 0.8]]):
+ with self.assertRaisesRegex(ValueError, "finite, unchanged"):
+ controls.scores(values, statistics, 1)
+
+ def test_separate_controls_preserve_core_and_bind_access(self):
+ with tempfile.TemporaryDirectory() as temporary, ExitStack() as stack:
+ root = Path(temporary)
+ run, output = root / "study", root / "controls"
+ run.mkdir()
+ inputs = {
+ "calibration": [{"id": "cal1"}, {"id": "cal2"}],
+ "test": [{"id": "test1"}, {"id": "test2"}],
+ }
+ probabilities = [[0.8, 0.9], [0.2, 0.6]]
+ native = {
+ "models": {
+ "reference_base": {
+ "trained_sha256": "b" * 64,
+ "splits": {
+ split: {"ids": [row["id"] for row in rows], "probabilities": probabilities}
+ for split, rows in inputs.items()
+ },
+ }
+ }
+ }
+ for name, value in {
+ "plan.json": {},
+ "native-scores.json": native,
+ "grades-calibration.json": {"rows": inputs["calibration"]},
+ "native-500/trained.json": {"size": 500},
+ "native-2000/trained.json": {"size": 2000},
+ }.items():
+ path = run / name
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(json.dumps(value))
+ core = {
+ "study_plan_sha256": controls.file_hash(run / "plan.json"),
+ "calibration_grades_sha256": controls.file_hash(run / "grades-calibration.json"),
+ "model_artifacts_sha256": {"native-scores.json": controls.file_hash(run / "native-scores.json")},
+ "source_code_sha256": {"unchanged": "source"},
+ "ids": {split: [row["id"] for row in rows] for split, rows in inputs.items()},
+ }
+ (run / "frozen.json").write_text(json.dumps(core))
+ original = (run / "frozen.json").read_bytes()
+ stack.enter_context(patch.object(controls.study, "checked_plan", return_value=({}, inputs, {})))
+ stack.enter_context(patch.object(controls.study, "code_hashes", return_value={"unchanged": "source"}))
+ stack.enter_context(
+ patch.object(
+ controls.study,
+ "checked_native",
+ side_effect=lambda source, run, size: {"conditional_cost_statistics": {"scale": size}},
+ )
+ )
+ stack.enter_context(
+ patch.object(
+ controls.study.outcomes,
+ "normalized_scores",
+ side_effect=lambda values, statistics: (
+ (values[:, 1] - values[:, 0]) / statistics["scale"],
+ None,
+ ),
+ )
+ )
+ stack.enter_context(
+ patch.object(
+ controls.study.solver_report,
+ "select_cutoff",
+ return_value={"mode": "always_strong", "cutoff": None},
+ )
+ )
+ result = controls.freeze(root, run, output)
+ self.assertEqual((run / "frozen.json").read_bytes(), original)
+ self.assertEqual(result["methods"]["base_arithmetic_500"]["training_rows"], 0)
+ self.assertEqual(result["methods"]["base_arithmetic_2000"]["cost_training_rows"], 2000)
+ self.assertNotEqual(
+ result["methods"]["base_arithmetic_500"]["scores"],
+ result["methods"]["base_arithmetic_2000"]["scores"],
+ )
+ controls.checked_frozen(root, run, output)
+ with self.assertRaises(FileExistsError):
+ controls.freeze(root, run, output)
+ (run / "test-access.json").write_text(
+ json.dumps({"frozen_sha256": controls.file_hash(run / "frozen.json")})
+ )
+ access = controls.authorize_test(root, run, output)
+ self.assertEqual(access["frozen_sha256"], controls.file_hash(output / "frozen.json"))
+ result["methods"]["base_arithmetic_500"]["scores"]["test"][0] += 0.5
+ (output / "frozen.json").write_text(json.dumps(result))
+ with self.assertRaisesRegex(ValueError, "score formula changed"):
+ controls.checked_frozen(root, run, output)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_arithmetic_data.py b/examples/train-a-router/tests/test_arithmetic_data.py
new file mode 100644
index 000000000..1a09c7b45
--- /dev/null
+++ b/examples/train-a-router/tests/test_arithmetic_data.py
@@ -0,0 +1,57 @@
+import unittest
+
+from arithmetic_data import assign, grouped_sources
+from data import render_request
+
+
+def row(identity, text, source="train"):
+ return {"id": identity, "problem": text, "gold": "1", "dataset": "arithmetic", "original_split": source}
+
+
+class ArithmeticDataTests(unittest.TestCase):
+ def test_families_do_not_cross_exposed_and_fresh_splits(self):
+ rows = [row(str(i), f"Distinct question number {i}.") for i in range(9)]
+ exposed = [{"id": str(i), "request": render_request("GSM8K", rows[i]["problem"])} for i in range(3)]
+ components = [[0, 3], [1, 2], [4], [5], [6], [7], [8]]
+ counts = {"train": 3, "train_small": 2, "calibration": 2, "test": 2}
+ selected, families, small, audit = assign(rows, components, exposed, counts)
+ self.assertEqual(set(small), {"1", "2"})
+ self.assertEqual(audit["available_unused_families"], 5)
+ self.assertNotIn("3", {r["id"] for split in ("calibration", "test") for r in selected[split]})
+ groups = [{families[r["id"]] for r in selected[split]} for split in ("train", "calibration", "test")]
+ self.assertTrue(all(not groups[i] & groups[j] for i in range(3) for j in range(i)))
+
+ def test_rejects_changed_exposed_request_and_insufficient_population(self):
+ rows = [row(str(i), f"A question {i}") for i in range(3)]
+ counts = {"train": 1, "train_small": 1, "calibration": 1, "test": 2}
+ exposed = [{"id": "0", "request": render_request("GSM8K", rows[0]["problem"])}]
+ with self.assertRaisesRegex(ValueError, "Not enough"):
+ assign(rows, [[0], [1], [2]], exposed, counts)
+ with self.assertRaisesRegex(ValueError, "match the pinned source"):
+ assign(rows, [[0], [1], [2]], [{"id": "0", "request": "Changed"}], counts)
+
+ def test_numeric_templates_are_grouped_without_answers(self):
+ rows = [
+ row(
+ "a",
+ "Mira buys 12 apples for 8 dollars and shares them equally with her 4 friends. How many apples remain?",
+ ),
+ row(
+ "b",
+ "Mira buys 24 apples for 16 dollars and shares them equally with her 8 friends. How many apples remain?",
+ ),
+ row("c", "A triangular garden has side lengths five seven and nine meters. What is its perimeter?"),
+ ]
+ components, _ = grouped_sources(rows)
+ self.assertIn([0, 1], components)
+ self.assertIn([2], components)
+
+ def test_exposed_competition_overlap_is_explicit(self):
+ rows = [row("known", "Same question"), {"id": "old", "problem": "Same question", "dataset": "competition"}]
+ exposed = [{"id": "known", "request": render_request("GSM8K", "Same question")}]
+ with self.assertRaisesRegex(ValueError, "overlaps competition"):
+ assign(rows, [[0, 1]], exposed, {"train": 1, "train_small": 1, "calibration": 0, "test": 0})
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_arithmetic_labels.py b/examples/train-a-router/tests/test_arithmetic_labels.py
new file mode 100644
index 000000000..8998a2005
--- /dev/null
+++ b/examples/train-a-router/tests/test_arithmetic_labels.py
@@ -0,0 +1,193 @@
+import json
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import arithmetic_labels as labels
+
+
+class ArithmeticLabelTests(unittest.TestCase):
+ def test_explicit_closure_preserves_incomplete_collection_and_exact_mask(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ directory = Path(temporary)
+ inputs = [{"id": "one", "request": "How many?"}, {"id": "two", "request": "How much?"}]
+ plan = {
+ "format": labels.arithmetic_teacher.FORMAT,
+ "settings": labels.arithmetic_teacher.protocol_for("numeric").SETTINGS,
+ "answer_format": "numeric",
+ "ids": ["one", "two"],
+ "questions_sha256": labels.digest(inputs),
+ "question_only": True,
+ "reference_answers_received": False,
+ "solver_answers_received": False,
+ }
+ (directory / "plan.json").write_text(json.dumps(plan))
+ attempt = directory / "attempts/one/attempt-01"
+ (attempt / "records").mkdir(parents=True)
+ record = {"status": "complete", "answer": "7", "raw_response_sha256": "a" * 64}
+ (attempt / "records/one.json").write_text(json.dumps(record))
+ collection = {
+ "plan_sha256": labels.file_hash(directory / "plan.json"),
+ "complete": 1,
+ "completed_attempts": {"one": "attempts/one/attempt-01"},
+ "pending": ["two"],
+ "physical_attempts": 2,
+ }
+ collection_path = directory / "collection.json"
+ collection_path.write_text(json.dumps(collection))
+ original_bytes = collection_path.read_bytes()
+ closure = {
+ "format": "sie-arithmetic-teacher-closure-v1",
+ "plan_sha256": labels.file_hash(directory / "plan.json"),
+ "collection_sha256": labels.file_hash(collection_path),
+ "completed_ids": ["one"],
+ "missing_ids": ["two"],
+ "physical_attempts": 2,
+ "acquisition_stopped": True,
+ "additional_requests_authorized": False,
+ "eligibility_basis": "Teacher completion and reference-only parsing; no solver or gold outcomes.",
+ "reason": "Provider quota exhausted; preserve all completed references.",
+ }
+ closure_path = directory / "closure.json"
+ closure_path.write_text(json.dumps(closure))
+ with self.assertRaisesRegex(ValueError, "pending references"):
+ labels.teacher_records(directory, inputs)
+ with patch.object(
+ labels.arithmetic_teacher,
+ "attempt_history",
+ side_effect=lambda _, row: [(attempt, {}, record)] if row["id"] == "one" else [],
+ ):
+ records, missing, _, _ = labels.teacher_records(directory, inputs, closure_path=closure_path)
+ self.assertEqual(list(records), ["one"])
+ self.assertEqual(missing, ["two"])
+ self.assertEqual(collection_path.read_bytes(), original_bytes)
+ closure["missing_ids"] = []
+ closure_path.write_text(json.dumps(closure))
+ with self.assertRaisesRegex(ValueError, "closure does not match"):
+ labels.teacher_records(directory, inputs, closure_path=closure_path)
+
+ def test_teacher_targets_do_not_fall_back_to_benchmark_reference(self):
+ item = labels._label_item(
+ (
+ "training",
+ "How many apples remain?",
+ "7",
+ {"cheap": r"\boxed{7}", "strong": r"\boxed{6}"},
+ "6",
+ )
+ )
+ self.assertTrue(item["usable"])
+ self.assertTrue(item["grades"]["cheap"]["correct"])
+ self.assertFalse(item["grades"]["strong"]["correct"])
+ self.assertFalse(item["audit"]["correct"])
+
+ def test_unusable_teacher_reference_masks_both_without_comparing_solvers(self):
+ with patch.object(
+ labels, "grade_teacher", return_value={"correct": False, "status": "gold_unparseable"}
+ ) as grade:
+ usable, _, comparisons = labels.label_pair(
+ "How many apples remain?",
+ "Unknown",
+ {
+ "cheap": r"\boxed{2}",
+ "strong": r"\boxed{3}",
+ },
+ )
+ self.assertFalse(usable)
+ self.assertEqual(comparisons, {})
+ self.assertEqual(grade.call_count, 1)
+
+ def test_solver_comparison_exception_cannot_change_reference_eligibility(self):
+ with patch.object(
+ labels,
+ "grade_teacher",
+ side_effect=[
+ {"correct": True, "status": "graded"},
+ ValueError("unsupported"),
+ {"correct": False, "status": "graded"},
+ ],
+ ):
+ usable, _, comparisons = labels.label_pair(
+ "How many apples remain?",
+ "2",
+ {
+ "cheap": r"\boxed{odd notation}",
+ "strong": r"\boxed{3}",
+ },
+ )
+ self.assertTrue(usable)
+ self.assertEqual(comparisons["cheap"]["status"], "grading_error")
+ self.assertFalse(comparisons["strong"]["correct"])
+
+ def test_clock_teacher_accepts_bare_hour_but_preserves_explicit_period_conflict(self):
+ question = "What time should the bus depart?"
+ self.assertTrue(labels.grade_teacher(question, "2:00 pm", r"\boxed{2}")["correct"])
+ self.assertTrue(labels.grade_teacher(question, "2:00 pm", r"\boxed{14:00}")["correct"])
+ self.assertFalse(labels.grade_teacher(question, "2:00 pm", r"\boxed{2:00 am}")["correct"])
+
+ def test_wrong_unit_reference_does_not_become_usable(self):
+ usable, _, _ = labels.label_pair(
+ "Walk 12 miles in 3 hours. How many hours did it take?",
+ "3 miles",
+ {"cheap": r"\boxed{3}", "strong": r"\boxed{3 hours}"},
+ )
+ self.assertFalse(usable)
+
+ def test_benchmark_audit_failure_cannot_change_teacher_targets(self):
+ with patch.object(
+ labels.arithmetic_answers,
+ "grade_reference",
+ side_effect=[
+ {"correct": True, "status": "graded"},
+ {"correct": True, "status": "graded"},
+ {"correct": False, "status": "graded"},
+ ValueError("unsupported benchmark"),
+ ],
+ ):
+ item = labels._label_item(
+ (
+ "training",
+ "How many apples remain?",
+ "7",
+ {"cheap": r"\boxed{7}", "strong": r"\boxed{6}"},
+ "unsupported source",
+ )
+ )
+ self.assertTrue(item["usable"])
+ self.assertEqual(item["audit"]["status"], "grading_error")
+ self.assertTrue(item["grades"]["cheap"]["correct"])
+
+ def test_numeric_contract_rejects_units_prose_and_nonfinite_values(self):
+ for value in ("23", "-3.5", "1/4", ".5"):
+ self.assertTrue(labels.numeric_reference(value), value)
+ for value in ("$23", "23 pens", "23 or 24", "NaN", "1/0", "1e6", "1 000"):
+ self.assertFalse(labels.numeric_reference(value), value)
+
+ def test_numeric_reference_is_eligible_without_inferred_requested_unit(self):
+ usable, contract, grades = labels.label_pair(
+ "An agent buys ads. How much did he spend on buying all the ads?",
+ "1375",
+ {"cheap": r"\boxed{1375}", "strong": r"\boxed{1374}"},
+ answer_format="numeric",
+ )
+ self.assertTrue(usable)
+ self.assertEqual(contract["unit_contract"]["unit_status"], "numeric_requested_units")
+ self.assertTrue(grades["cheap"]["correct"])
+ self.assertFalse(grades["strong"]["correct"])
+
+ def test_numeric_contract_failure_never_uses_solver_answers(self):
+ with patch.object(labels, "grade_teacher", return_value={"correct": True, "status": "graded"}) as grade:
+ usable, _, grades = labels.label_pair(
+ "How many apples remain?",
+ "7 apples",
+ {"cheap": r"\boxed{7}", "strong": r"\boxed{7}"},
+ answer_format="numeric",
+ )
+ self.assertFalse(usable)
+ self.assertEqual(grades, {})
+ self.assertEqual(grade.call_count, 1)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_arithmetic_optimizer.py b/examples/train-a-router/tests/test_arithmetic_optimizer.py
new file mode 100644
index 000000000..14cf2d6ee
--- /dev/null
+++ b/examples/train-a-router/tests/test_arithmetic_optimizer.py
@@ -0,0 +1,60 @@
+import unittest
+
+import numpy as np
+
+import arithmetic_optimizer as stable
+import benefit_outcomes as original
+
+
+class ArithmeticOptimizerTests(unittest.TestCase):
+ def setUp(self):
+ generator = np.random.default_rng(89)
+ self.matrix = generator.normal(size=(80, 4))
+ self.targets = (self.matrix[:, 0] + 0.4 * generator.normal(size=80) > 0).astype(float)
+
+ def test_same_convex_solution_and_translation_invariance(self):
+ expected = original.fit_logistic(self.matrix, self.targets, 0.1)
+ actual = stable.fit_logistic(self.matrix, self.targets, 0.1)
+ shifted = stable.fit_logistic(self.matrix + 13, self.targets, 0.1)
+ np.testing.assert_allclose(
+ original.probability(self.matrix, expected),
+ original.probability(self.matrix, actual),
+ atol=1e-7,
+ rtol=0,
+ )
+ np.testing.assert_allclose(
+ original.probability(self.matrix, actual),
+ original.probability(self.matrix + 13, shifted),
+ atol=1e-7,
+ rtol=0,
+ )
+ self.assertTrue(actual["optimizer"]["success"])
+
+ def test_rejects_nonbinary_and_missing_class(self):
+ for targets in (np.full(80, 0.5), np.ones(80)):
+ with self.assertRaises(ValueError):
+ stable.fit_logistic(self.matrix, targets, 0.1)
+
+ def test_head_preserves_candidate_grid_folds_targets_and_costs(self):
+ rows = [
+ {"id": str(i), "cheap_correct": bool(self.targets[i]), "strong_correct": bool(self.targets[-i - 1])}
+ for i in range(80)
+ ]
+ groups = [f"family-{i // 2}" for i in range(80)]
+ folds = [(i // 2) % 5 for i in range(80)]
+ costs = [[50.0 + i, 200.0 + i] for i in range(80)]
+ head = stable.fit_head(self.matrix, rows, groups, costs, fold_assignments=folds)
+ self.assertEqual(head["folds"], folds)
+ self.assertEqual(head["training_cost_proxies"], costs)
+ self.assertEqual([item["penalty"] for item in head["cv"]], original.PROTOCOL["penalties"])
+ self.assertEqual(
+ head["conditional_cost_statistics"], original.conditional_costs(np.array(head["targets"]), costs)
+ )
+ winner = min(head["cv"], key=lambda item: (item["summed_binary_logloss"], -item["penalty"]))
+ self.assertEqual(head["penalty"], winner["penalty"])
+ with self.assertRaises(ValueError):
+ stable.fit_head(self.matrix, rows, ["one-family"] * 80, costs, fold_assignments=folds)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_arithmetic_replay.py b/examples/train-a-router/tests/test_arithmetic_replay.py
new file mode 100644
index 000000000..84422eb75
--- /dev/null
+++ b/examples/train-a-router/tests/test_arithmetic_replay.py
@@ -0,0 +1,39 @@
+import unittest
+
+import arithmetic_replay as replay
+
+
+class PublicReplayTests(unittest.TestCase):
+ def setUp(self):
+ self.rows = [
+ {"id": "easy", "cheap_correct": True, "strong_correct": True, "cheap_credits": 1, "strong_credits": 10},
+ {"id": "hard", "cheap_correct": False, "strong_correct": True, "cheap_credits": 1, "strong_credits": 10},
+ ]
+ self.model = {
+ "ids": {"test": ["easy", "hard"]},
+ "scores": {"test": [0, 2]},
+ "router_credits": {"test": [1, 1]},
+ "policies": {"1": {"mode": "threshold", "cutoff": 1}},
+ "method": "embedding_2000", "design": "embedding", "training_rows": 2000,
+ }
+
+ def test_good_decisions_save_money_after_embedding_cost(self):
+ metrics = replay.method_report(self.model, self.rows)["policies"]["1"]["metrics"]
+ self.assertEqual(metrics["answer_accuracy"], 1)
+ self.assertEqual(metrics["gap_vs_strong_pp"], 0)
+ self.assertAlmostEqual(metrics["gross_solver_savings_pct"], 45)
+ self.assertAlmostEqual(metrics["net_savings_pct"], 35)
+
+ def test_static_policy_skips_embedding_charge(self):
+ self.model["policies"]["1"] = {"mode": "always_strong", "cutoff": None}
+ metrics = replay.method_report(self.model, self.rows)["policies"]["1"]["metrics"]
+ self.assertEqual(metrics["net_savings_pct"], 0)
+ self.assertEqual(metrics["router_cost_usd_per1000"], 0)
+
+ def test_reordered_outcomes_cannot_be_used_for_saved_decisions(self):
+ with self.assertRaisesRegex(ValueError, "question order"):
+ replay.method_report(self.model, list(reversed(self.rows)))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_arithmetic_study.py b/examples/train-a-router/tests/test_arithmetic_study.py
new file mode 100644
index 000000000..fe17a9530
--- /dev/null
+++ b/examples/train-a-router/tests/test_arithmetic_study.py
@@ -0,0 +1,221 @@
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import arithmetic_study as study
+from data import digest, file_hash, write
+
+
+def replace(path, value):
+ path.unlink(missing_ok=True)
+ write(path, value)
+
+
+class ArithmeticStudyTests(unittest.TestCase):
+ def setUp(self):
+ self.temporary = tempfile.TemporaryDirectory()
+ self.root = Path(self.temporary.name)
+ self.source, self.run = self.root / "source", self.root / "run"
+ self.counts = {"train": 10, "train_small": 5, "calibration": 2, "test": 2}
+ self.count_patch = patch.object(study, "COUNTS", self.counts)
+ self.count_patch.start()
+ self.code_patch = patch.object(
+ study,
+ "code_hashes",
+ return_value={
+ name: "a" * 64
+ for name in ("arithmetic_study.py", "arithmetic_answers.py", "teacher_format.py", "solver_answers.py")
+ },
+ )
+ self.code_patch.start()
+ self.inputs = {
+ split: [{"id": f"{split}-{i}", "request": f"How many apples are in box {i}?"} for i in range(n)]
+ for split, n in self.counts.items()
+ if split != "train_small"
+ }
+ self.metadata = {
+ split: [{"id": row["id"], "family": row["id"], "original_split": "train"} for row in rows]
+ for split, rows in self.inputs.items()
+ }
+ write(self.source / "inputs.json", self.inputs)
+ write(self.source / "rows.json", self.metadata)
+ write(self.source / "train-small-ids.json", [row["id"] for row in self.inputs["train"][:5]])
+ write(
+ self.source / "gold-for-scoring-only.json",
+ {row["id"]: "3" for rows in self.inputs.values() for row in rows},
+ )
+ self.reseal_source()
+ self.embedding, self.native = self.root / "embedding", self.root / "native"
+ for directory, names in (
+ (self.embedding, ("selected-model.json", "frozen.json")),
+ (self.native, ("trained.json", "frozen.json")),
+ ):
+ for name in names:
+ write(directory / name, {})
+ study.prepare(self.source, self.run, self.embedding, self.native)
+
+ def tearDown(self):
+ self.count_patch.stop()
+ self.code_patch.stop()
+ self.temporary.cleanup()
+
+ def reseal_source(self):
+ replace(
+ self.source / "source-plan.json",
+ {
+ "format": study.SOURCE_FORMAT,
+ "counts": self.counts,
+ "exposed_source_plan_sha256": "b" * 64,
+ "files_sha256": {
+ name: file_hash(self.source / name)
+ for name in ("inputs.json", "rows.json", "train-small-ids.json", "gold-for-scoring-only.json")
+ },
+ },
+ )
+
+ def test_source_mutation_and_cross_role_family_are_rejected(self):
+ changed = study.read(self.source / "inputs.json")
+ changed["train"][0]["request"] = "Different question"
+ replace(self.source / "inputs.json", changed)
+ with self.assertRaisesRegex(ValueError, "source changed"):
+ study.checked_source(self.source)
+ replace(self.source / "inputs.json", self.inputs)
+ self.metadata["calibration"][0]["family"] = self.metadata["train"][0]["family"]
+ replace(self.source / "rows.json", self.metadata)
+ self.reseal_source()
+ with self.assertRaisesRegex(ValueError, "family crosses"):
+ study.checked_source(self.source)
+
+ def test_nested_training_cannot_split_a_family(self):
+ self.metadata["train"][5]["family"] = self.metadata["train"][0]["family"]
+ replace(self.source / "rows.json", self.metadata)
+ self.reseal_source()
+ with self.assertRaisesRegex(ValueError, "splits a source family"):
+ study.checked_source(self.source)
+
+ def test_prepared_fold_assignment_and_subset_cannot_change(self):
+ plan = study.read(self.run / "plan.json")
+ plan["training_folds"][0] = (plan["training_folds"][0] + 1) % 5
+ replace(self.run / "plan.json", plan)
+ with self.assertRaisesRegex(ValueError, "folds changed"):
+ study.checked_plan(self.source, self.run)
+
+ def test_fresh_test_cannot_be_read_before_model_freeze(self):
+ with self.assertRaises(FileNotFoundError) as failure:
+ study.grade(self.source, self.run, self.root / "missing-answers", self.root / "prices", "test")
+ self.assertEqual(Path(failure.exception.filename), self.run / "test-access.json")
+
+ def test_all_unresolved_arms_and_all_physical_calls_are_preserved(self):
+ record = {
+ "answer": r"\boxed{3}",
+ "status": "complete",
+ "has_unknown_charges": False,
+ "calls": [
+ {"tokens": {"prompt_tokens": 10, "completion_tokens": 4}},
+ {"tokens": {"prompt_tokens": 15, "completion_tokens": 2}},
+ ],
+ }
+ rate = {"input_tokens": {"numerator": 1, "denominator": 1}, "output_tokens": {"numerator": 2, "denominator": 1}}
+ with patch.object(
+ study.arithmetic_answers,
+ "grade_reference",
+ side_effect=[
+ {"correct": False, "status": "unit_contract_unresolved"},
+ {"correct": False, "status": "grading_error"},
+ ],
+ ):
+ row, flags = study._grade_pair(
+ (
+ {"id": "one", "request": "Question"},
+ {"family": "f"},
+ "3",
+ {arm: record for arm in study.outcomes.ARMS},
+ {arm: "c" * 64 for arm in study.outcomes.ARMS},
+ {arm: rate for arm in study.outcomes.ARMS},
+ )
+ )
+ self.assertEqual(len(flags), 2)
+ self.assertEqual(row["cheap_credits"], 37)
+ self.assertEqual(len(row["strong_cost_calls"]), 2)
+
+ def test_grading_replays_verified_records_without_optional_acquisition_files(self):
+ collection = self.root / "answers"
+ manifest = {}
+ for question in self.inputs["train"]:
+ for arm in study.outcomes.ARMS:
+ path = collection / f"{question['id']}-{arm}.json"
+ write(path, {"answer": r"\boxed{3}"})
+ manifest[path.name] = file_hash(path)
+ write(
+ collection / "collection-plan.json",
+ {
+ "source_plan_sha256": "b" * 64,
+ "settings_sha256": digest(study.solver.SETTINGS),
+ "records_sha256": manifest,
+ },
+ )
+ write(
+ collection / "collection.json",
+ {
+ "status": "complete",
+ "completed": 20,
+ "missing": 0,
+ "unknown_execution_attempts": 0,
+ "receipt_sha256": {"strong/model.yaml": "f" * 64},
+ },
+ )
+ prices = self.root / "prices.json"
+ write(prices, {})
+ with (
+ patch.object(study.solver, "validate_record"),
+ patch.object(study.benefit_costs, "rates_from_snapshot", return_value=({"cheap": {}, "strong": {}}, [])),
+ patch.object(study.benefit_costs, "physical_costs", return_value=[{"credits_proxy": 3}]),
+ patch.object(
+ study.arithmetic_answers, "grade_reference", return_value={"status": "graded", "correct": True}
+ ),
+ ):
+ grades = study.grade(self.source, self.run, collection, prices, "train", workers=1)
+ self.assertEqual(len(grades["rows"]), 10)
+ self.assertEqual(grades["acquisition_receipt_mapping_sha256"], digest({"strong/model.yaml": "f" * 64}))
+ self.assertTrue(all(row["cheap_correct"] and row["strong_correct"] for row in grades["rows"]))
+
+ def test_embedding_overhead_is_allocated_once_per_batch(self):
+ directory = self.root / "embeddings"
+ write(
+ directory / "collection.json", {"calls": [{"ids": ["a", "b"], "credits": 5}, {"ids": ["c"], "credits": 2}]}
+ )
+ self.assertEqual(study.overhead(directory, ["c", "b", "a"]), [2, 2.5, 2.5])
+ replace(
+ directory / "collection.json", {"calls": [{"ids": ["a", "b"], "credits": 5}, {"ids": ["b"], "credits": 2}]}
+ )
+ with self.assertRaisesRegex(ValueError, "multiple batches"):
+ study.overhead(directory, ["a"])
+
+ def test_native_context_preserves_shared_labels_and_family_fold(self):
+ rows = [
+ {"id": "a", "cheap_correct": True, "strong_correct": False, "cheap_credits": 2, "strong_credits": 5},
+ {"id": "b", "cheap_correct": False, "strong_correct": True, "cheap_credits": 4, "strong_credits": 7},
+ ]
+ grades = {"prices_sha256": "d" * 64, "cost_scope": "token proxy", "rows": rows}
+ write(self.run / "grades-train.json", grades)
+ context = {
+ "inputs": [{"id": "a", "request": "A"}, {"id": "b", "request": "B"}],
+ "rows": rows,
+ "folds": [0, 1],
+ "grades": grades,
+ }
+ with (
+ patch.object(study, "training", return_value=context),
+ patch.object(study.native, "_train_cuda", return_value={"complete": True}) as trainer,
+ ):
+ study.train_native(self.source, self.run, 500)
+ passed = trainer.call_args.args[2]
+ self.assertEqual(passed["targets"].tolist(), [[1, 0], [0, 1]])
+ self.assertEqual(passed["held"].tolist(), [True, False])
+ self.assertEqual(passed["plan"]["selection_validation_ids"], ["a"])
+ self.assertEqual(passed["costs"], [[2, 5], [4, 7]])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_arithmetic_teacher.py b/examples/train-a-router/tests/test_arithmetic_teacher.py
new file mode 100644
index 000000000..e8f5664eb
--- /dev/null
+++ b/examples/train-a-router/tests/test_arithmetic_teacher.py
@@ -0,0 +1,162 @@
+import asyncio
+import json
+import tempfile
+import unittest
+from pathlib import Path
+
+import arithmetic_teacher as acquisition
+import arithmetic_teacher_protocol as numeric
+import teacher_collect as teacher
+from data import write
+
+
+class TeacherAcquisitionTests(unittest.TestCase):
+ def setUp(self):
+ self.temporary = tempfile.TemporaryDirectory()
+ self.addCleanup(self.temporary.cleanup)
+ self.root = Path(self.temporary.name)
+ self.inputs = self.root / "inputs.json"
+ self.rows = [{"id": "training-1", "request": "What is 3 plus 4?"}]
+ write(self.inputs, self.rows)
+
+ def prepare(self, **kwargs):
+ run = self.root / "run"
+ acquisition.prepare(self.inputs, run, workers=1, **kwargs)
+ return run
+
+ @staticmethod
+ async def success(client, body, secret, seconds):
+ return {
+ "payload": {
+ "id": "fake-response",
+ "model": teacher.MODEL,
+ "status": "completed",
+ "service_tier": "default",
+ "usage": {"input_tokens": 100, "output_tokens": 10},
+ "output": [{"type": "message", "content": [{"type": "output_text", "text": '{"answer":"7"}'}]}],
+ }
+ }
+
+ def test_question_only_boundary_rejects_gold_and_duplicate_ids(self):
+ self.inputs.write_text(json.dumps([self.rows[0] | {"gold": "7"}]))
+ with self.assertRaisesRegex(ValueError, "question-only"):
+ acquisition.questions(self.inputs)
+ self.inputs.write_text(json.dumps(self.rows * 2))
+ with self.assertRaises(ValueError):
+ acquisition.questions(self.inputs)
+
+ def test_reuses_completed_answers_without_resending_or_reading_gold(self):
+ run = self.prepare()
+ called = []
+
+ async def request(client, body, secret, seconds):
+ called.append(body)
+ self.assertEqual(body["input"], self.rows[0]["request"])
+ self.assertNotIn("gold", body)
+ return await self.success(client, body, secret, seconds)
+
+ first = asyncio.run(acquisition.collect(run, secret="test-only", request_fn=request))
+ second = asyncio.run(acquisition.collect(run, secret="test-only", request_fn=request))
+ self.assertEqual(len(called), 1)
+ self.assertEqual(first["complete"], 1)
+ self.assertEqual(first["known_list_price_usd"], "0.0015")
+ self.assertEqual(first["budget"]["settled_or_unknown_usd"], "0.00175")
+ self.assertEqual(second["new_attempts"], 0)
+
+ def test_budget_prevents_dispatch(self):
+ run = self.prepare(max_usd="0.001")
+
+ async def forbidden(*args):
+ raise AssertionError("No budget means no request")
+
+ result = asyncio.run(acquisition.collect(run, secret="test-only", request_fn=forbidden))
+ self.assertEqual(result["new_attempts"], 0)
+ self.assertEqual(result["pending"], ["training-1"])
+
+ def test_interrupted_attempt_is_not_silently_repeated(self):
+ run = self.prepare()
+ write(
+ run / "attempts/training-1/attempt-01/intent.json",
+ {
+ "request_sha256": acquisition.digest(numeric.body_for(self.rows[0])),
+ "reservation_usd": str(numeric.reservation(self.rows[0])),
+ },
+ )
+ result = asyncio.run(acquisition.collect(run, secret="test-only", request_fn=self.success))
+ self.assertEqual(result["new_attempts"], 0)
+ self.assertEqual(result["unavailable"], {"training-1": "uncertain_interrupted_attempt"})
+ self.assertGreater(float(result["budget"]["settled_or_unknown_usd"]), 0)
+
+ def test_unparseable_answer_is_retained_without_retry(self):
+ run = self.prepare()
+
+ async def malformed(client, body, secret, seconds):
+ result = await self.success(client, body, secret, seconds)
+ result["payload"]["output"][0]["content"][0]["text"] = "bad JSON"
+ return result
+
+ result = asyncio.run(acquisition.collect(run, secret="test-only", request_fn=malformed))
+ self.assertEqual(result["physical_attempts"], 1)
+ self.assertEqual(result["complete"], 0)
+ self.assertEqual(result["known_list_price_usd"], "0.0015")
+ raw = json.loads((run / "attempts/training-1/attempt-01/responses/training-1.json").read_text())
+ self.assertEqual(raw["output"][0]["content"][0]["text"], "bad JSON")
+
+ def test_retry_policy_never_retries_answer_content(self):
+ self.assertTrue(acquisition.retryable({"error": {"kind": "http_error", "status": 429}}))
+ self.assertTrue(acquisition.retryable({"error": {"kind": "ReadTimeout"}}))
+ self.assertFalse(acquisition.retryable({"error": {"kind": "http_error", "status": 401}}))
+ self.assertFalse(acquisition.retryable({"error": {"kind": "JSONDecodeError"}}))
+ self.assertFalse(acquisition.retryable({"error": {"kind": "incomplete_or_changed_model"}}))
+
+ def test_checkpoint_finishes_before_dispatch(self):
+ run = self.prepare()
+ persisted = []
+
+ async def checkpoint():
+ await asyncio.sleep(0)
+ persisted.append((run / "attempts/training-1/attempt-01/intent.json").is_file())
+
+ async def request(client, body, secret, seconds):
+ self.assertEqual(persisted, [True])
+ return await self.success(client, body, secret, seconds)
+
+ result = asyncio.run(acquisition.collect(run, secret="test-only", request_fn=request, checkpoint=checkpoint))
+ self.assertEqual(result["complete"], 1)
+
+ def test_corrupt_raw_receipt_stops_resume_without_resending(self):
+ run = self.prepare()
+ asyncio.run(acquisition.collect(run, secret="test-only", request_fn=self.success))
+ raw = run / "attempts/training-1/attempt-01/responses/training-1.json"
+ raw.write_text("{}")
+ with self.assertRaisesRegex(ValueError, "raw teacher response"):
+ asyncio.run(acquisition.collect(run, secret="test-only", request_fn=self.success))
+
+ def test_numeric_mode_binds_actual_request_and_preserves_non_numeric_answers(self):
+ run = self.prepare()
+ bodies = []
+
+ async def units(client, body, secret, seconds):
+ bodies.append(body)
+ result = await self.success(client, body, secret, seconds)
+ result["payload"]["output"][0]["content"][0]["text"] = '{"answer":"7 pens"}'
+ return result
+
+ result = asyncio.run(acquisition.collect(run, secret="test-only", request_fn=units))
+ self.assertEqual(bodies, [numeric.body_for(self.rows[0])])
+ self.assertEqual(result["physical_attempts"], 1)
+ record = acquisition.read(run / "attempts/training-1/attempt-01/records/training-1.json")
+ self.assertEqual(record["answer"], "7 pens")
+ self.assertFalse(record["numeric_answer"])
+ self.assertEqual(record["request_sha256"], acquisition.digest(bodies[0]))
+ self.assertEqual(acquisition.read(run / "plan.json")["settings"], numeric.SETTINGS)
+
+ def test_numeric_contract_rejects_units_and_zero_denominators(self):
+ for value in ("42", "-3.5", "7/2", "+.25", "0"):
+ self.assertTrue(numeric.numeric_answer(value))
+ for value in ("7 pens", "$42", "1,234", "1/0", "NaN", "2e3", ""):
+ self.assertFalse(numeric.numeric_answer(value))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benchmark_arithmetic_native.py b/examples/train-a-router/tests/test_benchmark_arithmetic_native.py
new file mode 100644
index 000000000..2a6fdacf6
--- /dev/null
+++ b/examples/train-a-router/tests/test_benchmark_arithmetic_native.py
@@ -0,0 +1,120 @@
+"""Timing boundaries and artifact checks without Torch or model downloads."""
+
+import json
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import Mock, patch
+
+import benchmark_arithmetic_native as benchmark
+import benefit
+from data import file_hash, write
+
+
+def fixture(root):
+ directory, inputs = root / "study/native-2000", root / "inputs.json"
+ rows = [{"id": f"training-{index}", "request": f"Question {index}?"} for index in range(21)]
+ study = {
+ "source_plan_sha256": "a" * 64,
+ "warm_probe_ids": [row["id"] for row in rows],
+ "training_ids": [row["id"] for row in rows],
+ }
+ write(directory.parent / "plan.json", study)
+ write(inputs, {"salt": "arithmetic-warm-v1", "warmup": 1, "timed": 20, "rows": rows,
+ "source_plan_sha256": study["source_plan_sha256"]})
+ plan = {
+ "protocol": benchmark.native.PROTOCOL,
+ "head_plan_sha256": file_hash(directory.parent / "plan.json"),
+ "training_ids": study["training_ids"],
+ }
+ write(directory / "plan.json", plan)
+ checkpoint = directory / "lora"
+ checkpoint.mkdir()
+ (checkpoint / "model.safetensors").write_bytes(b"test weights")
+ (checkpoint / "rl_agent_config.json").write_text("{}")
+ write(directory / "trained.json", benefit.seal({
+ "format": benchmark.native.TRAINED_FORMAT,
+ "plan_sha256": file_hash(directory / "plan.json"),
+ "head_plan_sha256": plan["head_plan_sha256"],
+ "conditional_cost_statistics": {},
+ "checkpoints": {"lora": {
+ "parameters": 12,
+ "files_sha256": {path.name: file_hash(path) for path in checkpoint.iterdir()},
+ }},
+ }))
+ return inputs, directory, rows
+
+
+class ArithmeticNativeTimingTests(unittest.TestCase):
+ def test_only_selected_export_is_needed_and_warmup_is_excluded(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ inputs, directory, rows = fixture(root)
+ router = Mock()
+ router.score_one.return_value = {"score": 0.1, "margins": [1.0, 2.0], "route": "strong"}
+ with (
+ patch.object(benchmark, "_runtime", return_value=(Mock(), Mock(), "cpu", {})),
+ patch.object(benchmark, "WarmRouter", return_value=router) as factory,
+ patch.object(benchmark.finetune, "empty_cache"),
+ patch.object(benchmark.finetune, "synchronize"),
+ ):
+ result = benchmark.run(inputs, directory, root / "output.json", device="cpu")
+ factory.assert_called_once()
+ self.assertEqual(factory.call_args.args[1], directory / "lora")
+ self.assertEqual(factory.call_args.args[-1], {"mode": "threshold", "cutoff": 0.0})
+ self.assertEqual(result["status"], "complete")
+ self.assertEqual(result["models"]["lora"]["n"], 20)
+ self.assertEqual([call.args[0] for call in router.score_one.call_args_list], [rows[20], *rows[:20]])
+ self.assertEqual([row["id"] for row in result["models"]["lora"]["requests"]], result["timed_ids"])
+ self.assertNotIn(result["warmup_id"], result["timed_ids"])
+
+ def test_changed_checkpoint_is_rejected_before_runtime_import(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ inputs, directory, _ = fixture(root)
+ (directory / "lora/model.safetensors").write_bytes(b"different weights")
+ with patch.object(benchmark, "_runtime") as runtime, self.assertRaisesRegex(ValueError, "bytes changed"):
+ benchmark.run(inputs, directory, root / "output.json", device="cpu")
+ runtime.assert_not_called()
+ self.assertFalse((root / "output.json").exists())
+
+ def test_changed_sample_order_or_answer_fields_are_rejected(self):
+ for change in ("order", "answers", "duplicate"):
+ with self.subTest(change=change), tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ inputs, directory, _ = fixture(root)
+ sample = json.loads(inputs.read_text())
+ if change == "order":
+ sample["rows"].reverse()
+ elif change == "answers":
+ sample["rows"][0]["correct"] = True
+ else:
+ sample["rows"][-1] = sample["rows"][0]
+ inputs.write_text(json.dumps(sample))
+ with patch.object(benchmark, "_runtime") as runtime, self.assertRaisesRegex(ValueError, "question-only"):
+ benchmark.run(inputs, directory, root / "output.json", device="cpu")
+ runtime.assert_not_called()
+
+ def test_existing_output_is_preserved_before_any_input_read(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ output = Path(temporary) / "output.json"
+ output.write_text("preserved")
+ with patch.object(benchmark, "inputs_and_checkpoint") as check, self.assertRaises(FileExistsError):
+ benchmark.run("missing", "missing", output, device="cpu")
+ check.assert_not_called()
+ self.assertEqual(output.read_text(), "preserved")
+
+ def test_runtime_failure_retains_incomplete_receipt(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ inputs, directory, _ = fixture(root)
+ with patch.object(benchmark, "_runtime", side_effect=RuntimeError("backend failed")), self.assertRaises(RuntimeError):
+ benchmark.run(inputs, directory, root / "output.json", device="cpu")
+ result = json.loads((root / "output.json").read_text())
+ self.assertEqual(result["status"], "incomplete")
+ self.assertEqual(result["error_type"], "RuntimeError")
+ self.assertEqual(result["models"], {})
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benchmark_outcomes_native.py b/examples/train-a-router/tests/test_benchmark_outcomes_native.py
new file mode 100644
index 000000000..9b37bd9c5
--- /dev/null
+++ b/examples/train-a-router/tests/test_benchmark_outcomes_native.py
@@ -0,0 +1,119 @@
+"""Sampling, warm timing and resident-model behavior without model downloads."""
+
+import tempfile
+import unittest
+from pathlib import Path
+from types import SimpleNamespace
+from unittest.mock import Mock, patch
+
+import test_benefit_outcomes_native as native_fixtures
+
+import benchmark_outcomes_native as benchmark
+import benefit_outcomes as outcomes
+
+
+def source_rows():
+ return [{"id": f"id-{index}", "request": f"problem{index}"} for index in range(512)]
+
+
+def result():
+ return {"score": 0.01, "margins": [1.0, 2.0], "route": "strong", "state_tokens": 3}
+
+
+class NativeWarmBenchmarkTests(unittest.TestCase):
+ def test_hash_sample_is_shared_between_training_sizes_and_source_orders(self):
+ source = source_rows()
+ eligible = [row["id"] for row in source[2:]]
+ timed, warmup = benchmark.select_inputs(source, eligible)
+ expanded, expanded_warmup = benchmark.select_inputs(list(reversed(source)), eligible + ["extra-training"])
+ self.assertEqual(timed, expanded)
+ self.assertEqual(warmup, expanded_warmup)
+ self.assertEqual(len({row["id"] for row in [*timed, warmup]}), 21)
+ self.assertTrue(all(row["id"] in eligible for row in [*timed, warmup]))
+ self.assertNotEqual(timed, source[2:22])
+
+ def test_wrong_population_or_duplicate_source_ids_are_rejected(self):
+ source = source_rows()
+ for pool, eligible in (
+ (source[:-1], [row["id"] for row in source[2:]]),
+ (source, [row["id"] for row in source[:20]]),
+ (source, [row["id"] for row in source[2:]] + [source[2]["id"]]),
+ ([source[0], *source[:-1]], [row["id"] for row in source[2:]]),
+ ):
+ with self.subTest(n=len(pool)), self.assertRaises(ValueError):
+ benchmark.select_inputs(pool, eligible)
+
+ def test_reviewed_exclusions_are_applied_before_shared_hash_sampling(self):
+ source = source_rows()
+ earlier, _ = benchmark.select_inputs(source, [row["id"] for row in source[2:]])
+ excluded = {earlier[0]["id"], earlier[3]["id"], *[row["id"] for row in source[:12]]}
+ eligible = [row["id"] for row in source if row["id"] not in excluded]
+ original = benchmark.select_inputs(source, eligible)
+ expanded = benchmark.select_inputs(source, eligible + ["expanded-only"])
+ self.assertEqual(original, expanded)
+ timed, warmup = original
+ self.assertFalse(excluded & {row["id"] for row in [*timed, warmup]})
+ self.assertEqual(len(timed), 20)
+
+ def test_only_one_warmup_is_excluded_and_all_requests_are_synchronized(self):
+ rows, warmup = source_rows()[:20], source_rows()[20]
+ clock_values, now = [0.0, 3.0], 3.0
+ for index in range(20):
+ clock_values.extend((now, now + (index + 1) / 100))
+ now = clock_values[-1]
+ clock = iter(clock_values)
+ score, synchronize = Mock(side_effect=lambda row: result()), Mock()
+ measured = benchmark.measure_requests(score, rows, warmup, synchronize, clock=lambda: next(clock))
+ self.assertEqual(
+ [call.args[0]["id"] for call in score.call_args_list], [warmup["id"], *[row["id"] for row in rows]]
+ )
+ self.assertEqual(synchronize.call_count, 42)
+ self.assertEqual(measured["warmup_seconds_excluded"], 3.0)
+ self.assertEqual(measured["n"], 20)
+ self.assertAlmostEqual(measured["p50_ms"], 105.0)
+ self.assertAlmostEqual(measured["p95_ms"], 190.5)
+ self.assertEqual([row["id"] for row in measured["requests"]], [row["id"] for row in rows])
+
+ def test_nonfinite_result_cannot_become_a_latency_measurement(self):
+ score = Mock(return_value={**result(), "score": float("nan")})
+ with self.assertRaisesRegex(ValueError, "finite"):
+ benchmark.measure_requests(score, source_rows()[:20], source_rows()[20], lambda: None)
+ self.assertEqual(score.call_count, 1)
+
+ def test_checkpoint_is_loaded_once_for_all_two_question_requests(self):
+ lt = native_fixtures.fake_laya()
+ model = Mock()
+ model.encoder = SimpleNamespace(config=SimpleNamespace(reference_compile=True))
+ model.to.return_value = model
+ model.eval.return_value = model
+ lt.load_checkpoint = Mock(return_value=(model, None, {}))
+
+ def records(_model, _tokenizer, items, _device, _max_len, _head_len, *, batch_size):
+ self.assertEqual(len(items), 2)
+ self.assertEqual(batch_size, 2)
+ return [(0, [0, float(index + 2)], item["target"], 2) for index, item in enumerate(items)]
+
+ lt.calibration_records = Mock(side_effect=records)
+ statistics = outcomes.conditional_costs([[0, 0], [1, 1]], [[2, 4], [1, 3]])
+ router = benchmark.WarmRouter(lt, "checkpoint", "cpu", statistics, {"mode": "threshold", "cutoff": 0})
+ for row in source_rows()[:21]:
+ measured = router.score_one(row)
+ self.assertEqual(measured["margins"], [2.0, 3.0])
+ self.assertEqual(measured["route"], "strong")
+ lt.load_checkpoint.assert_called_once_with("checkpoint")
+ self.assertEqual(lt.calibration_records.call_count, 21)
+ self.assertFalse(model.encoder.config.reference_compile)
+ model.eval.assert_called_once()
+
+ def test_existing_output_is_rejected_before_loading_or_verifying_models(self):
+ with tempfile.TemporaryDirectory() as directory:
+ output = Path(directory) / "result.json"
+ output.write_text("preserved")
+ with patch.object(benchmark.native, "checked_frozen") as check, self.assertRaises(FileExistsError):
+ benchmark.run("missing", "missing", "missing", output)
+ check.assert_not_called()
+ self.assertEqual(output.read_text(), "preserved")
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benefit.py b/examples/train-a-router/tests/test_benefit.py
new file mode 100644
index 000000000..21a1fca5e
--- /dev/null
+++ b/examples/train-a-router/tests/test_benefit.py
@@ -0,0 +1,434 @@
+"""Synthetic outcome learning, paper separation and pre-test freeze checks."""
+
+import gzip
+import io
+import json
+import tempfile
+import unittest
+from contextlib import redirect_stdout
+from pathlib import Path
+from unittest.mock import patch
+
+import numpy as np
+
+import benefit
+import solver_policies
+from data import digest, file_hash
+from serving import INSTRUCTION, MODEL, REVISION
+
+
+def save(path, document):
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(json.dumps(document, allow_nan=False) + "\n")
+ return path
+
+
+def outcomes(split, count):
+ return [
+ {
+ "id": f"{split}-{i}",
+ "cheap_correct": i % 4 < 2,
+ "strong_correct": True,
+ "cheap_credits": 10,
+ "strong_credits": 100,
+ }
+ for i in range(count)
+ ]
+
+
+class BenefitLearningTests(unittest.TestCase):
+ def test_signed_targets_preserve_cheap_only_rescues_and_both_failures(self):
+ rows = [
+ {"id": str(i), "cheap_correct": cheap, "strong_correct": strong}
+ for i, (cheap, strong) in enumerate(((True, False), (True, True), (False, False), (False, True)))
+ ]
+ np.testing.assert_array_equal(benefit.signed_targets(rows), [-1, 0, 0, 1])
+ rows[0]["cheap_correct"] = 1
+ with self.assertRaises(TypeError):
+ benefit.signed_targets(rows)
+
+ def test_cv_excludes_every_held_out_paper_and_refits_all_training_rows(self):
+ rows = outcomes("train", 20)
+ matrix = np.array([[i, i % 4, 1] for i in range(20)], dtype=float)
+ groups = [f"paper-{i // 4}" for i in range(20)]
+ calls = []
+ ridge = benefit.policies.ridge
+
+ def record(x, y, penalty):
+ calls.append((x.copy(), y.copy(), penalty))
+ return ridge(x, y, penalty)
+
+ with patch.object(benefit.policies, "ridge", side_effect=record):
+ head = benefit.fit_head(matrix, rows, groups)
+ self.assertEqual(len(calls), 4 * 5 + 1)
+ folds = np.array(head["folds"])
+ targets = benefit.signed_targets(rows)
+ for index, penalty in enumerate(benefit.PROTOCOL["penalties"]):
+ for fold in range(5):
+ x, y, observed_penalty = calls[5 * index + fold]
+ np.testing.assert_array_equal(x, matrix[folds != fold])
+ np.testing.assert_array_equal(y, targets[folds != fold])
+ self.assertEqual(observed_penalty, penalty)
+ fit_groups = {groups[int(row[0])] for row in x}
+ held_groups = {group for group, assignment in zip(groups, folds, strict=True) if assignment == fold}
+ self.assertFalse(fit_groups & held_groups)
+ np.testing.assert_array_equal(calls[-1][0], matrix)
+ np.testing.assert_array_equal(calls[-1][1], targets)
+ self.assertEqual(head["penalty"], calls[-1][2])
+ self.assertEqual(benefit.group_folds(groups), head["folds"])
+
+ def test_exact_cv_ties_choose_stronger_regularization_and_predictions_stay_finite(self):
+ rows = outcomes("train", 10)
+ for row in rows:
+ row["cheap_correct"] = row["strong_correct"]
+ head = benefit.fit_head(np.zeros((10, 3)), rows, [f"paper-{i // 2}" for i in range(10)])
+ self.assertEqual(head["penalty"], 10)
+ self.assertEqual(head["weights"], [0, 0, 0])
+ self.assertEqual(head["bias"], 0)
+ with self.assertRaisesRegex(ValueError, "five distinct"):
+ benefit.fit_head(np.zeros((10, 3)), rows, ["one-paper"] * 10)
+ with self.assertRaisesRegex(ValueError, "finite"):
+ benefit.fit_head(np.full((10, 3), np.nan), rows, [f"paper-{i // 2}" for i in range(10)])
+
+
+class BenefitLifecycleTests(unittest.TestCase):
+ def setUp(self):
+ temporary = tempfile.TemporaryDirectory()
+ self.addCleanup(temporary.cleanup)
+ self.root = Path(temporary.name)
+ self.baseline, self.original_dir, self.training, self.run = (
+ self.root / name for name in ("baseline", "original", "training", "head")
+ )
+ # Exercise the full lifecycle on small vectors without downloading models.
+ protocol = patch.dict(
+ benefit.PROTOCOL,
+ {"training_pool_rows": 20, "training_rows": 19, "excluded_training_ids": ["train-0"], "dimensions": 3},
+ )
+ protocol.start()
+ self.addCleanup(protocol.stop)
+ verify = patch.object(benefit.evidence, "verify")
+ verify.start()
+ self.addCleanup(verify.stop)
+ self.dataset = {
+ "splits": {
+ split: [
+ {
+ "id": f"{split}-{i}",
+ "request": f"{split} problem number {i}",
+ "family": f"{split}-family-{i}",
+ "paper_group": f"{split}-paper-{i // 4}",
+ "label": i % 2,
+ "year": {"train": 2017, "validation": 2018, "calibration": 2019, "test": 2020}[split],
+ }
+ for i in range(count)
+ ]
+ for split, count in (("train", 20), ("validation", 4), ("calibration", 8), ("test", 8))
+ }
+ }
+ save(self.baseline / "data.json", self.dataset)
+ save(self.baseline / "manifest.json", {})
+ for directory, splits in (
+ ("development-embeddings", ("train", "validation", "calibration")),
+ ("test-embeddings", ("test",)),
+ ):
+ rows = [row for split in splits for row in self.dataset["splits"][split]]
+ path = self.baseline / directory
+ save(
+ path / "plan.json",
+ {
+ "model": MODEL,
+ "revision": REVISION,
+ "instruction": INSTRUCTION,
+ "ids": [row["id"] for row in rows],
+ "input_sha256": {row["id"]: digest(row["request"]) for row in rows},
+ },
+ )
+ with gzip.open(path / "vectors.jsonl.gz", "wt") as handle:
+ for row in rows:
+ index = int(row["id"].rsplit("-", 1)[1])
+ handle.write(
+ json.dumps({"id": row["id"], "vector": [1, index % 4 - 1.5, 0.01 * (index // 4)]}) + "\n"
+ )
+ self.original_plan = save(self.original_dir / "plan.json", {"settings_sha256": "1" * 64})
+ score_splits = {split: self.dataset["splits"][split] for split in ("calibration", "test")}
+ scores = solver_policies.seal(
+ {
+ "format": solver_policies.SCORE_FORMAT,
+ "dataset_sha256": digest(self.dataset),
+ "ids": {split: [row["id"] for row in rows] for split, rows in score_splits.items()},
+ "input_sha256": {
+ split: {row["id"]: digest(row["request"]) for row in rows} for split, rows in score_splits.items()
+ },
+ "stages": {
+ name: {
+ "family": "embedding" if name.startswith("ridge") else "native",
+ "original_threshold": 0.5,
+ "scores": {split: [i % 2 for i in range(len(rows))] for split, rows in score_splits.items()},
+ }
+ for name in solver_policies.STAGES
+ },
+ }
+ )
+ self.grading = {
+ "grader_source_sha256": "2" * 64,
+ "gold_review_source_sha256": "3" * 64,
+ "gold_review_sha256": "4" * 64,
+ }
+ self.original = save(
+ self.original_dir / "policies-frozen.json",
+ solver_policies.seal(
+ {
+ "format": solver_policies.FROZEN_FORMAT,
+ "analysis_mode": "cost",
+ "plan_sha256": file_hash(self.original_plan),
+ "source_scores": scores,
+ "source_code_sha256": {
+ name: file_hash(Path(benefit.__file__).with_name(name))
+ for name in ("solver_policies.py", "solver_report.py")
+ },
+ "calibration_grades_sha256": "5" * 64,
+ "calibration_rows": outcomes("calibration", 8),
+ "embedding_calibration_credits": [2] * 8,
+ "embedding_calibration_credits_sha256": "6" * 64,
+ "credit_usd": 1e-5,
+ **self.grading,
+ }
+ ),
+ )
+ self.eligible_rows = self.dataset["splits"]["train"][1:]
+ self.eligibility = save(
+ self.training / "eligibility.json",
+ {
+ "format": "sie-benefit-training-eligibility-v1",
+ "original_pool_ids": [row["id"] for row in self.dataset["splits"]["train"]],
+ "eligible_train_ids": [row["id"] for row in self.eligible_rows],
+ "exclusions": [{"id": "train-0", "reason": "Missing choices leave the source question ambiguous"}],
+ "source_only": True,
+ "decided_before_training_outcomes": True,
+ "source_revision": benefit.SOURCE_REVISION,
+ "source_hashes": benefit.SOURCE_HASHES,
+ "dataset_sha256": digest(self.dataset),
+ "training_gold_review_source_sha256": "9" * 64,
+ },
+ )
+ inputs = {"train": [{key: row[key] for key in ("id", "request")} for row in self.eligible_rows]}
+ self.training_inputs = save(self.training / "inputs.json", inputs)
+ self.training_plan = save(
+ self.training / "plan.json",
+ {
+ "dataset_sha256": digest(self.dataset),
+ "settings_sha256": "1" * 64,
+ "inputs_sha256": digest(inputs),
+ "source_counts": {"train": 20},
+ "counts": {"train": 19},
+ "eligible_train_ids": [row["id"] for row in self.eligible_rows],
+ "excluded_train_ids": ["train-0"],
+ "eligibility_sha256": file_hash(self.eligibility),
+ **self.grading,
+ "gold_review_sha256": "7" * 64,
+ "training_gold_rules_sha256": "8" * 64,
+ "training_gold_review_source_sha256": "9" * 64,
+ },
+ )
+ self.training_grades = save(
+ self.training / "grades-train.json",
+ {
+ "split": "train",
+ "plan_sha256": file_hash(self.training_plan),
+ **self.grading,
+ "gold_review_sha256": "7" * 64,
+ "training_gold_rules_sha256": "8" * 64,
+ "training_gold_review_source_sha256": "9" * 64,
+ "rows": outcomes("train", 20)[1:],
+ },
+ )
+ plan = benefit.prepare(self.baseline, self.original, self.eligibility, attest_test_outcomes_unopened=True)
+ save(self.run / "plan.json", plan)
+
+ def freeze(self):
+ frozen = benefit.freeze(
+ self.run, self.baseline, self.original, self.training_plan, self.training_inputs, self.training_grades
+ )
+ save(self.run / "frozen.json", frozen)
+ return frozen
+
+ def test_freeze_reads_training_and_calibration_only_even_with_held_test_marker(self):
+ marker = save(self.original_dir / "grades-test.json", {"held": "test outcomes remain unopened"})
+ read = benefit.read
+ opened = []
+
+ def guarded(path):
+ opened.append(Path(path))
+ if Path(path) == marker:
+ self.fail("Freeze attempted to read test answers")
+ return read(path)
+
+ with patch.object(benefit, "read", side_effect=guarded):
+ frozen = self.freeze()
+ self.assertNotIn(marker, opened)
+ self.assertEqual(frozen["head"]["training_ids"], [row["id"] for row in self.eligible_rows])
+ self.assertEqual(frozen["training_pool_rows"], 20)
+ self.assertEqual(frozen["training_rows"], 19)
+ self.assertEqual(len(frozen["plan"]["populations"]["train_pool"]), 20)
+ self.assertEqual(frozen["plan"]["eligibility_sha256"], file_hash(self.eligibility))
+ self.assertEqual(frozen["head"]["folds"], frozen["plan"]["training_folds"])
+ self.assertEqual(tuple(frozen["selections"]), ("1", "0", "2", "5"))
+ self.assertEqual(frozen["training_grade_provenance"]["training_gold_rules_sha256"], "8" * 64)
+ self.assertEqual(frozen["training_artifacts_sha256"]["grades"], file_hash(self.training_grades))
+ self.assertEqual(len(frozen["scores"]["test"]), 8)
+ self.assertFalse((self.run / "test-access.json").exists())
+
+ def test_changed_training_ids_settings_and_review_hashes_fail_before_fit(self):
+ original_plan, original_grades = benefit.read(self.training_plan), benefit.read(self.training_grades)
+ changes = [
+ ("grades", {"rows": list(reversed(original_grades["rows"]))}),
+ ("grades", {"grader_source_sha256": "a" * 64}),
+ ("grades", {"training_gold_rules_sha256": None}),
+ ("grades", {"training_gold_rules_sha256": "e" * 64}),
+ ("grades", {"gold_review_sha256": "e" * 64}),
+ ("plan", {"settings_sha256": "b" * 64}),
+ ("plan", {"eligibility_sha256": "c" * 64}),
+ ("plan", {"excluded_train_ids": ["train-0", "train-1"]}),
+ ("plan", {"training_gold_review_source_sha256": "e" * 64}),
+ ]
+ for kind, change in changes:
+ with self.subTest(kind=kind, change=list(change)):
+ save(self.training_plan, original_plan)
+ save(self.training_grades, original_grades)
+ save(
+ self.training_plan if kind == "plan" else self.training_grades,
+ (original_plan if kind == "plan" else original_grades) | change,
+ )
+ if kind == "plan":
+ save(self.training_grades, original_grades | {"plan_sha256": file_hash(self.training_plan)})
+ with patch.object(benefit, "fit_head") as fit:
+ with self.assertRaises(ValueError):
+ self.freeze()
+ fit.assert_not_called()
+
+ def test_modified_cached_features_are_rejected_before_fitting(self):
+ path = self.baseline / "development-embeddings/vectors.jsonl.gz"
+ with path.open("ab") as handle:
+ handle.write(b"changed")
+ with patch.object(benefit, "fit_head") as fit:
+ with self.assertRaisesRegex(ValueError, "vectors changed"):
+ self.freeze()
+ fit.assert_not_called()
+
+ def test_report_cannot_open_test_grades_without_matching_access_receipt(self):
+ self.freeze()
+ test = self.root / "must-not-open-test.json"
+ read = benefit.read
+
+ def guarded(path):
+ if Path(path) == test:
+ self.fail("Test opened before access chain was validated")
+ return read(path)
+
+ with patch.object(benefit, "read", side_effect=guarded):
+ with self.assertRaises(FileNotFoundError):
+ benefit.report(self.run, test)
+ access = benefit.authorize_test(self.run)
+ access.pop("sha256")
+ save(self.run / "test-access.json", benefit.seal(access | {"frozen_sha256": "a" * 64}))
+ with self.assertRaisesRegex(ValueError, "Test-access receipt"):
+ benefit.report(self.run, test)
+
+ def test_report_keeps_frozen_policy_when_test_quality_loss_exceeds_target(self):
+ frozen = self.freeze()
+ save(self.run / "test-access.json", benefit.authorize_test(self.run))
+ rows = outcomes("test", 8)
+ for row in rows:
+ row["cheap_correct"] = False
+ grades = save(
+ self.root / "test-grades.json",
+ {
+ "split": "test",
+ "plan_sha256": file_hash(self.original_plan),
+ "policies_frozen_sha256": file_hash(self.original),
+ **self.grading,
+ "rows": rows,
+ },
+ )
+ select = benefit.solver_report.select_cutoff
+
+ def only_calibration(rows, *args, **kwargs):
+ self.assertTrue(all(row["id"].startswith("calibration-") for row in rows))
+ return select(rows, *args, **kwargs)
+
+ with patch.object(benefit.solver_report, "select_cutoff", side_effect=only_calibration):
+ measured = benefit.report(self.run, grades)
+ self.assertEqual(measured["policies"]["1"]["cutoff"], frozen["selections"]["1"]["cutoff"])
+ self.assertGreater(measured["policies"]["1"]["metrics"]["gap_vs_strong_pp"], 1)
+ self.assertNotIn("net_total_cost_usd_per1000", measured["policies"]["1"]["metrics"])
+ credits = save(
+ self.root / "test-router-credits.json",
+ {
+ "plan_sha256": file_hash(self.original_plan),
+ "model": MODEL,
+ "revision": REVISION,
+ "batch_size": 1,
+ "test": [{"id": row["id"], "credits": 2} for row in rows],
+ },
+ )
+ priced = benefit.report(self.run, grades, credits)["policies"]["1"]["metrics"]
+ self.assertAlmostEqual(priced["router_cost_usd_per1000"], 0.02)
+ self.assertAlmostEqual(priced["net_total_cost_usd_per1000"], priced["total_solver_cost_usd_per1000"] + 0.02)
+ save(grades, benefit.read(grades) | {"gold_review_sha256": "c" * 64})
+ with self.assertRaisesRegex(ValueError, "grading experiment"):
+ benefit.report(self.run, grades)
+
+ def test_resealed_cutoff_tamper_and_second_freeze_are_rejected(self):
+ frozen = self.freeze()
+ with self.assertRaises(FileExistsError):
+ self.freeze()
+ frozen.pop("sha256")
+ frozen["selections"]["1"]["cutoff"] += 1
+ save(self.run / "frozen.json", benefit.seal(frozen))
+ with self.assertRaisesRegex(ValueError, "calibration-only selection"):
+ benefit.authorize_test(self.run)
+
+ def test_prepare_requires_unopened_test_attestation(self):
+ with self.assertRaisesRegex(ValueError, "explicit assertion"):
+ benefit.prepare(self.baseline, self.original, self.eligibility, attest_test_outcomes_unopened=False)
+
+ def test_eligibility_cannot_drop_more_rows_reorder_or_follow_training_outcomes(self):
+ original = benefit.read(self.eligibility)
+ for change in (
+ {"source_only": False},
+ {"decided_before_training_outcomes": False},
+ {"eligible_train_ids": original["eligible_train_ids"][1:]},
+ {"eligible_train_ids": list(reversed(original["eligible_train_ids"]))},
+ {"exclusions": original["exclusions"] + [{"id": "train-1", "reason": "Additional drop"}]},
+ {"source_revision": "other"},
+ ):
+ with self.subTest(change=list(change)):
+ save(self.eligibility, original | change)
+ with self.assertRaisesRegex(ValueError, "Eligibility"):
+ benefit.prepare(self.baseline, self.original, self.eligibility, attest_test_outcomes_unopened=True)
+
+ def test_existing_cli_output_is_preserved_before_loading_data(self):
+ output = self.run / "plan.json"
+ before = output.read_bytes()
+ args = [
+ "benefit.py",
+ "prepare",
+ "--run",
+ str(self.run),
+ "--baseline",
+ "absent",
+ "--original-frozen",
+ "absent",
+ "--eligibility",
+ "absent",
+ "--attest-test-outcomes-unopened",
+ ]
+ with patch("sys.argv", args), patch.object(benefit, "prepare") as prepare, redirect_stdout(io.StringIO()):
+ with self.assertRaises(FileExistsError):
+ benefit.main()
+ prepare.assert_not_called()
+ self.assertEqual(output.read_bytes(), before)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benefit_collect.py b/examples/train-a-router/tests/test_benefit_collect.py
new file mode 100644
index 000000000..2cb85639c
--- /dev/null
+++ b/examples/train-a-router/tests/test_benefit_collect.py
@@ -0,0 +1,157 @@
+"""Check that training collection never repeats a completed paid call."""
+
+import copy
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import benefit_collect as collector
+import solver
+from data import digest, write
+
+
+class MetadataClient:
+ def __init__(self, **kwargs):
+ pass
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *args):
+ return False
+
+ def get_model(self, model):
+ return {"name": model, "revision": next(a["revision"] for a in solver.ARMS.values() if a["model"] == model)}
+
+
+class BenefitCollectionTests(unittest.TestCase):
+ def setUp(self):
+ self.temp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.temp.cleanup)
+ self.directory = Path(self.temp.name)
+ self.rows = [{"id": "row0", "request": "Compute 1+1."}, {"id": "row1", "request": "Compute 2+2."}]
+ self.settings = copy.deepcopy(solver.SETTINGS)
+ write(self.directory / "plan.json", {"source": "synthetic"})
+ self.sent = []
+ checked = patch.object(collector, "checked_training", return_value=({}, {"train": self.rows}, self.settings))
+ metadata = patch.object(collector, "SingleSendClient", MetadataClient)
+ stage = patch.object(collector, "stage_call", side_effect=self.success)
+ for item in (checked, metadata, stage):
+ item.start()
+ self.addCleanup(item.stop)
+
+ def success(self, row, arm, settings, phase, messages, self_hosted):
+ self.sent.append((row["id"], arm, phase, messages))
+ answer = '{"answer":"4"}' if phase == "final" else ("\\boxed{2}" if row["id"] == "row0" else "Some work")
+ call = {
+ "messages_sha256": digest(messages),
+ "answer": answer,
+ "finish_reason": "stop",
+ "response_valid": True,
+ "billing_known": True,
+ "billing": "self_hosted" if self_hosted else "cloud",
+ "receipt": {"credits": None if self_hosted else 5},
+ "elapsed_seconds": 0.1,
+ "execution_revision": "fixture",
+ "tokens": {"completion_tokens": 10},
+ }
+ return {
+ "id": row["id"],
+ "arm": arm,
+ "phase": phase,
+ "request_sha256": digest(row["request"]),
+ "settings_sha256": digest(settings),
+ "messages_sha256": digest(messages),
+ "status": "complete",
+ "call": call,
+ "observed_credits": 0 if self_hosted else 5,
+ "execution_unknown": False,
+ "safe_to_resume": False,
+ "http_failures": [],
+ }
+
+ def collect(self, arm="cheap", phase="initial", **kwargs):
+ return collector.collect(self.directory, arm, phase, workers=1, max_seconds=60, self_hosted=True, **kwargs)
+
+ def test_pilot_resume_and_finalization_preserve_initial_calls(self):
+ self.collect(limit=1)
+ self.collect(resume=True)
+ self.collect(phase="final", resume=True)
+ self.collect(phase="final", resume=True)
+ self.assertEqual(
+ [(i, a, p) for i, a, p, _ in self.sent],
+ [
+ ("row0", "cheap", "initial"),
+ ("row1", "cheap", "initial"),
+ ("row1", "cheap", "final"),
+ ],
+ )
+ record = solver.read(self.directory / "benefit-arms/cheap/records/row1-cheap.json")
+ self.assertEqual(record["answer"], "\\boxed{4}")
+ self.assertEqual(len(record["calls"]), 2)
+ self.assertEqual(
+ self.sent[-1][-1], collector.messages_for(self.rows[1], "cheap", self.settings, record["calls"][0])
+ )
+
+ def test_interrupted_intent_requires_explicit_bounded_unknown_retry(self):
+ self.collect(limit=1)
+ output = self.directory / "benefit-arms/cheap"
+ write(output / "attempts/row1/initial-0001.intent.json", {"id": "row1"})
+ with self.assertRaisesRegex(ValueError, "Attempt ceiling"):
+ self.collect(resume=True)
+ with self.assertRaisesRegex(ValueError, "execution is unknown"):
+ self.collect(resume=True, max_attempts=2)
+ self.collect(resume=True, max_attempts=2, retry_unknown=True)
+ self.assertTrue((output / "attempts/row1/initial-0001.intent.json").exists())
+ self.assertTrue((output / "attempts/row1/initial-0002.json").exists())
+ self.assertEqual(len(self.sent), 2)
+
+ def test_failure_stops_new_calls_and_retains_receipt(self):
+ def failure(*args):
+ result = self.success(*args)
+ result.update(status="incomplete", execution_unknown=True, error={"type": "RuntimeError"})
+ return result
+
+ with (
+ patch.object(collector, "stage_call", side_effect=failure),
+ self.assertRaisesRegex(RuntimeError, "Collection phase incomplete"),
+ ):
+ self.collect()
+ self.assertEqual(len(self.sent), 1)
+ receipt = self.directory / "benefit-arms/cheap/attempts/row0/initial-0001.json"
+ self.assertEqual(solver.read(receipt)["call"]["answer"], "\\boxed{2}")
+ with self.assertRaisesRegex(ValueError, "Attempt ceiling"):
+ self.collect(resume=True)
+ self.assertEqual(len(self.sent), 1)
+
+ def test_gather_requires_both_arms_and_retains_all_attempts(self):
+ for arm in solver.ARMS:
+ self.collect(arm=arm)
+ self.collect(arm=arm, phase="final", resume=True)
+ summary = collector.gather(self.directory)
+ self.assertEqual(summary["completed"], 4)
+ self.assertEqual(summary["status"], "complete")
+ self.assertFalse(summary["acquisition_cost_complete"])
+ self.assertEqual(len(summary["receipt_sha256"]), 12)
+ collector.gather(self.directory)
+ self.assertEqual(len(self.sent), 6)
+ plan = solver.read(self.directory / "answers-train/collection-plan.json")
+ solver.collection_binding(plan, self.directory, "train", self.rows, self.settings, None, True)
+
+ def test_changed_endpoint_cannot_resume_existing_arm(self):
+ self.collect(limit=1)
+ with (
+ patch.dict("os.environ", {"SIE_BASE_URL": "http://different.example"}),
+ self.assertRaisesRegex(ValueError, "identical inputs, endpoint and runtime"),
+ ):
+ self.collect(resume=True)
+ self.assertEqual(len(self.sent), 1)
+
+ def test_truncated_box_still_requires_finalizer(self):
+ self.assertTrue(collector.needs_final({"answer": "\\boxed{2}", "finish_reason": "length"}))
+ self.assertFalse(collector.needs_final({"answer": "\\boxed{2}", "finish_reason": "stop"}))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benefit_costs.py b/examples/train-a-router/tests/test_benefit_costs.py
new file mode 100644
index 000000000..97da1e707
--- /dev/null
+++ b/examples/train-a-router/tests/test_benefit_costs.py
@@ -0,0 +1,216 @@
+import json
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import benefit_costs
+import solver
+from data import digest, file_hash, write
+
+
+class CostProxyTests(unittest.TestCase):
+ def setUp(self):
+ self.temp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.temp.cleanup)
+ self.root = Path(self.temp.name)
+ self.answers = self.root / "answers-train"
+ self.answers.mkdir()
+ self.inputs = {
+ "train": [{"id": identity, "request": "Synthetic " + identity} for identity in ("first", "second")]
+ }
+ write(self.root / "inputs.json", self.inputs)
+ write(self.root / "settings.json", solver.SETTINGS)
+ write(
+ self.root / "plan.json", {"inputs_sha256": digest(self.inputs), "settings_sha256": digest(solver.SETTINGS)}
+ )
+ write(
+ self.answers / "collection-plan.json",
+ {
+ "ids": [row["id"] for row in self.inputs["train"]],
+ "split": "train",
+ "plan_sha256": file_hash(self.root / "plan.json"),
+ "settings_sha256": digest(solver.SETTINGS),
+ "policies_frozen_sha256": None,
+ "self_hosted": True,
+ },
+ )
+ write(self.answers / "collection.json", {"status": "complete", "completed": 4, "failed": 0})
+ rows = []
+ for identity, correct in (("first", True), ("second", False)):
+ row = {"id": identity}
+ for arm, model in benefit_costs.MODELS.items():
+ call = {
+ "tokens": {
+ "prompt_tokens": 1 if arm == "cheap" else 5,
+ "completion_tokens": 1 if arm == "cheap" else 5,
+ },
+ "finish_reason": "stop",
+ "billing": "self_hosted",
+ }
+ record = {
+ "id": identity,
+ "arm": arm,
+ "model": model,
+ "status": "complete",
+ "has_unknown_charges": False,
+ "calls": [call, call] if arm == "cheap" else [call],
+ "request_sha256": digest("Synthetic " + identity),
+ "settings_sha256": digest(solver.SETTINGS),
+ "catalog_revision": solver.SETTINGS["arms"][arm]["revision"],
+ "answer": "Synthetic answer",
+ "observed_credits": 0,
+ "credits": None,
+ }
+ path = self.answers / f"{identity}-{arm}.json"
+ write(path, record)
+ row[f"{arm}_record_sha256"] = file_hash(path)
+ row[f"{arm}_correct"] = correct
+ rows.append(row)
+ write(
+ self.root / "grades-train.json",
+ {
+ "split": "train",
+ "plan_sha256": file_hash(self.root / "plan.json"),
+ "collection_summary_sha256": file_hash(self.answers / "collection.json"),
+ "collection_plan_sha256": file_hash(self.answers / "collection-plan.json"),
+ "rows": rows,
+ },
+ )
+ items = []
+ for arm, model in benefit_costs.MODELS.items():
+ for unit, rate in (
+ ("input_tokens", 2 if arm == "cheap" else 4),
+ ("output_tokens", 3 if arm == "cheap" else 6),
+ ):
+ items.append(
+ {
+ "operation": "generate",
+ "model": model,
+ "profile": "default",
+ "unit": unit,
+ "region": "us",
+ "currency": "usd",
+ "usd_per_unit": {"numerator": str(rate), "denominator": "1000000"},
+ }
+ )
+ self.prices = self.root / "prices.json"
+ write(
+ self.prices,
+ {"currency": "usd", "pricing_version": "fixture-v1", "rate_book_sha256": "a" * 64, "items": items},
+ )
+
+ def test_round_each_physical_call_and_include_finalizer(self):
+ result = benefit_costs.build(self.root, self.prices)
+ self.assertEqual(result["n"], 2)
+ self.assertEqual(result["rows"][0]["cheap_credits_proxy"], 2)
+ self.assertEqual(result["rows"][0]["strong_credits_proxy"], 5)
+ self.assertEqual(result["conditional_means"]["cheap_wrong"]["mean_credits_proxy"], 2)
+ self.assertEqual(result["price_source"]["sha256"], file_hash(self.prices))
+
+ def test_changed_raw_answer_is_rejected(self):
+ path = self.answers / "first-cheap.json"
+ record = benefit_costs.read(path)
+ record["calls"][0]["tokens"]["completion_tokens"] += 1
+ path.unlink()
+ write(path, record)
+ with self.assertRaisesRegex(ValueError, "changed after grading"):
+ benefit_costs.build(self.root, self.prices)
+
+ def test_ambiguous_or_missing_tokens_are_rejected(self):
+ rates, _ = benefit_costs.rates_from_snapshot(benefit_costs.read(self.prices))
+ record = benefit_costs.read(self.answers / "first-cheap.json")
+ record["has_unknown_charges"] = True
+ with self.assertRaisesRegex(ValueError, "unambiguous"):
+ benefit_costs.physical_costs(record, rates["cheap"])
+ record["has_unknown_charges"] = False
+ record["calls"][0]["tokens"]["completion_tokens"] = None
+ with self.assertRaisesRegex(ValueError, "integer token"):
+ benefit_costs.physical_costs(record, rates["cheap"])
+
+ def test_ambiguous_price_rows_are_rejected(self):
+ prices = benefit_costs.read(self.prices)
+ prices["items"].append(dict(prices["items"][0]))
+ with self.assertRaisesRegex(ValueError, "exactly one"):
+ benefit_costs.rates_from_snapshot(prices)
+
+ def filtered_fixture(self):
+ """Add one collected/excluded row without creating readable answer files."""
+ self.inputs["train"].insert(1, {"id": "excluded", "request": "Source missing required information"})
+ (self.root / "inputs.json").write_text(json.dumps(self.inputs))
+ plan = benefit_costs.read(self.root / "plan.json") | {"inputs_sha256": digest(self.inputs)}
+ (self.root / "plan.json").write_text(json.dumps(plan))
+ binding = benefit_costs.read(self.answers / "collection-plan.json") | {
+ "ids": ["first", "excluded", "second"],
+ "plan_sha256": file_hash(self.root / "plan.json"),
+ }
+ (self.answers / "collection-plan.json").write_text(json.dumps(binding))
+ summary = benefit_costs.read(self.answers / "collection.json") | {"completed": 6}
+ (self.answers / "collection.json").write_text(json.dumps(summary))
+ grades = benefit_costs.read(self.root / "grades-train.json") | {
+ "plan_sha256": file_hash(self.root / "plan.json"),
+ "collection_plan_sha256": file_hash(self.answers / "collection-plan.json"),
+ "collection_summary_sha256": file_hash(self.answers / "collection.json"),
+ "collected_ids": ["first", "excluded", "second"],
+ "excluded_ids": ["excluded"],
+ }
+ path = self.root / "reviewed-grades.json"
+ write(path, grades)
+ return path
+
+ def test_filtered_grades_price_only_retained_rows_and_preserve_default_file(self):
+ original_hash = file_hash(self.root / "grades-train.json")
+ grades = self.filtered_fixture()
+ real_read = benefit_costs.read
+
+ def reject_excluded(path):
+ self.assertNotIn("excluded-", Path(path).name)
+ return real_read(path)
+
+ with patch.object(benefit_costs, "read", side_effect=reject_excluded):
+ result = benefit_costs.build(self.root, self.prices, grades_path=grades)
+ self.assertEqual(result["n"], 2)
+ self.assertEqual(result["collected_ids"], ["first", "excluded", "second"])
+ self.assertEqual(result["excluded_ids"], ["excluded"])
+ self.assertEqual(
+ result["conditional_means"]["cheap_correct"], {"n": 1, "sum_credits_proxy": 2, "mean_credits_proxy": 2}
+ )
+ self.assertEqual(result["conditional_means"]["strong_wrong"]["mean_credits_proxy"], 5)
+ self.assertEqual(result["training_grades_sha256"], file_hash(grades))
+ self.assertEqual(file_hash(self.root / "grades-train.json"), original_hash)
+
+ def test_filtered_partition_and_complete_original_count_are_required(self):
+ grades = self.filtered_fixture()
+ original = benefit_costs.read(grades)
+ cases = (
+ original | {"excluded_ids": []},
+ original | {"excluded_ids": ["excluded", "excluded"]},
+ original | {"collected_ids": ["first", "second"]},
+ original | {"rows": list(reversed(original["rows"]))},
+ )
+ for invalid in cases:
+ grades.write_text(json.dumps(invalid))
+ with self.assertRaisesRegex(ValueError, "partition"):
+ benefit_costs.build(self.root, self.prices, grades_path=grades)
+ grades.write_text(json.dumps(original))
+ summary = benefit_costs.read(self.answers / "collection.json") | {"completed": 4}
+ (self.answers / "collection.json").write_text(json.dumps(summary))
+ original["collection_summary_sha256"] = file_hash(self.answers / "collection.json")
+ grades.write_text(json.dumps(original))
+ with self.assertRaisesRegex(ValueError, "complete paired collection"):
+ benefit_costs.build(self.root, self.prices, grades_path=grades)
+
+ def test_resealed_raw_record_with_wrong_request_is_rejected(self):
+ path = self.answers / "first-cheap.json"
+ record = benefit_costs.read(path) | {"request_sha256": digest("Another question")}
+ path.write_text(json.dumps(record))
+ grades_path = self.root / "grades-train.json"
+ grades = benefit_costs.read(grades_path)
+ grades["rows"][0]["cheap_record_sha256"] = file_hash(path)
+ grades_path.write_text(json.dumps(grades))
+ with self.assertRaisesRegex(ValueError, "frozen request"):
+ benefit_costs.build(self.root, self.prices)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benefit_data.py b/examples/train-a-router/tests/test_benefit_data.py
new file mode 100644
index 000000000..8776317c8
--- /dev/null
+++ b/examples/train-a-router/tests/test_benefit_data.py
@@ -0,0 +1,236 @@
+"""Source-only grading and input-boundary checks; no solver outputs or calls."""
+
+import importlib.util
+import io
+import json
+import tempfile
+import unittest
+from contextlib import redirect_stdout
+from pathlib import Path
+from unittest import mock
+
+import benefit_data as benefit
+import solver
+from data import digest, file_hash, render_request, write
+
+
+@unittest.skipUnless(importlib.util.find_spec("math_verify"), "Install the evaluation dependency group")
+class TrainingReviewTests(unittest.TestCase):
+ def grade(self, identity, answer):
+ return benefit.grade_training(identity, benefit.RULES[identity][0], "\\boxed{" + answer + "}")
+
+ def test_all_canonical_rules_grade_and_source_text_is_bound(self):
+ for identity, (original, canonical, _) in benefit.RULES.items():
+ with self.subTest(identity=identity):
+ self.assertTrue(self.grade(identity, canonical)["correct"])
+ with self.assertRaisesRegex(ValueError, "original source"):
+ benefit.grade_training(identity, original + " changed", canonical)
+ self.assertNotIn("e2h-amc-eval-661", benefit.RULES)
+
+ def test_two_clear_source_transcription_corrections(self):
+ for identity, good, wrong in [
+ ("e2h-amc-train-966", "251750000", "2517500"),
+ ("e2h-amc-eval-2430", r"\frac27", r"\sqrt{\frac27}"),
+ ]:
+ self.assertTrue(self.grade(identity, good)["correct"])
+ self.assertFalse(self.grade(identity, wrong)["correct"])
+
+ def test_prime_approximation_uses_independent_count_and_declared_source_score(self):
+ count, logarithm = benefit.prime_compositions()
+ self.assertEqual(logarithm, benefit.RULES["e2h-amc-eval-2451"][1])
+ # Even the subset using just2 and3 exceeds the supplied614.519 bits.
+ subset = [1] + [0] * 2013
+ for index in range(1, 2014):
+ subset[index] = sum(subset[index - part] for part in (2, 3) if part <= index)
+ self.assertGreater(subset[-1].bit_length(), 615)
+ self.assertGreater(int(count), subset[-1])
+ for answer in ("1129.418", "1129", "1130", "1120", "1140"):
+ self.assertTrue(self.grade("e2h-amc-eval-2451", answer)["correct"], answer)
+ for answer in ("614.519", "1100", "1150", "0", "-1129", "1/0", "x", "8=1129"):
+ self.assertFalse(self.grade("e2h-amc-eval-2451", answer)["correct"], answer)
+
+ def test_approximate_percent_accepts_source_rounding_and_correctly_stated_precision(self):
+ for answer in ("80", r"80\%", "83", r"82.9\%", "82.93", r"\frac{3400}{41}\%", "3400/41"):
+ self.assertTrue(self.grade("e2h-amc-eval-1984", answer)["correct"], answer)
+ for answer in ("82", "82.5", "82.94", "80.1", r"0.83\%", "8=80", "0", "100"):
+ self.assertFalse(self.grade("e2h-amc-eval-1984", answer)["correct"], answer)
+
+ def test_percentage_questions_do_not_change_percentage_data_into_answers(self):
+ for identity, (_, answer, kind) in benefit.RULES.items():
+ if kind == "percentage":
+ self.assertTrue(self.grade(identity, answer)["correct"])
+ self.assertTrue(self.grade(identity, answer.removesuffix(r"\%"))["correct"])
+ self.assertFalse(benefit.grade_training("e2h-amc-eval-1935", "37", r"\boxed{37\%}")["correct"])
+ self.assertFalse(self.grade("e2h-amc-eval-759", r"0.75\%")["correct"])
+
+ def test_compact_plus_minus_expands_both_ordered_complex_points(self):
+ identity = "e2h-amc-train-230"
+ self.assertTrue(self.grade(identity, benefit.RULES[identity][0])["correct"])
+ self.assertTrue(self.grade(identity, benefit.RULES[identity][1])["correct"])
+ self.assertFalse(self.grade(identity, r"(1,2),(4,2),(\frac52,2+\frac32 i)")["correct"])
+ self.assertFalse(self.grade(identity, r"(2,1),(4,2),(\frac52,2\pm\frac32 i)")["correct"])
+ self.assertFalse(self.grade(identity, r"(1,2),(4,2),(\frac52,2\pm\frac32 i),99")["correct"])
+
+ def test_unreviewed_ambiguity_stays_visible(self):
+ sources = {"new": {"problem": "Unknown meaning", "solution": "Unknown"}}
+ _, _, review = benefit.review_training({"new": "(1,2)"}, sources)
+ self.assertFalse(review["ready"])
+ self.assertEqual(review["unresolved_ids"], ["new"])
+
+
+class TrainingBoundaryTests(unittest.TestCase):
+ def test_exact_source_join_exposes_only_id_and_request(self):
+ source = {"problem": "What is2+2?", "answer": "4", "tag": "AMC12", "year": 2000, "solution": "Add."}
+ row = {
+ "id": "e2h-amc-train-0",
+ "request": render_request(source["tag"], source["problem"]),
+ "contest": source["tag"],
+ "year": source["year"],
+ }
+ dataset = {"splits": {"train": [row]}}
+ inputs, gold, _ = benefit.join_training(dataset, {"train": [source]})
+ self.assertEqual(set(inputs["train"][0]), {"id", "request"})
+ self.assertEqual(gold, {row["id"]: "4"})
+ row["request"] += " Changed"
+ with self.assertRaisesRegex(ValueError, "source/request join"):
+ benefit.join_training(dataset, {"train": [source]})
+
+ def fixture(self, directory):
+ ids = ["synthetic-" + str(index) for index in range(511)]
+ excluded = list(benefit.EXCLUSIONS)
+ inputs = {"train": [{"id": identity, "request": "A synthetic problem."} for identity in ids]}
+ eligibility = {
+ "format": "sie-benefit-training-eligibility-v1",
+ "source_only": True,
+ "decided_before_training_outcomes": True,
+ "original_pool_ids": ids[:20] + excluded + ids[20:],
+ "eligible_train_ids": ids,
+ "exclusions": [{"id": identity, **benefit.EXCLUSIONS[identity]} for identity in excluded],
+ }
+ write(directory / "inputs.json", inputs)
+ write(directory / "settings.json", solver.SETTINGS)
+ write(directory / "eligibility.json", eligibility)
+ write(directory / "gold-preflight.json", {"synthetic": True})
+ write(directory / "training-gold-adjudications.json", {"synthetic": True})
+ write(
+ directory / "gold-review.json",
+ {"ready": True, "approved_ids": ids, "preflight_sha256": file_hash(directory / "gold-preflight.json")},
+ )
+ plan = {
+ "format": "sie-benefit-training-source-v1",
+ "source_counts": {"train": 512},
+ "counts": {"train": 511},
+ "eligible_train_ids": ids,
+ "excluded_train_ids": excluded,
+ "inputs_sha256": digest(inputs),
+ "settings_sha256": digest(solver.SETTINGS),
+ "eligibility_sha256": file_hash(directory / "eligibility.json"),
+ "gold_review_sha256": file_hash(directory / "gold-review.json"),
+ "training_gold_rules_sha256": file_hash(directory / "training-gold-adjudications.json"),
+ **benefit.source_hashes(),
+ }
+ write(directory / "plan.json", plan)
+ return plan
+
+ def test_collection_validation_needs_no_answer_key_and_rejects_input_tampering(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ directory = Path(temporary)
+ self.fixture(directory)
+ # Neither gold-for-scoring-only.json nor model answers exist.
+ benefit.checked_training(directory)
+ inputs = json.loads((directory / "inputs.json").read_text())
+ inputs["train"][0]["request"] = "Tampered"
+ (directory / "inputs.json").write_text(json.dumps(inputs))
+ with self.assertRaisesRegex(ValueError, "inputs or settings changed"):
+ benefit.checked_training(directory)
+
+ def test_collection_rejects_changed_review_and_runtime_source(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ directory = Path(temporary)
+ self.fixture(directory)
+ with (
+ mock.patch.object(benefit, "source_hashes", return_value={"grader_source_sha256": "0" * 64}),
+ self.assertRaisesRegex(ValueError, "source changed"),
+ ):
+ benefit.checked_training(directory)
+ (directory / "eligibility.json").write_text("{}")
+ with self.assertRaisesRegex(ValueError, "review changed"):
+ benefit.checked_training(directory)
+
+ def test_grade_refuses_overwriting_before_reading_any_outcome(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ directory = Path(temporary)
+ output = directory / "grades-train.json"
+ output.write_text("preserve")
+ with self.assertRaises(FileExistsError):
+ benefit.grade(directory)
+ self.assertEqual(output.read_text(), "preserve")
+
+ def completed_fixture(self, directory):
+ plan = self.fixture(directory)
+ inputs = benefit.read(directory / "inputs.json")
+ gold = {row["id"]: "42" for row in inputs["train"]}
+ write(directory / "gold-for-scoring-only.json", gold)
+ plan.update({"gold_sha256": digest(gold), "grader": benefit.grader_metadata()})
+ (directory / "plan.json").write_text(json.dumps(plan))
+ row = inputs["train"][0]
+ collection = directory / "answers-train"
+ write(
+ collection / "collection-plan.json",
+ {
+ "ids": [row["id"]],
+ "split": "train",
+ "plan_sha256": file_hash(directory / "plan.json"),
+ "settings_sha256": digest(solver.SETTINGS),
+ "policies_frozen_sha256": None,
+ "self_hosted": False,
+ },
+ )
+ write(collection / "collection.json", {"status": "complete", "completed": 2, "failed": 0})
+ for arm in solver.SETTINGS["arms"]:
+ answer = solver.initial_record(row, arm, solver.SETTINGS) | {
+ "status": "complete",
+ "observed_credits": 1,
+ "credits": 1,
+ "has_unknown_charges": False,
+ "answer": r"\boxed{42}" if arm == "cheap" else r"\boxed{41}",
+ "calls": [{"billing": "cloud", "receipt": {"credits": 1}}],
+ }
+ write(collection / f"{row['id']}-{arm}.json", answer)
+ return row, plan
+
+ @unittest.skipUnless(importlib.util.find_spec("math_verify"), "Install the evaluation dependency group")
+ def test_grade_preserves_negative_benefit_and_binds_every_provenance_record(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ directory = Path(temporary)
+ row, plan = self.completed_fixture(directory)
+ with redirect_stdout(io.StringIO()):
+ benefit.grade(directory)
+ grades = benefit.read(directory / "grades-train.json")
+ self.assertEqual(grades["split"], "train")
+ self.assertEqual(grades["plan_sha256"], file_hash(directory / "plan.json"))
+ self.assertEqual(grades["training_gold_rules_sha256"], plan["training_gold_rules_sha256"])
+ self.assertEqual(grades["gold_review_source_sha256"], benefit.source_hashes()["gold_review_source_sha256"])
+ self.assertTrue(grades["rows"][0]["cheap_correct"])
+ self.assertFalse(grades["rows"][0]["strong_correct"])
+ path = directory / "answers-train" / f"{row['id']}-cheap.json"
+ self.assertEqual(grades["rows"][0]["cheap_record_sha256"], file_hash(path))
+
+ @unittest.skipUnless(importlib.util.find_spec("math_verify"), "Install the evaluation dependency group")
+ def test_grade_rejects_wrong_request_and_unbound_answer_key(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ directory = Path(temporary)
+ row, _ = self.completed_fixture(directory)
+ path = directory / "answers-train" / f"{row['id']}-cheap.json"
+ answer = benefit.read(path)
+ answer["request_sha256"] = "0" * 64
+ path.write_text(json.dumps(answer))
+ with self.assertRaisesRegex(ValueError, "frozen request"):
+ benefit.grade(directory)
+ (directory / "gold-for-scoring-only.json").write_text("{}")
+ with self.assertRaisesRegex(ValueError, "Answer key|answer key"):
+ benefit.grade(directory)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benefit_expanded.py b/examples/train-a-router/tests/test_benefit_expanded.py
new file mode 100644
index 000000000..2bd90ec75
--- /dev/null
+++ b/examples/train-a-router/tests/test_benefit_expanded.py
@@ -0,0 +1,141 @@
+"""Expanded inference gates bind source review without reading answer keys."""
+
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import benefit_expanded as expanded
+import solver
+from data import digest, file_hash, write
+
+
+class ExpandedInputTests(unittest.TestCase):
+ def setUp(self):
+ self.temp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.temp.cleanup)
+ self.directory = Path(self.temp.name)
+ self.inputs = {"train": [{"id": "old0", "request": "One"}, {"id": "new0", "request": "Two"}]}
+ self.eligibility = {
+ "source_only": True,
+ "decided_before_new_training_outcomes": True,
+ "reused_training_outcomes_already_available": True,
+ "original_pool_ids": ["old0", "old1", "new0"],
+ "eligible_train_ids": ["old0", "new0"],
+ "exclusions": [{"id": "old1", "reason": "Source-only template overlap"}],
+ }
+ write(self.directory / "inputs.json", self.inputs)
+ write(self.directory / "settings.json", solver.SETTINGS)
+ write(self.directory / "source-leakage-review.json", {"source_only": True})
+ self.eligibility["leakage_review_sha256"] = file_hash(self.directory / "source-leakage-review.json")
+ write(self.directory / "eligibility.json", self.eligibility)
+ write(self.directory / "reuse.json", {"ids": ["old0"]})
+ write(self.directory / "training-rows.json", [])
+ write(self.directory / "expansion-audit.json", {})
+ write(self.directory / "training-gold-adjudications.json", {"answer_bearing": True})
+ # The inference gate must hash this file, not parse its answer-bearing contents.
+ (self.directory / "gold-preflight.json").write_bytes(b"not valid JSON or an answer key")
+ write(
+ self.directory / "gold-review.json",
+ {
+ "preflight_sha256": file_hash(self.directory / "gold-preflight.json"),
+ "ready": True,
+ "approved_ids": ["old0", "new0"],
+ },
+ )
+ self.plan = {
+ "format": expanded.FORMAT,
+ "inputs_sha256": digest(self.inputs),
+ "settings_sha256": digest(solver.SETTINGS),
+ "counts": {"train": 2},
+ "source_counts": {"train": 3},
+ "eligible_train_ids": ["old0", "new0"],
+ "excluded_train_ids": ["old1"],
+ "new_train_ids": ["new0"],
+ "grader_source_sha256": "a" * 64,
+ }
+ for key, name in (
+ ("eligibility_sha256", "eligibility.json"),
+ ("leakage_review_sha256", "source-leakage-review.json"),
+ ("reuse_sha256", "reuse.json"),
+ ("training_rows_sha256", "training-rows.json"),
+ ("expansion_audit_sha256", "expansion-audit.json"),
+ ("training_gold_rules_sha256", "training-gold-adjudications.json"),
+ ("gold_review_sha256", "gold-review.json"),
+ ):
+ self.plan[key] = file_hash(self.directory / name)
+ write(self.directory / "plan.json", self.plan)
+ mock = patch.object(expanded, "source_hashes", return_value={"grader_source_sha256": "a" * 64})
+ mock.start()
+ self.addCleanup(mock.stop)
+
+ def replace_eligibility(self):
+ (self.directory / "eligibility.json").unlink()
+ write(self.directory / "eligibility.json", self.eligibility)
+ self.plan["eligibility_sha256"] = file_hash(self.directory / "eligibility.json")
+ (self.directory / "plan.json").unlink()
+ write(self.directory / "plan.json", self.plan)
+
+ def test_inference_needs_no_answer_key_or_parsed_preflight(self):
+ _, inputs, _ = expanded.checked_training(self.directory)
+ self.assertEqual(inputs, self.inputs)
+ self.assertFalse((self.directory / "gold-for-scoring-only.json").exists())
+
+ def test_revised_eligibility_must_acknowledge_existing_outcomes(self):
+ self.eligibility.pop("reused_training_outcomes_already_available")
+ self.eligibility["decided_before_training_outcomes"] = True
+ self.replace_eligibility()
+ with self.assertRaisesRegex(ValueError, "eligibility"):
+ expanded.checked_training(self.directory)
+
+ def test_resealed_duplicate_population_is_rejected(self):
+ self.eligibility["original_pool_ids"] = ["old0", "old0", "new0"]
+ self.replace_eligibility()
+ with self.assertRaisesRegex(ValueError, "eligibility"):
+ expanded.checked_training(self.directory)
+
+ def test_leakage_receipt_bytes_are_required(self):
+ (self.directory / "source-leakage-review.json").write_text("{}")
+ with self.assertRaisesRegex(ValueError, "evidence changed"):
+ expanded.checked_training(self.directory)
+
+
+class SourceDecisionTests(unittest.TestCase):
+ def setUp(self):
+ self.gold = {identity: "1" for identity in ("old", "new1", "new2", "invalid", "leak")}
+ self.sources = {identity: {"problem": identity, "solution": "Solution"} for identity in self.gold}
+ self.leakage = {
+ "format": "sie-expanded-source-leakage-review-v1",
+ "source_only": True,
+ "solver_outcomes_read_for_decisions": False,
+ "decided_before_new_training_outcomes": True,
+ "reused_training_outcomes_already_available": True,
+ "exclusions": {"leak": {"reason": "Held-out question template"}},
+ "duplicate_groups": [["new1", "new2"], ["old", "new2"]],
+ "candidate_reviews": [],
+ }
+ self.mandatory = {"invalid": {"original_gold": "1", "reason": "Missing choices"}}
+ mock = patch.object(expanded.benefit_data, "EXCLUSIONS", {})
+ mock.start()
+ self.addCleanup(mock.stop)
+
+ def test_connected_duplicates_keep_original_before_new_rows(self):
+ excluded, decisions = expanded.reviewed_exclusions(
+ self.gold, self.sources, {"old"}, self.leakage, self.mandatory
+ )
+ self.assertEqual(set(excluded), {"new1", "new2", "invalid", "leak"})
+ self.assertEqual(decisions[0]["retained_id"], "old")
+
+ def test_new_duplicate_representative_uses_fixed_digest(self):
+ _, decisions = expanded.reviewed_exclusions(self.gold, self.sources, set(), self.leakage, self.mandatory)
+ expected = min(("old", "new1", "new2"), key=lambda identity: digest(["expanded-source-duplicate-v1", identity]))
+ self.assertEqual(decisions[0]["retained_id"], expected)
+
+ def test_changed_source_cannot_reuse_review(self):
+ self.leakage["candidate_reviews"] = [{"source_problem_sha256": {"old": digest("Different question")}}]
+ with self.assertRaisesRegex(ValueError, "problem differs"):
+ expanded.reviewed_exclusions(self.gold, self.sources, {"old"}, self.leakage, self.mandatory)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benefit_expanded_collect.py b/examples/train-a-router/tests/test_benefit_expanded_collect.py
new file mode 100644
index 000000000..044f45f0a
--- /dev/null
+++ b/examples/train-a-router/tests/test_benefit_expanded_collect.py
@@ -0,0 +1,186 @@
+"""Disjoint collection jobs preserve old answers and never retry submitted calls."""
+
+import copy
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import benefit_collect as stages
+import benefit_expanded_collect as collector
+import solver
+from data import digest, file_hash, write
+
+
+class MetadataClient:
+ def __init__(self, **kwargs):
+ pass
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *args):
+ return False
+
+ def get_model(self, model):
+ return {
+ "name": model,
+ "revision": next(item["revision"] for item in solver.ARMS.values() if item["model"] == model),
+ }
+
+
+class ExpandedCollectionTests(unittest.TestCase):
+ def setUp(self):
+ self.temp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.temp.cleanup)
+ self.root = Path(self.temp.name)
+ self.directory, self.prior = self.root / "expanded", self.root / "prior"
+ self.rows = [
+ {"id": identity, "request": f"Question {identity}"} for identity in ("old0", "new0", "new1", "new2")
+ ]
+ self.settings = copy.deepcopy(solver.SETTINGS)
+ self.plan = {"new_train_ids": [row["id"] for row in self.rows[1:]], "reuse_sha256": "pending"}
+ write(self.prior / "plan.json", {"prior": True})
+ write(self.prior / "answers-train/collection-plan.json", {"historical": True})
+ write(self.prior / "answers-train/collection.json", {"status": "complete", "completed": 4})
+ hashes = {}
+ for identity in ("old0", "old1"):
+ row = {"id": identity, "request": f"Question {identity}"}
+ for arm in solver.ARMS:
+ message = stages.messages_for(row, arm, self.settings)
+ attempt = self.success(row, arm, self.settings, "initial", message, True, record_send=False)
+ record = solver.initial_record(row, arm, self.settings) | {
+ "status": "complete",
+ "calls": [attempt["call"]],
+ "answer": "\\boxed{2}",
+ "observed_credits": 0,
+ "credits": None,
+ "has_unknown_charges": False,
+ }
+ path = self.prior / "answers-train" / f"{identity}-{arm}.json"
+ write(path, record)
+ if identity == "old0":
+ hashes[path.name] = file_hash(path)
+ reuse = {
+ "ids": ["old0"],
+ "records_sha256": hashes,
+ "plan_sha256": file_hash(self.prior / "plan.json"),
+ "collection_plan_sha256": file_hash(self.prior / "answers-train/collection-plan.json"),
+ "collection_sha256": file_hash(self.prior / "answers-train/collection.json"),
+ }
+ write(self.directory / "reuse.json", reuse)
+ self.plan["reuse_sha256"] = file_hash(self.directory / "reuse.json")
+ write(self.directory / "plan.json", self.plan)
+ self.sent = []
+ for mocked in (
+ patch.object(
+ collector,
+ "checked_training",
+ side_effect=lambda directory: (self.plan, {"train": self.rows}, self.settings),
+ ),
+ patch.object(collector, "SingleSendClient", MetadataClient),
+ patch.object(stages, "stage_call", side_effect=self.success),
+ ):
+ mocked.start()
+ self.addCleanup(mocked.stop)
+
+ def success(self, row, arm, settings, phase, messages, self_hosted, record_send=True):
+ if record_send:
+ self.sent.append((row["id"], arm, phase))
+ answer = '{"answer":"4"}' if phase == "final" else "Some work" if row["id"] == "new1" else "\\boxed{2}"
+ call = {
+ "messages_sha256": digest(messages),
+ "answer": answer,
+ "finish_reason": "stop",
+ "response_valid": True,
+ "billing_known": True,
+ "billing": "self_hosted",
+ "receipt": {"credits": None},
+ "tokens": {"prompt_tokens": 10, "completion_tokens": 10},
+ }
+ return {
+ "id": row["id"],
+ "arm": arm,
+ "phase": phase,
+ "request_sha256": digest(row["request"]),
+ "settings_sha256": digest(settings),
+ "messages_sha256": digest(messages),
+ "status": "complete",
+ "call": call,
+ "observed_credits": 0,
+ "execution_unknown": False,
+ "safe_to_resume": False,
+ "http_failures": [],
+ }
+
+ def run_job(self, name, phase="initial", **kwargs):
+ return collector.collect(self.directory, name, phase, workers=1, max_seconds=60, self_hosted=True, **kwargs)
+
+ def test_shards_and_resume_do_not_repeat_old_or_completed_calls(self):
+ jobs = collector.prepare_jobs(self.directory)
+ self.assertEqual(jobs["jobs"]["strong-0"]["ids"], ["new0", "new2"])
+ self.assertEqual(jobs["jobs"]["strong-1"]["ids"], ["new1"])
+ for name in jobs["jobs"]:
+ self.run_job(name, limit=1)
+ self.run_job(name, resume=True)
+ self.run_job(name, "final", resume=True)
+ self.run_job(name, "final", resume=True)
+ self.assertEqual(len(self.sent), 8)
+ self.assertEqual(len(set(self.sent)), 8)
+ self.assertFalse(any(identity.startswith("old") for identity, _, _ in self.sent))
+ summary = collector.gather(self.directory, self.prior)
+ self.assertEqual(summary["completed"], 8)
+ self.assertEqual(summary["reused_records"], 2)
+ self.assertEqual(summary["new_completed_records"], 6)
+ self.assertEqual(summary["unknown_execution_attempts"], 0)
+ self.assertFalse((self.directory / "answers-train/old1-cheap.json").exists())
+ for arm in solver.ARMS:
+ self.assertEqual(
+ file_hash(self.directory / f"answers-train/old0-{arm}.json"),
+ file_hash(self.prior / f"answers-train/old0-{arm}.json"),
+ )
+ collector.gather(self.directory, self.prior)
+ self.assertEqual(len(self.sent), 8)
+
+ def test_duplicate_shard_rejected_before_any_call(self):
+ plan = collector.prepare_jobs(self.directory)
+ plan["jobs"]["strong-1"]["ids"] = ["new0", "new1"]
+ (self.directory / "collection-jobs.json").unlink()
+ write(self.directory / "collection-jobs.json", plan)
+ with self.assertRaisesRegex(ValueError, "exactly once"):
+ self.run_job("strong-0")
+ self.assertEqual(self.sent, [])
+
+ def test_unknown_intent_blocks_resume_without_retry(self):
+ collector.prepare_jobs(self.directory)
+ self.run_job("cheap", limit=1)
+ path = self.directory / "collection-jobs/cheap/attempts/new1/initial-0001.intent.json"
+ write(path, {"id": "new1"})
+ with self.assertRaisesRegex(ValueError, "no retries"):
+ self.run_job("cheap", resume=True)
+ self.assertEqual(len(self.sent), 1)
+ self.assertTrue(path.exists())
+
+ def test_original_only_control_needs_no_calls(self):
+ self.rows = self.rows[:1]
+ self.plan["new_train_ids"] = []
+ (self.directory / "plan.json").unlink()
+ write(self.directory / "plan.json", self.plan)
+ self.assertEqual(collector.prepare_jobs(self.directory)["jobs"], {})
+ summary = collector.gather(self.directory, self.prior)
+ self.assertEqual(summary["completed"], 2)
+ self.assertEqual(summary["new_completed_records"], 0)
+ self.assertEqual(self.sent, [])
+
+ def test_changed_reused_record_is_rejected(self):
+ self.rows = self.rows[:1]
+ self.plan["new_train_ids"] = []
+ collector.prepare_jobs(self.directory)
+ path = self.prior / "answers-train/old0-cheap.json"
+ path.write_text(path.read_text() + "\n")
+ with self.assertRaisesRegex(ValueError, "Original completed answer changed"):
+ collector.gather(self.directory, self.prior)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benefit_expanded_gold.py b/examples/train-a-router/tests/test_benefit_expanded_gold.py
new file mode 100644
index 000000000..646b02d9e
--- /dev/null
+++ b/examples/train-a-router/tests/test_benefit_expanded_gold.py
@@ -0,0 +1,162 @@
+"""Source-only expanded-answer semantics; no solver outputs or model calls."""
+
+import importlib.util
+import unittest
+from decimal import Decimal, localcontext
+from itertools import combinations, permutations
+from math import comb, factorial
+
+import benefit_data
+import benefit_expanded_gold as expanded
+from data import digest
+
+
+@unittest.skipUnless(importlib.util.find_spec("math_verify"), "Install the evaluation dependency group")
+class ExpandedGoldTests(unittest.TestCase):
+ def grade(self, identity, answer):
+ return expanded.grade_expanded(identity, expanded.RULES[identity][0], r"\boxed{" + answer + "}")
+
+ def test_every_reviewed_canonical_answer_and_original_binding(self):
+ self.assertFalse(set(expanded.RULES) & set(benefit_data.RULES))
+ for identity, (original, canonical, _) in expanded.RULES.items():
+ with self.subTest(identity=identity):
+ self.assertTrue(self.grade(identity, canonical)["correct"])
+ with self.assertRaisesRegex(ValueError, "original answer"):
+ expanded.grade_expanded(identity, original + " changed", canonical)
+
+ def test_original_training_rules_delegate_unchanged(self):
+ for identity, (original, canonical, _) in benefit_data.RULES.items():
+ response = r"\boxed{" + canonical + "}"
+ with self.subTest(identity=identity):
+ self.assertEqual(
+ expanded.grade_expanded(identity, original, response),
+ benefit_data.grade_training(identity, original, response),
+ )
+
+ def test_missing_zero_and_secret_santa_corrections(self):
+ for identity, correct, wrong in (
+ ("e2h-amc-train-774", "400000", "4000"),
+ ("e2h-amc-eval-870", "400000", "400"),
+ ("e2h-amc-eval-1456", "1008016", "1816"),
+ ("e2h-amc-eval-1412", "0.96040", "0.96039"),
+ ):
+ self.assertTrue(self.grade(identity, correct)["correct"])
+ self.assertFalse(self.grade(identity, wrong)["correct"])
+
+ def test_ratios_accept_equivalent_fraction_without_losing_direction(self):
+ identity = "e2h-amc-eval-1533"
+ for answer in (r"6:\pi", r"12:2\pi", r"\frac{6}{\pi}", r"\frac{1350}{225\pi}"):
+ self.assertTrue(self.grade(identity, answer)["correct"], answer)
+ for answer in (r"\pi:6", "6:0", "6:3", "6", "6:1:2"):
+ self.assertFalse(self.grade(identity, answer)["correct"], answer)
+ self.assertTrue(self.grade("e2h-amc-eval-2110", "4:2")["correct"])
+
+ def test_morning_time_is_not_a_ratio_or_evening_time(self):
+ identity = "e2h-amc-eval-910"
+ for answer in ("11", "11:0", "11:00", "11 AM", "11:00 a.m.", r"\text{11:00 AM}"):
+ self.assertTrue(self.grade(identity, answer)["correct"], answer)
+ for answer in ("11 PM", "23:00", "11:60", "22:0", "11/0", "10:59", "1.1"):
+ self.assertFalse(self.grade(identity, answer)["correct"], answer)
+
+ def test_division_symbol_and_words_not_missing_choice_letter(self):
+ identity = "e2h-amc-eval-946"
+ for answer in (r"\div", "÷", "/", "division", r"\text{division operator}"):
+ self.assertTrue(self.grade(identity, answer)["correct"], answer)
+ for answer in ("A", "2", "+", "multiplication", r"\frac63"):
+ self.assertFalse(self.grade(identity, answer)["correct"], answer)
+
+ def test_both_officially_accepted_seating_interpretations(self):
+ identity = "e2h-amc-eval-2670"
+ for answer in ("2", str(2 * factorial(15) ** 2), r"2\cdot15 !^{2}", r"2(15!)^2", r"\{2,2(15!)^2\}"):
+ self.assertTrue(self.grade(identity, answer)["correct"], answer)
+ for answer in ("1", "4", "15!", "30!", r"2\cdot(14!)^2", "3=2"):
+ self.assertFalse(self.grade(identity, answer)["correct"], answer)
+
+ def test_plus_minus_requires_every_solution_and_keeps_tuple_order(self):
+ self.assertTrue(self.grade("e2h-amc-eval-2564", r"\frac12,\frac{-1\pm\sqrt{13}}4")["correct"])
+ self.assertFalse(self.grade("e2h-amc-eval-2564", r"\frac12,\frac{-1+\sqrt{13}}4")["correct"])
+ self.assertTrue(self.grade("e2h-amc-train-965", r"(-1006,\pm1006\sqrt3)")["correct"])
+ self.assertFalse(self.grade("e2h-amc-train-965", r"(\pm1006\sqrt3,-1006)")["correct"])
+ self.assertFalse(self.grade("e2h-amc-eval-799", r"\{(3,1),(\frac13,3)\}")["correct"])
+
+ def test_named_answers_preserve_order(self):
+ identity = "e2h-amc-eval-1138"
+ for answer in ("MLS", "M,L,S", "medium, large, small", "Medium > Large > Small"):
+ self.assertTrue(self.grade(identity, answer)["correct"], answer)
+ for answer in ("SML", "small, large, medium", "M*S*L"):
+ self.assertFalse(self.grade(identity, answer)["correct"], answer)
+ for answer in ("Friday", "friday", "day 5", "fifth day"):
+ self.assertTrue(self.grade("e2h-amc-eval-2777", answer)["correct"], answer)
+ self.assertFalse(self.grade("e2h-amc-eval-2777", "Thursday")["correct"])
+
+ def test_percentage_context_does_not_change_counts_or_fractions(self):
+ for identity, (_, canonical, kind) in expanded.RULES.items():
+ if kind == "percentage":
+ self.assertTrue(self.grade(identity, canonical.removesuffix(r"\%"))["correct"])
+ self.assertFalse(self.grade("e2h-amc-eval-558", r"0.125\%")["correct"])
+ # This question asks a fraction, even though the source prints25%.
+ self.assertTrue(expanded.grade_expanded("e2h-amc-train-944", r"25\%", r"\boxed{\frac14}")["correct"])
+ self.assertFalse(expanded.grade_expanded("e2h-amc-train-944", r"25\%", r"\boxed{25}")["correct"])
+ self.assertFalse(expanded.grade_expanded("e2h-amc-eval-64", "5", r"\boxed{5\%}")["correct"])
+ self.assertFalse(self.grade("e2h-amc-eval-1782", "26.6666667")["correct"])
+
+ def test_source_invalid_questions_cannot_be_collected_or_graded(self):
+ for identity, rule in expanded.EXCLUSIONS.items():
+ self.assertEqual(len(rule["problem_sha256"]), 64)
+ self.assertEqual(len(rule["solution_sha256"]), 64)
+ with self.assertRaisesRegex(ValueError, "excluded before collection"):
+ expanded.grade_expanded(identity, rule["original_gold"], r"\boxed{1}")
+ with self.assertRaisesRegex(ValueError, "source-invalid"):
+ expanded.review_expanded({identity: rule["original_gold"]}, {})
+
+ def test_review_binds_metadata_and_keeps_unknown_ambiguity_visible(self):
+ identity = "e2h-amc-train-774"
+ source = {"answer": "400,0", "problem": "Reviewed source question", "solution": "Reviewed derivation"}
+ corrected, _, review = expanded.review_expanded({identity: "400,0"}, {identity: source})
+ self.assertTrue(review["ready"])
+ self.assertEqual(corrected[identity], "400000")
+ self.assertEqual(review["correction_records"][0]["problem_sha256"], digest(source["problem"]))
+ self.assertEqual(review["canonical_gold_sha256"], digest(corrected))
+ self.assertEqual(len(review["expanded_gold_review_source_sha256"]), 64)
+ _, _, unknown = expanded.review_expanded({"unknown": "(1,2)"}, {})
+ self.assertFalse(unknown["ready"])
+ self.assertEqual(unknown["unresolved_ids"], ["unknown"])
+
+
+class IndependentSourceProofTests(unittest.TestCase):
+ @staticmethod
+ def santa_counts(n):
+ edges = ((0, 1), (0, 2), (1, 0), (2, 0))
+ valid = 0
+ for count in range(5):
+ for chosen in combinations(edges, count):
+ rows = {left for left, _ in chosen}
+ columns = {right for _, right in chosen}
+ if len(rows) != count or len(columns) != count:
+ continue
+ diagonal_constraints = n - len(rows | columns)
+ completions = sum(
+ (-1) ** j * comb(diagonal_constraints, j) * factorial(n - count - j)
+ for j in range(diagonal_constraints + 1)
+ )
+ valid += (-1) ** count * completions
+ all_derangements = sum((-1) ** j * comb(n, j) * factorial(n - j) for j in range(n + 1))
+ return valid, all_derangements
+
+ def test_secret_santa_count_matches_small_exhaustive_permutations(self):
+ for n in range(3, 8):
+ derangements = [p for p in permutations(range(n)) if all(value != i for i, value in enumerate(p))]
+ good = sum(p[0] not in (1, 2) and p[1] != 0 and p[2] != 0 for p in derangements)
+ self.assertEqual(self.santa_counts(n), (good, len(derangements)))
+
+ def test_exact101_person_probability_changes_fifth_decimal(self):
+ valid, total = self.santa_counts(101)
+ with localcontext() as context:
+ context.prec = 60
+ rounded = (Decimal(valid) / total).quantize(Decimal("0.00001"))
+ self.assertEqual(str(rounded), "0.96040")
+ self.assertEqual(expanded.RULES["e2h-amc-eval-1412"][1], str(rounded))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benefit_expanded_gold_v2.py b/examples/train-a-router/tests/test_benefit_expanded_gold_v2.py
new file mode 100644
index 000000000..738deca61
--- /dev/null
+++ b/examples/train-a-router/tests/test_benefit_expanded_gold_v2.py
@@ -0,0 +1,200 @@
+"""Exact arithmetic and question-only teacher semantics, without model outputs."""
+
+import unittest
+from fractions import Fraction
+from math import comb
+from unittest.mock import patch
+
+import benefit_expanded_gold as previous
+import benefit_expanded_gold_v2 as amended
+from data import digest
+
+
+class ExactArithmeticTests(unittest.TestCase):
+ def test_equivalent_rational_forms_are_exact(self):
+ expected = Fraction(1 + 2**1005, 2**1007) ** 2
+ for value in (
+ r"\left(\frac{1+2^{1005}}{2^{1007}}\right)^2",
+ r"\frac{1}{16}(1+\frac{1}{2^{1005}})^2",
+ r"\frac{1}{16}+2^{-1008}+2^{-2014}",
+ r"p=\frac{(1+2^{1005})^2}{2^{2014}}",
+ ):
+ with self.subTest(value=value):
+ self.assertEqual(amended.exact_rational(value), expected)
+
+ def test_nearby_values_cannot_match_by_numeric_tolerance(self):
+ expected = r"\frac{1}{16}(1+2^{-1005})^2"
+ for wrong in (
+ "0.0625",
+ r"\frac{1}{16}",
+ r"\frac{1}{16}(1+2^{-1006})^2",
+ r"\frac{1}{16}+2^{-1008}",
+ r"\frac{1}{16}(1-2^{-1005})^2",
+ ):
+ self.assertFalse(amended.exact_comparison(expected, wrong)["correct"], wrong)
+ self.assertFalse(amended.exact_comparison("1/3", "0.333333333333333333333333333333")["correct"])
+ self.assertTrue(amended.exact_comparison("1/8", ".125")["correct"])
+
+ def test_precedence_decimal_percent_and_braced_arguments(self):
+ for expression, expected in (
+ ("-2^2", -4),
+ ("(-2)^2", 4),
+ ("2^3^2", 512),
+ ("2^-3", Fraction(1, 8)),
+ (r"\frac34", Fraction(3, 4)),
+ (r"\frac 34", Fraction(3, 4)),
+ (r"\dfrac{12}{16}", Fraction(3, 4)),
+ ("2(1+3)", 8),
+ (r"75\%", Fraction(3, 4)),
+ ("0.1+0.2", Fraction(3, 10)),
+ (r"\sqrt{\frac{4}{9}}", Fraction(2, 3)),
+ (r"x=\frac34", Fraction(3, 4)),
+ ):
+ self.assertEqual(amended.exact_rational(expression), expected, expression)
+
+ def test_unsupported_expressions_and_resource_bounds_fail_closed(self):
+ for expression in (
+ "__import__('os').system('bad')",
+ "2=3",
+ "1/0",
+ "0^-1",
+ "0^0",
+ "2^1000000000",
+ "2^(2^20)",
+ "2^(1/2)",
+ r"\sqrt2",
+ r"\pi",
+ "x+1",
+ "1,2",
+ "3:4",
+ "0.1.2",
+ "1e-6",
+ "",
+ "(" * 100 + "1" + ")" * 100,
+ "1" * 4097,
+ ):
+ self.assertIsNone(amended.exact_rational(expression), expression[:60])
+
+ def test_guard_is_generic_and_overrides_a_mocked_tolerant_fallback(self):
+ with patch.object(previous, "grade_expanded", return_value={"correct": True, "status": "graded"}):
+ result = amended.grade_expanded("synthetic", "1/3", r"\boxed{0.3333333333333}")
+ self.assertFalse(result["correct"])
+ self.assertEqual(result["exact_numeric_guard"], "applied")
+
+ def test_mixed_numbers_and_digit_spacing_do_not_become_products_or_concatenation(self):
+ for expression in (r"1\frac{1}{5}", r"1\frac15", "1 1/5", "1 000", r"1\,1/5", r"\frac1 5"):
+ self.assertIsNone(amended.exact_rational(expression), expression)
+ self.assertEqual(amended.exact_rational(r"2*\frac12"), 1)
+ self.assertEqual(amended.exact_rational(r"2\cdot\frac12"), 1)
+ for original in (r"1\frac{1}{5}", "1 1/5"):
+ response = r"\boxed{1.2}"
+ result = amended.grade_expanded("synthetic", original, response)
+ self.assertEqual(result["correct"], previous.grade_expanded("synthetic", original, response)["correct"])
+ self.assertIn("unsupported", result["exact_numeric_guard"])
+ self.assertTrue(amended.grade_expanded("synthetic", r"1\frac{1}{5}", r"\boxed{1.2}")["correct"])
+
+ def test_symbolic_fallback_and_existing_approximation_are_explicit(self):
+ result = amended.grade_expanded("synthetic", r"\sqrt2", r"\boxed{\sqrt{8}/2}")
+ self.assertTrue(result["correct"])
+ self.assertIn("unsupported", result["exact_numeric_guard"])
+ scientific = amended.grade_expanded("synthetic", r"10^{-6}", r"\boxed{1e-6}")
+ self.assertIn("unsupported", scientific["exact_numeric_guard"])
+ original, _, _ = amended.benefit_data.RULES["e2h-amc-eval-1984"]
+ result = amended.grade_expanded("e2h-amc-eval-1984", original, r"\boxed{82.9\%}")
+ self.assertTrue(result["correct"])
+ self.assertIn("non-exact", result["exact_numeric_guard"])
+
+
+class SourceAmendmentTests(unittest.TestCase):
+ def grade(self, identity, answer):
+ return amended.grade_expanded(identity, amended.RULES[identity][0], r"\boxed{" + answer + "}")
+
+ def test_pentagon_binomial_count_and_independent_walk(self):
+ counts = [1, 0, 0, 0, 0]
+ for _ in range(10):
+ counts = [counts[(i - 1) % 5] + counts[(i + 1) % 5] for i in range(5)]
+ ways = sum(comb(10, k) for k in range(11) if (2 * k - 10) % 5 == 0)
+ self.assertEqual(ways, counts[0])
+ self.assertEqual(Fraction(ways, sum(counts)), Fraction(127, 512))
+ self.assertTrue(self.grade("e2h-amc-eval-1181", r"\frac{127}{512}")["correct"])
+ self.assertFalse(self.grade("e2h-amc-eval-1181", r"\sqrt{\frac{127}{512}}")["correct"])
+
+ def test_chessboard_independent_integer_recurrence(self):
+ counts = [1] + [0] * 7
+ for _ in range(2012):
+ counts = [counts[(i - 1) % 8] + counts[(i + 1) % 8] for i in range(8)]
+ observed = Fraction(counts[0], sum(counts)) ** 2
+ self.assertEqual(observed, Fraction(1 + 2**1005, 2**1007) ** 2)
+ self.assertGreater(observed, Fraction(1, 16))
+ self.assertTrue(self.grade("e2h-amc-eval-776", r"\frac1{16}(1+2^{-1005})^2")["correct"])
+ self.assertFalse(self.grade("e2h-amc-eval-776", "0.0625")["correct"])
+ self.assertFalse(self.grade("e2h-amc-eval-776", "p")["correct"])
+
+ def test_ratio_and_count_units_without_altering_numeric_value(self):
+ for answer in ("3:4", "6:8", "3/4"):
+ self.assertTrue(self.grade("e2h-amc-eval-2735", answer)["correct"])
+ for answer in ("4:3", "3:0", "3:4:5"):
+ self.assertFalse(self.grade("e2h-amc-eval-2735", answer)["correct"])
+ self.assertEqual(2 * 2 * comb(6, 2) + 6 * 6, 96)
+ for answer in ("96", "96 games", r"96\text{ games}", r"96\text{games}", "96games", "96 conference games"):
+ self.assertTrue(self.grade("e2h-amc-eval-911", answer)["correct"], answer)
+ for answer in ("192 games", "96 teams", "96 games per team", "96+1 games"):
+ self.assertFalse(self.grade("e2h-amc-eval-911", answer)["correct"], answer)
+
+ def test_review_binds_both_versions_and_new_source_correction(self):
+ gold = {identity: rule[0] for identity, rule in amended.RULES.items()}
+ sources = {
+ identity: {"answer": value, "problem": "Bound source", "solution": "Exact proof"}
+ for identity, value in gold.items()
+ }
+ corrected, _, review = amended.review_expanded(gold, sources)
+ self.assertTrue(review["ready"])
+ self.assertEqual(corrected["e2h-amc-eval-1181"], r"\frac{127}{512}")
+ self.assertEqual(review["canonical_gold_sha256"], digest(corrected))
+ self.assertEqual(review["outcome_grader_protocol_sha256"], digest(amended.OUTCOME_PROTOCOL))
+ self.assertEqual(len(review["previous_review_source_sha256"]), 64)
+ self.assertEqual(sum(row["source_answer_correction"] for row in review["correction_records"]), 1)
+ with self.assertRaisesRegex(ValueError, "original answer"):
+ amended.grade_expanded("e2h-amc-eval-911", "95", r"\boxed{96}")
+
+
+class TeacherContextTests(unittest.TestCase):
+ def grade(self, question, reference, answer):
+ return amended.grade_teacher_reference(question, reference, r"\boxed{" + answer + "}")
+
+ def test_percentage_normalization_uses_request_not_reference_value(self):
+ for question, teacher, observed in (
+ ("What is the percent of yellow tint?", r"40\%", "40"),
+ ("What percentage of males listen?", "75", r"75\%"),
+ ("By what percent did the price increase?", "17", r"17\%"),
+ ):
+ self.assertTrue(self.grade(question, teacher, observed)["correct"])
+ self.assertFalse(self.grade("What percentage of males listen?", "75", r"0.75\%")["correct"])
+ self.assertFalse(self.grade("25 percent of them are red. How many are red?", "5", r"5\%")["correct"])
+ self.assertTrue(self.grade("What fraction is red?", r"25\%", "1/4")["correct"])
+
+ def test_ratio_context_and_game_suffix_are_limited(self):
+ self.assertTrue(self.grade("What is the ratio of x to y?", "3:4", "3/4")["correct"])
+ self.assertFalse(self.grade("What is the ratio of x to y?", "3:4", "4/3")["correct"])
+ self.assertTrue(self.grade("How many conference games are scheduled?", "96 games", "96")["correct"])
+ self.assertFalse(self.grade("How many teams are scheduled?", "96", "96 games")["correct"])
+
+ def test_teacher_reference_never_uses_known_gold_or_row_rules(self):
+ with patch.object(amended, "RULES", {}), patch.object(amended, "REASONS", {}):
+ result = self.grade("What is the probability?", "1/2", "1/2")
+ self.assertTrue(result["correct"])
+ self.assertEqual(result["reference_mode"], "teacher_final_and_question_context_only")
+ self.assertFalse(self.grade("What is the probability?", "1/3", "0.333333333333")["correct"])
+
+ def test_no_intermediate_or_missing_final_answer_is_selected(self):
+ result = amended.grade_teacher_reference(
+ "What percentage?", "40", "We derive 40 while checking another quantity."
+ )
+ self.assertFalse(result["correct"])
+ self.assertEqual(result["status"], "no_final_answer")
+ result = amended.grade_teacher_reference("How many games?", "96 games", "\\boxed{96}\nFinal answer: 97 games")
+ self.assertFalse(result["correct"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benefit_native.py b/examples/train-a-router/tests/test_benefit_native.py
new file mode 100644
index 000000000..b253d21d8
--- /dev/null
+++ b/examples/train-a-router/tests/test_benefit_native.py
@@ -0,0 +1,297 @@
+"""Soft-outcome training, paper selection, and native test-access contracts."""
+
+import copy
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import numpy as np
+import test_benefit as fixtures
+
+import benefit
+import benefit_native as native
+from data import file_hash
+
+
+def reseal(document):
+ return benefit.seal({key: value for key, value in document.items() if key != "sha256"})
+
+
+def history(targets, winner=1):
+ result = []
+ for epoch in range(native.PROTOCOL["selection_epochs"] + 1):
+ scores = np.asarray(targets) * (0.6 if epoch == winner else 0.0)
+ margins = 2 * np.arctanh(scores)
+ result.append(
+ {
+ "epoch": epoch,
+ "margins": margins.tolist(),
+ "benefit_mse": float(np.mean((scores - targets) ** 2)),
+ "train_loss": None if epoch == 0 else 0.5,
+ }
+ )
+ return result
+
+
+class NativeSoftTargetTests(unittest.TestCase):
+ def test_soft_targets_keep_rescues_and_both_types_of_tie(self):
+ inputs = [{"request": f"Problem {i}"} for i in range(4)]
+ rows = native.soft_rows(inputs, [-1, 0, 0, 1])
+ self.assertEqual(
+ [row["gold"]["route"]["probabilities"] for row in rows],
+ [
+ {"cheap": 1.0, "strong": 0.0},
+ {"cheap": 0.5, "strong": 0.5},
+ {"cheap": 0.5, "strong": 0.5},
+ {"cheap": 0.0, "strong": 1.0},
+ ],
+ )
+ self.assertEqual(list(rows[0]["questions"]["route"]["criteria"]), ["cheap", "strong"])
+ for targets in ([0], [0, 0, 0, 2], [0, 0, 0, np.nan]):
+ with self.subTest(targets=targets), self.assertRaises(ValueError):
+ native.soft_rows(inputs, targets)
+
+ def test_canonical_records_produce_signed_benefit_without_argmax(self):
+ records = [(0, [0, margin], target, 2) for margin, target in ((-1000, [1, 0]), (0, [0.5, 0.5]), (1000, [0, 1]))]
+ margins, scores = native.records_to_scores(records, [-1, 0, 1])
+ np.testing.assert_array_equal(margins, [-1000, 0, 1000])
+ np.testing.assert_array_equal(scores, [-1, 0, 1])
+ for bad in (
+ [(0, [0, 1], [1, 0], 2)],
+ [(0, [0, np.nan], [0.5, 0.5], 2)],
+ [(0, [0, 1], [0.5, 0.5], 3)],
+ ):
+ with self.subTest(records=bad), self.assertRaises(ValueError):
+ native.records_to_scores(bad, [0])
+
+ def test_epoch_selection_recomputes_soft_mse_and_includes_epoch_zero(self):
+ targets = np.array([-1, 0, 1])
+ complete = history(targets, winner=3)
+ self.assertEqual(native.selected_epoch(complete, targets), 3)
+ self.assertEqual(native.selected_epoch(history(targets, winner=0), targets), 0)
+ self.assertEqual(native.selected_epoch(history(np.zeros(3)), np.zeros(3)), 0)
+ complete[2]["benefit_mse"] = 0
+ with self.assertRaisesRegex(ValueError, "MSE"):
+ native.selected_epoch(complete, targets)
+ with self.assertRaisesRegex(ValueError, "every declared"):
+ native.selected_epoch(history(targets)[:-1], targets)
+ with self.assertRaisesRegex(ValueError, "signed paired"):
+ native.selected_epoch(history(targets), [0.5, 0, 1])
+
+ def test_inference_rejects_missing_local_weights_before_optional_imports(self):
+ with self.assertRaisesRegex(ValueError, "local complete"):
+ native.score_checkpoint(Path(__file__).parent / "no-such-checkpoint", [{"request": "x"}], "cpu")
+
+
+class NativeLifecycleTests(unittest.TestCase):
+ def setUp(self):
+ # Reuse a complete public synthetic experiment, not real solver outcomes.
+ self.fixture = fixtures.BenefitLifecycleTests()
+ self.fixture.setUp()
+ self.addCleanup(self.fixture.doCleanups)
+ self.run = self.fixture.root / "native"
+ self.head = self.fixture.run
+ fixtures.save(self.run / "plan.json", native.prepare(self.head))
+
+ def preflight(self):
+ f = self.fixture
+ return native.preflight(
+ self.run, self.head, f.baseline, f.original, f.training_plan, f.training_inputs, f.training_grades
+ )
+
+ def freeze(self):
+ context = self.preflight()
+ self.fixture.freeze()
+ targets = context["targets"][context["held"]]
+ selection_history = history(targets)
+ chosen = native.selected_epoch(selection_history, targets)
+ start = {
+ "plan_sha256": file_hash(self.run / "plan.json"),
+ "training_artifacts_sha256": context["training_artifacts_sha256"],
+ "eager_flags": native.PROTOCOL["eager_flags"],
+ "device": "mps",
+ "runtime": native.PROTOCOL["runtime_profiles"]["mps"],
+ "versions": native.PROTOCOL["versions"],
+ }
+ fixtures.save(self.run / "training-start.json", start)
+ models = {}
+ for variant in ("base", "lora"):
+ destination = self.run / variant
+ destination.mkdir()
+ (destination / "model.safetensors").write_bytes(b"synthetic weights; never loaded")
+ margins = {
+ split: [-2 if i % 4 < 2 else 2 for i in range(len(context[split]))] for split in ("calibration", "test")
+ }
+ scores = {split: native.benefit_scores(values).tolist() for split, values in margins.items()}
+ models[variant] = {
+ "checkpoint_files_sha256": {"model.safetensors": file_hash(destination / "model.safetensors")},
+ "parameters": 100,
+ "margins": margins,
+ "scores": scores,
+ "max_merge_margin_error": 0.0,
+ "max_export_margin_error": 0.0,
+ "policies": native._calibrated(context["original"]["calibration_rows"], scores["calibration"], 1e-5),
+ }
+ frozen = benefit.seal(
+ {
+ "format": native.FROZEN_FORMAT,
+ "plan_sha256": file_hash(self.run / "plan.json"),
+ "head_plan_sha256": file_hash(self.head / "plan.json"),
+ "training_start_sha256": file_hash(self.run / "training-start.json"),
+ "training_artifacts_sha256": context["training_artifacts_sha256"],
+ "training_grade_provenance": context["training_grade_provenance"],
+ "selection_history": selection_history,
+ "validation_signed_targets": targets.tolist(),
+ "selected_epoch": chosen,
+ "refit_losses": [0.5] * chosen,
+ "grading_provenance": self.fixture.grading,
+ "calibration_rows": context["original"]["calibration_rows"],
+ "calibration_grades_sha256": context["original"]["calibration_grades_sha256"],
+ "credit_usd": 1e-5,
+ "models": models,
+ }
+ )
+ fixtures.save(self.run / "frozen.json", frozen)
+ return frozen
+
+ def test_prepare_uses_only_one_training_paper_fold_and_preflight_leaves_test_marker_unopened(self):
+ context = self.preflight()
+ plan = context["plan"]
+ self.assertEqual(plan["training_pool_rows"], 20)
+ self.assertEqual(plan["training_rows"], 19)
+ fitting, held = set(plan["selection_fit_ids"]), set(plan["selection_validation_ids"])
+ self.assertFalse(fitting & held)
+ self.assertEqual(fitting | held, {row["id"] for row in self.fixture.eligible_rows})
+ by_id = {row["id"]: row for row in self.fixture.eligible_rows}
+ self.assertFalse(
+ {by_id[ident]["paper_group"] for ident in fitting} & {by_id[ident]["paper_group"] for ident in held}
+ )
+ marker = fixtures.save(self.fixture.original_dir / "grades-test.json", {"held": True})
+ original_read = benefit.read
+
+ def guarded(path):
+ self.assertNotEqual(Path(path), marker)
+ return original_read(path)
+
+ with patch.object(benefit, "read", side_effect=guarded):
+ self.preflight()
+
+ def test_shared_access_refuses_new_training_before_optional_model_work(self):
+ fixtures.save(self.head / "test-access.json", {"access": "already opened"})
+ with self.assertRaisesRegex(ValueError, "already authorized"):
+ native.prepare(self.head)
+ f = self.fixture
+ with patch.object(native, "_train_gpu") as model_work:
+ with self.assertRaisesRegex(ValueError, "already authorized"):
+ native.train(
+ self.run,
+ self.head,
+ f.baseline,
+ f.original,
+ f.training_plan,
+ f.training_inputs,
+ f.training_grades,
+ device="mps",
+ )
+ model_work.assert_not_called()
+
+ def test_training_review_provenance_mismatch_and_repeated_attempt_fail(self):
+ grades = benefit.read(self.fixture.training_grades)
+ fixtures.save(self.fixture.training_grades, grades | {"training_gold_rules_sha256": "a" * 64})
+ with self.assertRaisesRegex(ValueError, "review"):
+ self.preflight()
+ fixtures.save(self.fixture.training_grades, grades)
+ fixtures.save(self.run / "training-start.json", {"previous": "attempt"})
+ with self.assertRaises(FileExistsError):
+ self.preflight()
+
+ def test_freeze_recomputes_selection_scoring_and_calibration(self):
+ original = self.freeze()
+ native.checked_frozen(self.run, self.head, weights=True)
+ mutations = (
+ lambda item: item.update(selected_epoch=8),
+ lambda item: item["selection_history"][1].update(benefit_mse=100.0),
+ lambda item: item["models"]["lora"]["scores"]["test"].__setitem__(0, 0.0),
+ lambda item: item["models"]["lora"]["policies"]["1"].update(cutoff=100.0),
+ lambda item: item["models"]["lora"].update(max_export_margin_error=0.031),
+ )
+ for mutate in mutations:
+ changed = copy.deepcopy(original)
+ mutate(changed)
+ fixtures.save(self.run / "frozen.json", reseal(changed))
+ with self.subTest(change=mutate), self.assertRaises(ValueError):
+ native.checked_frozen(self.run, self.head)
+ fixtures.save(self.run / "frozen.json", original)
+ start = benefit.read(self.run / "training-start.json")
+ start["runtime"] = {**start["runtime"], "gradient_checkpointing": True}
+ fixtures.save(self.run / "training-start.json", start)
+ original["training_start_sha256"] = file_hash(self.run / "training-start.json")
+ fixtures.save(self.run / "frozen.json", reseal(original))
+ with self.assertRaises(ValueError):
+ native.checked_frozen(self.run, self.head)
+
+ def test_weights_optional_for_metadata_replay_but_checked_when_requested(self):
+ self.freeze()
+ (self.run / "lora" / "model.safetensors").write_bytes(b"changed weights")
+ native.checked_frozen(self.run, self.head)
+ with self.assertRaisesRegex(ValueError, "checkpoint bytes"):
+ native.authorize_test(self.run, self.head, verify_weights=True)
+ (self.run / "lora" / "model.safetensors").unlink()
+ access = native.authorize_test(self.run, self.head)
+ self.assertEqual(access["checkpoint_verification"], "manifest-only")
+ self.assertEqual(
+ access["checkpoint_manifest_sha256"],
+ native.checkpoint_manifest_hash(benefit.read(self.run / "frozen.json")),
+ )
+
+ def test_report_requires_both_access_receipts_before_reading_test_grades(self):
+ self.freeze()
+ grade_path = self.fixture.original_dir / "grades-test.json"
+ with patch.object(native.solver_policies, "_grade_rows") as reader:
+ with self.assertRaises(FileNotFoundError):
+ native.report(self.run, self.head, grade_path)
+ reader.assert_not_called()
+ fixtures.save(self.run / "test-access.json", native.authorize_test(self.run, self.head))
+ with self.assertRaises(FileNotFoundError):
+ native.report(self.run, self.head, grade_path)
+ reader.assert_not_called()
+ access = benefit.authorize_test(self.head)
+ fixtures.save(self.head / "test-access.json", access)
+ access["plan_sha256"] = "f" * 64
+ fixtures.save(self.head / "test-access.json", reseal(access))
+ with patch.object(native.solver_policies, "_grade_rows") as reader:
+ with self.assertRaisesRegex(ValueError, "test access"):
+ native.report(self.run, self.head, grade_path)
+ reader.assert_not_called()
+
+ def test_report_preserves_cheap_rescue_negative_gap_and_unknown_native_cost(self):
+ self.freeze()
+ fixtures.save(self.run / "test-access.json", native.authorize_test(self.run, self.head))
+ fixtures.save(self.head / "test-access.json", benefit.authorize_test(self.head))
+ rows = fixtures.outcomes("test", 8)
+ rows[0]["strong_correct"] = False # Cheap-only success is a real rescue.
+ path = fixtures.save(
+ self.fixture.original_dir / "grades-test.json",
+ {
+ "split": "test",
+ "plan_sha256": file_hash(self.fixture.original_plan),
+ "policies_frozen_sha256": file_hash(self.fixture.original),
+ **self.fixture.grading,
+ "rows": rows,
+ },
+ )
+ result = native.report(self.run, self.head, path)
+ self.assertEqual(result["ids"], [row["id"] for row in rows])
+ self.assertEqual(result["training_rows"], 19)
+ for variant in ("base", "lora"):
+ primary = result["models"][variant]["1"]
+ self.assertEqual(primary["mode"], "threshold")
+ self.assertEqual(primary["metrics"]["gap_vs_strong_pp"], -12.5)
+ self.assertEqual(primary["metrics"]["rescues"], 1)
+ self.assertFalse(primary["metrics"]["router_overhead_known"])
+ self.assertNotIn("router_cost_usd_per1000", primary["metrics"])
+ self.assertNotIn("net_cost_usd_per1000", primary["metrics"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benefit_outcomes.py b/examples/train-a-router/tests/test_benefit_outcomes.py
new file mode 100644
index 000000000..f1d52ccc0
--- /dev/null
+++ b/examples/train-a-router/tests/test_benefit_outcomes.py
@@ -0,0 +1,584 @@
+"""Outcome-head math, grouped selection, price binding and lifecycle guards."""
+
+import importlib.util
+import io
+import json
+import math
+import unittest
+from contextlib import redirect_stdout
+from pathlib import Path
+from unittest.mock import patch
+
+import numpy as np
+import test_benefit as fixtures
+
+import benefit
+import benefit_outcomes as outcomes
+from data import digest, file_hash
+
+
+def paired_rows(count=20):
+ return [{"id": f"row-{i}", "cheap_correct": i % 4 < 2, "strong_correct": i % 3 != 0} for i in range(count)]
+
+
+def fake_fit(matrix, targets, penalty):
+ del penalty
+ frequency = float(targets.mean())
+ return {
+ "weights": [0.0] * matrix.shape[1],
+ "bias": math.log(frequency / (1 - frequency)),
+ "optimizer": {
+ "success": True,
+ "iterations": 0,
+ "function_evaluations": 1,
+ "objective": 0.0,
+ "gradient_absolute_max": 0.0,
+ "message": "Synthetic test fit",
+ },
+ }
+
+
+class OutcomeMathTests(unittest.TestCase):
+ def test_signed_benefit_and_train_only_cost_mixture_include_rescues_and_floor(self):
+ targets = [[0, 0], [0, 1], [1, 0], [1, 1]]
+ costs = [[20, 80], [20, 40], [10, 80], [10, 40]]
+ means = outcomes.conditional_costs(targets, costs)
+ probabilities = np.array([[0.8, 0.2], [0.2, 0.8], [0.5, 0.5]])
+ scores, expected = outcomes.normalized_scores(probabilities, means)
+ np.testing.assert_allclose(expected, [[12, 72], [18, 48], [15, 60]])
+ np.testing.assert_allclose(scores, [-0.6 / 60, 0.6 / 30, 0])
+ # A negative incremental predicted cost uses the predeclared1credit
+ # floor; it never reverses the meaning of strong-minus-cheap benefit.
+ for state in ("correct", "wrong"):
+ means[f"cheap_{state}"]["mean_credits_proxy"] = 100
+ means[f"strong_{state}"]["mean_credits_proxy"] = 1
+ np.testing.assert_allclose(outcomes.normalized_scores(probabilities, means)[0], [-0.6, 0.6, 0])
+ for bad in ([[1.1, 0]], [[np.nan, 0]], [[0, 1, 0]]):
+ with self.assertRaises(ValueError):
+ outcomes.normalized_scores(bad, means)
+
+ def test_cv_uses_only_other_papers_for_both_heads_and_ties_prefer_larger_penalty(self):
+ rows = paired_rows()
+ matrix = np.array([[i, i % 3] for i in range(20)], dtype=float)
+ groups = [f"paper-{i // 4}" for i in range(20)]
+ folds = np.array(benefit.group_folds(groups))
+ targets = np.array([[row["cheap_correct"], row["strong_correct"]] for row in rows], dtype=float)
+ calls = []
+
+ def traced(x, y, penalty):
+ calls.append((x.copy(), y.copy(), penalty))
+ return fake_fit(x, y, penalty)
+
+ with patch.object(outcomes, "fit_logistic", side_effect=traced):
+ head = outcomes.fit_head(matrix, rows, groups, [[10, 30]] * 20)
+ self.assertEqual(len(calls), 42)
+ self.assertEqual(head["penalty"], 10)
+ for index, penalty in enumerate(outcomes.PROTOCOL["penalties"]):
+ for fold in range(5):
+ for arm in range(2):
+ x, y, observed_penalty = calls[index * 10 + fold * 2 + arm]
+ np.testing.assert_array_equal(x, matrix[folds != fold])
+ np.testing.assert_array_equal(y, targets[folds != fold, arm])
+ self.assertEqual(observed_penalty, penalty)
+ self.assertFalse(
+ {groups[int(row[0])] for row in x} & {groups[i] for i in np.flatnonzero(folds == fold)}
+ )
+ for arm, (x, y, penalty) in enumerate(calls[-2:]):
+ np.testing.assert_array_equal(x, matrix)
+ np.testing.assert_array_equal(y, targets[:, arm])
+ self.assertEqual(penalty, head["penalty"])
+ for candidate in head["cv"]:
+ p = np.array(candidate["oof_probabilities"])
+ expected = float((-targets * np.log(p) - (1 - targets) * np.log1p(-p)).mean(axis=0).sum())
+ self.assertEqual(candidate["summed_binary_logloss"], expected)
+
+ def test_bad_labels_features_and_cost_classes_fail_before_optimization(self):
+ rows, groups = paired_rows(), [f"paper-{i // 4}" for i in range(20)]
+ with patch.object(outcomes, "fit_logistic") as fitting:
+ with self.assertRaisesRegex(ValueError, "align"):
+ outcomes.fit_head(np.full((20, 2), np.nan), rows, groups, [[10, 30]] * 20)
+ rows[0]["cheap_correct"] = 1
+ with self.assertRaises(TypeError):
+ outcomes.fit_head(np.zeros((20, 2)), rows, groups, [[10, 30]] * 20)
+ fitting.assert_not_called()
+ with self.assertRaisesRegex(ValueError, "correct and incorrect"):
+ outcomes.conditional_costs([[1, 1], [0, 1]], [[2, 4], [3, 5]])
+
+ @unittest.skipUnless(
+ importlib.util.find_spec("scipy") is not None, "Optional SciPy fitting dependency not installed"
+ )
+ def test_real_optimizer_recovers_intercept_and_informative_logistic_direction(self):
+ x = np.zeros((20, 2))
+ y = np.array([0] * 5 + [1] * 15, dtype=float)
+ model = outcomes.fit_logistic(x, y, 0.1)
+ np.testing.assert_allclose(outcomes.probability(x, model), 0.75, atol=1e-8)
+ x[:, 0] = np.linspace(-2, 2, 20)
+ model = outcomes.fit_logistic(x, y, 0.1)
+ self.assertGreater(model["weights"][0], 0)
+ self.assertLess(outcomes._nll(y, outcomes.probability(x, model)), outcomes._nll(y, np.full(20, 0.75)))
+
+
+class OutcomeLifecycleTests(unittest.TestCase):
+ def setUp(self):
+ fixture = fixtures.BenefitLifecycleTests()
+ fixture.setUp()
+ self.addCleanup(fixture.doCleanups)
+ self.fixture = fixture
+ self.root, self.baseline, self.original = fixture.root, fixture.baseline, fixture.original
+ self.training = fixture.training
+ self.run = self.root / "outcome-head"
+ v2 = {"outcome_grader_source_sha256": "b" * 64, "outcome_grader_protocol_sha256": "c" * 64}
+ fixture.grading.update(v2)
+ original = outcomes.read(self.original)
+ original.pop("sha256")
+ fixtures.save(self.original, benefit.seal(original | v2))
+ fixtures.save(fixture.training_plan, outcomes.read(fixture.training_plan) | v2)
+ fixtures.save(
+ fixture.training_grades,
+ outcomes.read(fixture.training_grades) | v2 | {"plan_sha256": file_hash(fixture.training_plan)},
+ )
+ protocol = patch.dict(
+ outcomes.PROTOCOL,
+ {"training_pool_sizes": [20, 30], "required_original_exclusions": ["train-0"], "dimensions": 3},
+ )
+ protocol.start()
+ self.addCleanup(protocol.stop)
+ self.rows_path = fixtures.save(self.training / "training-rows.json", fixture.dataset["splits"]["train"])
+ grades = outcomes.read(fixture.training_grades)
+ for index, row in enumerate(grades["rows"]):
+ row["strong_correct"] = index % 3 != 0
+ for arm in outcomes.ARMS:
+ row[f"{arm}_record_sha256"] = digest([arm, row["id"]])
+ grades["collection_summary_sha256"] = "d" * 64
+ fixtures.save(fixture.training_grades, grades)
+ rates = {
+ "cheap": {
+ "input_tokens": {"numerator": 1, "denominator": 10},
+ "output_tokens": {"numerator": 1, "denominator": 5},
+ },
+ "strong": {
+ "input_tokens": {"numerator": 3, "denominator": 10},
+ "output_tokens": {"numerator": 1, "denominator": 2},
+ },
+ }
+ price_rows = []
+ for arm, model in zip(outcomes.ARMS, ("Qwen/Qwen3.5-4B", "Qwen/Qwen3.8-27B-FP8"), strict=True):
+ for unit, rate in rates[arm].items():
+ price_rows.append(
+ {
+ "model": model,
+ "unit": unit,
+ "profile": "default",
+ "region": "us",
+ "usd_per_unit": {"numerator": rate["numerator"], "denominator": rate["denominator"] * 100000},
+ }
+ )
+ self.price_path = fixtures.save(
+ self.root / "prices.json",
+ {"currency": "usd", "pricing_version": "synthetic-v1", "rate_book_sha256": "f" * 64, "items": price_rows},
+ )
+ records, cost_rows = [], []
+ for row in grades["rows"]:
+ record = {"id": row["id"]}
+ values = []
+ for arm, initial_cost in zip(outcomes.ARMS, (2, 4), strict=True):
+ calls = [{"prompt_tokens": 3, "completion_tokens": 5, "credits_proxy": initial_cost}]
+ if not row[f"{arm}_correct"]:
+ calls.append({"prompt_tokens": 1, "completion_tokens": 1, "credits_proxy": 1})
+ charge = sum(call["credits_proxy"] for call in calls)
+ record.update(
+ {
+ f"{arm}_correct": row[f"{arm}_correct"],
+ f"{arm}_record_sha256": row[f"{arm}_record_sha256"],
+ f"{arm}_calls": calls,
+ f"{arm}_credits_proxy": charge,
+ }
+ )
+ values.append(charge)
+ records.append(record)
+ cost_rows.append(values)
+ targets = [[int(row[f"{arm}_correct"]) for arm in outcomes.ARMS] for row in grades["rows"]]
+ self.cost_path = fixtures.save(
+ self.root / "cost-proxy.json",
+ {
+ "format": "sie-training-uncached-cloud-cost-proxy-v1",
+ "split": "train",
+ "n": len(records),
+ "credit_usd": 1e-5,
+ "training_plan_sha256": file_hash(fixture.training_plan),
+ "training_grades_sha256": file_hash(fixture.training_grades),
+ "training_collection_sha256": grades["collection_summary_sha256"],
+ "price_source": {
+ "sha256": file_hash(self.price_path),
+ "public_rows": price_rows,
+ "pricing_version": "synthetic-v1",
+ "rate_book_sha256": "f" * 64,
+ },
+ "rates_credits_per_token": rates,
+ "rows": records,
+ "conditional_means": outcomes.conditional_costs(targets, cost_rows),
+ },
+ )
+ fixtures.save(self.run / "plan.json", self.prepare())
+
+ def prepare(self):
+ fixture = self.fixture
+ return outcomes.prepare(
+ self.baseline,
+ self.original,
+ fixture.training_plan,
+ fixture.training_inputs,
+ self.rows_path,
+ fixture.eligibility,
+ attest_test_outcomes_unopened=True,
+ )
+
+ def fit(self):
+ fixture = self.fixture
+ with patch.object(outcomes, "fit_logistic", side_effect=fake_fit):
+ selected = outcomes.fit(
+ self.run,
+ self.baseline,
+ fixture.training_plan,
+ fixture.training_inputs,
+ self.rows_path,
+ fixture.training_grades,
+ self.baseline / "development-embeddings",
+ self.cost_path,
+ self.price_path,
+ )
+ fixtures.save(self.run / "selected-model.json", selected)
+ return selected
+
+ def freeze(self):
+ self.fit()
+ result = outcomes.freeze(self.run, self.baseline, self.original)
+ fixtures.save(self.run / "frozen.json", result)
+ return result
+
+ def test_fit_cannot_open_calibration_or_test_and_freeze_requires_durable_selected_model(self):
+ with self.assertRaises(FileNotFoundError):
+ outcomes.freeze(self.run, self.baseline, self.original)
+ read = outcomes.read
+ original = self.original
+
+ def training_only(path):
+ self.assertNotEqual(Path(path), original)
+ self.assertNotIn(Path(path).name, ("grades-calibration.json", "grades-test.json"))
+ return read(path)
+
+ with patch.object(outcomes, "read", side_effect=training_only):
+ selected = self.fit()
+ self.assertEqual(selected["head"]["training_ids"], [row["id"] for row in self.fixture.eligible_rows])
+ self.assertFalse((self.run / "frozen.json").exists())
+ frozen = outcomes.freeze(self.run, self.baseline, self.original)
+ self.assertEqual(frozen["selected_model_sha256"], file_hash(self.run / "selected-model.json"))
+ self.assertEqual(tuple(frozen["selections"]), ("1", "0", "2", "5"))
+
+ def test_cost_proxy_recomputes_every_call_and_rejects_swapped_labels_prices_or_ids(self):
+ fixture = self.fixture
+ grades = outcomes.read(fixture.training_grades)
+ original = outcomes.read(self.cost_path)
+ costs, means, _ = outcomes.cost_proxy(
+ self.cost_path, self.price_path, fixture.training_plan, fixture.training_grades, grades
+ )
+ self.assertEqual(means, original["conditional_means"])
+ self.assertEqual(len(costs), 19)
+ cases = []
+ bad = json.loads(json.dumps(original))
+ bad["rows"][0]["cheap_calls"][0]["credits_proxy"] += 1
+ cases.append(bad)
+ bad = json.loads(json.dumps(original))
+ bad["rows"][0]["cheap_calls"][0]["prompt_tokens"] = True
+ cases.append(bad)
+ bad = json.loads(json.dumps(original))
+ bad["rows"][0]["cheap_correct"] = not bad["rows"][0]["cheap_correct"]
+ cases.append(bad)
+ bad = json.loads(json.dumps(original))
+ bad["rows"].reverse()
+ cases.append(bad)
+ bad = json.loads(json.dumps(original))
+ bad["price_source"]["public_rows"][0]["usd_per_unit"]["numerator"] = 9
+ cases.append(bad)
+ for value in cases:
+ with self.subTest(value=cases.index(value)):
+ fixtures.save(self.cost_path, value)
+ with self.assertRaises(ValueError):
+ outcomes.cost_proxy(
+ self.cost_path, self.price_path, fixture.training_plan, fixture.training_grades, grades
+ )
+
+ def test_changed_plan_review_hashes_or_vectors_fail_before_optimizer(self):
+ fixture = self.fixture
+ grades = outcomes.read(fixture.training_grades)
+ for key in outcomes.GRADE_HASHES:
+ with self.subTest(field=key):
+ fixtures.save(fixture.training_grades, grades | {key: "a" * 64})
+ with patch.object(outcomes, "fit_head") as fitting:
+ with self.assertRaises(ValueError):
+ self.fit()
+ fitting.assert_not_called()
+ fixtures.save(fixture.training_grades, grades)
+ with (self.baseline / "development-embeddings/vectors.jsonl.gz").open("ab") as stream:
+ stream.write(b"changed")
+ with patch.object(outcomes, "fit_head") as fitting:
+ with self.assertRaisesRegex(ValueError, "vectors changed"):
+ self.fit()
+ fitting.assert_not_called()
+
+ def test_resealed_cv_objective_cost_means_and_normalized_score_tampering_fail(self):
+ self.freeze()
+ original_model = outcomes.read(self.run / "selected-model.json")
+ original_frozen = outcomes.read(self.run / "frozen.json")
+ for kind in ("objective", "mean", "penalty"):
+ changed = json.loads(json.dumps(original_model))
+ changed.pop("sha256")
+ if kind == "objective":
+ changed["head"]["cv"][0]["summed_binary_logloss"] = 0
+ elif kind == "mean":
+ changed["head"]["conditional_cost_statistics"]["cheap_correct"]["mean_credits_proxy"] += 1
+ else:
+ changed["head"]["penalty"] = 0.1
+ fixtures.save(self.run / "selected-model.json", benefit.seal(changed))
+ with self.subTest(kind=kind), self.assertRaises(ValueError):
+ outcomes.checked_model(self.run)
+ fixtures.save(self.run / "selected-model.json", original_model)
+ changed = json.loads(json.dumps(original_frozen))
+ changed.pop("sha256")
+ changed["scores"]["test"][0] += 1
+ fixtures.save(self.run / "frozen.json", benefit.seal(changed))
+ with self.assertRaisesRegex(ValueError, "cost-normalized"):
+ outcomes.authorize_test(self.run)
+
+ def test_report_checks_access_chain_before_opening_test_and_never_reselects_on_test(self):
+ frozen = self.freeze()
+ test = self.root / "must-not-open.json"
+ with self.assertRaises(FileNotFoundError):
+ outcomes.report(self.run, test)
+ fixtures.save(self.run / "test-access.json", outcomes.authorize_test(self.run))
+ access = outcomes.read(self.run / "test-access.json")
+ access.pop("sha256")
+ fixtures.save(self.run / "test-access.json", benefit.seal(access | {"frozen_sha256": "a" * 64}))
+ with self.assertRaisesRegex(ValueError, "Test-access"):
+ outcomes.report(self.run, test)
+ fixtures.save(self.run / "test-access.json", benefit.seal(access))
+ rows = fixtures.outcomes("test", 8)
+ for row in rows:
+ row["cheap_correct"] = False
+ fixtures.save(
+ test,
+ {
+ "split": "test",
+ "plan_sha256": file_hash(self.fixture.original_plan),
+ "policies_frozen_sha256": file_hash(self.original),
+ **self.fixture.grading,
+ "rows": rows,
+ },
+ )
+ selector = outcomes.solver_report.select_cutoff
+
+ def only_calibration(rows, *args, **kwargs):
+ self.assertTrue(all(row["id"].startswith("calibration-") for row in rows))
+ return selector(rows, *args, **kwargs)
+
+ with patch.object(outcomes.solver_report, "select_cutoff", side_effect=only_calibration):
+ report = outcomes.report(self.run, test)
+ self.assertEqual(report["ids"], [row["id"] for row in rows])
+ self.assertEqual(report["policies"]["1"]["cutoff"], frozen["selections"]["1"]["cutoff"])
+ self.assertEqual(report["training_rows"], 19)
+ with self.assertRaises(FileExistsError):
+ outcomes.fit(self.run, None, None, None, None, None, None, None, None)
+
+ def test_new_source_only_review_explicitly_acknowledges_reused_outcomes(self):
+ eligibility = outcomes.read(self.fixture.eligibility)
+ eligibility.pop("decided_before_training_outcomes")
+ eligibility.update(decided_before_new_training_outcomes=True, reused_training_outcomes_already_available=True)
+ fixtures.save(self.fixture.eligibility, eligibility)
+ plan = outcomes.read(self.fixture.training_plan)
+ fixtures.save(self.fixture.training_plan, plan | {"eligibility_sha256": file_hash(self.fixture.eligibility)})
+ revised = self.prepare()
+ self.assertTrue(revised["eligibility"]["reused_training_outcomes_already_available"])
+ eligibility["source_only"] = False
+ fixtures.save(self.fixture.eligibility, eligibility)
+ with self.assertRaisesRegex(ValueError, "Eligibility"):
+ self.prepare()
+
+ def test_post_collection_review_filters_training_and_held_rows_without_rewriting_collection(self):
+ fixture = self.fixture
+ source_review = outcomes.source_review
+ excluded = {"train-1", "calibration-0", "test-0"}
+ records = [
+ {
+ "id": row["id"],
+ "split": split,
+ "question": row["request"],
+ "question_sha256": digest(row["request"]),
+ "request_sha256": digest(row["request"]),
+ "original_reference": "1",
+ "reference": "1",
+ "semantic_kind": "scalar",
+ "decision": "exclude" if row["id"] in excluded else "retain",
+ "reason": "Synthetic source-only missing-input review",
+ }
+ for split, rows in fixture.dataset["splits"].items()
+ if split != "validation"
+ for row in rows
+ ]
+ review_path = fixtures.save(
+ self.root / "source-review.json",
+ {
+ "format": source_review.FORMAT,
+ "protocol": source_review.PROTOCOL,
+ "finalized": True,
+ "source_revision": outcomes.SOURCE_REVISION,
+ "source_hashes": outcomes.SOURCE_HASHES,
+ "training_outcomes_already_available": True,
+ "solver_test_outcomes_accessed": False,
+ "records": records,
+ },
+ )
+ metadata = source_review.metadata(review_path)
+ original = outcomes.read(self.original)
+ original.pop("sha256")
+ original["calibration_rows"] = original["calibration_rows"][1:]
+ original["embedding_calibration_credits"] = original["embedding_calibration_credits"][1:]
+ fixtures.save(self.original, benefit.seal(original | metadata))
+ original_collection_bytes = fixture.training_plan.read_bytes()
+ plan = outcomes.prepare(
+ self.baseline,
+ self.original,
+ fixture.training_plan,
+ fixture.training_inputs,
+ self.rows_path,
+ fixture.eligibility,
+ source_review_path=review_path,
+ attest_test_outcomes_unopened=True,
+ )
+ self.assertEqual(fixture.training_plan.read_bytes(), original_collection_bytes)
+ self.assertEqual(len(plan["source_populations"]["train"]), 19)
+ self.assertEqual(len(plan["populations"]["train"]), 18)
+ self.assertEqual(outcomes.checked_plan(plan), plan)
+ graded = outcomes.read(fixture.training_grades)
+ graded.update(
+ metadata, collected_ids=[row["id"] for row in plan["source_populations"]["train"]], excluded_ids=["train-1"]
+ )
+ graded["rows"] = [row for row in graded["rows"] if row["id"] != "train-1"]
+ fixtures.save(fixture.training_grades, graded)
+ rows, _ = outcomes._training(
+ plan, fixture.training_plan, fixture.training_inputs, self.rows_path, fixture.training_grades
+ )
+ self.assertEqual([row["id"] for row in rows], [row["id"] for row in plan["populations"]["train"]])
+ for split in ("calibration", "test"):
+ held = outcomes.held_rows(plan, fixture.dataset, split)
+ self.assertEqual([row["id"] for row in held], [row["id"] for row in plan["populations"][split]])
+ self.assertNotIn(f"{split}-0", [row["id"] for row in held])
+ changed = json.loads(json.dumps(plan))
+ changed.pop("sha256")
+ changed["populations"]["test"].pop()
+ with self.assertRaisesRegex(ValueError, "nonexcluded"):
+ outcomes.checked_plan(benefit.seal(changed))
+
+ def test_prepare_rejects_unaudited_expansion_or_held_population_overlap(self):
+ original = outcomes.read(self.rows_path)
+ extra = [
+ {**row, "id": f"extra-{i}", "request": f"new training question {i}", "family": f"extra-family-{i}"}
+ for i, row in enumerate(original[:10])
+ ]
+ fixtures.save(self.rows_path, original + extra)
+ with self.assertRaisesRegex(ValueError, "expansion audit"):
+ self.prepare()
+ extra[0]["family"] = self.fixture.dataset["splits"]["calibration"][0]["family"]
+ fixtures.save(self.rows_path, original + extra)
+ with self.assertRaises(ValueError):
+ self.prepare()
+
+ def test_expanded_plan_keeps_source_prefix_and_same_whole_paper_folds(self):
+ original = outcomes.read(self.rows_path)
+ pool = original + [
+ {**row, "id": f"extra-{i}", "request": f"new training question {i}", "family": f"extra-family-{i}"}
+ for i, row in enumerate(original[:10])
+ ]
+ fixtures.save(self.rows_path, pool)
+ eligibility = outcomes.read(self.fixture.eligibility)
+ eligibility["original_pool_ids"] = [row["id"] for row in pool]
+ eligibility["eligible_train_ids"] = [row["id"] for row in pool if row["id"] != "train-0"]
+ fixtures.save(self.fixture.eligibility, eligibility)
+ inputs = {"train": [{key: row[key] for key in ("id", "request")} for row in pool if row["id"] != "train-0"]}
+ fixtures.save(self.fixture.training_inputs, inputs)
+ collection = outcomes.read(self.fixture.training_plan)
+ collection.update(
+ source_counts={"train": 30},
+ counts={"train": 29},
+ inputs_sha256=digest(inputs),
+ eligible_train_ids=eligibility["eligible_train_ids"],
+ eligibility_sha256=file_hash(self.fixture.eligibility),
+ )
+ fixtures.save(self.fixture.training_plan, collection)
+ expansion = fixtures.save(
+ self.root / "expansion-audit.json",
+ {
+ "source_revision": outcomes.SOURCE_REVISION,
+ "source_hashes": outcomes.SOURCE_HASHES,
+ "original_dataset_sha256": digest(self.fixture.dataset),
+ "expanded_rows_sha256": digest(pool),
+ "original_prefix_rows": 20,
+ "training_rows": 30,
+ "ids": [row["id"] for row in pool],
+ "families": [row["family"] for row in pool],
+ },
+ )
+ plan = outcomes.prepare(
+ self.baseline,
+ self.original,
+ self.fixture.training_plan,
+ self.fixture.training_inputs,
+ self.rows_path,
+ self.fixture.eligibility,
+ expansion_audit_path=expansion,
+ attest_test_outcomes_unopened=True,
+ )
+ self.assertEqual(len(plan["populations"]["train"]), 29)
+ old = outcomes.read(self.run / "plan.json")
+ old_assignments = {
+ row["paper_group"]: fold
+ for row, fold in zip(old["populations"]["train"], old["training_folds"], strict=True)
+ }
+ self.assertTrue(
+ all(
+ old_assignments[row["paper_group"]] == fold
+ for row, fold in zip(plan["populations"]["train"], plan["training_folds"], strict=True)
+ )
+ )
+ self.assertEqual(plan["training_artifacts_sha256"]["expansion_audit"], file_hash(expansion))
+
+ def test_cli_preserves_existing_output_before_reading_any_inputs(self):
+ destination = self.run / "plan.json"
+ before = destination.read_bytes()
+ args = [
+ "benefit_outcomes.py",
+ "prepare",
+ "--run",
+ str(self.run),
+ "--baseline",
+ "absent",
+ "--original-frozen",
+ "absent",
+ "--training-plan",
+ "absent",
+ "--training-inputs",
+ "absent",
+ "--training-rows",
+ "absent",
+ "--eligibility",
+ "absent",
+ "--attest-test-outcomes-unopened",
+ ]
+ with patch("sys.argv", args), patch.object(outcomes, "prepare") as preparing, redirect_stdout(io.StringIO()):
+ with self.assertRaises(FileExistsError):
+ outcomes.main()
+ preparing.assert_not_called()
+ self.assertEqual(destination.read_bytes(), before)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benefit_outcomes_native.py b/examples/train-a-router/tests/test_benefit_outcomes_native.py
new file mode 100644
index 000000000..3bfaa15a4
--- /dev/null
+++ b/examples/train-a-router/tests/test_benefit_outcomes_native.py
@@ -0,0 +1,384 @@
+"""Two-question target mapping and native training/calibration/test boundaries."""
+
+import json
+import unittest
+from types import SimpleNamespace
+from unittest.mock import patch
+
+import numpy as np
+import test_benefit as fixtures
+import test_benefit_outcomes as outcome_fixtures
+
+import benefit
+import benefit_outcomes as outcomes
+import benefit_outcomes_native as native
+from data import file_hash
+
+
+def fake_laya():
+ def state(_tokenizer, text, _max_len):
+ return list(text.encode())
+
+ def items(_tokenizer, rows, _max_len, _head_len, label_smoothing):
+ if label_smoothing != 0:
+ raise AssertionError("Outcome targets must remain hard labels")
+ result = []
+ for row in rows:
+ for name, question in row["questions"].items():
+ result.append(
+ {
+ "state_ids": list(row["state"].encode()),
+ "q": {"ins": question["instructions"], "crit": question["criteria"]},
+ "k": 2,
+ "target": [row["gold"][name]["probabilities"][option] for option in native.OPTIONS],
+ }
+ )
+ return result, {}
+
+ return SimpleNamespace(encode_state=state, items_from_rows=items)
+
+
+class NativeOutcomeMathTests(unittest.TestCase):
+ def test_float64_replay_rejects_broadcast_nonfinite_and_meaningful_drift(self):
+ values = np.array([[1e-5, 500.0], [0.0, 0.7]])
+ self.assertTrue(native._matches_recorded(values, np.nextafter(values, np.inf)))
+ for changed in (values[0], np.full_like(values, np.nan), np.full_like(values, np.inf), values + 1e-6):
+ with self.subTest(changed=changed.tolist()):
+ self.assertFalse(native._matches_recorded(values, changed))
+ self.assertFalse(native._matches_recorded([], []))
+
+ def test_known_replay_source_does_not_relax_any_other_dependency(self):
+ current = native.code_hashes()
+ historical = current | {"benefit_outcomes_native.py": next(iter(native.REPLAY_COMPATIBLE_NATIVE_SOURCES))}
+ self.assertTrue(native._source_matches(historical, replay=True))
+ self.assertFalse(native._source_matches(historical))
+ for changed in (
+ historical | {"benefit_outcomes_native.py": "f" * 64},
+ historical | {"finetune.py": "f" * 64},
+ historical | {"extra.py": "f" * 64},
+ {key: value for key, value in historical.items() if key != "benefit_outcomes.py"},
+ ):
+ self.assertFalse(native._source_matches(changed, replay=True))
+
+ def test_two_questions_preserve_request_arm_and_incorrect_correct_order(self):
+ rows = [{"id": "a", "request": "first"}, {"id": "b", "request": "second"}]
+ targets = np.array([[0, 1], [1, 0]])
+ items = native.encode_rows(fake_laya(), None, rows, targets)
+ self.assertEqual([item["target"] for item in items], [[1, 0], [0, 1], [0, 1], [1, 0]])
+ records = [(0, [0, value], item["target"], 2) for value, item in zip((1.0, 2.0, 3.0, 4.0), items, strict=True)]
+ np.testing.assert_array_equal(native.records_to_margins(records, targets), [[1, 2], [3, 4]])
+ records[0], records[1] = records[1], records[0]
+ with self.assertRaisesRegex(ValueError, "order"):
+ native.records_to_margins(records, targets)
+
+ def test_encoder_rejects_dropped_reordered_or_reversed_option_items(self):
+ rows, targets = [{"request": "a"}, {"request": "b"}], [[0, 1], [1, 0]]
+ original = fake_laya()
+ for change in ("drop", "questions", "options", "requests"):
+
+ def changed(*args, change=change, **kwargs):
+ items, skipped = original.items_from_rows(*args, **kwargs)
+ if change == "drop":
+ return items[:-1], {"options_collapsed": 1}
+ if change == "questions":
+ items[0], items[1] = items[1], items[0]
+ elif change == "options":
+ items[0]["q"]["crit"] = dict(reversed(list(items[0]["q"]["crit"].items())))
+ else:
+ items[0]["state_ids"] = items[2]["state_ids"]
+ return items, skipped
+
+ with self.subTest(change=change), self.assertRaises(ValueError):
+ native.encode_rows(
+ SimpleNamespace(encode_state=original.encode_state, items_from_rows=changed), None, rows, targets
+ )
+
+ def test_epoch_selection_uses_both_outcome_losses_and_includes_zero(self):
+ targets = np.array([[0, 1], [1, 0]])
+ with patch.dict(native.PROTOCOL, {"selection_epochs": 2}):
+ # Epoch1 improves one arm while badly harming the other: choose0.
+ values = [np.zeros((2, 2)), np.array([[2, 1], [-2, -1]]), np.zeros((2, 2))]
+ history = [
+ {"epoch": epoch, "margins": margins.tolist(), "summed_binary_nll": native.outcome_nll(margins, targets)}
+ for epoch, margins in enumerate(values)
+ ]
+ self.assertEqual(native.selected_epoch(history, targets), 0)
+ self.assertAlmostEqual(history[0]["summed_binary_nll"], 2 * np.log(2))
+ history[0]["summed_binary_nll"] = 0
+ with self.assertRaisesRegex(ValueError, "NLL"):
+ native.selected_epoch(history, targets)
+ self.assertTrue(np.isfinite(native.outcome_nll([[1000, -1000]], [[1, 0]])))
+
+ def test_epoch_tie_uses_recorded_losses_after_numeric_validation(self):
+ targets = [[0, 1]]
+ loss = native.outcome_nll([[0, 0]], targets)
+ history = [{"epoch": epoch, "margins": [[0, 0]], "summed_binary_nll": loss} for epoch in range(3)]
+ with (
+ patch.dict(native.PROTOCOL, {"selection_epochs": 2}),
+ patch.object(
+ native, "outcome_nll", side_effect=[np.nextafter(loss, np.inf), loss, np.nextafter(loss, -np.inf)]
+ ),
+ ):
+ self.assertEqual(native.selected_epoch(history, targets), 0)
+
+
+class NativeOutcomeLifecycleTests(unittest.TestCase):
+ def setUp(self):
+ fixture = outcome_fixtures.OutcomeLifecycleTests()
+ fixture.setUp()
+ self.addCleanup(fixture.doCleanups)
+ self.fixture = fixture
+ self.head_run, self.run = fixture.run, fixture.root / "native-outcomes"
+ protocol = patch.dict(native.PROTOCOL, {"selection_epochs": 2})
+ protocol.start()
+ self.addCleanup(protocol.stop)
+ fixtures.save(self.run / "plan.json", native.prepare(self.head_run))
+
+ def trained(self):
+ fixture = self.fixture
+ fixture.freeze()
+ embedding = outcomes.checked_model(self.head_run)
+ plan = native.checked_plan(self.run, self.head_run)
+ targets = np.array(embedding["head"]["targets"])
+ held = np.array(plan["head_plan"]["training_folds"]) == 0
+ history = []
+ for epoch, strength in enumerate((0, 0.8, 0.4)):
+ margins = strength * (2 * targets[held] - 1)
+ history.append(
+ {
+ "epoch": epoch,
+ "train_loss": None if epoch == 0 else 0.5,
+ "margins": margins.tolist(),
+ "summed_binary_nll": native.outcome_nll(margins, targets[held]),
+ }
+ )
+ artifacts = {
+ **{key: plan["head_plan"]["training_artifacts_sha256"][key] for key in ("plan", "inputs", "rows")},
+ "grades": embedding["training_grades_sha256"],
+ "cost_proxy": embedding["cost_proxy_sha256"],
+ "price_snapshot": embedding["price_snapshot_sha256"],
+ }
+ fixtures.save(
+ self.run / "training-start.json",
+ {
+ "plan_sha256": file_hash(self.run / "plan.json"),
+ "versions": native.PROTOCOL["versions"],
+ "device": "cuda",
+ "runtime": native.PROTOCOL["runtime"],
+ "eager_flags": native.PROTOCOL["eager_flags"],
+ "training_artifacts_sha256": artifacts,
+ },
+ )
+ fixtures.save(self.run / "selection-config.json", native._config_values(2))
+ fixtures.save(self.run / "refit-config.json", native._config_values(1))
+ fixtures.save(self.run / "selection-history.json", history)
+ fixtures.save(
+ self.run / "epoch-selected.json",
+ {
+ "epoch": 1,
+ "history_sha256": file_hash(self.run / "selection-history.json"),
+ "validation_ids": plan["selection_validation_ids"],
+ "targets": targets[held].tolist(),
+ "criterion": native.PROTOCOL["selection"],
+ },
+ )
+ checkpoints = {}
+ for variant in ("base", "lora"):
+ path = self.run / variant
+ fixtures.save(path / "rl_agent_config.json", {})
+ (path / "model.safetensors").write_bytes(b"synthetic weight bytes; never loaded")
+ margins = np.zeros((int(held.sum()), 2)).tolist()
+ checkpoints[variant] = {
+ "files_sha256": {
+ name: file_hash(path / name) for name in ("rl_agent_config.json", "model.safetensors")
+ },
+ "parameters": 10,
+ "reference_margins": margins,
+ "merged_margins": margins,
+ "exported_margins": margins,
+ "max_merge_margin_error": 0.0,
+ "max_export_margin_error": 0.0,
+ }
+ trained = benefit.seal(
+ {
+ "format": native.TRAINED_FORMAT,
+ "plan_sha256": file_hash(self.run / "plan.json"),
+ "head_plan_sha256": plan["head_plan_sha256"],
+ "training_start_sha256": file_hash(self.run / "training-start.json"),
+ "training_artifacts_sha256": artifacts,
+ "training_grade_provenance": embedding["training_grade_provenance"],
+ "training_targets": targets.tolist(),
+ "training_cost_proxies": embedding["head"]["training_cost_proxies"],
+ "conditional_cost_statistics": embedding["head"]["conditional_cost_statistics"],
+ "cost_proxy_provenance": embedding["cost_proxy_provenance"],
+ "selection_history": history,
+ "selected_epoch": 1,
+ "refit_losses": [0.5],
+ "checkpoints": checkpoints,
+ "receipts_sha256": {
+ name: file_hash(self.run / name)
+ for name in (
+ "selection-config.json",
+ "selection-history.json",
+ "epoch-selected.json",
+ "refit-config.json",
+ )
+ },
+ }
+ )
+ fixtures.save(self.run / "trained.json", trained)
+ return trained
+
+ def freeze(self):
+ trained = self.trained()
+
+ def infer(_checkpoint, rows, _device):
+ # The first request prefers cheap; all others prefer strong. These
+ # synthetic features never consult calibration or test correctness.
+ result = np.tile([-2.0, 2.0], (len(rows), 1))
+ result[0] = [2.0, -2.0]
+ return result
+
+ with patch.object(native, "checkpoint_margins", side_effect=infer):
+ frozen = native.freeze(self.run, self.head_run, self.fixture.baseline, self.fixture.original)
+ fixtures.save(self.run / "frozen.json", frozen)
+ return trained, frozen
+
+ def test_preflight_uses_training_only_and_rejects_non_cuda_before_gpu_imports(self):
+ fixture, data = self.fixture, self.fixture.fixture
+ context = native.preflight(
+ self.run,
+ self.head_run,
+ fixture.baseline,
+ data.training_plan,
+ data.training_inputs,
+ fixture.rows_path,
+ data.training_grades,
+ fixture.cost_path,
+ fixture.price_path,
+ )
+ self.assertEqual(context["targets"].shape, (19, 2))
+ self.assertFalse(set(context["plan"]["selection_fit_ids"]) & set(context["plan"]["selection_validation_ids"]))
+ self.assertNotIn("calibration", context)
+ with self.assertRaisesRegex(ValueError, "CUDA only"):
+ native.train(None, None, None, None, None, None, None, None, None, device="mps")
+
+ def test_replay_is_metadata_only_and_weight_bytes_can_be_verified_separately(self):
+ self.freeze()
+ native.checked_frozen(self.run, self.head_run, weights=True)
+ (self.run / "base/model.safetensors").unlink()
+ (self.run / "lora/model.safetensors").unlink()
+ native.checked_frozen(self.run, self.head_run)
+ with self.assertRaises(FileNotFoundError):
+ native.checked_frozen(self.run, self.head_run, weights=True)
+ receipt = native.authorize_test(self.run, self.head_run)
+ self.assertEqual(receipt["checkpoint_verification"], "manifest-only")
+
+ def test_historical_training_replays_but_cannot_train_or_freeze_under_new_source(self):
+ historical = native.code_hashes() | {
+ "benefit_outcomes_native.py": next(iter(native.REPLAY_COMPATIBLE_NATIVE_SOURCES))
+ }
+ with patch.object(native, "code_hashes", return_value=historical):
+ fixtures.save(self.run / "plan.json", native.prepare(self.head_run))
+ self.freeze()
+ native.checked_frozen(self.run, self.head_run)
+ receipt = native.authorize_test(self.run, self.head_run)
+ self.assertEqual(receipt["replay_validation"]["training_source_code_sha256"], historical)
+ self.assertEqual(receipt["replay_validation"]["validator_source_code_sha256"], native.code_hashes())
+ (self.run / "frozen.json").unlink()
+ (self.run / "scoring-start.json").unlink()
+ for action in (
+ lambda: native.checked_plan(self.run, self.head_run),
+ lambda: native.freeze(self.run, self.head_run, self.fixture.baseline, self.fixture.original),
+ ):
+ with self.assertRaisesRegex(ValueError, "source"):
+ action()
+
+ def test_numeric_replay_preserves_stored_scores_and_cutoff_ties(self):
+ _, frozen = self.freeze()
+ original = outcomes.normalized_scores
+
+ def rounded(*args):
+ scores, costs = original(*args)
+ return np.nextafter(scores, -np.inf), np.nextafter(costs, np.inf)
+
+ with patch.object(outcomes, "normalized_scores", side_effect=rounded):
+ _, _, replayed = native.checked_frozen(self.run, self.head_run)
+ self.assertEqual(replayed, frozen)
+ thresholds = 0
+ for model in replayed["models"].values():
+ for policy in model["policies"].values():
+ if policy["mode"] == "threshold":
+ thresholds += 1
+ saved = np.asarray(model["scores"]["calibration"])
+ self.assertTrue((saved == policy["cutoff"]).any())
+ self.assertEqual((saved >= policy["cutoff"]).tolist(), policy["calibration_routes"])
+ self.assertGreater(thresholds, 0)
+ for field in ("scores", "costs"):
+
+ def changed(*args, field=field):
+ scores, costs = original(*args)
+ return (scores + 1e-9, costs) if field == "scores" else (scores, costs + 1e-6)
+
+ with (
+ self.subTest(field=field),
+ patch.object(outcomes, "normalized_scores", side_effect=changed),
+ self.assertRaisesRegex(ValueError, "ratio scores"),
+ ):
+ native.checked_frozen(self.run, self.head_run)
+
+ def test_resealed_export_parity_and_epoch_selection_tampering_are_rejected(self):
+ trained = self.trained()
+ for field in ("epoch", "parity", "costs"):
+ changed = json.loads(json.dumps(trained))
+ changed.pop("sha256")
+ if field == "epoch":
+ changed["selected_epoch"] = 2
+ elif field == "parity":
+ changed["checkpoints"]["lora"]["exported_margins"][0][0] = 1
+ else:
+ changed["conditional_cost_statistics"]["cheap_correct"]["mean_credits_proxy"] += 1
+ fixtures.save(self.run / "trained.json", benefit.seal(changed))
+ with self.subTest(field=field), self.assertRaises(ValueError):
+ native.checked_trained(self.run, self.head_run)
+
+ def test_test_access_is_required_and_native_router_overhead_stays_unknown(self):
+ _, frozen = self.freeze()
+ test = self.fixture.root / "held-test-grades.json"
+ with self.assertRaises(FileNotFoundError):
+ native.report(self.run, self.head_run, test)
+ fixtures.save(self.run / "test-access.json", native.authorize_test(self.run, self.head_run))
+ fixtures.save(self.head_run / "test-access.json", outcomes.authorize_test(self.head_run))
+ rows = fixtures.outcomes("test", 8)
+ for row in rows:
+ row["cheap_correct"] = False
+ fixture = self.fixture.fixture
+ fixtures.save(
+ test,
+ {
+ "split": "test",
+ "plan_sha256": file_hash(fixture.original_plan),
+ "policies_frozen_sha256": file_hash(self.fixture.original),
+ **fixture.grading,
+ "rows": rows,
+ },
+ )
+ measured = native.report(self.run, self.head_run, test)
+ self.assertEqual(measured["ids"], [row["id"] for row in rows])
+ for variant in ("base", "lora"):
+ policy = measured["models"][variant]["1"]
+ self.assertEqual(policy["cutoff"], frozen["models"][variant]["policies"]["1"]["cutoff"])
+ self.assertGreater(policy["metrics"]["gap_vs_strong_pp"], 1)
+ self.assertFalse(policy["metrics"]["router_overhead_known"])
+ self.assertNotIn("net_total_cost_usd_per1000", policy["metrics"])
+ with self.assertRaises(FileExistsError):
+ native.prepare(self.head_run)
+ grades = outcomes.read(test)
+ grades["outcome_grader_protocol_sha256"] = "e" * 64
+ fixtures.save(test, grades)
+ with self.assertRaisesRegex(ValueError, "grading study"):
+ native.report(self.run, self.head_run, test)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benefit_route.py b/examples/train-a-router/tests/test_benefit_route.py
new file mode 100644
index 000000000..cb61b638b
--- /dev/null
+++ b/examples/train-a-router/tests/test_benefit_route.py
@@ -0,0 +1,181 @@
+"""The live path must obey its frozen cutoff and call only the selected solver."""
+
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import numpy as np
+
+import benefit_route
+from data import digest, file_hash, write
+
+
+class BenefitRouteTests(unittest.TestCase):
+ def setUp(self):
+ self.frozen = {
+ "head": {"weights": [0.4, 0.6], "bias": 0.1},
+ "selections": {"1": {"mode": "threshold", "cutoff": 0.5}},
+ "plan": {"solver_settings_sha256": digest(benefit_route.solver.SETTINGS)},
+ }
+
+ def test_cutoff_equality_selects_strong_and_static_rules_skip_embedding(self):
+ with patch.object(benefit_route, "embed", return_value=(np.array([1, 0]), {"credits": 2})) as encode:
+ result = benefit_route.choose(self.frozen, "problem", 1)
+ self.assertEqual(result["arm"], "strong")
+ self.assertEqual(result["score"], 0.5)
+ encode.assert_called_once_with("problem", None)
+ for arm in ("cheap", "strong"):
+ self.frozen["selections"]["1"] = {"mode": f"always_{arm}", "cutoff": None}
+ with patch.object(benefit_route, "embed") as encode:
+ self.assertEqual(benefit_route.choose(self.frozen, "problem", 1)["arm"], arm)
+ encode.assert_not_called()
+
+ def test_cloud_requires_receipt_and_self_hosted_keeps_unknown_cost(self):
+ with patch.object(benefit_route, "embed", return_value=(np.array([0, 1]), {"credits": None})):
+ with self.assertRaisesRegex(ValueError, "charge receipt"):
+ benefit_route.choose(self.frozen, "problem", 1)
+ self.assertIsNone(
+ benefit_route.choose(self.frozen, "problem", 1, self_hosted=True)["embedding_receipt"]["credits"]
+ )
+
+ def test_outcome_route_uses_both_probabilities_and_expected_cost(self):
+ head = {
+ "models": {
+ "cheap": {"weights": [0, 0], "bias": -float(np.log(9))},
+ "strong": {"weights": [0, 0], "bias": float(np.log(9))},
+ },
+ "conditional_cost_statistics": {
+ f"{arm}_{state}": {"mean_credits_proxy": cost}
+ for arm, cost in (("cheap", 10), ("strong", 100))
+ for state in ("correct", "wrong")
+ },
+ }
+ frozen = {"selections": {"1": {"mode": "threshold", "cutoff": 0.008}}}
+ with patch.object(benefit_route, "embed", return_value=(np.array([1, 0]), {"credits": 2})) as encode:
+ result = benefit_route.choose(frozen, "question", 1, outcome_head=head)
+ self.assertAlmostEqual(result["score"], 0.8 / 90)
+ self.assertEqual(result["arm"], "strong")
+ self.assertEqual(result["router"], "qwen_embedding_outcomes")
+ encode.assert_called_once()
+ frozen["selections"]["1"] = {"mode": "always_cheap", "cutoff": None}
+ with patch.object(benefit_route, "embed") as encode:
+ self.assertEqual(benefit_route.choose(frozen, "question", 1, outcome_head=head)["arm"], "cheap")
+ encode.assert_not_called()
+
+ def test_live_path_preserves_receipts_and_never_runs_both_solvers(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ write(root / "model/frozen.json", self.frozen)
+ selected = benefit_route.solver.SETTINGS["arms"]["cheap"]
+
+ def solve(row, arm, settings, **kwargs):
+ self.assertEqual(arm, "cheap")
+ result = benefit_route.solver.initial_record(row, arm, settings)
+ return result | {
+ "status": "complete",
+ "answer": "\\boxed{2}",
+ "credits": 8,
+ "observed_credits": 8,
+ "has_unknown_charges": False,
+ "calls": [{"billing": "cloud", "receipt": {"credits": 8}}],
+ }
+
+ with (
+ patch.object(benefit_route.benefit, "_checked_frozen", return_value=self.frozen),
+ patch.object(benefit_route, "embed", return_value=(np.array([0, 0]), {"credits": 2})),
+ patch.object(benefit_route, "SingleSendClient") as client,
+ patch.object(benefit_route.solver, "solve", side_effect=solve) as generate,
+ ):
+ client.return_value.__enter__.return_value.get_model.return_value = {
+ "name": selected["model"],
+ "revision": selected["revision"],
+ }
+ summary = benefit_route.answer(root / "model", root / "request", "AMC8", "What is 1+1?")
+ generate.assert_called_once()
+ self.assertEqual(summary["total_api_credits"], 10)
+ self.assertEqual(summary["embedding_calls"], 1)
+ self.assertTrue((root / "request/intent.json").exists())
+ self.assertTrue((root / "request/answer.json").exists())
+ with self.assertRaises(FileExistsError):
+ benefit_route.answer(root / "model", root / "request", "AMC8", "What is 1+1?")
+ generate.assert_called_once()
+
+ def test_invalid_paid_embedding_preserves_its_receipt_before_raising(self):
+ with tempfile.TemporaryDirectory() as temporary, patch.object(benefit_route, "SingleSendClient") as factory:
+ client = factory.return_value.__enter__.return_value
+ client.get_model.return_value = {"name": benefit_route.MODEL, "revision": benefit_route.REVISION}
+ client.encode.return_value = {
+ "dense": [0, 0],
+ "request": {"id": "paid-embedding-request", "credits_debited": 3},
+ }
+ path = Path(temporary) / "embedding.json"
+ with self.assertRaises(ValueError):
+ benefit_route.embed("question", path)
+ self.assertEqual(benefit_route.benefit.read(path)["credits"], 3)
+ self.assertEqual(benefit_route.benefit.read(path)["request_id"], "paid-embedding-request")
+
+ def test_native_uses_frozen_cutoff_and_verifies_weights_without_embedding_calls(self):
+ with tempfile.TemporaryDirectory() as temporary:
+ root = Path(temporary)
+ (root / "lora").mkdir()
+ weights = root / "lora/model.safetensors"
+ weights.write_bytes(b"only the selected checkpoint is downloaded")
+ frozen = {
+ "models": {
+ "base": {"checkpoint_files_sha256": {"model.safetensors": "0" * 64}},
+ "lora": {
+ "policies": {"1": {"mode": "threshold", "cutoff": 0.2}},
+ "checkpoint_files_sha256": {"model.safetensors": file_hash(weights)},
+ },
+ }
+ }
+ with (
+ patch.object(benefit_route.benefit_native, "checked_frozen", return_value=({}, frozen)) as check,
+ patch.object(benefit_route.benefit_native, "score_checkpoint", return_value=[0.2]) as score,
+ patch.object(benefit_route, "embed") as embed,
+ ):
+ result = benefit_route.choose_native(root, Path("head"), "question", 1, "lora", "mps")
+ self.assertEqual(result["arm"], "strong")
+ self.assertEqual(result["native_model_calls"], 1)
+ check.assert_called_once_with(root, Path("head"))
+ score.assert_called_once_with(root / "lora", [{"request": "question"}], "mps")
+ embed.assert_not_called()
+ self.assertGreaterEqual(result["local_checkpoint_verification_seconds"], 0)
+ self.assertGreaterEqual(
+ result["local_router_end_to_end_seconds"],
+ result["local_checkpoint_verification_seconds"] + result["local_model_load_and_score_seconds"],
+ )
+ weights.write_bytes(b"tampered selected weights")
+ with self.assertRaisesRegex(ValueError, "checkpoint bytes"):
+ benefit_route.choose_native(root, Path("head"), "question", 1, "lora", "mps")
+ score.assert_called_once()
+ weights.unlink()
+ with self.assertRaises(FileNotFoundError):
+ benefit_route.choose_native(root, Path("head"), "question", 1, "lora", "mps")
+ score.assert_called_once()
+
+ def test_native_static_bypass_needs_no_checkpoint_files(self):
+ frozen = {
+ "models": {
+ "lora": {
+ "policies": {"1": {"mode": "always_cheap", "cutoff": None}},
+ "checkpoint_files_sha256": {"model.safetensors": "0" * 64},
+ }
+ }
+ }
+ with (
+ patch.object(benefit_route.benefit_native, "checked_frozen", return_value=({}, frozen)),
+ patch.object(benefit_route.benefit_native, "score_checkpoint") as score,
+ patch.object(benefit_route, "file_hash") as hash_file,
+ ):
+ result = benefit_route.choose_native(Path("native"), Path("head"), "question", 1, "lora", "cpu")
+ self.assertEqual(result["arm"], "cheap")
+ self.assertEqual(result["native_model_calls"], 0)
+ self.assertIsNone(result["local_checkpoint_verification_seconds"])
+ hash_file.assert_not_called()
+ score.assert_not_called()
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_benefit_training_review.py b/examples/train-a-router/tests/test_benefit_training_review.py
new file mode 100644
index 000000000..6222c09a4
--- /dev/null
+++ b/examples/train-a-router/tests/test_benefit_training_review.py
@@ -0,0 +1,34 @@
+"""Check that symbolic review cannot silently relabel other predictions."""
+
+import unittest
+
+from benefit_training_review import REVIEW, reviewed_grade
+
+
+class TrainingReviewTests(unittest.TestCase):
+ def test_monotonic_power_comparison(self):
+ identity = "e2h-amc-eval-1621"
+ for final, correct in REVIEW[identity]["answers"].items():
+ result = reviewed_grade(identity, REVIEW[identity]["original"], r"\boxed{" + final + "}")
+ self.assertIs(result["correct"], correct)
+ self.assertEqual(result["status"], "graded")
+
+ def test_tent_map_length_exact_square(self):
+ self.assertEqual(4**2012, 2**4024)
+ self.assertLess((2**2012) ** 2, 4**2012 + 1)
+ identity = "e2h-amc-eval-1775"
+ for final, correct in REVIEW[identity]["answers"].items():
+ result = reviewed_grade(identity, REVIEW[identity]["original"], r"\boxed{" + final + "}")
+ self.assertIs(result["correct"], correct)
+ self.assertTrue(result["training_adjudication"]["source_correction"])
+
+ def test_unreviewed_reference_or_answer_rejected(self):
+ identity = "e2h-amc-eval-1775"
+ with self.assertRaises(ValueError):
+ reviewed_grade(identity, "5", "7")
+ with self.assertRaises(ValueError):
+ reviewed_grade(identity, REVIEW[identity]["original"], "7")
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_evidence.py b/examples/train-a-router/tests/test_evidence.py
new file mode 100644
index 000000000..f3d288909
--- /dev/null
+++ b/examples/train-a-router/tests/test_evidence.py
@@ -0,0 +1,120 @@
+"""Verify the arithmetic evidence download without network access."""
+
+import hashlib
+import io
+import json
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import evidence
+
+
+def manifest(artifacts):
+ body = json.dumps(
+ {
+ "files": {
+ name: {"bytes": len(data), "sha256": hashlib.sha256(data).hexdigest(), "group": group}
+ for name, (group, data) in artifacts.items()
+ }
+ }
+ ).encode()
+ return body, hashlib.sha256(body).hexdigest()
+
+
+class EvidenceTests(unittest.TestCase):
+ def setUp(self):
+ directory = tempfile.TemporaryDirectory()
+ self.addCleanup(directory.cleanup)
+ self.directory = Path(directory.name)
+ self.artifacts = {
+ "arithmetic/study/report.json": ("core", b"{}\n"),
+ "arithmetic/embeddings/train/vectors.jsonl.gz": ("core", b"cached vectors"),
+ "arithmetic/study/native-2000/lora/model.safetensors": ("native-weights", b"native weights"),
+ "arithmetic/records/fresh.tar.gz": ("answers", b"raw records"),
+ }
+ self.body, self.digest = manifest(self.artifacts)
+ self.base = f"{evidence.BASE_URL}arithmetic-revision/{evidence.ARITHMETIC_FOLDER}/"
+ self.requested = []
+ for patcher in (
+ patch.object(evidence, "ARITHMETIC_REVISION", "arithmetic-revision"),
+ patch.object(evidence, "ARITHMETIC_MANIFEST_SHA256", self.digest),
+ patch.object(evidence.urllib.request, "urlopen", side_effect=self.open_url),
+ ):
+ patcher.start()
+ self.addCleanup(patcher.stop)
+
+ def open_url(self, url, **kwargs):
+ self.assertTrue(url.startswith(self.base))
+ name = url.removeprefix(self.base)
+ self.requested.append(name)
+ return io.BytesIO(self.body if name == "manifest.json" else self.artifacts[name][1])
+
+ def test_default_download_has_cached_data_but_no_weights_or_raw_answers(self):
+ evidence.fetch(self.directory, arithmetic_study=True)
+ core = {name for name, (group, _) in self.artifacts.items() if group == "core"}
+ self.assertEqual(set(self.requested), {"manifest.json", *core})
+ evidence.verify(self.directory, manifest_sha256=self.digest)
+ self.assertFalse((self.directory / "arithmetic/study/native-2000/lora/model.safetensors").exists())
+ self.assertFalse((self.directory / "arithmetic/records/fresh.tar.gz").exists())
+
+ def test_weights_and_raw_records_are_independent_optional_downloads(self):
+ evidence.fetch(self.directory, arithmetic_study=True, solver_weights=True)
+ self.assertEqual((self.directory / "arithmetic/study/native-2000/lora/model.safetensors").read_bytes(), b"native weights")
+ self.assertFalse((self.directory / "arithmetic/records/fresh.tar.gz").exists())
+ self.requested.clear()
+ evidence.fetch(self.directory, arithmetic_study=True, solver_records=True)
+ self.assertEqual(self.requested, ["manifest.json", "arithmetic/records/fresh.tar.gz"])
+ self.assertEqual((self.directory / "manifest.json").read_bytes(), self.body)
+
+ def test_different_manifest_is_rejected_before_network_access(self):
+ (self.directory / "manifest.json").write_bytes(b"another experiment")
+ with self.assertRaisesRegex(ValueError, "different experiment"):
+ evidence.fetch(self.directory, arithmetic_study=True)
+ self.assertEqual(self.requested, [])
+ self.assertEqual((self.directory / "manifest.json").read_bytes(), b"another experiment")
+
+ def test_changed_cached_file_is_rejected_without_overwriting_it(self):
+ evidence.fetch(self.directory, arithmetic_study=True)
+ path = self.directory / "arithmetic/study/report.json"
+ path.write_bytes(b"changed")
+ self.requested.clear()
+ with self.assertRaisesRegex(ValueError, "Artifact hash/size differs"):
+ evidence.fetch(self.directory, arithmetic_study=True)
+ self.assertEqual(self.requested, ["manifest.json"])
+ self.assertEqual(path.read_bytes(), b"changed")
+
+ def test_corrupt_remote_file_is_not_installed_and_temporary_file_is_removed(self):
+ self.artifacts["arithmetic/study/report.json"] = ("core", b"[]\n")
+ with self.assertRaisesRegex(ValueError, "Artifact hash/size differs"):
+ evidence.fetch(self.directory, arithmetic_study=True)
+ self.assertFalse((self.directory / "arithmetic/study/report.json").exists())
+ self.assertFalse(list(self.directory.rglob(".download-*")))
+
+ def test_oversized_remote_file_is_not_installed(self):
+ self.artifacts["arithmetic/study/report.json"] = ("core", b"too much data")
+ with self.assertRaisesRegex(ValueError, "exceeds its declared size"):
+ evidence.fetch(self.directory, arithmetic_study=True)
+ self.assertFalse((self.directory / "arithmetic/study/report.json").exists())
+ self.assertFalse(list(self.directory.rglob(".download-*")))
+
+ def test_unsafe_manifest_paths_are_rejected(self):
+ for name in ("../outside", "/absolute", "folder\\outside", "folder/../outside"):
+ body, digest = manifest({name: ("core", b"contents")})
+ with self.subTest(name=name), self.assertRaisesRegex(ValueError, "Unsafe artifact path"):
+ evidence.entries(body, manifest_sha256=digest)
+
+ def test_manifest_allows_float32_checkpoint_but_keeps_size_bound(self):
+ for size in (1_686_000_000, 2_000_000_001):
+ body = json.dumps({"files": {"weights": {"bytes": size}}}).encode()
+ digest = hashlib.sha256(body).hexdigest()
+ if size <= 2_000_000_000:
+ evidence.entries(body, manifest_sha256=digest)
+ else:
+ with self.assertRaisesRegex(ValueError, "Invalid artifact size"):
+ evidence.entries(body, manifest_sha256=digest)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_mps_training.py b/examples/train-a-router/tests/test_mps_training.py
new file mode 100644
index 000000000..a4fe5ea69
--- /dev/null
+++ b/examples/train-a-router/tests/test_mps_training.py
@@ -0,0 +1,51 @@
+"""Optional Apple GPU check for dropout gradient correctness; no models are downloaded."""
+
+import unittest
+
+from finetune import runtime_settings
+
+
+class MPSTrainingTests(unittest.TestCase):
+ def test_runtime_profile_preserves_dropout_gradients(self):
+ try:
+ import torch
+ except ModuleNotFoundError as error:
+ if error.name != "torch":
+ raise
+ self.skipTest("Install the optional training dependencies to check MPS gradients")
+ if not torch.backends.mps.is_available():
+ self.skipTest("An available Apple MPS GPU is required")
+
+ from torch.utils.checkpoint import checkpoint
+
+ def forward_backward(use_checkpoint):
+ torch.manual_seed(89)
+ model = torch.nn.Sequential(
+ torch.nn.Linear(32, 32),
+ torch.nn.Tanh(),
+ torch.nn.Dropout(0.5),
+ torch.nn.Linear(32, 8),
+ ).to(device="mps", dtype=torch.float32)
+ inputs = torch.linspace(-1, 1, 256, dtype=torch.float32).reshape(8, 32).to("mps").requires_grad_()
+ # Reset dropout RNG after identical parameter initialization in both runs.
+ torch.manual_seed(137)
+ output = checkpoint(model, inputs, use_reentrant=False) if use_checkpoint else model(inputs)
+ output.square().mean().backward()
+ gradients = {"inputs": inputs.grad.detach().cpu()}
+ gradients.update({name: parameter.grad.detach().cpu() for name, parameter in model.named_parameters()})
+ return output.detach().cpu(), gradients
+
+ expected_output, expected_gradients = forward_backward(False)
+ actual_output, actual_gradients = forward_backward(runtime_settings("mps")["gradient_checkpointing"])
+ torch.testing.assert_close(actual_output, expected_output, rtol=1e-5, atol=1e-6)
+ for name, expected in expected_gradients.items():
+ with self.subTest(gradient=name):
+ self.assertTrue(torch.isfinite(expected).all().item())
+ self.assertGreater(expected.abs().sum().item(), 0)
+ # Torch 2.9.1 can reproduce the forward output while checkpoint
+ # recomputation uses different MPS dropout masks in backward.
+ torch.testing.assert_close(actual_gradients[name], expected, rtol=1e-5, atol=1e-6)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_protocol.py b/examples/train-a-router/tests/test_protocol.py
new file mode 100644
index 000000000..b4e61e1e9
--- /dev/null
+++ b/examples/train-a-router/tests/test_protocol.py
@@ -0,0 +1,76 @@
+"""Check split isolation and statistical operations independently of model scores."""
+
+import copy
+import unittest
+
+import numpy as np
+
+from data import audit, duplicate_components
+from finetune import auc, better
+from finetune import cutoff as native_cutoff
+from policies import cutoff, ridge
+
+
+class ProtocolTests(unittest.TestCase):
+ def test_families_include_number_substitutions(self):
+ text = "Alice has {} apples in a basket and Bob gives her three more apples in the morning"
+ rows = [{"problem": text.format(n)} for n in (12, 24)]
+ rows.append({"problem": "Find the roots of the polynomial x squared minus five"})
+ self.assertEqual(duplicate_components(rows), [[0, 1], [2]])
+
+ def test_audit_rejects_cross_split_families(self):
+ splits = {}
+ for split, year in zip(("train", "validation", "calibration", "test"), (2017, 2018, 2019, 2020)):
+ splits[split] = [
+ {
+ "id": f"{split}{i}",
+ "request": f"{split} text {i}",
+ "label": i,
+ "family": f"{split}{i}",
+ "paper_group": str(year),
+ "year": year,
+ }
+ for i in (0, 1)
+ ]
+ audit({"splits": splits})
+ bad = copy.deepcopy(splits)
+ bad["test"][0]["family"] = "train0"
+ with self.assertRaisesRegex(ValueError, "leakage"):
+ audit({"splits": bad})
+ bad = copy.deepcopy(splits)
+ bad["test"][0]["year"] = 2017
+ with self.assertRaisesRegex(ValueError, "year"):
+ audit({"splits": bad})
+
+ def test_ridge_matches_primal_solution(self):
+ rng = np.random.default_rng(13)
+ x, y = rng.normal(size=(20, 4)), rng.integers(0, 2, 20)
+ centered = x - x.mean(axis=0)
+ expected = np.linalg.solve(centered.T @ centered + 0.3 * np.eye(4), centered.T @ (y - y.mean()))
+ weights, bias = ridge(x, y, 0.3)
+ np.testing.assert_allclose(weights, expected)
+ self.assertAlmostEqual(bias, y.mean() - x.mean(axis=0) @ expected)
+
+ def test_calibration_handles_imbalance(self):
+ values, labels = np.array([0.1, 0.2, 0.3, 0.4]), np.array([0, 0, 1, 0])
+ self.assertAlmostEqual(cutoff(values, labels), 0.25)
+ self.assertAlmostEqual(native_cutoff(values, labels), 0.25)
+ with self.assertRaises(ValueError):
+ cutoff(values, np.zeros(4))
+
+ def test_ranking_selection_ignores_confidence_scale_and_keeps_early_ties(self):
+ labels = [0, 0, 1, 1]
+ scores = np.array([0.0, 0.5, 0.5, 1.0])
+ self.assertAlmostEqual(auc(scores, labels), 0.875)
+ self.assertEqual(auc(scores, labels), auc(scores * 100 + 5, labels))
+ best = {"auc": 0.875, "cross_entropy": 0.5}
+ self.assertFalse(better({"auc": 0.875, "cross_entropy": 0.4}, best, "auc"))
+ candidate = {"auc": 0.9, "cross_entropy": 0.6}
+ self.assertTrue(better(candidate, best, "auc"))
+ self.assertFalse(better(candidate, best, "cross-entropy"))
+ with self.assertRaises(ValueError):
+ auc(scores, [0, 0, 0, 0])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_recordings.py b/examples/train-a-router/tests/test_recordings.py
new file mode 100644
index 000000000..4324affe7
--- /dev/null
+++ b/examples/train-a-router/tests/test_recordings.py
@@ -0,0 +1,40 @@
+import gzip
+import json
+import tempfile
+import unittest
+from pathlib import Path
+
+from data import digest
+from serving import INSTRUCTION, MODEL, REVISION, error_receipt, vectors
+
+
+class RecordingTests(unittest.TestCase):
+ def test_vectors_reject_changed_input_with_same_id(self):
+ with tempfile.TemporaryDirectory() as directory:
+ path = Path(directory)
+ row = {"id": "one", "request": "original problem"}
+ (path / "plan.json").write_text(
+ json.dumps(
+ {
+ "model": MODEL,
+ "revision": REVISION,
+ "instruction": INSTRUCTION,
+ "ids": ["one"],
+ "input_sha256": {"one": digest(row["request"])},
+ }
+ )
+ )
+ with gzip.open(path / "vectors.jsonl.gz", "wt") as handle:
+ handle.write(json.dumps({"id": "one", "vector": [1.0, 0.0]}) + "\n")
+ self.assertEqual(vectors([row], path).tolist(), [[1.0, 0.0]])
+ with self.assertRaisesRegex(ValueError, "different input"):
+ vectors([dict(row, request="changed problem")], path)
+
+ def test_error_receipt_does_not_publish_exception_text(self):
+ error = RuntimeError("private request text")
+ self.assertNotIn("private", json.dumps(error_receipt(error)))
+ self.assertIsNone(error_receipt(error)["credits"])
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_solver.py b/examples/train-a-router/tests/test_solver.py
new file mode 100644
index 000000000..261133a9a
--- /dev/null
+++ b/examples/train-a-router/tests/test_solver.py
@@ -0,0 +1,276 @@
+import json
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+import httpx
+
+import solver
+from data import digest, file_hash, write
+
+
+def make_run(directory, split="calibration", n=2):
+ rows = [{"id": f"problem-{index}", "request": f"What is {index} + 1?"} for index in range(n)]
+ inputs = {split: rows}
+ gold = {row["id"]: "1" for row in rows}
+ write(directory / "inputs.json", inputs)
+ write(directory / "settings.json", solver.SETTINGS)
+ write(directory / "gold-for-scoring-only.json", gold)
+ write(directory / "gold-preflight.json", {"needs_review": {}})
+ write(
+ directory / "gold-review.json",
+ {
+ "preflight_sha256": file_hash(directory / "gold-preflight.json"),
+ "approved_ids": [row["id"] for row in rows],
+ "ready": True,
+ },
+ )
+ write(
+ directory / "plan.json",
+ {
+ "inputs_sha256": digest(inputs),
+ "settings_sha256": digest(solver.SETTINGS),
+ "gold_sha256": digest(gold),
+ },
+ )
+ return rows
+
+
+def record(row, arm, settings=solver.SETTINGS, *, complete=True, known_rejection=False):
+ return {
+ "id": row["id"],
+ "arm": arm,
+ "model": settings["arms"][arm]["model"],
+ "catalog_revision": settings["arms"][arm]["revision"],
+ "settings_sha256": digest(settings),
+ "request_sha256": digest(row["request"]),
+ "status": "complete" if complete else "incomplete",
+ "answer": r"\boxed{1}" if complete else "",
+ "calls": [{"billing": "cloud", "billing_known": True, "receipt": {"credits": 2}}] if complete else [],
+ "credits": 2 if complete else None,
+ "observed_credits": 2 if complete else 0,
+ "has_unknown_charges": not complete and not known_rejection,
+ "safe_to_resume": known_rejection,
+ }
+
+
+class CatalogClient:
+ def __init__(self, **kwargs):
+ pass
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *args):
+ return None
+
+ def get_model(self, name):
+ return next(arm for arm in solver.ARMS.values() if arm["model"] == name)
+
+
+class SolverCollectionTests(unittest.TestCase):
+ def collect(self, directory, **kwargs):
+ return solver.collect(directory, "calibration", None, 1, 1000, 30, **kwargs)
+
+ def test_sdk_retry_is_blocked_after_http_diagnostics_are_captured(self):
+ requests = []
+
+ def respond(request):
+ requests.append(request)
+ return httpx.Response(
+ 429,
+ headers={"retry-after": "1", "x-sie-request-id": "request-123", "x-sie-error-code": "RATE_LIMIT"},
+ json={
+ "error": {"code": "rate_limit_exceeded", "message": "secret response text"},
+ },
+ )
+
+ with solver.RecordingClient(
+ base_url="https://test.invalid", api_key="", http_transport=httpx.MockTransport(respond)
+ ) as client:
+ with self.assertRaisesRegex(RuntimeError, "Automatic retries"):
+ solver.call(client, solver.ARMS["cheap"], [{"role": "user", "content": "1+1"}])
+ self.assertEqual(len(requests), 1)
+ failure = client.http_failures[0]
+ self.assertEqual(failure["http_status"], 429)
+ self.assertEqual(failure["error_code"], "RATE_LIMIT")
+ self.assertTrue(failure["known_pre_execution_rejection"])
+ self.assertNotIn("secret", json.dumps(failure))
+
+ def test_error_code_precedence_and_openai_normalization_fail_closed(self):
+ for status, header, body, expected, safe in [
+ (429, "RATE_LIMIT", {"error": {"code": "rate_limit_exceeded"}}, "RATE_LIMIT", True),
+ (503, "RESOURCE_EXHAUSTED", {"error": {"code": "QUEUE_FULL"}}, "RESOURCE_EXHAUSTED", False),
+ (429, "private-value", {"error": {"code": "RATE_LIMIT"}}, "unrecognized", False),
+ (503, None, {"detail": {"code": "QUEUE_FULL"}}, "QUEUE_FULL", True),
+ (503, None, {"error": {"code": "provisioning"}}, "PROVISIONING", True),
+ (429, None, {"error": {"code": "rate_limit_exceeded"}}, "RATE_LIMIT", False),
+ (503, None, {"error": "malformed", "detail": {"code": "QUEUE_FULL"}}, "unrecognized", False),
+ (403, "INSUFFICIENT_SCOPE", {}, "INSUFFICIENT_SCOPE", False),
+ ]:
+ with self.subTest(status=status, header=header, body=body):
+ response = httpx.Response(status, json=body, headers={"x-sie-error-code": header} if header else {})
+ failure = solver.safe_http_failure(response)
+ self.assertEqual(failure["error_code"], expected)
+ self.assertEqual(failure["known_pre_execution_rejection"], safe)
+ self.assertNotIn("private-value", json.dumps(failure))
+
+ def test_transport_failure_needs_retry_after_to_be_known_pre_execution(self):
+ for status, code, hint, expected in [
+ (503, "transport_failure", None, False),
+ (503, "transport_failure", "1", True),
+ (504, "transport_failure", "1", False),
+ (503, "QUEUE_FULL", None, True),
+ (503, {"bad": "code"}, None, False),
+ ]:
+ with self.subTest(status=status, code=code, hint=hint):
+ response = httpx.Response(
+ status, json={"error": {"code": code}}, headers={"retry-after": hint} if hint else {}
+ )
+ self.assertEqual(solver.safe_http_failure(response)["known_pre_execution_rejection"], expected)
+
+ def test_missing_credit_receipt_preserves_answer_and_blocks_unknown_retry(self):
+ class MissingReceipt(CatalogClient):
+ http_failures = ()
+ last_model_revision = "b" * 64
+
+ def chat_completions(self, **kwargs):
+ return {"choices": [{"message": {"content": r"\boxed{1}"}, "finish_reason": "stop"}], "usage": {}}
+
+ with patch.object(solver, "RecordingClient", MissingReceipt):
+ result = solver.solve({"id": "one", "request": "1+0"}, "cheap", solver.SETTINGS)
+ self.assertEqual(result["status"], "incomplete")
+ self.assertEqual(result["calls"][0]["answer"], r"\boxed{1}")
+ self.assertEqual(result["calls"][0]["execution_revision"], "b" * 64)
+ self.assertTrue(result["has_unknown_charges"])
+ self.assertFalse(result["safe_to_resume"])
+ self.assertIsNone(result["credits"])
+
+ def test_resume_preserves_history_and_does_not_resend_completed_arm(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ directory = Path(tmp)
+ make_run(directory)
+ first_calls = []
+
+ def initial(row, arm, settings, **kwargs):
+ first_calls.append((row["id"], arm))
+ return record(row, arm, settings, complete=arm == "cheap")
+
+ with (
+ patch.object(solver, "SingleSendClient", CatalogClient),
+ patch.object(solver, "solve", initial),
+ self.assertRaisesRegex(RuntimeError, "incomplete"),
+ ):
+ self.collect(directory)
+ output = directory / "answers-calibration"
+ original_complete = file_hash(output / "problem-0-cheap.json")
+ original_failed = file_hash(output / "problem-0-strong.json")
+ with patch.object(solver, "SingleSendClient", CatalogClient), patch.object(solver, "solve") as solve:
+ with self.assertRaisesRegex(ValueError, "retry-unknown"):
+ self.collect(directory, resume=True)
+ solve.assert_not_called()
+ resumed_calls = []
+
+ def resumed(row, arm, settings, **kwargs):
+ resumed_calls.append((row["id"], arm))
+ return record(row, arm, settings)
+
+ with patch.object(solver, "SingleSendClient", CatalogClient), patch.object(solver, "solve", resumed):
+ self.collect(directory, resume=True, retry_unknown=True)
+ self.assertEqual(first_calls, [("problem-0", "cheap"), ("problem-0", "strong")])
+ self.assertNotIn(("problem-0", "cheap"), resumed_calls)
+ self.assertEqual(len(resumed_calls), 3)
+ self.assertEqual(file_hash(output / "problem-0-cheap.json"), original_complete)
+ self.assertEqual(file_hash(output / "attempts/problem-0-strong/attempt-0001.json"), original_failed)
+ summary = solver.read(output / "collection.json")
+ self.assertEqual(summary["status"], "complete")
+ self.assertEqual(summary["completed"], 4)
+ self.assertEqual(summary["failed"], 0)
+ self.assertEqual(summary["failed_attempts"], 1)
+ self.assertEqual(summary["unknown_charge_attempts"], 1)
+ self.assertFalse(summary["acquisition_cost_complete"])
+ self.assertEqual(summary["observed_credits"], 8)
+ self.assertEqual(len(list((output / "sessions").glob("*/collection.json"))), 2)
+
+ def test_known_rejection_can_resume_without_unknown_flag(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ directory = Path(tmp)
+ make_run(directory, n=1)
+
+ def rejected(row, arm, settings, **kwargs):
+ return record(row, arm, settings, complete=False, known_rejection=True)
+
+ with (
+ patch.object(solver, "SingleSendClient", CatalogClient),
+ patch.object(solver, "solve", rejected),
+ self.assertRaises(RuntimeError),
+ ):
+ self.collect(directory)
+ with (
+ patch.object(solver, "SingleSendClient", CatalogClient),
+ patch.object(
+ solver, "solve", side_effect=lambda row, arm, settings, **kwargs: record(row, arm, settings)
+ ),
+ ):
+ self.collect(directory, resume=True)
+ self.assertTrue(solver.read(directory / "answers-calibration/collection.json")["acquisition_cost_complete"])
+
+ def test_resume_checks_limits_and_saved_input_order_before_requests(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ directory = Path(tmp)
+ make_run(directory, n=1)
+ with (
+ patch.object(solver, "SingleSendClient", CatalogClient),
+ patch.object(
+ solver,
+ "solve",
+ side_effect=lambda row, arm, settings, **kwargs: record(row, arm, settings, complete=False),
+ ),
+ self.assertRaises(RuntimeError),
+ ):
+ self.collect(directory)
+ with patch.object(solver, "SingleSendClient") as client:
+ with self.assertRaisesRegex(ValueError, "Attempt limit"):
+ self.collect(directory, resume=True, retry_unknown=True, max_attempts=1)
+ plan_path = directory / "answers-calibration/collection-plan.json"
+ plan = solver.read(plan_path)
+ plan["ids"] = ["different-problem"]
+ plan_path.write_text(json.dumps(plan))
+ with self.assertRaisesRegex(ValueError, "prepared split"):
+ self.collect(directory, resume=True, retry_unknown=True)
+ client.assert_not_called()
+
+ def test_complete_credit_receipts_and_input_bindings_are_validated(self):
+ row = {"id": "one", "request": "1+0"}
+ valid = record(row, "cheap")
+ solver.validate_record(valid, row, "cheap", solver.SETTINGS)
+ for field, value in [("credits", 0), ("observed_credits", 0), ("request_sha256", "changed"), ("id", "two")]:
+ with self.subTest(field=field), self.assertRaises(ValueError):
+ solver.validate_record(valid | {field: value}, row, "cheap", solver.SETTINGS)
+
+ def test_regrade_to_new_file_preserves_archived_grades(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ directory = Path(tmp)
+ make_run(directory, n=1)
+ with (
+ patch.object(solver, "SingleSendClient", CatalogClient),
+ patch.object(
+ solver, "solve", side_effect=lambda row, arm, settings, **kwargs: record(row, arm, settings)
+ ),
+ ):
+ self.collect(directory)
+ solver.grade(directory, "calibration")
+ original = (directory / "grades-calibration.json").read_bytes()
+ reproduced = directory / "regraded/calibration.json"
+ solver.grade(directory, "calibration", reproduced)
+ self.assertEqual(reproduced.read_bytes(), original)
+ self.assertEqual((directory / "grades-calibration.json").read_bytes(), original)
+ with patch.object(solver, "grade_reviewed") as grade_reviewed:
+ with self.assertRaises(FileExistsError):
+ solver.grade(directory, "calibration", reproduced)
+ grade_reviewed.assert_not_called()
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_solver_answers.py b/examples/train-a-router/tests/test_solver_answers.py
new file mode 100644
index 000000000..f4ff7cb7e
--- /dev/null
+++ b/examples/train-a-router/tests/test_solver_answers.py
@@ -0,0 +1,171 @@
+"""Check source binding and answer grading without generation calls."""
+
+import copy
+import importlib.util
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+from data import SOURCE_REVISION, file_hash, render_request
+from solver_answers import extract_final_answer, grade_answer, preflight_gold, prepare_inputs
+
+
+class FinalAnswerTests(unittest.TestCase):
+ def test_nested_box_and_last_explicit_answer(self):
+ self.assertEqual(extract_final_answer(r"Work: 5. Final answer: \boxed{\frac{1}{2}}"), r"\frac{1}{2}")
+ self.assertEqual(extract_final_answer("First \\boxed{4}.\nFinal answer: 5."), "5")
+ self.assertEqual(extract_final_answer(r"Answer: \fbox{\{1,2\}}"), r"\{1,2\}")
+ self.assertEqual(extract_final_answer(r"Final answer: \boxed{1+\boxed{2}}"), r"1+\boxed{2}")
+
+ def test_unfinished_final_box_cannot_reuse_an_intermediate(self):
+ self.assertEqual(extract_final_answer(r"First \boxed{4}. Finally \boxed{\frac{1}{"), "")
+
+ def test_later_final_clause_overrides_an_earlier_box(self):
+ for text in (
+ r"An intermediate result is \boxed{42}. Therefore, the final answer is 43.",
+ r"An intermediate result is \boxed{42}. Thus the answer is 43.",
+ r"An intermediate result is \boxed{42}, the final answer is 43.",
+ r"An intermediate result is \boxed{42}; hence the final answer is 43.",
+ r"Answer: \boxed{42}. Final answer: 43.",
+ "Final answer: 42. Final answer: 43.",
+ ):
+ with self.subTest(text=text):
+ self.assertEqual(extract_final_answer(text), "43")
+ self.assertEqual(extract_final_answer(r"The result is \boxed{42}. I am confident in this result."), "42")
+
+ def test_unmarked_reasoning_is_not_an_answer(self):
+ self.assertEqual(extract_final_answer("We first find 42.\nThe calculation is unfinished."), "")
+ self.assertEqual(extract_final_answer("We first find 42."), "")
+ self.assertEqual(extract_final_answer("42"), "42")
+ self.assertEqual(extract_final_answer(r"$\frac12$"), r"\frac12")
+
+
+@unittest.skipUnless(importlib.util.find_spec("math_verify"), "Install the evaluation dependency group")
+class GradingTests(unittest.TestCase):
+ def test_equivalent_fractions_radicals_and_cosmetic_boxing(self):
+ for gold, answer in [
+ (r"\frac12", r"\boxed{\frac{2}{4}}"),
+ (r"2\sqrt{3}", r"\boxed{\sqrt{12}}"),
+ ("240", "Final answer: 240."),
+ (r"\sqrt{113}-\sqrt{65}.", r"\boxed{\sqrt{113}-\sqrt{65}}"),
+ (r"2 \frac{265}{357}", r"\boxed{\frac{979}{357}}"),
+ (r"320^{\circ}", r"\boxed{320}"),
+ ]:
+ with self.subTest(gold=gold):
+ result = grade_answer(gold, answer)
+ self.assertEqual(result["status"], "graded")
+ self.assertTrue(result["correct"])
+
+ def test_tuple_order_and_cardinality_are_preserved(self):
+ self.assertTrue(grade_answer("(7,9)", r"\boxed{(7,9)}")["correct"])
+ self.assertFalse(grade_answer("(7,9)", r"\boxed{(9,7)}")["correct"])
+ self.assertFalse(grade_answer("(7,9)", r"\boxed{7}")["correct"])
+ self.assertTrue(grade_answer(r"(\frac12,9)", r"\boxed{(0.5,9)}")["correct"])
+
+ def test_named_text_is_exact_and_not_an_intermediate(self):
+ self.assertTrue(grade_answer(r"\text{Aaron}", r"\boxed{\text{Aaron}}")["correct"])
+ self.assertTrue(grade_answer(r"\text{Aaron}", "Final answer: aaron")["correct"])
+ self.assertTrue(grade_answer("Yes", "Final answer: yes")["correct"])
+ self.assertFalse(grade_answer("Yes", "Final answer: no")["correct"])
+ self.assertFalse(grade_answer(r"\text{Aaron}", "Aaron was considered.\nFinal answer: Dan")["correct"])
+ self.assertEqual(grade_answer("42", "We first find 42.")["status"], "no_final_answer")
+
+ def test_multiple_tuples_preserve_coordinates_but_not_set_order(self):
+ gold = "(2,2),(12,17),(17,12)"
+ self.assertTrue(grade_answer(gold, r"\boxed{(17,12),(2,2),(12,17)}")["correct"])
+ self.assertTrue(grade_answer(gold, r"\boxed{\{(17,12),(2,2),(12,17)\}}")["correct"])
+ self.assertFalse(grade_answer(gold, r"\boxed{(17,12),(2,2),(17,12)}")["correct"])
+
+ def test_math_verify_timeouts_are_recorded(self):
+ from math_verify.errors import TimeoutException
+
+ with patch("solver_answers._parse", side_effect=TimeoutException("bounded")):
+ self.assertEqual(grade_answer("42", "42")["status"], "gold_unparseable")
+ self.assertFalse(preflight_gold({"problem": "42"})["ready"])
+
+ def test_wrong_and_unparseable_answers_remain_distinct(self):
+ self.assertFalse(grade_answer("42", r"\boxed{43}")["correct"])
+ self.assertEqual(grade_answer("42", r"\boxed{\frac{}{}}")["status"], "prediction_unparseable")
+ self.assertEqual(grade_answer(r"\frac{}{A}", r"\boxed{42}")["status"], "gold_unparseable")
+
+ def test_percentages_have_mathematical_scale(self):
+ self.assertFalse(grade_answer("42", r"\boxed{42\%}")["correct"])
+ self.assertFalse(grade_answer(r"42\%", r"\boxed{42}")["correct"])
+ self.assertTrue(grade_answer(r"\frac12", r"\boxed{50\%}")["correct"])
+ self.assertTrue(grade_answer("7", r"\boxed{6+100\%}")["correct"])
+ self.assertTrue(grade_answer(r"80\%", r"\boxed{0.8}")["correct"])
+
+ def test_false_equations_cannot_supply_only_the_correct_rhs(self):
+ for answer in ("8=7", "x=8=7", "x+1=7", "7=8"):
+ self.assertFalse(grade_answer("7", "\\boxed{" + answer + "}")["correct"], answer)
+ for answer in ("x=7", "x=6+1=7", "6+1=7"):
+ self.assertTrue(grade_answer("7", "\\boxed{" + answer + "}")["correct"], answer)
+
+ def test_preflight_keeps_ambiguous_rows_visible(self):
+ gold = {"integer": "42", "ellipsis": r"0,3,6,\ldots,6048", "pair": "(7,9)", "ratio": "6:0"}
+ audit = preflight_gold(gold)
+ self.assertFalse(audit["ready"])
+ self.assertEqual(audit["n"], 4)
+ self.assertEqual(audit["counts"], {"ready": 1, "needs_review": 3})
+ self.assertEqual(set(audit["needs_review"]), {"ellipsis", "pair", "ratio"})
+ self.assertFalse(grade_answer("6:0", r"\boxed{3:0}")["correct"])
+
+
+@unittest.skipUnless(importlib.util.find_spec("pyarrow"), "Install the evaluation dependency group")
+class SourceBindingTests(unittest.TestCase):
+ def setUp(self):
+ import pyarrow as pa
+ import pyarrow.parquet as pq
+
+ self.temporary = tempfile.TemporaryDirectory()
+ self.addCleanup(self.temporary.cleanup)
+ self.source = Path(self.temporary.name)
+ self.dataset = {"source_revision": SOURCE_REVISION, "splits": {}}
+ originals = []
+ for split, year in zip(("train", "validation", "calibration", "test"), (2017, 2018, 2019, 2020)):
+ self.dataset["splits"][split] = []
+ for label in (0, 1):
+ index = len(originals)
+ original = {"tag": "AMC10", "problem": f"Problem {index}", "answer": str(index), "year": year}
+ originals.append(original)
+ self.dataset["splits"][split].append(
+ {
+ "id": f"e2h-amc-train-{index}",
+ "request": render_request(original["tag"], original["problem"]),
+ "contest": "AMC10",
+ "year": year,
+ "label": label,
+ "family": str(index),
+ "paper_group": str(year),
+ }
+ )
+ table = pa.Table.from_pylist(originals)
+ pq.write_table(table, self.source / "train.parquet")
+ pq.write_table(table.slice(0, 0), self.source / "eval.parquet")
+ pins = {name: file_hash(self.source / name) for name in ("train.parquet", "eval.parquet")}
+ self.pins = patch("solver_answers.SOURCE_HASHES", pins)
+ self.pins.start()
+ self.addCleanup(self.pins.stop)
+
+ def test_solver_inputs_do_not_contain_gold_or_difficulty_metadata(self):
+ inputs, gold, provenance = prepare_inputs(self.source, self.dataset)
+ self.assertEqual(set(inputs), {"validation", "calibration", "test"})
+ self.assertEqual(len(gold), 6)
+ self.assertTrue(all(set(row) == {"id", "request"} for rows in inputs.values() for row in rows))
+ self.assertEqual(gold["e2h-amc-train-2"], "2")
+ self.assertEqual(provenance["counts"], {"validation": 2, "calibration": 2, "test": 2})
+
+ def test_changed_source_or_request_is_rejected(self):
+ altered = copy.deepcopy(self.dataset)
+ altered["splits"]["test"][0]["request"] += " changed"
+ with self.assertRaisesRegex(ValueError, "join differs"):
+ prepare_inputs(self.source, altered)
+ with (self.source / "train.parquet").open("ab") as handle:
+ handle.write(b"changed")
+ with self.assertRaisesRegex(ValueError, "hash differs"):
+ prepare_inputs(self.source, self.dataset)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/examples/train-a-router/tests/test_solver_gold.py b/examples/train-a-router/tests/test_solver_gold.py
new file mode 100644
index 000000000..9b046b067
--- /dev/null
+++ b/examples/train-a-router/tests/test_solver_gold.py
@@ -0,0 +1,108 @@
+"""Exercise source-only adjudications without consulting model outputs."""
+
+import importlib.util
+import unittest
+
+from solver_gold import RULES, grade_reviewed, reviewed_gold
+
+
+@unittest.skipUnless(importlib.util.find_spec("math_verify"), "Install the evaluation dependency group")
+class ReviewedGoldTests(unittest.TestCase):
+ def grade(self, identity, final):
+ return grade_reviewed(identity, RULES[identity][0], "\\boxed{" + final + "}")["correct"]
+
+ def test_all_declared_rules_are_gradeable_without_excluding_rows(self):
+ original = {identity: rule[0] for identity, rule in RULES.items()} | {"ordinary": "42"}
+ corrected, audit = reviewed_gold(original)
+ self.assertTrue(audit["ready"])
+ self.assertEqual(set(corrected), set(original))
+ self.assertEqual(audit["n"], len(RULES) + 1)
+ self.assertEqual(audit["excluded"], [])
+ self.assertEqual(set(audit["approved_ids"]), set(RULES))
+ self.assertEqual(sum(r["source_answer_correction"] for r in audit["correction_records"]), 2)
+
+ def test_source_corrections_are_exact_and_bound_to_original_text(self):
+ self.assertTrue(self.grade("e2h-amc-eval-1694", "1010000"))
+ self.assertFalse(self.grade("e2h-amc-eval-1694", "1100"))
+ self.assertTrue(self.grade("e2h-amc-eval-620", "21000"))
+ self.assertFalse(self.grade("e2h-amc-eval-620", "210"))
+ with self.assertRaisesRegex(ValueError, "original source"):
+ grade_reviewed("e2h-amc-eval-620", "changed", r"\boxed{21000}")
+ with self.assertRaisesRegex(ValueError, "Source gold changed"):
+ reviewed_gold({"e2h-amc-eval-620": "changed"})
+
+ def test_ratio_equivalence_does_not_turn_time_into_a_ratio(self):
+ for identity, positive, negative in [
+ ("e2h-amc-train-684", ["1:2", "2:4", "0.5"], ["2:1", "1:0"]),
+ ("e2h-amc-eval-1815", ["2:3", "4:6", r"\frac23"], ["3:2"]),
+ ("e2h-amc-eval-2906", ["4:1", "8:2", "4"], ["1:4"]),
+ ]:
+ for value in positive:
+ self.assertTrue(self.grade(identity, value), value)
+ for value in negative:
+ self.assertFalse(self.grade(identity, value), value)
+
+ def test_clock_supports_equivalent_time_notations(self):
+ for value in ["6:0", "6:00", "6 pm", "6:00 PM", "18:00", r"6\text{ p.m.}", "6", "6:00 in the evening"]:
+ self.assertTrue(self.grade("e2h-amc-train-977", value), value)
+ for value in ["6 am", "7:00 pm", "18:30", "6:60", "24:00", "1.8:00", ".6", "6:00 in the morning"]:
+ self.assertFalse(self.grade("e2h-amc-train-977", value), value)
+
+ def test_progression_accepts_only_the_complete_discrete_set(self):
+ positive = [
+ r"0,3,6,9,\ldots,6048",
+ r"\{0,3,\dots,6048\}",
+ r"\{3k:k=0,1,\ldots,2016\}",
+ r"\{3k\mid k\in\mathbb{Z},0\leq k\leq2016\}",
+ r"\{3n:0\leq n<2017,n\in\mathbb{Z}\}",
+ r"\{n\in\mathbb{Z}:0\le n\le6048,3\mid n\}",
+ "all multiples of 3 from 0 to 6048 inclusive",
+ "all multiples of 3 from 0 to 6048",
+ ]
+ negative = [
+ "0",
+ "{0,3,6,9,6048}",
+ r"0,4,8,\ldots,6048",
+ r"0,3,\ldots,6045",
+ r"\{3k:0\leq k\leq2016\}",
+ r"\{3k:k=0,1,\ldots,2015\}",
+ "[0,6048]",
+ r"\{n\in\mathbb{Z}:0= 2:
+ raise ValueError("Two-attempt ceiling reached")
+ todo.append((row, messages, len(previous) + 1))
+ sessions = output / "sessions"
+ session = sessions / f"session-{len(list(sessions.glob('session-*'))) + 1:04d}"
+ write(
+ session / "plan.json",
+ {
+ "job_plan_sha256": file_hash(plan_path),
+ "phase": phase,
+ "workers": workers,
+ "pending_ids": [row["id"] for row, _, _ in todo],
+ "max_seconds": max_seconds,
+ "max_credits": max_credits,
+ "limit": limit,
+ "max_generation_attempts": 2,
+ "budget_scope": "New observed charges; in-flight calls drain; external watchdog bounds infrastructure",
+ },
+ )
+ started, sent, failed, observed, fatal = time.monotonic(), 0, 0, 0.0, None
+ try:
+ if todo:
+ with SingleSendClient(timeout_s=60) as client:
+ model = settings["arms"][arm]
+ metadata = client.get_model(model["model"])
+ if (
+ metadata.get("revision") != model["revision"]
+ or metadata.get("name", metadata.get("id")) != model["model"]
+ ):
+ raise ValueError("Endpoint model identity or HF revision differs")
+ write(session / "model.json", {key: metadata.get(key) for key in ("name", "revision", "profiles")})
+ tasks = iter(todo)
+ with ThreadPoolExecutor(max_workers=workers) as pool:
+ pending = {}
+
+ def schedule():
+ nonlocal sent
+ if (
+ failed
+ or time.monotonic() - started >= max_seconds
+ or (max_credits is not None and observed >= max_credits)
+ ):
+ return
+ task = next(tasks, None)
+ if task is None:
+ return
+ row, messages, attempt_number = task
+ target = output / "attempts" / row["id"] / f"{phase}-{attempt_number:04d}.json"
+ write(
+ target.with_name(target.stem + ".intent.json"),
+ {
+ "id": row["id"],
+ "arm": arm,
+ "phase": phase,
+ "job": job,
+ "session": session.name,
+ "request_sha256": digest(row["request"]),
+ "settings_sha256": digest(settings),
+ "messages_sha256": digest(messages),
+ },
+ )
+ future = pool.submit(stages.stage_call, row, arm, settings, phase, messages, self_hosted)
+ pending[future] = row, messages, target
+ sent += 1
+
+ for _ in range(workers):
+ schedule()
+ while pending:
+ done, _ = wait(pending, timeout=10, return_when=FIRST_COMPLETED)
+ for future in done:
+ row, messages, target = pending.pop(future)
+ result = future.result()
+ stages.checked_stage(result, row, arm, settings, phase, messages)
+ write(target, result)
+ observed += result["observed_credits"]
+ failed += result["status"] != "complete"
+ stages.materialize(output, row, arm, settings, self_hosted)
+ print(f"{job}/{phase}: {sent - len(pending)}/{len(todo)} returned, {failed} failed", flush=True)
+ for _ in done:
+ schedule()
+ except Exception as error: # noqa: BLE001 -- retain all intent and response evidence
+ fatal = error_receipt(error)
+ summary = {
+ "status": "complete" if sent == len(todo) and not failed and fatal is None else "incomplete",
+ "phase": phase,
+ "sent": sent,
+ "failed": failed,
+ "observed_credits": observed,
+ "elapsed_seconds": time.monotonic() - started,
+ "fatal_error": fatal,
+ }
+ write(session / "collection.json", summary)
+ if summary["status"] != "complete":
+ raise RuntimeError("Collection phase incomplete; preserve receipts and inspect before continuing")
+ return summary
+
+
+def gather(directory, locations):
+ directory = Path(directory)
+ jobs, populations, settings = checked_jobs(directory)
+ if set(locations) != set(populations):
+ raise ValueError("Need exactly the declared transfer jobs")
+ output = directory / "answers"
+ bindings, receipts, records, missing = {}, {}, {}, 0
+ failed_attempts, unknown, observed = 0, 0, 0.0
+ for name, location in locations.items():
+ location = Path(location)
+ binding = solver.read(location / "job-plan.json")
+ if (
+ binding.get("job") != name
+ or binding.get("arm") != jobs["jobs"][name]["arm"]
+ or binding.get("ids") != jobs["jobs"][name]["ids"]
+ or binding.get("jobs_sha256") != file_hash(directory / "collection-jobs.json")
+ ):
+ raise ValueError("Transfer job binding differs")
+ if (
+ binding.get("runtime_receipt_sha256")
+ and file_hash(location / "runtime-receipt.json") != binding["runtime_receipt_sha256"]
+ ):
+ raise ValueError("Transfer runtime receipt changed")
+ bindings[name] = binding
+ arm = binding["arm"]
+ for row in populations[name]:
+ record = stages.materialize(location, row, arm, settings, binding["self_hosted"])
+ if record is None:
+ missing += 1
+ continue
+ filename = f"{row['id']}-{arm}.json"
+ archives.copy_exact(location / "records" / filename, output / filename)
+ records[filename] = file_hash(output / filename)
+ for path in location.rglob("*"):
+ if (
+ not path.is_file()
+ or path.suffix not in (".json", ".yaml")
+ or "records" in path.relative_to(location).parts
+ ):
+ continue
+ relative = Path(name) / path.relative_to(location)
+ archives.copy_exact(path, output / "acquisition" / relative)
+ receipts[relative.as_posix()] = file_hash(path)
+ if "attempts" not in path.relative_to(location).parts:
+ continue
+ if path.name.endswith(".intent.json"):
+ unknown += not path.with_name(path.name.replace(".intent", "")).exists()
+ else:
+ attempt = solver.read(path)
+ failed_attempts += attempt["status"] != "complete"
+ unknown += attempt.get("execution_unknown", True)
+ observed += attempt["observed_credits"]
+ plan = {
+ "format": FORMAT,
+ "source_plan_sha256": file_hash(directory / "source-plan.json"),
+ "jobs_sha256": file_hash(directory / "collection-jobs.json"),
+ "settings_sha256": digest(settings),
+ "jobs": bindings,
+ "records_sha256": records,
+ }
+ target = output / "collection-plan.json"
+ if target.exists() and solver.read(target) != plan:
+ raise ValueError("Completed transfer collection cannot be overwritten")
+ if not target.exists():
+ write(target, plan)
+ summary = {
+ "status": "complete" if not missing and not unknown else "incomplete",
+ "completed": len(records),
+ "missing": missing,
+ "failed_attempts": failed_attempts,
+ "unknown_execution_attempts": unknown,
+ "observed_credits": observed,
+ "receipt_sha256": receipts,
+ "graded": False,
+ }
+ solver.save_latest(output / "collection.json", summary)
+ if summary["status"] != "complete":
+ raise RuntimeError("Transfer collection is incomplete; retain all acquisition evidence")
+ return summary
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ sub = parser.add_subparsers(dest="command", required=True)
+ prep = sub.add_parser("prepare")
+ prep.add_argument("--directory", type=Path, required=True)
+ prep.add_argument("--shards", type=int, default=2)
+ acquire = sub.add_parser("collect")
+ acquire.add_argument("--directory", type=Path, required=True)
+ acquire.add_argument("--job", required=True)
+ acquire.add_argument("--phase", choices=stages.PHASES, required=True)
+ acquire.add_argument("--workers", type=int, default=4)
+ acquire.add_argument("--max-seconds", type=float, required=True)
+ acquire.add_argument("--max-credits", type=float)
+ acquire.add_argument("--self-hosted", action="store_true")
+ acquire.add_argument("--resume", action="store_true")
+ acquire.add_argument("--runtime-receipt", type=Path)
+ args = parser.parse_args()
+ if args.command == "prepare":
+ prepare(args.directory, args.shards)
+ else:
+ collect(
+ args.directory,
+ args.job,
+ args.phase,
+ workers=args.workers,
+ max_seconds=args.max_seconds,
+ max_credits=args.max_credits,
+ self_hosted=args.self_hosted,
+ resume=args.resume,
+ runtime_receipt=args.runtime_receipt,
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/train-a-router/uv.lock b/examples/train-a-router/uv.lock
new file mode 100644
index 000000000..59e52bdc4
--- /dev/null
+++ b/examples/train-a-router/uv.lock
@@ -0,0 +1,1239 @@
+version = 1
+requires-python = "==3.12.*"
+
+[[package]]
+name = "accelerate"
+version = "1.15.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "huggingface-hub" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "psutil" },
+ { name = "pyyaml" },
+ { name = "safetensors" },
+ { name = "torch" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f5/b5/1d3ed029ac71d3f2961346829a268da923698e9fd63f218f78841f216bfd/accelerate-1.15.0.tar.gz", hash = "sha256:5654f8c5eaa0d4fa68b33e287a97765da6849bf6d51dcac874e73fbbddfb6134", size = 422615 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8a/4c/34f0450479d01195027260da68d8a3880683f1640c3ca5adf64acb3185f1/accelerate-1.15.0-py3-none-any.whl", hash = "sha256:97eacca0b73e45cb867dbf8c5d5d4dc32219544300e0c8992c7334dc2ef33cec", size = 394295 },
+]
+
+[[package]]
+name = "aiohappyeyeballs"
+version = "2.7.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ce/f4/eec0465c2f67b2664688d0240b3212d5196fd89e741df67ddb81f8d35658/aiohappyeyeballs-2.7.1.tar.gz", hash = "sha256:065665c041c42a5938ed220bdcd7230f22527fbec085e1853d2402c8a3615d9d", size = 24757 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/71/43/1947f06babed6b3f1d7f38b0c767f52df66bfb2bc10b468c4a7de9eceff2/aiohappyeyeballs-2.7.1-py3-none-any.whl", hash = "sha256:9243213661e29250eb41368e5daa826fc017156c3b8a11440826b2e3ed376472", size = 15038 },
+]
+
+[[package]]
+name = "aiohttp"
+version = "3.14.5"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiohappyeyeballs" },
+ { name = "aiosignal" },
+ { name = "attrs" },
+ { name = "frozenlist" },
+ { name = "multidict" },
+ { name = "propcache" },
+ { name = "typing-extensions" },
+ { name = "yarl" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/6c/4c/bdccd81e9ee225b69c60e7766c9a5b05364f118f4d383713b89a682d772d/aiohttp-3.14.5.tar.gz", hash = "sha256:5558a7f5a05af9ecf744af91e5baefc436f93c9333e656c27ec253f9a6bbe178", size = 8078052 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d5/94/6ba86efddcb616c811b40e6a0dfdd862738f647e4e3860a961075d5e9ed8/aiohttp-3.14.5-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:df37b620684e19b5e25724412518ccafc3b1a49cdac706fdbd2f983fad943450", size = 817499 },
+ { url = "https://files.pythonhosted.org/packages/5e/e1/7bca6d84dabd228aa8eb4b7f9feac2586aaa9be5505d7d65bf287a615c43/aiohttp-3.14.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ef60869969180ec2464f1349aff07138ae35ca2200f0946cb3552e49e8f301a8", size = 549467 },
+ { url = "https://files.pythonhosted.org/packages/64/91/11b89f45ca486252dd67dd5f3231fec04bf5518da39a95cb3997619f17fb/aiohttp-3.14.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d079c0a0135c36e7beb6f1c88087c8f108dc5891cdd0b5eafa778421bda70ed2", size = 548067 },
+ { url = "https://files.pythonhosted.org/packages/22/ff/c6615806c14aab34f82b9424ccde8ce6e417315fd57ce1c5b4d4747888e1/aiohttp-3.14.5-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:abfda5cb094a829f7bc25216a32f7db2e85cc65bd59910f8e7b40b3d9b224764", size = 1896245 },
+ { url = "https://files.pythonhosted.org/packages/39/b2/25a8c971ae6a92c8394d77e42422d7f38989e05cf41a5ceb92d73d67ab7e/aiohttp-3.14.5-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:9cc882cf8619109583c906b4d4a85d6a111a98afa34b7a450d1e08118d016820", size = 1872180 },
+ { url = "https://files.pythonhosted.org/packages/2f/d5/99f93ea36cc5205e47c1e5a803e087f2ad21b5430b5db2e942cb6e988a37/aiohttp-3.14.5-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7457580535e019e1247ea35d6a02bf081ad30c26d0cbc210c93f6c3ab67a0835", size = 1937261 },
+ { url = "https://files.pythonhosted.org/packages/40/a6/9ac9c9e6695040bd73d2584a1b59a9388f6433c76d5294a7bf591e21ffe5/aiohttp-3.14.5-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c5ed596aedb9c42afd3fe0aae3117725378ac73d2cc5ddc735056fbdb96c5d02", size = 2064994 },
+ { url = "https://files.pythonhosted.org/packages/da/e4/aa172eb534b7f02f1f8ff1c3213347eaf3cf218db91a727c6863c22f1035/aiohttp-3.14.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:20f085697d7e911f1f73c43ed03fafbed1e7121797e2eb5428efa80398060584", size = 1919630 },
+ { url = "https://files.pythonhosted.org/packages/06/7d/4eedafc5bababa8932636c141e346806966eade12c0b7e5946d43bf8218b/aiohttp-3.14.5-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:74b0a9c8270f9b0a11410e124ff8d4f18bfc1f1837440ec84da5ae7b50927b5d", size = 1710016 },
+ { url = "https://files.pythonhosted.org/packages/b2/94/eee018537ba19da0ceb2ac79cab83faed4ac49568e08376e2799043f2538/aiohttp-3.14.5-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:19e2ba471507c34f8252402ab50f5ab512398b9ea8c8f1cb26beb3f75793ba30", size = 1858112 },
+ { url = "https://files.pythonhosted.org/packages/68/76/354653a306547238f3427283905972d796ba7c292ba9977abec9f2b6f260/aiohttp-3.14.5-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:d418ce2af40c6bb685b3f663e9e8de27cb0a22431d8e88a167348d7f01878073", size = 1896371 },
+ { url = "https://files.pythonhosted.org/packages/f2/ec/63e8c7136b570e356345ad3174e3820fdc973ea10712cb6c649bf875755a/aiohttp-3.14.5-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:70cb4008ac2ed1e0ca9e824deb4b53d3aa0d939109698ebf1e723a84337bd794", size = 1905836 },
+ { url = "https://files.pythonhosted.org/packages/57/d8/11365bda144b127928cd42533d0eff78a55613c9e28f81941bd6630ea887/aiohttp-3.14.5-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:a23fe35d776bc03cb495938b9594450d047e3bc08c5255315a82323e9cb7d2dd", size = 1696534 },
+ { url = "https://files.pythonhosted.org/packages/2f/3d/82df0461b18e00b2998f205c03e0d3010222478c43640aceb8e03dcd7e8f/aiohttp-3.14.5-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:3e0eb43bed3c6801a6cee315195377789e90b2a72c2277a475b578535312488d", size = 1933970 },
+ { url = "https://files.pythonhosted.org/packages/03/ad/6ddfe0aacd931c17b53533336d97e9d11a98b96d6ae815a9da0b19f82ccf/aiohttp-3.14.5-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3be7dd397d64ca3e1869626fa9318aaebb54b7bf93bc72d7a205448d83e4f748", size = 1899501 },
+ { url = "https://files.pythonhosted.org/packages/9e/8e/189bdd9ae4793059bb09f6dc880f211a6c6c7859cebcbf33912dbfab7dd7/aiohttp-3.14.5-cp312-cp312-win32.whl", hash = "sha256:eb324e2009fb54db30a071dad7caf6998ee2879c4704007efb244514dad1fec1", size = 480370 },
+ { url = "https://files.pythonhosted.org/packages/ae/ce/1f08114679d49655b30a6e0a29858375c94b82c1de1a0bd0a20c2fee8b02/aiohttp-3.14.5-cp312-cp312-win_amd64.whl", hash = "sha256:2cc38a4f2b516bef1714e690df87a0e043faf1a7693c82d860091684453d5111", size = 506141 },
+ { url = "https://files.pythonhosted.org/packages/79/d4/c7b4f60b16a1b7e43249fa9031ae05e7e8c341ba4b7d1866f914dafeaa0e/aiohttp-3.14.5-cp312-cp312-win_arm64.whl", hash = "sha256:a63afd1f757de949028387e65a7127b61ad0f775432dbb0e62816ae619fe69ac", size = 488108 },
+ { url = "https://files.pythonhosted.org/packages/68/30/173960c42b05a6c59f7558e4b12a4b0d9ba376cf6aa9bde7f9e08a30ca8d/aiohttp-3.14.5-py3-none-any.whl", hash = "sha256:efc21a454892828368b11c2c780de0ff8bc991f73f6b99c6b66e56205470929b", size = 279517 },
+]
+
+[[package]]
+name = "aiosignal"
+version = "1.4.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "frozenlist" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/61/62/06741b579156360248d1ec624842ad0edf697050bbaf7c3e46394e106ad1/aiosignal-1.4.0.tar.gz", hash = "sha256:f47eecd9468083c2029cc99945502cb7708b082c232f9aca65da147157b251c7", size = 25007 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl", hash = "sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e", size = 7490 },
+]
+
+[[package]]
+name = "antlr4-python3-runtime"
+version = "4.13.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/33/5f/2cdf6f7aca3b20d3f316e9f505292e1f256a32089bd702034c29ebde6242/antlr4_python3_runtime-4.13.2.tar.gz", hash = "sha256:909b647e1d2fc2b70180ac586df3933e38919c85f98ccc656a96cd3f25ef3916", size = 117467 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/89/03/a851e84fcbb85214dc637b6378121ef9a0dd61b4c65264675d8a5c9b1ae7/antlr4_python3_runtime-4.13.2-py3-none-any.whl", hash = "sha256:fe3835eb8d33daece0e799090eda89719dbccee7aa39ef94eed3818cafa5a7e8", size = 144462 },
+]
+
+[[package]]
+name = "anyio"
+version = "4.15.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "idna" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a9/d2/f4d173e22df740bc37b1db102b386ba719b66e95b0f0d751f556b387e6d2/anyio-4.15.1.tar.gz", hash = "sha256:9f28306018cbd6d329e64a36d58256edff76dd996fe423bc957326e578b82a94", size = 276966 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/12/b8/4bd346e22b28902df4d651910f5242c28d84e4a5c2435ca5c3f797ed7e2e/anyio-4.15.1-py3-none-any.whl", hash = "sha256:6152fdbbf9a77fdec97731721bebf7c4c44f7c29b424b0065826173efc7ed101", size = 132079 },
+]
+
+[[package]]
+name = "attrs"
+version = "26.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/9a/8e/82a0fe20a541c03148528be8cac2408564a6c9a0cc7e9171802bc1d26985/attrs-26.1.0.tar.gz", hash = "sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32", size = 952055 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548 },
+]
+
+[[package]]
+name = "certifi"
+version = "2026.10.18"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/7e/eb/a2aa1ac3ea30dd3f30ca57f40232138b6b0040719316ecb85daf61167771/certifi-2026.10.18.tar.gz", hash = "sha256:b9995d97b64d0431a5af837d8ca8cd6238104d1f319f04bd93ea43124d6c861e", size = 135227 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/08/ba/95dd616042484f7c83a06974388b37b837732740ee70bc5dc715b5fcd55a/certifi-2026.10.18-py3-none-any.whl", hash = "sha256:73191c47aaa79da394e8fee2dbc2ec827432cab56311122c9e47abf79e390f62", size = 134093 },
+]
+
+[[package]]
+name = "charset-normalizer"
+version = "3.5.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/33/1c/f41d4e74c28ab327ff3acd36053f7ea506c55872d7a90b0fa71aa3ab0c89/charset_normalizer-3.5.2.tar.gz", hash = "sha256:39de2a259fc954455c57274dc94c79d5842774e1247a016aff30bc0efed0f4ef", size = 172659 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e7/c8/693809898870237d82785a03f3b2b58fe4c9f14669f84a7d4e623c92a59e/charset_normalizer-3.5.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:ed2a239c0ea213acc1908150a3037257083c7c083128f1a4cec2ec4b97dca491", size = 367780 },
+ { url = "https://files.pythonhosted.org/packages/c9/87/2fea8c13dc24b3ca9c6f803a5b2dfdeae73eb4f9e12c7885ed908ff0433c/charset_normalizer-3.5.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b91363207bd9dc966a691e959bb47f64b30f7ac4b072be9968b366982f7db77c", size = 246730 },
+ { url = "https://files.pythonhosted.org/packages/a8/9e/09efac30b937722f46d3110ba30b875b24b2e3a266ed746cc4e376a94d80/charset_normalizer-3.5.2-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:38a873987f3be698494da8b2e3085e29da02da7b633dce73e79c699a113d7bf0", size = 237707 },
+ { url = "https://files.pythonhosted.org/packages/9e/18/70d76670b13686237863a379928d60bd10e021f17d243ab3d7014c4a5f4e/charset_normalizer-3.5.2-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:355ad8011081dec5412240c087a9a0c9d4d5039f3ed11a3f13e18c2b29b56c51", size = 273050 },
+ { url = "https://files.pythonhosted.org/packages/54/e2/77a8b09d5adc013ed07b95b01b8b8fa5441c4e810e83ee7e4aae2fa4d91a/charset_normalizer-3.5.2-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ee21e28f0430bd6dc9086c6e525d5e818a44a5ad19720c8a0ef766792f3eb5e5", size = 270345 },
+ { url = "https://files.pythonhosted.org/packages/7f/c5/38806a25ab5e65fc178f39affeda20858efafede2fce1ffc2556cfc9fe73/charset_normalizer-3.5.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3d31298449090ab8d47b7b1b2a555ff73cac7ed438a08b7ac160980c7ebed649", size = 257601 },
+ { url = "https://files.pythonhosted.org/packages/ae/8d/213565184708fdb263ae55e2c04ee1ff748129dd65d48ed0e3502da9c85a/charset_normalizer-3.5.2-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5cde776b7cc66e4f6c99612cea4aa7269aa65863f7a15841b2c264f103822f4e", size = 252222 },
+ { url = "https://files.pythonhosted.org/packages/7e/24/76d2cefc25472531e4c5c7dfff68865eb1c39b78482f0fdc15b46f047830/charset_normalizer-3.5.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ae4f5fea5b8b8ccff88238cc8569303e5ee95efae67fa62922a311397a71f346", size = 248482 },
+ { url = "https://files.pythonhosted.org/packages/7d/dc/65a801b66ab4c197e22c433ab25e7ac24324ac6f45a2269aca42cce309bf/charset_normalizer-3.5.2-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:f7d486c83842422badd511868fd8a9a20e9407ace71564b6af47ce7e60a336c1", size = 241206 },
+ { url = "https://files.pythonhosted.org/packages/a7/95/ca9b5eabde673002c6f1e7ada1b223916fe18f6d661da7aabd4d643718f1/charset_normalizer-3.5.2-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:11a4d68a6ecda3292cb1e50239e111543ba5d709bb62a6b4ea1afcfa729d8875", size = 273190 },
+ { url = "https://files.pythonhosted.org/packages/2d/8b/803b4d2a3f6e1740f63f1e87b04d14b42f3d4fdfe6ed7d4db2d34102b14f/charset_normalizer-3.5.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:d6734d2ef8a50fbf8445c139477da401f50d62a0606bf00e20ec6d87773fefb1", size = 253527 },
+ { url = "https://files.pythonhosted.org/packages/a9/55/93c0e5dbd085ae0471346026abbe7e0db9ea2d6fea74e51f0b5a46f233a7/charset_normalizer-3.5.2-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:a815775b6c38d4e0ff7bcffbeba67feded90202bb6a226b8dd35f1c855217413", size = 271285 },
+ { url = "https://files.pythonhosted.org/packages/95/69/0dbd0e0b9b16cfa816cdfcb3e2e3854a1f680dc07fb1245ea125e7448060/charset_normalizer-3.5.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:23851fb4e1b85ed3f6c2a27b777cdfe2e19fb5b38429a8faf38c7542b7665869", size = 260010 },
+ { url = "https://files.pythonhosted.org/packages/58/9d/e7b88e7b1bf403590c3b573277b5e1e488c68c7a6fbacca310a2c324e90c/charset_normalizer-3.5.2-cp312-cp312-win32.whl", hash = "sha256:db19d07e2e0129e974a0e65d0064fc222a446cd5122c2fd4184d2af9fc734a9e", size = 184125 },
+ { url = "https://files.pythonhosted.org/packages/eb/e6/e6e083884cbcfd49c64865af05027fe7011be7b2d9179524f099a1b611f3/charset_normalizer-3.5.2-cp312-cp312-win_amd64.whl", hash = "sha256:780fbe7cab297b81dad9fb8dc5eb003c0468ffb0d9e5f65068c53a34661a96bc", size = 207486 },
+ { url = "https://files.pythonhosted.org/packages/c4/e3/017aea0911ada7405a825c7d937eb3a13009664e2f5b38e8c4bbf2abf894/charset_normalizer-3.5.2-cp312-cp312-win_arm64.whl", hash = "sha256:e2af3aad578aa6bd1384bcf4750fc285e5a9de53f40b7d41e5a0bf748edeb2b3", size = 196734 },
+ { url = "https://files.pythonhosted.org/packages/8c/ab/176fbfd5b64939c55d652366aa5b9ef1d767af207a3aa6ebeb0d226c484d/charset_normalizer-3.5.2-cp37-abi3-macosx_10_9_universal2.whl", hash = "sha256:4275811936e2f06feff5e598fb42a1b7ae852da8e39605211892b56b81a34efd", size = 331815 },
+ { url = "https://files.pythonhosted.org/packages/7e/84/371eac6b30bdbcbf2d632a1a01809103459216fcaae61b8b8d922c1bfb8a/charset_normalizer-3.5.2-cp37-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:1c50fe28bbc2ced33386f298650d91218076c05420e6cbd790b913adc41659e7", size = 253276 },
+ { url = "https://files.pythonhosted.org/packages/43/6f/c4fbae58febff71709c51bc7e18fdfa55341dc382704740f9f0cbf03817b/charset_normalizer-3.5.2-cp37-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d19fbd981a488e22cd04883659ca6b08f50b5974f9fd7c95655ef6a043e5893f", size = 241239 },
+ { url = "https://files.pythonhosted.org/packages/61/71/458c3f42164a07d0c5210798e9e704b39e540a6793b05aba67f3a35243a9/charset_normalizer-3.5.2-cp37-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:0fed1d06615f022ee3b13caf5e8b180cfea32bb2c5aded8a9d44277afc040f93", size = 231121 },
+ { url = "https://files.pythonhosted.org/packages/09/54/ab9e89367076f6331bb6c65c4bf14a5361fa5191cb6561bf534f18504e1b/charset_normalizer-3.5.2-cp37-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:838dcc90063569a0448120554591a1d6c4a4ffe11babf048908793154ab86ade", size = 260350 },
+ { url = "https://files.pythonhosted.org/packages/7c/c1/061431ecc688d9d76602502cb57cc01e691e682c18f1beb45f9673b5bbd2/charset_normalizer-3.5.2-cp37-abi3-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2ce45c6627b22c47e390bc91a41c3d13032192e699fa0bea96e9671b373d69b0", size = 255430 },
+ { url = "https://files.pythonhosted.org/packages/8d/1f/20c8949f0676f7ab811abdeb7f4d7f1cbc6e61ff20bef08b44edeb092bc8/charset_normalizer-3.5.2-cp37-abi3-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0774bf9bf620249fee3e0b8b9fd3065de213be30f3aa94ce2494b3b638949e26", size = 250612 },
+ { url = "https://files.pythonhosted.org/packages/2b/9e/46f2fa4c431fc98c4ae76a8cb5bdca54e0341e3cfc3fcfd8e82740250818/charset_normalizer-3.5.2-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1db38f4c5496827c1a501846d64d14c3b80c7e6714e406cd7dc36a9899fa1011", size = 242083 },
+ { url = "https://files.pythonhosted.org/packages/bd/39/559be29a0c0f086e0bba6922babd38916cc5e0b58ced4de13ee01ea05508/charset_normalizer-3.5.2-cp37-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:304d8e4d493af723536393eee0c689eb7813f4a474c8b479dee63f1fdd98f621", size = 232738 },
+ { url = "https://files.pythonhosted.org/packages/ff/6c/387b0e4f756a282831c1d9fc6aeb6c51ca4507ca202767c8de15ce9b12e2/charset_normalizer-3.5.2-cp37-abi3-musllinux_1_2_ppc64le.whl", hash = "sha256:9b7f416ff0978e2f2249330527f0ad6fa02f4932e6199692d3b52da2048c19e4", size = 260703 },
+ { url = "https://files.pythonhosted.org/packages/96/92/1fdf015f09ef449f50d3ac4b67c90887c9c318b727daa95cc4f866e6521d/charset_normalizer-3.5.2-cp37-abi3-musllinux_1_2_riscv64.whl", hash = "sha256:01077390b03f7988f11d700a2194e69b119741a86b1a638b1db88891e3eced8e", size = 247622 },
+ { url = "https://files.pythonhosted.org/packages/dc/3c/8e7b8a5671ad5d433669fb2a76f1a0164df2d9b1718b0206bc2a16d840cc/charset_normalizer-3.5.2-cp37-abi3-musllinux_1_2_s390x.whl", hash = "sha256:7e841fb9010836c992c9f12fcbd43a831de93a5f726fc1ccd8ca1d0268c5014c", size = 257500 },
+ { url = "https://files.pythonhosted.org/packages/b4/f0/45b579df5cabc1d5d53ea1cc35e8437d3ca768c0acccc7041517cb6fbb32/charset_normalizer-3.5.2-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:9cae88599c7219005d879f98e5ed53341e9a122af585e1091200358a3003d2a0", size = 255100 },
+ { url = "https://files.pythonhosted.org/packages/31/68/fdec18a343f5fb3f310588dd478b09ac4799e0b187dbade3a8cd776f03ef/charset_normalizer-3.5.2-cp37-abi3-win32.whl", hash = "sha256:01b0c0d2262a9e28e8484a278c7e1b5d650e3ac8cf2683d2967e25899f208bdf", size = 174499 },
+ { url = "https://files.pythonhosted.org/packages/9d/8a/b618149cc5207943a0242068d7a27897f56a62947b5a039085f2a22029f8/charset_normalizer-3.5.2-cp37-abi3-win_amd64.whl", hash = "sha256:9f56f72050826f63dcee7a7f55b0a77168cb3bfc553fd405e7f8f9ece75a4036", size = 200092 },
+ { url = "https://files.pythonhosted.org/packages/03/cf/4c66866fa9e2b1c78e3c911516d1de497a677b7ac60f1eceda74ce777ca3/charset_normalizer-3.5.2-cp37-abi3-win_arm64.whl", hash = "sha256:40ab6bffa02ae10a0581e6c198be7d2d8ca5c2a0c64e4ed3465d766df457573e", size = 294363 },
+ { url = "https://files.pythonhosted.org/packages/fc/ad/d07d7862a62ffa6d79d68074d14823243dd235a77c45262acbf6adeb28bf/charset_normalizer-3.5.2-py3-none-any.whl", hash = "sha256:b6b751274acb69d77b3323d6b7dbaa3c7fdfc1eb829b7eb61d262f32e1af9685", size = 68872 },
+]
+
+[[package]]
+name = "colorama"
+version = "0.4.6"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335 },
+]
+
+[[package]]
+name = "contourpy"
+version = "1.4.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/83/5a/a55177dd22553a277388e8a1b3220e92de91bacb28356cdc73caa240121d/contourpy-1.4.0.tar.gz", hash = "sha256:20156f5a1ac4f8ce02656e39a61e82164a3d359796dc8026f75b062783d500e1", size = 13323726 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/97/ba/01bde9753bdbba04a6da9d2bff3881f021bc708f654655e95fae26d3b3a3/contourpy-1.4.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:186ba929df36d61b6127da2e89cd1357e4cb79aca647381ab1a6feb0b152877b", size = 297302 },
+ { url = "https://files.pythonhosted.org/packages/dd/f5/c5bf67522d49a2222f3154fe46879451d26f0e35ece41770758a64ea78cb/contourpy-1.4.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:c76f3a5318164db7d9401132fc1b5364b784c613c93fa506e3b5e6d1bf353ec8", size = 284921 },
+ { url = "https://files.pythonhosted.org/packages/0f/cc/cb989599eec12fda312e127eb8e04a8b21a7a6c94bf7ff1bc68273334a42/contourpy-1.4.0-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:43c3ccbb32c6294b183dcc8e8c46dacf5ecef809497e3d48a5be298eef185ad0", size = 356332 },
+ { url = "https://files.pythonhosted.org/packages/47/fb/6f620d7602507817b0b4c21dd790ed68688f1f4ae9c29aacf21f08db5cf1/contourpy-1.4.0-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:86d05cec773c9507a3950122e0e40ce77c23c75ceeb2fc189514e71893cbb34b", size = 405878 },
+ { url = "https://files.pythonhosted.org/packages/32/4e/693c6d6bdece679f0953775eef3a1b66d56be55478d936e7deb1f3004422/contourpy-1.4.0-cp312-cp312-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2deb580178ca19437a84bd77e4bc2cd91a8ccad212413a83c273900868b5978c", size = 408296 },
+ { url = "https://files.pythonhosted.org/packages/78/f9/b6831508960d559581c448532ba4df312217072975486dbf2b24fcc5b76f/contourpy-1.4.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:875f42444c9cf48d56f724f2637e60d0f73b3b12c9041e1484580a233edf9591", size = 383243 },
+ { url = "https://files.pythonhosted.org/packages/f2/21/52903825a0ae7bb625e8bd30a09816d4c3c5d6174a469c10a616166cf780/contourpy-1.4.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:f863c6100bf926cf47d13f3cd75f9bb8ebd98aaab230eeb4468b91f98f39a6b3", size = 1357306 },
+ { url = "https://files.pythonhosted.org/packages/d8/79/d68b1ce8539e4071518fed9523f558398f34dcd078b8927b109c72dad2ef/contourpy-1.4.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3863ef2e2b13fe93f8c0ebb08ee400cb07153b8b0e91c5acb26e4537f283634f", size = 1427243 },
+ { url = "https://files.pythonhosted.org/packages/ac/f0/b75a10e9d0616b97a30de277b6dfe83f1879880854c7330337309530eb48/contourpy-1.4.0-cp312-cp312-win32.whl", hash = "sha256:5450f091ac1be0be3ad3a2a3b3f23b5e443e78c670ced4fd347d626f92a28fd2", size = 347345 },
+ { url = "https://files.pythonhosted.org/packages/50/a9/dab08786bb4d77ef9a046b3c1be923be4e73c5d07d91f5a54e8ea9b09e41/contourpy-1.4.0-cp312-cp312-win_amd64.whl", hash = "sha256:6e697d94e69f499ff6bebb899cae97a58d5d14f0e1fe9568b43a0248d2f9af8c", size = 234083 },
+ { url = "https://files.pythonhosted.org/packages/47/b9/3ba509755a970dbde5948e7142ac21f266be72f38017cc4087a05fdceee1/contourpy-1.4.0-cp312-cp312-win_arm64.whl", hash = "sha256:0c7a4c2716a4e98342221954416a836cca77c996c14ddbf22b5f01d5d93ca09c", size = 559521 },
+]
+
+[[package]]
+name = "cycler"
+version = "0.12.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a9/95/a3dbbb5028f35eafb79008e7522a75244477d2838f38cbb722248dabc2a8/cycler-0.12.1.tar.gz", hash = "sha256:88bb128f02ba341da8ef447245a9e138fae777f6a23943da4540077d3601eb1c", size = 7615 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl", hash = "sha256:85cef7cff222d8644161529808465972e51340599459b8ac3ccbac5a854e0d30", size = 8321 },
+]
+
+[[package]]
+name = "filelock"
+version = "4.2.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d4/08/d8f81287742010e38daf3632380c8d80ede7b4df2716f8ca3d0a2b087d44/filelock-4.2.2.tar.gz", hash = "sha256:b29ff2ae9140bb17b3bc77c648dc4f010d9ac0955eccabff4f2b01dde859f787", size = 651339 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4e/53/35698fff350cb99c49ee01ae5cb1eb4408567792a286314ea984b53b5002/filelock-4.2.2-py3-none-any.whl", hash = "sha256:1851d2f14ca2901e98069d68333b33ad7409f56904e17b48cc0c32a0db8ed73a", size = 162415 },
+]
+
+[[package]]
+name = "fonttools"
+version = "4.67.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/94/36/102e180f8f5dbaee88b26595b01ca8aa80bf4e62128d9aa94265b3996c96/fonttools-4.67.0.tar.gz", hash = "sha256:3cb57e6600ca77c0b1729cf8adc23bc0652633a37f18cfa934d9c7bc3de25519", size = 3750028 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5b/50/f674402869f11a89868c4755ae86cd2fcfd67ca6193c6f5d1b479b1267b9/fonttools-4.67.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:fcb9743140419410161acfe7ec205fb0a8a703acfccb85b586becb5a97c047c9", size = 3108316 },
+ { url = "https://files.pythonhosted.org/packages/e3/c8/5963603c5f9bbc28bde3a29dd7cdbe0bfcbee414b0f7eccec04ae477e1b6/fonttools-4.67.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ad813967410ba6d24a52850df59b164ee17883f17b96a91b4b0ac6e9d7b5a118", size = 2598450 },
+ { url = "https://files.pythonhosted.org/packages/25/6d/f8e5924917a6b5c0296fb507f748c139a34972f66e91d89159d5c98e27b2/fonttools-4.67.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:768a33bbe6ec5ba8f19979f938752f06d4e614cb554fd47abd7830f2007660e3", size = 5427391 },
+ { url = "https://files.pythonhosted.org/packages/c1/e0/ec9e4cc868c514deb02233aa1047a6aeb9350d3ee012862f58eec10ef834/fonttools-4.67.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:eb3c98cac93aac4b9f6e3ce2008325340b234cc9b0338ca6b513f31962a1e278", size = 5408233 },
+ { url = "https://files.pythonhosted.org/packages/cd/4a/fe409cb3ab32f322de92e08e6362cd06bf6dd5f0cee5980d823849e9bd11/fonttools-4.67.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e0ca4c8438dd6320f5850c9bbee3b3980455ee3bac602a9a0299caf9e799a0e8", size = 5364778 },
+ { url = "https://files.pythonhosted.org/packages/de/5b/2a8dede092113be56329dd210deb6b34c55df2f3d7270934ffece8c7d0bb/fonttools-4.67.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:2a09d33a9264a6b29efca9dc633b53969aaedb250a9c8521d60f51280cef65ca", size = 5528522 },
+ { url = "https://files.pythonhosted.org/packages/6c/de/d3baf686e4ac5726a24819a670747c51571c774dcfa41cc0528e5e8c1a2d/fonttools-4.67.0-cp312-cp312-win32.whl", hash = "sha256:e8a8545cbd58bd29494ffe81e3cb35f8a29332a8e495c42bec334145ce8cd65b", size = 2446090 },
+ { url = "https://files.pythonhosted.org/packages/c1/3a/625a6dd0173e88dbea1826405b4bcbfa06c6ca095310ed720caba36b2e43/fonttools-4.67.0-cp312-cp312-win_amd64.whl", hash = "sha256:2bfab2f5d1d255dec82f4bd082a1c10e77df808e42210890f50a9c30bf91570e", size = 2497985 },
+ { url = "https://files.pythonhosted.org/packages/3d/61/4161946319472aaa9b897bd18ad5108a5b10f5ebaa503d921a001ac4fff9/fonttools-4.67.0-py3-none-any.whl", hash = "sha256:4304f03ed7f4ba000a8dcc941ad854bfa52e2f3b6112b8f099b6f431cf98e701", size = 1213142 },
+]
+
+[[package]]
+name = "frozenlist"
+version = "1.8.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/2d/f5/c831fac6cc817d26fd54c7eaccd04ef7e0288806943f7cc5bbf69f3ac1f0/frozenlist-1.8.0.tar.gz", hash = "sha256:3ede829ed8d842f6cd48fc7081d7a41001a56f1f38603f9d49bf3020d59a31ad", size = 45875 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/69/29/948b9aa87e75820a38650af445d2ef2b6b8a6fab1a23b6bb9e4ef0be2d59/frozenlist-1.8.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:78f7b9e5d6f2fdb88cdde9440dc147259b62b9d3b019924def9f6478be254ac1", size = 87782 },
+ { url = "https://files.pythonhosted.org/packages/64/80/4f6e318ee2a7c0750ed724fa33a4bdf1eacdc5a39a7a24e818a773cd91af/frozenlist-1.8.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:229bf37d2e4acdaf808fd3f06e854a4a7a3661e871b10dc1f8f1896a3b05f18b", size = 50594 },
+ { url = "https://files.pythonhosted.org/packages/2b/94/5c8a2b50a496b11dd519f4a24cb5496cf125681dd99e94c604ccdea9419a/frozenlist-1.8.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f833670942247a14eafbb675458b4e61c82e002a148f49e68257b79296e865c4", size = 50448 },
+ { url = "https://files.pythonhosted.org/packages/6a/bd/d91c5e39f490a49df14320f4e8c80161cfcce09f1e2cde1edd16a551abb3/frozenlist-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:494a5952b1c597ba44e0e78113a7266e656b9794eec897b19ead706bd7074383", size = 242411 },
+ { url = "https://files.pythonhosted.org/packages/8f/83/f61505a05109ef3293dfb1ff594d13d64a2324ac3482be2cedc2be818256/frozenlist-1.8.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96f423a119f4777a4a056b66ce11527366a8bb92f54e541ade21f2374433f6d4", size = 243014 },
+ { url = "https://files.pythonhosted.org/packages/d8/cb/cb6c7b0f7d4023ddda30cf56b8b17494eb3a79e3fda666bf735f63118b35/frozenlist-1.8.0-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3462dd9475af2025c31cc61be6652dfa25cbfb56cbbf52f4ccfe029f38decaf8", size = 234909 },
+ { url = "https://files.pythonhosted.org/packages/31/c5/cd7a1f3b8b34af009fb17d4123c5a778b44ae2804e3ad6b86204255f9ec5/frozenlist-1.8.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c4c800524c9cd9bac5166cd6f55285957fcfc907db323e193f2afcd4d9abd69b", size = 250049 },
+ { url = "https://files.pythonhosted.org/packages/c0/01/2f95d3b416c584a1e7f0e1d6d31998c4a795f7544069ee2e0962a4b60740/frozenlist-1.8.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d6a5df73acd3399d893dafc71663ad22534b5aa4f94e8a2fabfe856c3c1b6a52", size = 256485 },
+ { url = "https://files.pythonhosted.org/packages/ce/03/024bf7720b3abaebcff6d0793d73c154237b85bdf67b7ed55e5e9596dc9a/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:405e8fe955c2280ce66428b3ca55e12b3c4e9c336fb2103a4937e891c69a4a29", size = 237619 },
+ { url = "https://files.pythonhosted.org/packages/69/fa/f8abdfe7d76b731f5d8bd217827cf6764d4f1d9763407e42717b4bed50a0/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:908bd3f6439f2fef9e85031b59fd4f1297af54415fb60e4254a95f75b3cab3f3", size = 250320 },
+ { url = "https://files.pythonhosted.org/packages/f5/3c/b051329f718b463b22613e269ad72138cc256c540f78a6de89452803a47d/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:294e487f9ec720bd8ffcebc99d575f7eff3568a08a253d1ee1a0378754b74143", size = 246820 },
+ { url = "https://files.pythonhosted.org/packages/0f/ae/58282e8f98e444b3f4dd42448ff36fa38bef29e40d40f330b22e7108f565/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:74c51543498289c0c43656701be6b077f4b265868fa7f8a8859c197006efb608", size = 250518 },
+ { url = "https://files.pythonhosted.org/packages/8f/96/007e5944694d66123183845a106547a15944fbbb7154788cbf7272789536/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:776f352e8329135506a1d6bf16ac3f87bc25b28e765949282dcc627af36123aa", size = 239096 },
+ { url = "https://files.pythonhosted.org/packages/66/bb/852b9d6db2fa40be96f29c0d1205c306288f0684df8fd26ca1951d461a56/frozenlist-1.8.0-cp312-cp312-win32.whl", hash = "sha256:433403ae80709741ce34038da08511d4a77062aa924baf411ef73d1146e74faf", size = 39985 },
+ { url = "https://files.pythonhosted.org/packages/b8/af/38e51a553dd66eb064cdf193841f16f077585d4d28394c2fa6235cb41765/frozenlist-1.8.0-cp312-cp312-win_amd64.whl", hash = "sha256:34187385b08f866104f0c0617404c8eb08165ab1272e884abc89c112e9c00746", size = 44591 },
+ { url = "https://files.pythonhosted.org/packages/a7/06/1dc65480ab147339fecc70797e9c2f69d9cea9cf38934ce08df070fdb9cb/frozenlist-1.8.0-cp312-cp312-win_arm64.whl", hash = "sha256:fe3c58d2f5db5fbd18c2987cba06d51b0529f52bc3a6cdc33d3f4eab725104bd", size = 40102 },
+ { url = "https://files.pythonhosted.org/packages/9a/9a/e35b4a917281c0b8419d4207f4334c8e8c5dbf4f3f5f9ada73958d937dcc/frozenlist-1.8.0-py3-none-any.whl", hash = "sha256:0c18a16eab41e82c295618a77502e17b195883241c563b00f0aa5106fc4eaa0d", size = 13409 },
+]
+
+[[package]]
+name = "fsspec"
+version = "2026.9.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/77/cd/9be253869fc42e764de7f3dedd6969af7d44ff9c3375214a3442a6f3fc08/fsspec-2026.9.0.tar.gz", hash = "sha256:0f08147951c8cb31d844c3547d631053b127863b60be04cf06e121333ee0e2fe", size = 333545 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6c/c0/a98505f18594f1bce828bb159cec0fcf9860562f1a2c85913409fc8f3d9e/fsspec-2026.9.0-py3-none-any.whl", hash = "sha256:8dd6e646e99ea382bd85f97a45e6b526a442d79423a7dc673f1e2756d05fcb5f", size = 221738 },
+]
+
+[[package]]
+name = "h11"
+version = "0.16.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515 },
+]
+
+[[package]]
+name = "hf-xet"
+version = "1.7.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/9e/27/06d899ea7bd721d272f84aac98bdb238de98af4cc767a69056d967d68c71/hf_xet-1.7.0.tar.gz", hash = "sha256:d406ec79053c0871817f700c2ac8c36ba0d87f9c34b7458b0f0063bb218b0466", size = 985689 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9c/0b/b03be21ffaada749ba0d3197d8aefbf1aa698bac149580421c15239b299e/hf_xet-1.7.0-cp38-abi3-macosx_10_12_x86_64.whl", hash = "sha256:e3e88a7a75d7d95cbee1f37dc31341d6201124cf21c6c4b1dfab8ccba9b09e0f", size = 3796096 },
+ { url = "https://files.pythonhosted.org/packages/c3/47/a26ebdce7056a61e931f228439bc0ab08cbec239d1690f965e5e637cba79/hf_xet-1.7.0-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:59fba37039233c7fcbe196817d6cdcf1b40dfb17b410f229d85b0cf0a1848da4", size = 3560352 },
+ { url = "https://files.pythonhosted.org/packages/a3/4c/2bf3b66c215d409655f28de1622393dde04c9461280d48c7924bb3b2decd/hf_xet-1.7.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2814a6e999d13464c4d679b788cc5d784eb5a4edfc638a31f10e9a11ab531ef8", size = 4212180 },
+ { url = "https://files.pythonhosted.org/packages/49/0c/a2f703a5a78267556e89e03316fa0805c86b72b50829bc67665746e8ebf0/hf_xet-1.7.0-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:fcfd6c22418e57dd5b3aea649e813b2e2cfb2aebf317b210d90f1fe4b3018b52", size = 3990011 },
+ { url = "https://files.pythonhosted.org/packages/a4/77/e52e4201b1cbf571530a61cc57f70182045a39a230089ee5f1df182a4de2/hf_xet-1.7.0-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:80f79dae613ce9e0ea1fd1ae15616ca9ac74aed4c770aabc199c4f03ebecc863", size = 4190628 },
+ { url = "https://files.pythonhosted.org/packages/6c/dc/03a21b89f118664a0926ff25b0f8e44a519bf22724a6a8fc7a9abbc188b6/hf_xet-1.7.0-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:0a9e802f33bf50c851abe45fc5380e61f959e2d369647d6742b79ad9d6c27cab", size = 4418814 },
+ { url = "https://files.pythonhosted.org/packages/4d/59/b35106dfa71b6eef605dc88bd038fe99c7f86fb132a15b60d0bf2f235b2c/hf_xet-1.7.0-cp38-abi3-win_amd64.whl", hash = "sha256:2b7bb5727889b0f2436dbaaad8fc4c3e66b8240d992716989e0c086b4278b1bc", size = 3822644 },
+ { url = "https://files.pythonhosted.org/packages/48/cd/072313585f74fe9d441e2eb5e0a4703c30586cd709810ea369675f61b74e/hf_xet-1.7.0-cp38-abi3-win_arm64.whl", hash = "sha256:acc3851cf2576a8fb2ae926da863f4efabe21303cf292e9a44332802ab0dcc6a", size = 3662436 },
+]
+
+[[package]]
+name = "httpcore"
+version = "1.0.9"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "certifi" },
+ { name = "h11" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784 },
+]
+
+[[package]]
+name = "httpx"
+version = "0.28.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+ { name = "certifi" },
+ { name = "httpcore" },
+ { name = "idna" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517 },
+]
+
+[[package]]
+name = "huggingface-hub"
+version = "0.36.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "filelock" },
+ { name = "fsspec" },
+ { name = "hf-xet", marker = "platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'" },
+ { name = "packaging" },
+ { name = "pyyaml" },
+ { name = "requests" },
+ { name = "tqdm" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/7c/b7/8cb61d2eece5fb05a83271da168186721c450eb74e3c31f7ef3169fa475b/huggingface_hub-0.36.2.tar.gz", hash = "sha256:1934304d2fb224f8afa3b87007d58501acfda9215b334eed53072dd5e815ff7a", size = 649782 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a8/af/48ac8483240de756d2438c380746e7130d1c6f75802ef22f3c6d49982787/huggingface_hub-0.36.2-py3-none-any.whl", hash = "sha256:48f0c8eac16145dfce371e9d2d7772854a4f591bcb56c9cf548accf531d54270", size = 566395 },
+]
+
+[[package]]
+name = "idna"
+version = "3.20"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f5/08/8eea9d4b8302028f3abb2c0813953f7aec26d33b7a8960ed760e65ff29fa/idna-3.20.tar.gz", hash = "sha256:a7db850025b95ded1eae8a46181a1a6c56c92c96f0e2b005d9ff8dc0210cab44", size = 216463 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/58/a2/bb081bab032533a855d44de1d56f8e8426114ff1ba5d1f07a438a0a654f8/idna-3.20-py3-none-any.whl", hash = "sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c", size = 69583 },
+]
+
+[[package]]
+name = "jinja2"
+version = "3.1.6"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "markupsafe" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/df/bf/f7da0350254c0ed7c72f3e33cef02e048281fec7ecec5f032d4aac52226b/jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d", size = 245115 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899 },
+]
+
+[[package]]
+name = "kiwisolver"
+version = "1.5.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ba/07/bd78e6a8fae171ea041ef5bba3ed21a003522fa088834b069b1909981f30/kiwisolver-1.5.1.tar.gz", hash = "sha256:f1303ef2eec81262a4b708c3e858afe58d7c75ad91c1c05266eda7673369859a", size = 104395 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6b/9b/65b302742389c6f96f2956bef5decf26011309feb2fc5d79613af18adea4/kiwisolver-1.5.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:63fb7294b768f444eb4b068965f2662f28c2fd4161e23bd60fcf3ff27b74c046", size = 123876 },
+ { url = "https://files.pythonhosted.org/packages/71/74/c21f339956f6f691b2ed7e31d5f3ae767304df6c460192739fc830853051/kiwisolver-1.5.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:0ebdef3eae5336568147c39a55be6a2036ffde53faa9ca2d978989ae7c2da12c", size = 66487 },
+ { url = "https://files.pythonhosted.org/packages/84/e5/bdb34e21523e01dceda064d63713f3bdec91388af24fba1eca7ea5e85864/kiwisolver-1.5.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1798e83840c3f627246104c4d8a9639c60fa068adf9ce92b61791781fa8a68c1", size = 64660 },
+ { url = "https://files.pythonhosted.org/packages/fc/f4/dadfec469313c7f428efa7e84b4aba9732f813c13ea7131a24b7b008ef57/kiwisolver-1.5.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:34633ecf50d16187ab8e5528b7a2530f2feb4e23f300db4672538b51cfc5cd38", size = 1477929 },
+ { url = "https://files.pythonhosted.org/packages/6f/35/09c58daac34e6f6ea5c6dee0094b422118e5a7c265586008a95fd135ac5f/kiwisolver-1.5.1-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d27c2123977cb9269c30a49ba45f03a4323017ef693e19db4ec9dbe1299a3002", size = 1278499 },
+ { url = "https://files.pythonhosted.org/packages/19/32/739765e24fbad29d13f83e546ea4abc215a78cea9d677ca09025b027724d/kiwisolver-1.5.1-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6a797a1cefc8b9c93170db580337e1fe3d011ad18b1299943231279406342048", size = 1296677 },
+ { url = "https://files.pythonhosted.org/packages/df/32/03304d1010e2cc45e5b3b52cef7e43fed3a2a5cd6c87a89b4a88e1d85b5d/kiwisolver-1.5.1-cp312-cp312-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2551cf9917af48ee7c4b29cc82320489508cf96fd26a51f6fc124de661cd44c7", size = 1346037 },
+ { url = "https://files.pythonhosted.org/packages/3e/57/4c49377bfd274450dd72ecaa13eaac32ea804a03363e4d1db0c5aa999ceb/kiwisolver-1.5.1-cp312-cp312-manylinux_2_39_riscv64.whl", hash = "sha256:38f6e0deb4d0a4615efe0c4efc5990b06ae450ab50a0b321c0b078b6d238c083", size = 988248 },
+ { url = "https://files.pythonhosted.org/packages/cb/c3/38df144a08b6c5d75ca4504e5cc3141bb3bfef64c04f4ef48204f42711b6/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:bfd1de989b3330420e29de39352f5c049905c9e3ee67233a50d550e3d652c148", size = 2228722 },
+ { url = "https://files.pythonhosted.org/packages/e7/11/3221838a89cd64d9b386353e000cd8a296069a20fbe3584507fdfd5bebae/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1209042a623ddfda5497e4066c7b77651dde8e1d3a9dd97599dc7e97f3b9b78c", size = 2325216 },
+ { url = "https://files.pythonhosted.org/packages/83/d4/075c219230697bb5db910d37262b9bacf880f92b4811a02ab81ed073a253/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:26e8268480be5061d509e29669d59103c067a26377a56491630ece11762e3858", size = 1977689 },
+ { url = "https://files.pythonhosted.org/packages/bb/08/1d219c3c2dd960983d0d4da623d916e9de6385df2b0bab3d1af0e9b8fccc/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:d79308fa689fac89cbcfbd4dbfc80b5f95c54c5a7fd4d194be221f9d33d026e6", size = 2491443 },
+ { url = "https://files.pythonhosted.org/packages/ba/d3/024208ec1079d273f1047468d1bdffbf38bb75b7b268090fd3a0301b9d9a/kiwisolver-1.5.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:b03af77d77e50edba2030fd5f7c352ff209314b09030a3cba7c14edf9a09a444", size = 2295200 },
+ { url = "https://files.pythonhosted.org/packages/6e/7c/7b210498f9f92e1cd7855f260fa69ef056881087b199ee20c208f0e4189a/kiwisolver-1.5.1-cp312-cp312-win_amd64.whl", hash = "sha256:06a6917674de9e0fe3f66f5430787f59a9f2ddb64af9b714eaec547e29ef5c19", size = 70748 },
+ { url = "https://files.pythonhosted.org/packages/94/61/ef0daa157c8bb23672f7423e0d14c39db1dc6ef8ed47e6bc54c9c1bef3bf/kiwisolver-1.5.1-cp312-cp312-win_arm64.whl", hash = "sha256:ad8b9671348d7c8716715652ae11f85ed0eb99e265a2df2ca490577d69860b2c", size = 68324 },
+]
+
+[[package]]
+name = "latex2sympy2-extended"
+version = "1.11.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "antlr4-python3-runtime" },
+ { name = "sympy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/30/75/456da2da05f6380ea96e6ea804ab2c03e41fc3ed80052307fe8efe6ea20e/latex2sympy2_extended-1.11.0.tar.gz", hash = "sha256:9695657c81b50abba2636638638618db59f4663ed2a4a12d62cef74a40e28fec", size = 207023 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e9/61/f75cd1fa54d8434276126034aed54dd120747de9a8fa013cdd79545ccbeb/latex2sympy2_extended-1.11.0-py3-none-any.whl", hash = "sha256:aebb77d52ce269e25028e4bea89ddb14d242ba36bcf7b636496fb5fd9728d234", size = 209050 },
+]
+
+[package.optional-dependencies]
+antlr4-13-2 = [
+ { name = "antlr4-python3-runtime" },
+]
+
+[[package]]
+name = "laya"
+version = "0.3.29"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "huggingface-hub" },
+ { name = "numpy" },
+ { name = "safetensors" },
+ { name = "torch" },
+ { name = "transformers" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/17/2e/9dcc0e424ea8f9e6c0cbcc91d6dbf4a0e30aba33021517eebf2a5dda286d/laya-0.3.29.tar.gz", hash = "sha256:7a3c64163be195c8ba6ca65c9cf6abb439de3d70d890900f80f3d0e923e6d0b4", size = 1082279 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/42/56/c82db3e34fe507ab6cda40ee5b8357ad5e30c1196e887ea7351896c96cde/laya-0.3.29-py3-none-any.whl", hash = "sha256:41a51775eae4bd624688f13425d71ce66aa0f6b9a49a922a775a92dc4a85c885", size = 377251 },
+]
+
+[[package]]
+name = "markupsafe"
+version = "3.0.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/38/9b/e422a865e1d5d57d0e509b4e0bf1c1a70a7f6382c29a5aa428df994c8bc8/markupsafe-3.0.4.tar.gz", hash = "sha256:2e9ad7dd851bf45fab9f75cbff4cb493fee9979e8d8c7c9c3ee119022518edd6", size = 153777 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/81/09/4c59d56b8461ae8eb0d8ba34bb25b7e618547044679d58a82ef9b2479fc1/markupsafe-3.0.4-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:61631e08084be9e21a8967ec3139c7616ed7c5e9368e05c86d1b39562c8a57b6", size = 11658 },
+ { url = "https://files.pythonhosted.org/packages/a2/f0/d6613774d86fbf6d145751d43c59875e47a6f9f17daee0aef173bd36d90e/markupsafe-3.0.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:0930db9bdc62d22944e10b066448bb65dc9abe9112880c7cab8da54db4284d5f", size = 12050 },
+ { url = "https://files.pythonhosted.org/packages/0d/f2/8f18e0b806eb13c1f8d07d917a720831ead54253a6dec011fbc78098a6f8/markupsafe-3.0.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6a45c3d514f2436064db00d7fc8778d888f0236ebfed649b53d13a59e69ad51b", size = 24363 },
+ { url = "https://files.pythonhosted.org/packages/60/ce/fa07dbe8a5675558fa36dea033e19995bc783de2dec5f540ccb9030b06aa/markupsafe-3.0.4-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:1e1451fab512d1bcc3dc26988ec1edb0b82c2db909132872cd9356070a6b63df", size = 28525 },
+ { url = "https://files.pythonhosted.org/packages/85/40/be87c01f3868ec217f8a2015089d71c22c8c5a75324822e5ed1cdd87210d/markupsafe-3.0.4-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:bd3ce56ae2cbae3ba82b683bc425cd7e48d2ed8b10f3e818186b6f5646d9271c", size = 24733 },
+ { url = "https://files.pythonhosted.org/packages/4f/a7/aeedb5140afa41fc74c225e9184ab96723a6e873b6ee1c9fede7283456d8/markupsafe-3.0.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8e124f974786f831d6043728e38296969d3579db8896fe004682f5758e613581", size = 22985 },
+ { url = "https://files.pythonhosted.org/packages/c3/fc/e91352bb08c6a59da3ef0909d457bf95a5f5908fbf151b30a06d9dbcfbb4/markupsafe-3.0.4-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c02e8f18bdedba082cef725942ac823b9b60656db07f7e265cb31618dfd00d77", size = 22001 },
+ { url = "https://files.pythonhosted.org/packages/5d/f8/bffee5e7d2a3deb59748a797650a48af7e672025cf641a79344a771ad106/markupsafe-3.0.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:9f098115c247e11d138ab83a28fa0323c77015007ea2df73ba5fd714dfefd67c", size = 23793 },
+ { url = "https://files.pythonhosted.org/packages/ed/59/b853d6628ecb4d658e1d637224846d5e9bb4adf4f8df97f3be9f29dce2ec/markupsafe-3.0.4-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:d5f93ebbeb8032d47e349328ec8662d973d9b05a70b3c35df1f91fe419b84749", size = 22640 },
+ { url = "https://files.pythonhosted.org/packages/09/b2/1506df394f0f075797c418d0301498f49e43be194e3ffcb49e6fe6ccf022/markupsafe-3.0.4-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:64511c54db4e4987aef4c41923235927428729e8174c5dba488429be70a998ed", size = 24074 },
+ { url = "https://files.pythonhosted.org/packages/c7/81/5ed69cda630ac69ef60d06c09ba5a7f84ff66a2e28cf986fd5614ab3c6e6/markupsafe-3.0.4-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:e1a622f13970d81f95d0c72f9dc090dce9085fccfa4c9f2174377ee32bd15786", size = 21563 },
+ { url = "https://files.pythonhosted.org/packages/0c/fe/fb1e79be0fea60aa32602ebefc9c35a82bb42b4df157285ab7dfec12341a/markupsafe-3.0.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:c9a7f43c0b202b334cc9184af09bb8f21d3a209e038efaf106936fb69e6b026e", size = 23048 },
+ { url = "https://files.pythonhosted.org/packages/c8/52/7632a53360671a9b750cdbabaf9cdd89f18b42248b8e4cb42c0b0296e459/markupsafe-3.0.4-cp312-cp312-win32.whl", hash = "sha256:f0ec3b750b59375eab5b0fb2b9254810c00a3375be6d789899f1055a1d556237", size = 14108 },
+ { url = "https://files.pythonhosted.org/packages/3f/bf/62495e180b7000aaf30000fff849e933f74264638057176cf46852500adc/markupsafe-3.0.4-cp312-cp312-win_amd64.whl", hash = "sha256:11935df9bf455ed0c04eb87bcd720f02b1fe5e02128a9430f23aed6f93336fc7", size = 14303 },
+ { url = "https://files.pythonhosted.org/packages/c5/8e/4c24208776a65878d656996945aacfbfe010d3720d1a98fc0eb8491fc03b/markupsafe-3.0.4-cp312-cp312-win_arm64.whl", hash = "sha256:a4bbd2d87dd233b9fc5812160c3d0ffbe42edc22a26ce0469f58479ede633fe9", size = 14174 },
+]
+
+[[package]]
+name = "math-verify"
+version = "0.9.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "latex2sympy2-extended" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/4f/12/b8d13b581e110ac2f724a2351a8361a70fa36d057eb945d6379e8747c256/math_verify-0.9.0.tar.gz", hash = "sha256:45ac6c61344ba056b9e99a660a4bc8d044ed408f730aed68c60435aa5eec4645", size = 60329 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/62/76/6b4969bccc842b6567f7e6ee015684b9428a9b7fcbdf479e73716f43597f/math_verify-0.9.0-py3-none-any.whl", hash = "sha256:3703e7c4885354027fa84409d762a596a2906d1fd4deb78361876bd905a76194", size = 29967 },
+]
+
+[package.optional-dependencies]
+antlr4-13-2 = [
+ { name = "latex2sympy2-extended", extra = ["antlr4-13-2"] },
+]
+
+[[package]]
+name = "matplotlib"
+version = "3.11.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "contourpy" },
+ { name = "cycler" },
+ { name = "fonttools" },
+ { name = "kiwisolver" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "pillow" },
+ { name = "pyparsing" },
+ { name = "python-dateutil" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e7/c8/9aa712a0afb882649424dd8de8ad9aa6235e796e84c6052e8f6dc1598d0d/matplotlib-3.11.2.tar.gz", hash = "sha256:cec596316640f2b394b8f0daa0ea61a8eae82d017b620b9f202befb972a59ea4", size = 32660610 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/cb/ce/1bfcc4873b121597791ad74032943b123218c0613af0b97e8dd05e916fb1/matplotlib-3.11.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ef752769cd962f39ea0b6ffc82d1ea43a0012c5a6157c7a075212fa509cfcff2", size = 9476466 },
+ { url = "https://files.pythonhosted.org/packages/a6/c4/7f5f3601ee69baf072c0c7d3ce60c03e0618621c5a56c34a62a460e29d11/matplotlib-3.11.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ef31985c4dedb5f1424e1aec6849a47dd37689cb7fa3c20b1b82187f26806261", size = 9305583 },
+ { url = "https://files.pythonhosted.org/packages/b8/90/2b3fd67ee273163faeda6d514be70b5596eaa0fd77b60ffc294ad0b34f5f/matplotlib-3.11.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:df4f7784aca81a94f254c0a2767d592ee25f407e488f5fa7203e51093fb6ca27", size = 9860353 },
+ { url = "https://files.pythonhosted.org/packages/f4/84/32549e7a462dc311aed2ab62e5d2538028840b5c53a8be0195c789937c3d/matplotlib-3.11.2-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1b9a7ad579856284135e401ecc918c5f8a017ee30539298862a109f51b971710", size = 10670348 },
+ { url = "https://files.pythonhosted.org/packages/84/39/02e21b74f7439bd643d717ea846006d46e309e6d29b632b57751265311d6/matplotlib-3.11.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3aa4b8516fd26659e4363abbf317c703d9116496c5db2e9d0609a2866dd39dd2", size = 10810580 },
+ { url = "https://files.pythonhosted.org/packages/04/93/0d239bde12b308262265c2d98909b2f0ecad2b5deeae96241642e822e9c7/matplotlib-3.11.2-cp312-cp312-win_amd64.whl", hash = "sha256:c5c1c68ee401fc98271263410f0e5ce88285abacf7627132914e8adf3d70ff43", size = 9349409 },
+ { url = "https://files.pythonhosted.org/packages/49/a8/06baf901c02246c8a222b655cc4540ef9c15e1549a04e07927f9cde716c3/matplotlib-3.11.2-cp312-cp312-win_arm64.whl", hash = "sha256:643ff850d8e0f5b8319337f87ed3cb59506afb3df3cc48de777d85871233be7b", size = 9028084 },
+]
+
+[[package]]
+name = "mpmath"
+version = "1.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e0/47/dd32fa426cc72114383ac549964eecb20ecfd886d1e5ccf5340b55b02f57/mpmath-1.3.0.tar.gz", hash = "sha256:7a28eb2a9774d00c7bc92411c19a89209d5da7c4c9a9e227be8330a23a25b91f", size = 508106 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl", hash = "sha256:a0b2b9fe80bbcd81a6647ff13108738cfb482d481d826cc0e02f5b35e5c88d2c", size = 536198 },
+]
+
+[[package]]
+name = "msgpack"
+version = "1.2.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/0a/e7/bb605a7bab2d8425a64b3fa762b39dc1bf1c7e3f11ba6fb5413d6db0ff8c/msgpack-1.2.3.tar.gz", hash = "sha256:32edb81a2b5eb7cd7c9d941b2bfbbb082fd2cd09e0e725930316af6b708db186", size = 196517 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/af/12/4d7c6d6203416d9fbf0f59ebaa805e70fb929b93a41b611bc821ec5964a0/msgpack-1.2.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:89c930aece4e972b208ba589c8410b4167b05e411a5ea2cb25fd96f8bc47ee43", size = 91577 },
+ { url = "https://files.pythonhosted.org/packages/eb/c7/8576ad39f4ca42ddad26f68eb8621d2d0a60501193d480f504bd9d7f36c4/msgpack-1.2.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:905a189853d6bdb204c7ae5f4ab77fb857448abfff574d3d93c62e2815b24b4f", size = 90027 },
+ { url = "https://files.pythonhosted.org/packages/0a/3a/aa9c580aea1314529a0f3562461479780b0d254b064f0880956bfbcc74a8/msgpack-1.2.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f3d7b3d0018746b5997dd6b14a1870b07cc4c327d9101145d94a1fc264a51a06", size = 460343 },
+ { url = "https://files.pythonhosted.org/packages/3a/cf/9c2e4d6c179529d5bf4a64cff76fa581486569e9fbdd35bd98f51cb624bf/msgpack-1.2.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ede33b2892ceb976283e009ad12fa1834cfdf1f9c43ee9c97849fc588d00a618", size = 472998 },
+ { url = "https://files.pythonhosted.org/packages/7b/41/915c81fe6df2d3cbdb0dece4f1a5cd313e1cd2abd9f501d0f50c0582517e/msgpack-1.2.3-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:666ef5601ab0e6e345e47febc96aa81143cc932201543480cbb9499164f05ffb", size = 423216 },
+ { url = "https://files.pythonhosted.org/packages/a2/e7/7dda8b1039abfd9bba4c5068172c67135c9e33089f503512db9226f23c24/msgpack-1.2.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:87cf2ef05ff2f2493ba29fcdaef27e960ca64dacfd13460ae29e6f92e0ed05bb", size = 451218 },
+ { url = "https://files.pythonhosted.org/packages/16/5b/ce995c1ed4a0522b7f2d034bc2034fd63005f240b945961b70fb56fbaf3d/msgpack-1.2.3-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:b774ff994d844e541439ac5d2d49a14def4104830c3465e9394c153f86200ffb", size = 422453 },
+ { url = "https://files.pythonhosted.org/packages/d2/3f/ce191fb87e2650d0166b34c437e499ee4a7f9db9c1eb164f41725eb6160e/msgpack-1.2.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:eaf7e82249837e3aa97297b34a0bb9ff562027381631e057cea6e1367f10b438", size = 469003 },
+ { url = "https://files.pythonhosted.org/packages/42/35/539123407fe200fb16609c835675496fbeb6017ace9fc93909f0613223ae/msgpack-1.2.3-cp312-cp312-win32.whl", hash = "sha256:7c047250096f9fc19dba26e3d1639b5e7a84114003605c94def667149a70ced1", size = 68303 },
+ { url = "https://files.pythonhosted.org/packages/6f/4c/331b45f9b86fbda6b9e103244d189068e51f726d8c40021ed66e1f2c415e/msgpack-1.2.3-cp312-cp312-win_amd64.whl", hash = "sha256:3ec409b0d6aa8e9eec6eaf881b893caa215dbe68c5319ca96e8a271d81bb111d", size = 76744 },
+ { url = "https://files.pythonhosted.org/packages/13/9f/fb572dc42b9fac06c7ea848aaee6e140d84469743bd1402bc07089fc4566/msgpack-1.2.3-cp312-cp312-win_arm64.whl", hash = "sha256:59612b4ed48a04cf024584218e813562f3b30a3bafa5f55abe300b15da314751", size = 71580 },
+]
+
+[[package]]
+name = "msgpack-numpy"
+version = "0.4.8"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "msgpack" },
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/08/94/61e8aee142733ebfdc400a05bdac6e1763c4514bba3b42743d223f388450/msgpack-numpy-0.4.8.tar.gz", hash = "sha256:c667d3180513422f9c7545be5eec5d296dcbb357e06f72ed39cc683797556e69", size = 10923 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9b/5d/f25ac7d4fb77cbd53ddc6d05d833c6bf52b12770a44fa9a447eed470ca9a/msgpack_numpy-0.4.8-py2.py3-none-any.whl", hash = "sha256:773c19d4dfbae1b3c7b791083e2caf66983bb19b40901646f61d8731554ae3da", size = 6919 },
+]
+
+[[package]]
+name = "multidict"
+version = "7.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f9/79/84ddb5ba16c4eb2c69c71db76ae3c579fe546e511f7170c7e27eedbab7c1/multidict-7.1.0.tar.gz", hash = "sha256:61a4e5d81b8d4e4ad61964b230129e7a2b914793d96289029078fc9009f074ec", size = 362341 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/03/e1/215e7df354e2907f3af9d910c8136653cfec94796012a776359bc802a326/multidict-7.1.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:ccfb950359a80de0fcd2030ad60ac1b1a861462de3e2ef746697c9256659af21", size = 165426 },
+ { url = "https://files.pythonhosted.org/packages/a4/ec/461ba588b308ada2cd16907d6ea425ca4d417596b88c12814ad9a0bb7325/multidict-7.1.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:939d8cd2d8c35e3956f6bc858390b6ccb611e6152b4920d64ab5e98f3fcf39e4", size = 95943 },
+ { url = "https://files.pythonhosted.org/packages/c9/84/31444ef07ec13a33c42c2772d986129e137e69c4beb89ca64f2138988145/multidict-7.1.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f8e95c95039eab6a2dad8c83c38ab87fc5431d28849e0c8a7e2a4e70ba38710d", size = 98294 },
+ { url = "https://files.pythonhosted.org/packages/dc/10/aca13806d73d88b5b01e35e828e23a346cb1abad710a49dff0507702efc9/multidict-7.1.0-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:05d12b4bac53abe0c65f3163af2b45894e2e1c0cc55493ac784d52a350047d88", size = 453301 },
+ { url = "https://files.pythonhosted.org/packages/96/8c/382d771bfb3a9282d98332d0e1f27fd1f8b4ae0e7175bd7e008ebf934900/multidict-7.1.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0e79ed92b1dece6bb57e9b46effd74d7a5d3d00187c85466d880ed184239a698", size = 478570 },
+ { url = "https://files.pythonhosted.org/packages/52/9c/e81b0c92449da3a1950a575c520d957d7be618777d170717a71e00558d7f/multidict-7.1.0-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:fab380fcff8b3555eb2bd04304fa4330909a771a9a9b0dc07666cfc23148a711", size = 443699 },
+ { url = "https://files.pythonhosted.org/packages/63/72/f8f5fee6960d1b580a7b046c7c5abccf67aa26fa5647927a91c9c835c2f8/multidict-7.1.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4b5c41e44da74383c924cc5d75ef0a268f301d69305b3c42bd17af685d55e412", size = 497332 },
+ { url = "https://files.pythonhosted.org/packages/e2/92/a25d7db3ca451b588e743e067ee80f2b475edf6467a56908454faf6a714c/multidict-7.1.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3dbaa7f7c2f0ca8578895fc61fb8c8e50ebb405dad8982f92f4343285c7a3fda", size = 512457 },
+ { url = "https://files.pythonhosted.org/packages/7a/f3/374c0ab122bb98b1a62a8742b1e3f59563e4d609942005a4041861a0df67/multidict-7.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fed6b7705d49dd07e5e0dd5f5c873fc44047e92d714299b13245b5fecac49d01", size = 483000 },
+ { url = "https://files.pythonhosted.org/packages/46/1f/01c8522859771dc3cee840d5852233fe84c91aa982a1cd8ad594306d25cc/multidict-7.1.0-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:53daa47dd176db64bb35170e3d5d0ae2388c060121201883696278f055a0e70c", size = 430746 },
+ { url = "https://files.pythonhosted.org/packages/ab/f3/af90affc8cca6b4ee59b2ec354a20839005e829d14d155451009810ef1ce/multidict-7.1.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:542429c796430de924d03b68a6173bb6d79d5c4967d4e9a18de3e501cad55593", size = 466859 },
+ { url = "https://files.pythonhosted.org/packages/f8/97/1b6762f37f6331449e164af9d5500f0abb0f23670e81ba543441e0a6be1d/multidict-7.1.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:c44ca6d3cdf4cfcbcd4f928fdcbe87af5fd7319f6ad4169617b7fd6b4527c33c", size = 441952 },
+ { url = "https://files.pythonhosted.org/packages/bf/d2/4908177fbf22438799c04ba11a2853a99c69d028fccefe61f19e68caba0c/multidict-7.1.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:eb0228c809b2e7eb47921876050af0bc4214b351bad8d8112f70b6ed4288763c", size = 452810 },
+ { url = "https://files.pythonhosted.org/packages/8d/05/5031f44f680ec54fc182c4d71c9ab7c07216983946aa64ce6fdd56a52692/multidict-7.1.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:6ad60de1f4c702448fc8f1449f05e810f6b7957c08a5b3950c8a792dfb13b50a", size = 483934 },
+ { url = "https://files.pythonhosted.org/packages/da/3b/9b21d107dbe96fa7e6966ff9e5e10b9ac0d2d1c2cf1633098e4c700f865e/multidict-7.1.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:f79def86aee67b5ba01b2565f1610f262bf88ae53c379f93e5fa29c50fe793be", size = 427042 },
+ { url = "https://files.pythonhosted.org/packages/53/ce/5b01b1041580072866b30e39c6380bff269e72fb5ca41a7f2fb828ede943/multidict-7.1.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:248dabb89b5aa90b2f7e43e045f048f7e5392ec77b6446d80853ba7117d7bbdf", size = 492543 },
+ { url = "https://files.pythonhosted.org/packages/1e/6f/6508a23fcc7b1122e4f18409d7900ffeb3cd020cd080fe98aba3eabf9486/multidict-7.1.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:0747a83e7ae617793181a4763ee8b84863cec5c0bbbde70c4394e4c0276c36de", size = 475796 },
+ { url = "https://files.pythonhosted.org/packages/8f/b6/ebb6433f4aa55ac1fea4bf11e860e99611640d07cc92d21cb3f35608de22/multidict-7.1.0-cp312-cp312-win32.whl", hash = "sha256:1df055e51fe7491120cc84f3362bd43db186be78d0e4c476acad45e435af9ffb", size = 81250 },
+ { url = "https://files.pythonhosted.org/packages/61/0a/11240e5e7d2e986e288f4a6090f569ad00225b7c2c513d4096b36643f9ab/multidict-7.1.0-cp312-cp312-win_amd64.whl", hash = "sha256:10202ba98cfb3f7eb60da7ca87a2c458a69b7d0d6e4d4388cd6773ebbce89085", size = 93817 },
+ { url = "https://files.pythonhosted.org/packages/e9/9b/a04ffd76db7cca2a60d347e839315e1ada63017c6c343fa90a23a5a55433/multidict-7.1.0-cp312-cp312-win_arm64.whl", hash = "sha256:0aa1ba3ff7cdda05a1242490612976b2ae1c90fc6200903ef8f53815dcb35c5d", size = 92312 },
+ { url = "https://files.pythonhosted.org/packages/d0/86/a3de309c5e28ee85b314d0e3ba0e0dea6fd361c313322a05e67be4656e1e/multidict-7.1.0-py3-none-any.whl", hash = "sha256:d9ef29cfd98e17085b4f91bba8fa1570bec6787d5c52ce653ed33a58785585d0", size = 28087 },
+]
+
+[[package]]
+name = "networkx"
+version = "3.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/dc/76/3af777226b63a5e64a6b36b1ec5855c14e2b94a37096d4760e595fc43511/networkx-3.7.tar.gz", hash = "sha256:fd77a511bd90f39f3d016351345b52cf5319b813bdca01de3f755d3cca62e96a", size = 1866482 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7e/cd/fe58041e9011f307c490e3e17dd48cc516448f7c698a3f2d9d9d65d7e6a8/networkx-3.7-py3-none-any.whl", hash = "sha256:e3fd2c13a7814cee3746340d8d7f8598a67f16a58bf47fb7f8793fab6efca1b0", size = 2142205 },
+]
+
+[[package]]
+name = "numpy"
+version = "2.4.6"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d0/ad/fed0499ce6a338d2a03ebae59cd15093910c8875328855781952abf6c2fe/numpy-2.4.6.tar.gz", hash = "sha256:f3a3570c4a2a16746ac2c31a7c7c7b0c186b95ce902e33db6f28094ed7387dda", size = 20735807 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/95/2a/3d7b5ac8aac24feaf9ad7ed58f45b0bbc06d37e4338ae84c9f2298b570f9/numpy-2.4.6-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:001fbb8e08d942dd57599e781f2472269ee7f2755fae407b4f67b2f0b17da3f1", size = 16689119 },
+ { url = "https://files.pythonhosted.org/packages/ea/12/92c4c131527599e8288d6918e888d88726f84d805d784b771f32408aeaef/numpy-2.4.6-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ebfb099f8dcf083deef3ac1ca4c1503f387cf76296fcb3816b66f5ecb5f54fdb", size = 14699246 },
+ { url = "https://files.pythonhosted.org/packages/ad/fe/c0a6b7b2ca128a8fb228575147073b660656734b8ebe4d76c8fd748dcc79/numpy-2.4.6-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:3213d622a0283a39a93d188f3cf72b26862df52fbb4ca3697f51705016523d41", size = 5204410 },
+ { url = "https://files.pythonhosted.org/packages/f3/d4/9770d14ba719432bb90a421bfd443872ed0f70f7264b64bec12ea363d5fd/numpy-2.4.6-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:357cc07a6d7b0b182ff02249616a03742827ebb1277546b5c7cd7f7620a45698", size = 6551240 },
+ { url = "https://files.pythonhosted.org/packages/c9/c6/50a46a6205feba2343f1d6d17438107c5dc491ed1c736e6ea68689fd906b/numpy-2.4.6-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5f9fb9157b4ce2971008323afe46053787b526ef624fea915b261468a8421a0f", size = 15671012 },
+ { url = "https://files.pythonhosted.org/packages/99/60/14115e6364fa676c5397c2ad3004e527e9aa487abf5d0706ec81bbd08529/numpy-2.4.6-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:90f9849678c75fe7afa2d348ac842c168b0a4d3d61919687216dfc547976d853", size = 16645538 },
+ { url = "https://files.pythonhosted.org/packages/ae/c5/693cbe59e57db94d2231fa519ca3978dc9e19da5a8f088588f5c6e947ff2/numpy-2.4.6-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c1a2af6c6ef86344a6b0db6b97834208bf598db514f2b155042439b62605601a", size = 17020706 },
+ { url = "https://files.pythonhosted.org/packages/ef/fc/85b7c4eff9b4966ade25c2273cf7e7012e92366c032058653934b37de044/numpy-2.4.6-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e5805d5a22fd19c8ccff10a9561f9df94436b0545619ea579db2d3c35294bce2", size = 18368541 },
+ { url = "https://files.pythonhosted.org/packages/f6/81/e1b27545deedce7f4a0b348618c6b62d74e36a4dc9ccd42f3eb2f85eee32/numpy-2.4.6-cp312-cp312-win32.whl", hash = "sha256:e3eeb0aabd6bd5ce64faae67e9935203a6991b4bc2a485a767fbafb2c5125f45", size = 5962825 },
+ { url = "https://files.pythonhosted.org/packages/ab/ca/feab00bd44aa5fe1ad2c18f08b4d3bb92e26484b0b1d1443897809ed528c/numpy-2.4.6-cp312-cp312-win_amd64.whl", hash = "sha256:d8e8286dd7cea7895157318d1b91cdacac64c479f3cbc8dce548331728484751", size = 12321687 },
+ { url = "https://files.pythonhosted.org/packages/63/cf/5a6d34850a39d1093558564f77ee8e8e0bee5061151b8f05a55711001ec7/numpy-2.4.6-cp312-cp312-win_arm64.whl", hash = "sha256:4081eb135ac24158bd51cdfbef16f1c64df7063b1143f24731387137c092bec8", size = 10221482 },
+]
+
+[[package]]
+name = "nvidia-cublas-cu12"
+version = "12.8.4.1"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/dc/61/e24b560ab2e2eaeb3c839129175fb330dfcfc29e5203196e5541a4c44682/nvidia_cublas_cu12-12.8.4.1-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:8ac4e771d5a348c551b2a426eda6193c19aa630236b418086020df5ba9667142", size = 594346921 },
+]
+
+[[package]]
+name = "nvidia-cuda-cupti-cu12"
+version = "12.8.90"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f8/02/2adcaa145158bf1a8295d83591d22e4103dbfd821bcaf6f3f53151ca4ffa/nvidia_cuda_cupti_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ea0cb07ebda26bb9b29ba82cda34849e73c166c18162d3913575b0c9db9a6182", size = 10248621 },
+]
+
+[[package]]
+name = "nvidia-cuda-nvrtc-cu12"
+version = "12.8.93"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/05/6b/32f747947df2da6994e999492ab306a903659555dddc0fbdeb9d71f75e52/nvidia_cuda_nvrtc_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:a7756528852ef889772a84c6cd89d41dfa74667e24cca16bb31f8f061e3e9994", size = 88040029 },
+]
+
+[[package]]
+name = "nvidia-cuda-runtime-cu12"
+version = "12.8.90"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0d/9b/a997b638fcd068ad6e4d53b8551a7d30fe8b404d6f1804abf1df69838932/nvidia_cuda_runtime_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:adade8dcbd0edf427b7204d480d6066d33902cab2a4707dcfc48a2d0fd44ab90", size = 954765 },
+]
+
+[[package]]
+name = "nvidia-cudnn-cu12"
+version = "9.10.2.21"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nvidia-cublas-cu12" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ba/51/e123d997aa098c61d029f76663dedbfb9bc8dcf8c60cbd6adbe42f76d049/nvidia_cudnn_cu12-9.10.2.21-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:949452be657fa16687d0930933f032835951ef0892b37d2d53824d1a84dc97a8", size = 706758467 },
+]
+
+[[package]]
+name = "nvidia-cufft-cu12"
+version = "11.3.3.83"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nvidia-nvjitlink-cu12" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1f/13/ee4e00f30e676b66ae65b4f08cb5bcbb8392c03f54f2d5413ea99a5d1c80/nvidia_cufft_cu12-11.3.3.83-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4d2dd21ec0b88cf61b62e6b43564355e5222e4a3fb394cac0db101f2dd0d4f74", size = 193118695 },
+]
+
+[[package]]
+name = "nvidia-cufile-cu12"
+version = "1.13.1.3"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/bb/fe/1bcba1dfbfb8d01be8d93f07bfc502c93fa23afa6fd5ab3fc7c1df71038a/nvidia_cufile_cu12-1.13.1.3-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1d069003be650e131b21c932ec3d8969c1715379251f8d23a1860554b1cb24fc", size = 1197834 },
+]
+
+[[package]]
+name = "nvidia-curand-cu12"
+version = "10.3.9.90"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fb/aa/6584b56dc84ebe9cf93226a5cde4d99080c8e90ab40f0c27bda7a0f29aa1/nvidia_curand_cu12-10.3.9.90-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:b32331d4f4df5d6eefa0554c565b626c7216f87a06a4f56fab27c3b68a830ec9", size = 63619976 },
+]
+
+[[package]]
+name = "nvidia-cusolver-cu12"
+version = "11.7.3.90"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nvidia-cublas-cu12" },
+ { name = "nvidia-cusparse-cu12" },
+ { name = "nvidia-nvjitlink-cu12" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/85/48/9a13d2975803e8cf2777d5ed57b87a0b6ca2cc795f9a4f59796a910bfb80/nvidia_cusolver_cu12-11.7.3.90-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:4376c11ad263152bd50ea295c05370360776f8c3427b30991df774f9fb26c450", size = 267506905 },
+]
+
+[[package]]
+name = "nvidia-cusparse-cu12"
+version = "12.5.8.93"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nvidia-nvjitlink-cu12" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c2/f5/e1854cb2f2bcd4280c44736c93550cc300ff4b8c95ebe370d0aa7d2b473d/nvidia_cusparse_cu12-12.5.8.93-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1ec05d76bbbd8b61b06a80e1eaf8cf4959c3d4ce8e711b65ebd0443bb0ebb13b", size = 288216466 },
+]
+
+[[package]]
+name = "nvidia-cusparselt-cu12"
+version = "0.7.1"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/56/79/12978b96bd44274fe38b5dde5cfb660b1d114f70a65ef962bcbbed99b549/nvidia_cusparselt_cu12-0.7.1-py3-none-manylinux2014_x86_64.whl", hash = "sha256:f1bb701d6b930d5a7cea44c19ceb973311500847f81b634d802b7b539dc55623", size = 287193691 },
+]
+
+[[package]]
+name = "nvidia-nccl-cu12"
+version = "2.27.5"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6e/89/f7a07dc961b60645dbbf42e80f2bc85ade7feb9a491b11a1e973aa00071f/nvidia_nccl_cu12-2.27.5-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ad730cf15cb5d25fe849c6e6ca9eb5b76db16a80f13f425ac68d8e2e55624457", size = 322348229 },
+]
+
+[[package]]
+name = "nvidia-nvjitlink-cu12"
+version = "12.8.93"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f6/74/86a07f1d0f42998ca31312f998bd3b9a7eff7f52378f4f270c8679c77fb9/nvidia_nvjitlink_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:81ff63371a7ebd6e6451970684f916be2eab07321b73c9d244dc2b4da7f73b88", size = 39254836 },
+]
+
+[[package]]
+name = "nvidia-nvshmem-cu12"
+version = "3.3.20"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3b/6c/99acb2f9eb85c29fc6f3a7ac4dccfd992e22666dd08a642b303311326a97/nvidia_nvshmem_cu12-3.3.20-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d00f26d3f9b2e3c3065be895e3059d6479ea5c638a3f38c9fec49b1b9dd7c1e5", size = 124657145 },
+]
+
+[[package]]
+name = "nvidia-nvtx-cu12"
+version = "12.8.90"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a2/eb/86626c1bbc2edb86323022371c39aa48df6fd8b0a1647bc274577f72e90b/nvidia_nvtx_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5b17e2001cc0d751a5bc2c6ec6d26ad95913324a4adb86788c944f8ce9ba441f", size = 89954 },
+]
+
+[[package]]
+name = "packaging"
+version = "26.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/7d/fa/3944b40b07da9ce895c0e6303a5ab7d53da063554f534556b134a54d6093/packaging-26.3.tar.gz", hash = "sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79", size = 313412 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/63/34/ba1c580383c9eada3711951fef0795c80b829a078d72188184bcab9dd527/packaging-26.3-py3-none-any.whl", hash = "sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c", size = 129956 },
+]
+
+[[package]]
+name = "peft"
+version = "0.18.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "accelerate" },
+ { name = "huggingface-hub" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "psutil" },
+ { name = "pyyaml" },
+ { name = "safetensors" },
+ { name = "torch" },
+ { name = "tqdm" },
+ { name = "transformers" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d8/48/147b3ea999560b40a34fd78724c7777aa9d18409c2250bdcaf9c4f2db7fc/peft-0.18.1.tar.gz", hash = "sha256:2dd0d6bfce936d1850e48aaddbd250941c5c02fc8ef3237cd8fd5aac35e0bae2", size = 635030 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b3/14/b4e3f574acf349ae6f61f9c000a77f97a3b315b4bb6ad03791e79ae4a568/peft-0.18.1-py3-none-any.whl", hash = "sha256:0bf06847a3551e3019fc58c440cffc9a6b73e6e2962c95b52e224f77bbdb50f1", size = 556960 },
+]
+
+[[package]]
+name = "pillow"
+version = "13.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ce/a4/9cc30690a394a8305e8be4c9036aa16d0dfddadd09079860b582f562496d/pillow-13.0.0.tar.gz", hash = "sha256:cdcd1ba258178f7356d74170c9a81ba4eab6a5d55748bc402a5a2a6fe6be3510", size = 47058631 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7a/b8/58d79332229482eb0d27c82973850808c19d37e5bdc14443d3417bd373f5/pillow-13.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:67c8d44ad89fa7eb88de5d6ef6199b1e03dd24974f85a67639fc8badb801fe79", size = 5568643 },
+ { url = "https://files.pythonhosted.org/packages/39/4b/f2ed53d83aa44ac19599ac0a54897f867fafb95f406980d1e24083f8cbe3/pillow-13.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc481f742ccda20e9ed760b8f978dff50f2a4e9bc253ac167fb0a05b0783ef52", size = 4852329 },
+ { url = "https://files.pythonhosted.org/packages/3b/bb/9e30e8ed18c20800ee327a12107d5249ad8dc444867b3adfcc21654e3c71/pillow-13.0.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8bf295fe459343bc6093c8e5e0df60bfa189e0fcb775a22e2c2488fdeccc6c0d", size = 6519309 },
+ { url = "https://files.pythonhosted.org/packages/e6/99/fdf65cc539e266ab93546c261bdf98411fe165c6e4e13f9dbf597097192b/pillow-13.0.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:42d63be3af4f9b951b199b0a9d23365d37ab14687e6d77b74940432c8515c4ca", size = 7144542 },
+ { url = "https://files.pythonhosted.org/packages/26/9b/f38b7c4d8f599537e2d1a6d546cec9beef7536ff8f5c49a3f9c78179d8ac/pillow-13.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:4bd024d567eca931b668978d672dba914d4b8e2628e2c3073a2ad178de8a2620", size = 6618042 },
+ { url = "https://files.pythonhosted.org/packages/a8/7c/45ee28531f89ee4582967d8a17f7a26d71ca64cd902b93c269590cd60a00/pillow-13.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:23edf4ec4b6f4f240e8df6e4985e9ee2ced86474632433e55fb1e2548744bf40", size = 7228137 },
+ { url = "https://files.pythonhosted.org/packages/01/bc/9c2a969cb1b70201d04fa17fd5aedf02a732870b8da86bceaaa30b0ca385/pillow-13.0.0-cp312-cp312-win32.whl", hash = "sha256:02464da86f561f410e13b061ec37c3068a9cb557a095494157742d07b3c24a27", size = 6515423 },
+ { url = "https://files.pythonhosted.org/packages/ff/cd/73a79ecdc822076002eb24b5e1e4a530b0bb5310fcb6d239abbec877110b/pillow-13.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:ff3fe9eb621fbb9db4c4be33ab946f6bac5f98ab50ca8d3260503653656c961b", size = 7272166 },
+ { url = "https://files.pythonhosted.org/packages/4a/a3/be5a1236efd085b3984e4305da8f179ae9dab80c991d2ad78f2bd5c4c843/pillow-13.0.0-cp312-cp312-win_arm64.whl", hash = "sha256:c3bc8ee0271387e0b1e3ad0ff893645990a656e79bf9c42f4f734a3642b11d51", size = 2698943 },
+ { url = "https://files.pythonhosted.org/packages/60/da/f6d644e631c5b370c5d0b899f3ccadd2780aab070b29a110527cf4e1fd2c/pillow-13.0.0-pp312-pypy312_pp80-macosx_10_15_x86_64.whl", hash = "sha256:8619685cad8efc4c38c66d3b36aad854523297c43d748b3b6f0cc03c90ba3463", size = 5523188 },
+ { url = "https://files.pythonhosted.org/packages/e4/b0/2594f202d743a35d4fe80682bfd96f277e5ca7aecb27cf64ef5bfc1d000b/pillow-13.0.0-pp312-pypy312_pp80-macosx_11_0_arm64.whl", hash = "sha256:7e91e97a929232cdb828889ee6ad0e48d7f40f4ec38818f4e4a8968b2b1af22e", size = 4806808 },
+ { url = "https://files.pythonhosted.org/packages/8e/ed/d504165ef79af6fafe25c76726cf30f544cc236002f7ed054f1d51adc4ff/pillow-13.0.0-pp312-pypy312_pp80-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:93ff9c9c3663ce23aaa5d0b6b6abac55359e2a0c06c7e30f3771829af816be63", size = 5464358 },
+ { url = "https://files.pythonhosted.org/packages/7b/fe/a10aba5503ca13e9e009b8bd0a1a4ce5584bdce246245fd6f3a83701369c/pillow-13.0.0-pp312-pypy312_pp80-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e65dd6e75de67b2362a4fa3e16aa627595f65054dc4ec73d64680c84d592fa61", size = 6043514 },
+ { url = "https://files.pythonhosted.org/packages/c0/37/66b12dc476041fe523bca29c9d58fbf1ea92a4eaf73acfb9e993948f1191/pillow-13.0.0-pp312-pypy312_pp80-win_amd64.whl", hash = "sha256:567f20f1c75897d7663830da6ea8af719e03d712b1d9ee7f813319f9ece42e5a", size = 7366030 },
+]
+
+[[package]]
+name = "propcache"
+version = "0.5.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b3/9a/9fbf4e4ec0c2d7f1c32519fff782ef467859b8faa9fbc5331a96f6395d43/propcache-0.5.4.tar.gz", hash = "sha256:ff6b113f50bc066a698db5d944d2c6dc7507168dd3341e255a8892fd0715a558", size = 61545 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/71/cd/348d58f142aebc4873345c6b31087629182ca6e0f2b3caeaa528cf882eba/propcache-0.5.4-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:b28f41fa3b8c6900457f858ec5b03998f3a6d535fbc1bb2edec5961ea05ec429", size = 87285 },
+ { url = "https://files.pythonhosted.org/packages/df/f4/f3ffaee281b276da854ac1d7a6a506d26cbc62ea2e623756f1d0a4a1ba1a/propcache-0.5.4-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:dcbf346a318a5e30063f547630b02bb787ce2f45b6368d5da143660b6a3835d8", size = 50984 },
+ { url = "https://files.pythonhosted.org/packages/25/88/1d7df7201750b37765ef2b23bc1c526c028dadde80afa0f57a118fc01182/propcache-0.5.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:87a3caecf8095e48dc72f84bfa42e23a848cf410cc9cc13031fba4869b706a21", size = 52460 },
+ { url = "https://files.pythonhosted.org/packages/83/4f/48865bd02a16ee5236bc46166b2946f37b93e07b0eae355dac0be0b216ca/propcache-0.5.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:60a64cbccaa11b7760ce705a14ada17ba459e7ca9f23ba587eb013821032d7ef", size = 251768 },
+ { url = "https://files.pythonhosted.org/packages/b0/19/3742a5eed62317b03b4002ee865dc9fd720308bdd0da1f29a5786c630311/propcache-0.5.4-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a74bfa37147cc08fb29df10bd9c16f40fa7f860cd3a6d2fff853323a94f6e17f", size = 257723 },
+ { url = "https://files.pythonhosted.org/packages/cb/d5/ee6350fb0be9122bb6c67082a876d34b90d980d100c106af4b81023e04f4/propcache-0.5.4-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a4d7a54719b67338a305dca2ce6aafe366817df94ddfd4b5514374356f5ca546", size = 265597 },
+ { url = "https://files.pythonhosted.org/packages/85/9f/83a07b6ec0e043c050cfdd35fb0cf1b7897b91d554d6eea293740309afe7/propcache-0.5.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2814ecd8e818f487bee4b0f921bc4d1c176cc5fc71ac0f072d0fa67eda4ac14b", size = 250424 },
+ { url = "https://files.pythonhosted.org/packages/33/2c/a763a8251f50fba042af0fb1f02bfec4b31381e40aff760db2be7b2e1f84/propcache-0.5.4-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:6af4693716bfb03f1752ef1b30faa593db2c01d5272e9b8564a1549452a979ab", size = 216748 },
+ { url = "https://files.pythonhosted.org/packages/6a/e2/4d11bea8fd6a777149c6c20645f873952eab5de3a2497aa11648ec9ab6ab/propcache-0.5.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:4fbc1a15dc8cd1689508758d626b372b1f09d28d9577667feaf9e6bfcd8efcbc", size = 246533 },
+ { url = "https://files.pythonhosted.org/packages/9f/36/6683597de4907e70c717e3588c541202c66086a72ff3db58be49de66e72c/propcache-0.5.4-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:cdee8205a44d0be91bbac4c41b95d86641b72dfc7aef1279400e4fda3f26a937", size = 238173 },
+ { url = "https://files.pythonhosted.org/packages/85/84/cb08d79f1762daafeb2b030c470cd0c725c97b8ad67412457c6f35c53e9d/propcache-0.5.4-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:9a2a8a50a93dee0268a860a07fa3b4bd968f8ce4dbd794957da772f395368526", size = 251128 },
+ { url = "https://files.pythonhosted.org/packages/c2/0d/41b848036db6621370c1f2e5471a7da8149c730f8552a5257567721f4576/propcache-0.5.4-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:7ffafcbfc7b549ab940047e505c831eabac5e67de53e1bc174adbc5285c55944", size = 214821 },
+ { url = "https://files.pythonhosted.org/packages/f1/b7/adfae4bf9c63bccf12e2d9690a175c6579047a6eec3b5a6a5f51428c15e2/propcache-0.5.4-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:d1f5a500bfcbb2c0ab85e98a0dcd70f5899d34efe365a0187700369a79603031", size = 254793 },
+ { url = "https://files.pythonhosted.org/packages/51/6f/eeca9647245d5f92e87d53e5f14335bb42fce1a7e6842c8045b364eded8b/propcache-0.5.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:8a235f73d6e020855dc29dff012d920c02ee0feab8d73a24185a7569f4be1161", size = 247134 },
+ { url = "https://files.pythonhosted.org/packages/5d/a9/424e38838793d37160b4379c702f61c74c598fc6cd17204adbe3c554f7a8/propcache-0.5.4-cp312-cp312-win32.whl", hash = "sha256:b3083bfe87f95c756e610bd8025f26cbd1cd4aaa03a422f2d65efb7a97cd53d8", size = 43073 },
+ { url = "https://files.pythonhosted.org/packages/58/7b/6e8ef26f6d510a7916064fec68d55fcbfbdf7eb01e377480d66a122152d8/propcache-0.5.4-cp312-cp312-win_amd64.whl", hash = "sha256:98914de2c4d7f0f9f4a8c6ea4bf05841f4175796941e3ef7d47eb718f22311fb", size = 46190 },
+ { url = "https://files.pythonhosted.org/packages/08/b9/72028c5b56ced97f456de6aefa79435ca64d7f77af78ea8cf3c76fc5195f/propcache-0.5.4-cp312-cp312-win_arm64.whl", hash = "sha256:8876b39961e33d912afe3c1bee18ee564fdad0206f873cc15d522756b7f50737", size = 43075 },
+ { url = "https://files.pythonhosted.org/packages/f5/cd/785c64ed382f3f04201870267b02783f63b4678c2acfddc177a3ebcc2727/propcache-0.5.4-py3-none-any.whl", hash = "sha256:62c60aec739ed00124573cce1178138fd690c7676352d67a37328c1cf51d7468", size = 16338 },
+]
+
+[[package]]
+name = "psutil"
+version = "7.2.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/aa/c6/d1ddf4abb55e93cebc4f2ed8b5d6dbad109ecb8d63748dd2b20ab5e57ebe/psutil-7.2.2.tar.gz", hash = "sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372", size = 493740 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e7/36/5ee6e05c9bd427237b11b3937ad82bb8ad2752d72c6969314590dd0c2f6e/psutil-7.2.2-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486", size = 129090 },
+ { url = "https://files.pythonhosted.org/packages/80/c4/f5af4c1ca8c1eeb2e92ccca14ce8effdeec651d5ab6053c589b074eda6e1/psutil-7.2.2-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979", size = 129859 },
+ { url = "https://files.pythonhosted.org/packages/b5/70/5d8df3b09e25bce090399cf48e452d25c935ab72dad19406c77f4e828045/psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9", size = 155560 },
+ { url = "https://files.pythonhosted.org/packages/63/65/37648c0c158dc222aba51c089eb3bdfa238e621674dc42d48706e639204f/psutil-7.2.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e", size = 156997 },
+ { url = "https://files.pythonhosted.org/packages/8e/13/125093eadae863ce03c6ffdbae9929430d116a246ef69866dad94da3bfbc/psutil-7.2.2-cp36-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8", size = 148972 },
+ { url = "https://files.pythonhosted.org/packages/04/78/0acd37ca84ce3ddffaa92ef0f571e073faa6d8ff1f0559ab1272188ea2be/psutil-7.2.2-cp36-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc", size = 148266 },
+ { url = "https://files.pythonhosted.org/packages/b4/90/e2159492b5426be0c1fef7acba807a03511f97c5f86b3caeda6ad92351a7/psutil-7.2.2-cp37-abi3-win_amd64.whl", hash = "sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988", size = 137737 },
+ { url = "https://files.pythonhosted.org/packages/8c/c7/7bb2e321574b10df20cbde462a94e2b71d05f9bbda251ef27d104668306a/psutil-7.2.2-cp37-abi3-win_arm64.whl", hash = "sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee", size = 134617 },
+]
+
+[[package]]
+name = "pyarrow"
+version = "23.0.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/88/22/134986a4cc224d593c1afde5494d18ff629393d74cc2eddb176669f234a4/pyarrow-23.0.1.tar.gz", hash = "sha256:b8c5873e33440b2bc2f4a79d2b47017a89c5a24116c055625e6f2ee50523f019", size = 1167336 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9a/4b/4166bb5abbfe6f750fc60ad337c43ecf61340fa52ab386da6e8dbf9e63c4/pyarrow-23.0.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:f4b0dbfa124c0bb161f8b5ebb40f1a680b70279aa0c9901d44a2b5a20806039f", size = 34214575 },
+ { url = "https://files.pythonhosted.org/packages/e1/da/3f941e3734ac8088ea588b53e860baeddac8323ea40ce22e3d0baa865cc9/pyarrow-23.0.1-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:7707d2b6673f7de054e2e83d59f9e805939038eebe1763fe811ee8fa5c0cd1a7", size = 35832540 },
+ { url = "https://files.pythonhosted.org/packages/88/7c/3d841c366620e906d54430817531b877ba646310296df42ef697308c2705/pyarrow-23.0.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:86ff03fb9f1a320266e0de855dee4b17da6794c595d207f89bba40d16b5c78b9", size = 44470940 },
+ { url = "https://files.pythonhosted.org/packages/2c/a5/da83046273d990f256cb79796a190bbf7ec999269705ddc609403f8c6b06/pyarrow-23.0.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:813d99f31275919c383aab17f0f455a04f5a429c261cc411b1e9a8f5e4aaaa05", size = 47586063 },
+ { url = "https://files.pythonhosted.org/packages/5b/3c/b7d2ebcff47a514f47f9da1e74b7949138c58cfeb108cdd4ee62f43f0cf3/pyarrow-23.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:bf5842f960cddd2ef757d486041d57c96483efc295a8c4a0e20e704cbbf39c67", size = 48173045 },
+ { url = "https://files.pythonhosted.org/packages/43/b2/b40961262213beaba6acfc88698eb773dfce32ecdf34d19291db94c2bd73/pyarrow-23.0.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:564baf97c858ecc03ec01a41062e8f4698abc3e6e2acd79c01c2e97880a19730", size = 50621741 },
+ { url = "https://files.pythonhosted.org/packages/f6/70/1fdda42d65b28b078e93d75d371b2185a61da89dda4def8ba6ba41ebdeb4/pyarrow-23.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:07deae7783782ac7250989a7b2ecde9b3c343a643f82e8a4df03d93b633006f0", size = 27620678 },
+]
+
+[[package]]
+name = "pyparsing"
+version = "3.3.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e4/11/b213bebff182584360cb8d17c72c1677fec5c5c228de439e63bcf8ab1c8f/pyparsing-3.3.3.tar.gz", hash = "sha256:928ae7e20211f3b6f3915a72f06a0cfd29ab9d24279dd6346b6b1a7146397d36", size = 1050487 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/38/bb/d215ee7c73b61497b28a5503f9f53523f294fcc936762b7caf90e0c1c2b5/pyparsing-3.3.3-py3-none-any.whl", hash = "sha256:ece8c00a69cf01b45d0b1dedabb469c90d8caf996d4fda40f147627a122849a4", size = 126420 },
+]
+
+[[package]]
+name = "python-dateutil"
+version = "2.9.0.post0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "six" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892 },
+]
+
+[[package]]
+name = "pyyaml"
+version = "6.0.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d1/33/422b98d2195232ca1826284a76852ad5a86fe23e31b009c9886b2d0fb8b2/pyyaml-6.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196", size = 182063 },
+ { url = "https://files.pythonhosted.org/packages/89/a0/6cf41a19a1f2f3feab0e9c0b74134aa2ce6849093d5517a0c550fe37a648/pyyaml-6.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0", size = 173973 },
+ { url = "https://files.pythonhosted.org/packages/ed/23/7a778b6bd0b9a8039df8b1b1d80e2e2ad78aa04171592c8a5c43a56a6af4/pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28", size = 775116 },
+ { url = "https://files.pythonhosted.org/packages/65/30/d7353c338e12baef4ecc1b09e877c1970bd3382789c159b4f89d6a70dc09/pyyaml-6.0.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c", size = 844011 },
+ { url = "https://files.pythonhosted.org/packages/8b/9d/b3589d3877982d4f2329302ef98a8026e7f4443c765c46cfecc8858c6b4b/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc", size = 807870 },
+ { url = "https://files.pythonhosted.org/packages/05/c0/b3be26a015601b822b97d9149ff8cb5ead58c66f981e04fedf4e762f4bd4/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e", size = 761089 },
+ { url = "https://files.pythonhosted.org/packages/be/8e/98435a21d1d4b46590d5459a22d88128103f8da4c2d4cb8f14f2a96504e1/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea", size = 790181 },
+ { url = "https://files.pythonhosted.org/packages/74/93/7baea19427dcfbe1e5a372d81473250b379f04b1bd3c4c5ff825e2327202/pyyaml-6.0.3-cp312-cp312-win32.whl", hash = "sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5", size = 137658 },
+ { url = "https://files.pythonhosted.org/packages/86/bf/899e81e4cce32febab4fb42bb97dcdf66bc135272882d1987881a4b519e9/pyyaml-6.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b", size = 154003 },
+ { url = "https://files.pythonhosted.org/packages/1a/08/67bd04656199bbb51dbed1439b7f27601dfb576fb864099c7ef0c3e55531/pyyaml-6.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd", size = 140344 },
+]
+
+[[package]]
+name = "regex"
+version = "2026.9.29"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/fc/f2/af1da9d3ceed77bfcdce40427d49ba0be94e4fe84245e3bfef68c10e75b6/regex-2026.9.29.tar.gz", hash = "sha256:8b5fcc4771732191b2b7d1dd68d8f0353f47f8d90b6150f6dce58bf1112442cb", size = 419199 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/84/48/3fdcde9a0baa84d7d25571223265d6e434e114763b438601d54a8028bf3e/regex-2026.9.29-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:dc79d36d0618752265f0d575915bdc5c5130ecb9c9f6b3bcefeae32e4bdfafcf", size = 497903 },
+ { url = "https://files.pythonhosted.org/packages/2e/1c/4ee3e97c76f53940488dfe7a7e18705e78daac8cd7fb161d246b9e328449/regex-2026.9.29-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3a21a9509d0ee88e7a70e1ad228cd2f0e0fd1e187458db132e8a8d18c97daf9d", size = 296416 },
+ { url = "https://files.pythonhosted.org/packages/37/14/f3f0ba083d2094392d5eabf56db5ea6ba469fd6e927afd187042054ea68a/regex-2026.9.29-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f57dc6b8fef170f105d2cf5cdce254f47b137d7755086cf7050f47e16582abba", size = 293633 },
+ { url = "https://files.pythonhosted.org/packages/c9/72/67e7a8ce17f1aea49df215564048efb49cc8c2b31a0e0fc30f36838f8516/regex-2026.9.29-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f93bc1c3486ef3747e07c9d7c1d0a147b8fbaab975f80e348aed6f71309dfaca", size = 805885 },
+ { url = "https://files.pythonhosted.org/packages/f6/78/25436bcfd4d2260b4b4090094d55d7ab53ec8a1ab4865a0b8bcb33c7d5c0/regex-2026.9.29-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9e1d3a4cb7993b708f0ada8d0c84590efd853f169e7147d2202c9da503180242", size = 878344 },
+ { url = "https://files.pythonhosted.org/packages/97/e6/a09ec3a23ae41d6179880e67f0aace9284b2d95f2d7b326eff203f8eec5e/regex-2026.9.29-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:dabee8f4935e731fb46b2a3091bdda0d3d94b3bbfb907d2b4f12eefce4009619", size = 919181 },
+ { url = "https://files.pythonhosted.org/packages/26/83/d2fbd2e4e3afb1167daa825187d196f313cbaa1a4768f311fb041bb0e3d2/regex-2026.9.29-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:39ab5894d971f9ac68baa6eca5c50387db579cfcacf36ae8df3feceb1815e6d0", size = 807783 },
+ { url = "https://files.pythonhosted.org/packages/46/0b/eb429a7016610d44fc89a597163f8c9127505f0d7dc724dc9effbb6a3ac0/regex-2026.9.29-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c1a9a6651197fbed6f0212591418b9def774fc3f8324f78d1bf0e6a63e5f8aa1", size = 783465 },
+ { url = "https://files.pythonhosted.org/packages/1b/07/58a3c0153c7476898430f6a7cf3d9062a1d17fbea4f43399ecaf411c7b4c/regex-2026.9.29-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:87fb80cbe3557e27e7b28b995c2b2eedf689b8886f941ab93e0e288f0976518a", size = 793519 },
+ { url = "https://files.pythonhosted.org/packages/2a/e8/161b94d39164520e21a7befe0245569bf7fda4c7cf1fc4e2df2b5def49da/regex-2026.9.29-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:3c5c2ef13797466aa64170cbb66ad98a32351dd4127694cea7199f80f213750d", size = 869293 },
+ { url = "https://files.pythonhosted.org/packages/8f/07/3b02ed829aa2decdc1955d222bd1e2f99d1c8bb4873bbb9a66b2f0a36bff/regex-2026.9.29-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:59b49507f47479e299a9e1bc41b5cb83a7afda0540625f1dbae886615978acbf", size = 770239 },
+ { url = "https://files.pythonhosted.org/packages/42/5b/ba61f6fe062eb8562e742367d177bb75370434138ef6c9d2a27114f8d613/regex-2026.9.29-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:0dd8af32e9f7b56b7f95cc1fd79b23054c3bdc172392ae560acc24d57b7ffe71", size = 861973 },
+ { url = "https://files.pythonhosted.org/packages/cc/27/767259b20e8a842948990f5e99138d6c077248fd42f8b5468b1d9ca4b814/regex-2026.9.29-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:db5e82ba15c142425b8406690032df89e39cca4a2e8afbbb9a3d84edc2373ac3", size = 796470 },
+ { url = "https://files.pythonhosted.org/packages/a0/05/2566c4ba849b68a8ab81a6bf428fa79d20aae7ddee83979103c0381df254/regex-2026.9.29-cp312-cp312-win32.whl", hash = "sha256:d0c3082bf79bcd6a614d55916590ad4b8f93200e10b97f463ea5d9d07c9b5f23", size = 269327 },
+ { url = "https://files.pythonhosted.org/packages/93/19/489bc8db91196381c935752df01ba3f607140daece33b78d88573f028e64/regex-2026.9.29-cp312-cp312-win_amd64.whl", hash = "sha256:fdd88ed5e20b1bcdd234421e454962c971aa44b653bdb7f1ea9ef683e90fb649", size = 280334 },
+ { url = "https://files.pythonhosted.org/packages/0b/47/fb88ba779d0e5e7d4b0ec1aceeb13845948a2cb876bd572a2d1dfdba090b/regex-2026.9.29-cp312-cp312-win_arm64.whl", hash = "sha256:4fe97894d1b306c919b4e50def1e6f6c522f4d03a7283811f4d108f1ce5d3ac2", size = 279614 },
+]
+
+[[package]]
+name = "requests"
+version = "2.34.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "certifi" },
+ { name = "charset-normalizer" },
+ { name = "idna" },
+ { name = "urllib3" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ac/c3/e2a2b89f2d3e2179abd6d00ebd70bff6273f37fb3e0cc209f48b39d00cbf/requests-2.34.2.tar.gz", hash = "sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed", size = 142856 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075 },
+]
+
+[[package]]
+name = "ruff"
+version = "0.17.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f9/e7/e6d5749983b020e6cb1055abec0a6f361a8e94f0184cdcf56d4760995f02/ruff-0.17.1.tar.gz", hash = "sha256:5bb796c5112e9fb9527f2ef2faa0a130d6312d19de92bcfaddd70dbb2d41020f", size = 5323349 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2e/78/90c46ca6250709f7c37e397b60d06061adee8d0f7d13959ed6a8e49b14fd/ruff-0.17.1-py3-none-linux_armv6l.whl", hash = "sha256:1796c41eafec66f4635125dd1e074a7aa27b8d20c187d484360acf11d247fc4a", size = 10147528 },
+ { url = "https://files.pythonhosted.org/packages/aa/d2/a2e28bc35b083ae51fa07dfba5280a667c8b01ecaee7802fc8849dcd5c46/ruff-0.17.1-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:8eadb09a788010be29c895418bdfa513e4e1c65e64b94e5b19e4cd381f6cc7fb", size = 10225974 },
+ { url = "https://files.pythonhosted.org/packages/8e/71/7a9029a81c8206ee01a98f6cfeeb56e6c095af86857581a37045c46d7e94/ruff-0.17.1-py3-none-macosx_11_0_arm64.whl", hash = "sha256:29b454f18ca3eeaf37fbf140841dcf27d36109e29dee9aae8acecb68c26c6f74", size = 9270361 },
+ { url = "https://files.pythonhosted.org/packages/c0/8c/e075e9c917cb0575297880c96768304e881240171c870cbd8d9b7b54bec2/ruff-0.17.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:872e03dd231744d6c5ec5c8af6be52b71323fc6ee096e4c8b34d9f844ee3b978", size = 9938300 },
+ { url = "https://files.pythonhosted.org/packages/87/27/0b4bb926a49de2383f161272b4d863571a4c7ecdbf595a6bc58813a13367/ruff-0.17.1-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4c0d6cb5f8fbac77eec9af49f4fc0eaf856b05545e372b0694d24b4e76b1c640", size = 9948464 },
+ { url = "https://files.pythonhosted.org/packages/14/27/518d57e0e02aecb8ea4acac6b57b51a53173d25fa734952bfdd7d531e88e/ruff-0.17.1-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:78beced4a3c94dfedbee4b999b95b6a27b35820fdc0afbacc6d37a8da26c1cd1", size = 10638639 },
+ { url = "https://files.pythonhosted.org/packages/01/82/d89a4e498ade59e6cee52d67f8da569019372212b87468c1fdeae884601c/ruff-0.17.1-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7feefc63921202894f0e158b0e32f45a5950d70a7fb693ccb9264816c25b1c05", size = 11337238 },
+ { url = "https://files.pythonhosted.org/packages/a2/7c/3b0fdef9352656317c977086fe1d802902406e4f2a0d8578da90c0e6d1c5/ruff-0.17.1-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:684154861a819bf1f433669d2a0ce9c3e54677c65b8ebb866cc2a7ec86345785", size = 10896327 },
+ { url = "https://files.pythonhosted.org/packages/62/86/5566290219708ce2928f3a29c4f843a1185b341ec6111c4631a33c5ceced/ruff-0.17.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a6ef98e51de8ef38082f90653ceeb84944bd7673783eea13e1647c5b9b064189", size = 10350811 },
+ { url = "https://files.pythonhosted.org/packages/ad/d2/ca8462dabb3c413990d49703ba306c30c862195a75448d080fa8fcce9775/ruff-0.17.1-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:8f9aa5b5bd5fa1937803ea609c8f153b8de39bf3e0e149897ae34b0c5b52f467", size = 10569994 },
+ { url = "https://files.pythonhosted.org/packages/e3/ad/dd1864c1f1db17c8cfb297ce70b352388e6977620f66f7182c52855407df/ruff-0.17.1-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:80e53b0e5faa3c5667f035daaa032893ae42b7cfe708c886d39ed131e54688ab", size = 10059672 },
+ { url = "https://files.pythonhosted.org/packages/65/45/3f56cf1d2168c2165cdd5ac6f9668aee66a4cfdd4caf2b2cc5ccb2b1d568/ruff-0.17.1-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:056c5d5fa0ac790051ae165d7035c7bf956d02de2db557f2e9b31d877a505fca", size = 9947803 },
+ { url = "https://files.pythonhosted.org/packages/aa/4b/5457cb2be4c1d1ea5e919558131a0f197a4a454a0530d9e30820e8206e4c/ruff-0.17.1-py3-none-musllinux_1_2_i686.whl", hash = "sha256:bd8f5a55b10f371642eacd0e8665a42a8b97da15c9a7187c79e697b77b5e0ce4", size = 10351807 },
+ { url = "https://files.pythonhosted.org/packages/49/fb/db15b4f8fa24a1e36b99e1401f42ef6df16ee299a48670d5d262b771e7d7/ruff-0.17.1-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:ce05e6f3973611d9550a600e04bb531061e368baba31610eb57c1509a8f21a0a", size = 10765716 },
+ { url = "https://files.pythonhosted.org/packages/0a/e8/1e186578076aff7a80735f625005fde678412ab609cd323ed182eb7d6b87/ruff-0.17.1-py3-none-win32.whl", hash = "sha256:1de0030a8aa78c65c5fecb84b2e5c390ac684c0452875a02afc7c083d49a496a", size = 9895093 },
+ { url = "https://files.pythonhosted.org/packages/6a/2e/da3d9589bae8e8b58f3a48bd5bc0b980232fb14a83bf02c3be7660f87c1b/ruff-0.17.1-py3-none-win_amd64.whl", hash = "sha256:5df76f16580fab58d0f6c98a3ac254a4abceb7b2489b8b678335869cff77b3d7", size = 10097955 },
+ { url = "https://files.pythonhosted.org/packages/df/c9/0f2a4dea06a5e12c1191ee3d96b1e1233797d0d5e18f97a2b909a95a9836/ruff-0.17.1-py3-none-win_arm64.whl", hash = "sha256:42400b9a6ff8515ff27796f07f92e0195be011c86e81e381606f58ea5502fef4", size = 9853542 },
+]
+
+[[package]]
+name = "safetensors"
+version = "0.8.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/45/06/f955dbbb1859e3bd23c8ac6141af5106e7ad5fedec4a3a6e3d60f94b7001/safetensors-0.8.0.tar.gz", hash = "sha256:fabaf3e0f18a6618d9b36560682562157f77c2b71fcffc7b432be2baed9d753d", size = 325846 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/39/a0/f718cda65b05407d228f97602cf60dca269c979867aa5beb25410de26cd3/safetensors-0.8.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:c554f85858e05226d3c2828e32395e677434685d6d94594a41643361c5e837f0", size = 473568 },
+ { url = "https://files.pythonhosted.org/packages/f5/b1/fa7c600e7dceae12e9606c7578cbc9ff1e1ed55844883ee5c92205e86226/safetensors-0.8.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:c80201d22cbf405b80647a60ada77bba06c8fba2da2743ba1e89cdcc39a81f25", size = 484562 },
+ { url = "https://files.pythonhosted.org/packages/09/7d/65a7de0af421317bb36a067241e4235fff194eed60b961ed6d3f59a3fc60/safetensors-0.8.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7a46e5ff292c356d6991e60942ba7f79817682d3a2cef0702136448cb9c4d235", size = 502844 },
+ { url = "https://files.pythonhosted.org/packages/91/4f/3175c9d75634e0e0dda0082794193521035edd7c70a6f212bf33ca06ddf4/safetensors-0.8.0-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4124502b78f03534117c848f87a39b8f31e577b15eff423bf8bfb95f2a8c30d0", size = 511823 },
+ { url = "https://files.pythonhosted.org/packages/20/87/846c289e7aa2299eff406335717cf43ce8777194ece8aad75772e0411615/safetensors-0.8.0-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7bc0a787ba8a35be368ee3574edfa2b1ad389eebd0a72e482ae275490e3f6c98", size = 633461 },
+ { url = "https://files.pythonhosted.org/packages/76/22/8d64d9df2c45d5ded401df889d0ad90882804ca172d79ec4f0df8f727fe0/safetensors-0.8.0-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:040070828e36dc8e122178bbbd5830ff9e97920affb84cbe0f46442497bed358", size = 545148 },
+ { url = "https://files.pythonhosted.org/packages/28/50/f203ff3a3ddfe19308efc83c5a3a29ed02bf786732ec35e68bf9162f3365/safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fd6f3f93c9a0a7cc2788ee63fb763353d4bd2e89b0751bc78fcf7dda00bea774", size = 516040 },
+ { url = "https://files.pythonhosted.org/packages/46/fb/cdaed17ceb2948784fd9c36b6fd3e951b608547cea81a48e8ee6f8cfdfcb/safetensors-0.8.0-cp310-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:fcdd41ec4628fee5799f807c73c353629130fbd942aa23d83c623dd6c9d52d78", size = 513832 },
+ { url = "https://files.pythonhosted.org/packages/0d/49/1e15de264dcc3b77943d2d0c56a95809956883b1c2d6d585c792523f180b/safetensors-0.8.0-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:8e9f537aa183a38ace122d27303dcd986b26bd2a7591f9181d7f0c396f4677ca", size = 559930 },
+ { url = "https://files.pythonhosted.org/packages/2a/43/bf38443278eab4b1be1fce2931e2b012ad9cb7df52ada751d0aab8f7659a/safetensors-0.8.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:87eec7ffed2b809f05a398a8becb7d013f19f7837cd15d9748580d6cf30dbaf4", size = 678670 },
+ { url = "https://files.pythonhosted.org/packages/72/e3/68cd3fa5b48488e84add63e04cb12f3bc28ae4638c06d4508c6e88823d0e/safetensors-0.8.0-cp310-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:4a95ae2b05d7726d751da4ebf626a2ca782b706e101bd894c95bc2450b1cffcc", size = 786679 },
+ { url = "https://files.pythonhosted.org/packages/29/4b/1c19c509d56e01f4fbb3d0a2e597450f6cc04d1d56cf52defb0a62dfd715/safetensors-0.8.0-cp310-abi3-musllinux_1_2_i686.whl", hash = "sha256:3ae091f16662658bdc019a4ff6cb4c085bb7d725eb5978b183ffd265863b6d2d", size = 765683 },
+ { url = "https://files.pythonhosted.org/packages/27/43/41c1621732edd934d868a00d1b891584c892a7b62a9aab82ea5a0a5623ee/safetensors-0.8.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8e080062fcde23be189565e1c3305d16751a218ecf9412c8601e64204eb6f846", size = 722361 },
+ { url = "https://files.pythonhosted.org/packages/8e/3f/73ccf82579412b4a71c4ca673f10b5f1f888d7cf5af7fe24f27d30307be4/safetensors-0.8.0-cp310-abi3-win32.whl", hash = "sha256:2ddf52eac562eda224f99acfa7889d02968c1fd59a5b011ae7d8137c37e9c02d", size = 342401 },
+ { url = "https://files.pythonhosted.org/packages/1b/6d/3fba214c1e5e0f69991677ec3bc17023f0421776975e1de0c682dca475e2/safetensors-0.8.0-cp310-abi3-win_amd64.whl", hash = "sha256:096ec1a98435df7beb08853bb5aa9081a84f23d0adc67ed1a0a10550f608373f", size = 355540 },
+ { url = "https://files.pythonhosted.org/packages/8d/fc/7eedc3510d97878876e32774eebbeb61c43f148a96e915c84229a3e967aa/safetensors-0.8.0-cp310-abi3-win_arm64.whl", hash = "sha256:f7838e5135a406ad3e02efdcb8cf2e5397d368b0154537c4fec682dbc544d452", size = 340500 },
+]
+
+[[package]]
+name = "scipy"
+version = "1.17.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/56/3e/9cca699f3486ce6bc12ff46dc2031f1ec8eb9ccc9a320fdaf925f1417426/scipy-1.17.0.tar.gz", hash = "sha256:2591060c8e648d8b96439e111ac41fd8342fdeff1876be2e19dea3fe8930454e", size = 30396830 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0b/11/7241a63e73ba5a516f1930ac8d5b44cbbfabd35ac73a2d08ca206df007c4/scipy-1.17.0-cp312-cp312-macosx_10_14_x86_64.whl", hash = "sha256:0d5018a57c24cb1dd828bcf51d7b10e65986d549f52ef5adb6b4d1ded3e32a57", size = 31364580 },
+ { url = "https://files.pythonhosted.org/packages/ed/1d/5057f812d4f6adc91a20a2d6f2ebcdb517fdbc87ae3acc5633c9b97c8ba5/scipy-1.17.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:88c22af9e5d5a4f9e027e26772cc7b5922fab8bcc839edb3ae33de404feebd9e", size = 27969012 },
+ { url = "https://files.pythonhosted.org/packages/e3/21/f6ec556c1e3b6ec4e088da667d9987bb77cc3ab3026511f427dc8451187d/scipy-1.17.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:f3cd947f20fe17013d401b64e857c6b2da83cae567adbb75b9dcba865abc66d8", size = 20140691 },
+ { url = "https://files.pythonhosted.org/packages/7a/fe/5e5ad04784964ba964a96f16c8d4676aa1b51357199014dce58ab7ec5670/scipy-1.17.0-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:e8c0b331c2c1f531eb51f1b4fc9ba709521a712cce58f1aa627bc007421a5306", size = 22463015 },
+ { url = "https://files.pythonhosted.org/packages/4a/69/7c347e857224fcaf32a34a05183b9d8a7aca25f8f2d10b8a698b8388561a/scipy-1.17.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5194c445d0a1c7a6c1a4a4681b6b7c71baad98ff66d96b949097e7513c9d6742", size = 32724197 },
+ { url = "https://files.pythonhosted.org/packages/d1/fe/66d73b76d378ba8cc2fe605920c0c75092e3a65ae746e1e767d9d020a75a/scipy-1.17.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9eeb9b5f5997f75507814ed9d298ab23f62cf79f5a3ef90031b1ee2506abdb5b", size = 35009148 },
+ { url = "https://files.pythonhosted.org/packages/af/07/07dec27d9dc41c18d8c43c69e9e413431d20c53a0339c388bcf72f353c4b/scipy-1.17.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:40052543f7bbe921df4408f46003d6f01c6af109b9e2c8a66dd1cf6cf57f7d5d", size = 34798766 },
+ { url = "https://files.pythonhosted.org/packages/81/61/0470810c8a093cdacd4ba7504b8a218fd49ca070d79eca23a615f5d9a0b0/scipy-1.17.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:0cf46c8013fec9d3694dc572f0b54100c28405d55d3e2cb15e2895b25057996e", size = 37405953 },
+ { url = "https://files.pythonhosted.org/packages/92/ce/672ed546f96d5d41ae78c4b9b02006cedd0b3d6f2bf5bb76ea455c320c28/scipy-1.17.0-cp312-cp312-win_amd64.whl", hash = "sha256:0937a0b0d8d593a198cededd4c439a0ea216a3f36653901ea1f3e4be949056f8", size = 36328121 },
+ { url = "https://files.pythonhosted.org/packages/9d/21/38165845392cae67b61843a52c6455d47d0cc2a40dd495c89f4362944654/scipy-1.17.0-cp312-cp312-win_arm64.whl", hash = "sha256:f603d8a5518c7426414d1d8f82e253e454471de682ce5e39c29adb0df1efb86b", size = 24314368 },
+]
+
+[[package]]
+name = "setuptools"
+version = "84.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/6d/44/f5da03a8ef95d369145c5bb53050e7877c9f3d312e128605fd9504829143/setuptools-84.0.0.tar.gz", hash = "sha256:f4695c21257f0d9b537ec2692c941d02ee143b7cc1276941349a546573b2ef73", size = 1168449 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/95/9c/c510029fc6ef33a6275cd2c5d3cecd6613dfd6aa401d57c54f1c18852ccf/setuptools-84.0.0-py3-none-any.whl", hash = "sha256:51a52592b3b99e102b609654876bd65f19f999935166d1352678931132b0c670", size = 818216 },
+]
+
+[[package]]
+name = "sie-sdk"
+version = "0.10.0"
+source = { git = "https://github.com/superlinked/sie?subdirectory=packages%2Fsie_sdk&rev=d680237f02c2b67fd9afd91f30b73a29cb18fae0#d680237f02c2b67fd9afd91f30b73a29cb18fae0" }
+dependencies = [
+ { name = "aiohttp" },
+ { name = "httpx" },
+ { name = "msgpack" },
+ { name = "msgpack-numpy" },
+ { name = "numpy" },
+ { name = "pillow" },
+ { name = "pyyaml" },
+ { name = "websockets" },
+]
+
+[[package]]
+name = "sie-train-a-router-example"
+version = "0.1.0"
+source = { virtual = "." }
+dependencies = [
+ { name = "httpx" },
+ { name = "numpy" },
+ { name = "sie-sdk" },
+]
+
+[package.dev-dependencies]
+data = [
+ { name = "pyarrow" },
+]
+dev = [
+ { name = "ruff" },
+]
+evaluation = [
+ { name = "math-verify", extra = ["antlr4-13-2"] },
+ { name = "pyarrow" },
+]
+outcomes = [
+ { name = "scipy" },
+]
+plots = [
+ { name = "matplotlib" },
+]
+training = [
+ { name = "laya" },
+ { name = "peft" },
+ { name = "torch" },
+ { name = "transformers" },
+]
+
+[package.metadata]
+requires-dist = [
+ { name = "httpx", specifier = ">=0.28,<1" },
+ { name = "numpy", specifier = "==2.4.6" },
+ { name = "sie-sdk", git = "https://github.com/superlinked/sie?subdirectory=packages%2Fsie_sdk&rev=d680237f02c2b67fd9afd91f30b73a29cb18fae0" },
+]
+
+[package.metadata.requires-dev]
+data = [{ name = "pyarrow", specifier = ">=18,<24" }]
+dev = [{ name = "ruff", specifier = ">=0.14" }]
+evaluation = [
+ { name = "math-verify", extras = ["antlr4-13-2"], specifier = "==0.9.0" },
+ { name = "pyarrow", specifier = ">=18,<24" },
+]
+outcomes = [{ name = "scipy", specifier = "==1.17.0" }]
+plots = [{ name = "matplotlib", specifier = ">=3.10,<4" }]
+training = [
+ { name = "laya", specifier = "==0.3.29" },
+ { name = "peft", specifier = "==0.18.1" },
+ { name = "torch", specifier = "==2.9.1" },
+ { name = "transformers", specifier = "==4.57.6" },
+]
+
+[[package]]
+name = "six"
+version = "1.17.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050 },
+]
+
+[[package]]
+name = "sympy"
+version = "1.14.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "mpmath" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/83/d3/803453b36afefb7c2bb238361cd4ae6125a569b4db67cd9e79846ba2d68c/sympy-1.14.0.tar.gz", hash = "sha256:d3d3fe8df1e5a0b42f0e7bdf50541697dbe7d23746e894990c030e2b05e72517", size = 7793921 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", size = 6299353 },
+]
+
+[[package]]
+name = "tokenizers"
+version = "0.22.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "huggingface-hub" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/73/6f/f80cfef4a312e1fb34baf7d85c72d4411afde10978d4657f8cdd811d3ccc/tokenizers-0.22.2.tar.gz", hash = "sha256:473b83b915e547aa366d1eee11806deaf419e17be16310ac0a14077f1e28f917", size = 372115 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/92/97/5dbfabf04c7e348e655e907ed27913e03db0923abb5dfdd120d7b25630e1/tokenizers-0.22.2-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:544dd704ae7238755d790de45ba8da072e9af3eea688f698b137915ae959281c", size = 3100275 },
+ { url = "https://files.pythonhosted.org/packages/2e/47/174dca0502ef88b28f1c9e06b73ce33500eedfac7a7692108aec220464e7/tokenizers-0.22.2-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:1e418a55456beedca4621dbab65a318981467a2b188e982a23e117f115ce5001", size = 2981472 },
+ { url = "https://files.pythonhosted.org/packages/d6/84/7990e799f1309a8b87af6b948f31edaa12a3ed22d11b352eaf4f4b2e5753/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2249487018adec45d6e3554c71d46eb39fa8ea67156c640f7513eb26f318cec7", size = 3290736 },
+ { url = "https://files.pythonhosted.org/packages/78/59/09d0d9ba94dcd5f4f1368d4858d24546b4bdc0231c2354aa31d6199f0399/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:25b85325d0815e86e0bac263506dd114578953b7b53d7de09a6485e4a160a7dd", size = 3168835 },
+ { url = "https://files.pythonhosted.org/packages/47/50/b3ebb4243e7160bda8d34b731e54dd8ab8b133e50775872e7a434e524c28/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:bfb88f22a209ff7b40a576d5324bf8286b519d7358663db21d6246fb17eea2d5", size = 3521673 },
+ { url = "https://files.pythonhosted.org/packages/e0/fa/89f4cb9e08df770b57adb96f8cbb7e22695a4cb6c2bd5f0c4f0ebcf33b66/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1c774b1276f71e1ef716e5486f21e76333464f47bece56bbd554485982a9e03e", size = 3724818 },
+ { url = "https://files.pythonhosted.org/packages/64/04/ca2363f0bfbe3b3d36e95bf67e56a4c88c8e3362b658e616d1ac185d47f2/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:df6c4265b289083bf710dff49bc51ef252f9d5be33a45ee2bed151114a56207b", size = 3379195 },
+ { url = "https://files.pythonhosted.org/packages/2e/76/932be4b50ef6ccedf9d3c6639b056a967a86258c6d9200643f01269211ca/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:369cc9fc8cc10cb24143873a0d95438bb8ee257bb80c71989e3ee290e8d72c67", size = 3274982 },
+ { url = "https://files.pythonhosted.org/packages/1d/28/5f9f5a4cc211b69e89420980e483831bcc29dade307955cc9dc858a40f01/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:29c30b83d8dcd061078b05ae0cb94d3c710555fbb44861139f9f83dcca3dc3e4", size = 9478245 },
+ { url = "https://files.pythonhosted.org/packages/6c/fb/66e2da4704d6aadebf8cb39f1d6d1957df667ab24cff2326b77cda0dcb85/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:37ae80a28c1d3265bb1f22464c856bd23c02a05bb211e56d0c5301a435be6c1a", size = 9560069 },
+ { url = "https://files.pythonhosted.org/packages/16/04/fed398b05caa87ce9b1a1bb5166645e38196081b225059a6edaff6440fac/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:791135ee325f2336f498590eb2f11dc5c295232f288e75c99a36c5dbce63088a", size = 9899263 },
+ { url = "https://files.pythonhosted.org/packages/05/a1/d62dfe7376beaaf1394917e0f8e93ee5f67fea8fcf4107501db35996586b/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:38337540fbbddff8e999d59970f3c6f35a82de10053206a7562f1ea02d046fa5", size = 10033429 },
+ { url = "https://files.pythonhosted.org/packages/fd/18/a545c4ea42af3df6effd7d13d250ba77a0a86fb20393143bbb9a92e434d4/tokenizers-0.22.2-cp39-abi3-win32.whl", hash = "sha256:a6bf3f88c554a2b653af81f3204491c818ae2ac6fbc09e76ef4773351292bc92", size = 2502363 },
+ { url = "https://files.pythonhosted.org/packages/65/71/0670843133a43d43070abeb1949abfdef12a86d490bea9cd9e18e37c5ff7/tokenizers-0.22.2-cp39-abi3-win_amd64.whl", hash = "sha256:c9ea31edff2968b44a88f97d784c2f16dc0729b8b143ed004699ebca91f05c48", size = 2747786 },
+ { url = "https://files.pythonhosted.org/packages/72/f4/0de46cfa12cdcbcd464cc59fde36912af405696f687e53a091fb432f694c/tokenizers-0.22.2-cp39-abi3-win_arm64.whl", hash = "sha256:9ce725d22864a1e965217204946f830c37876eee3b2ba6fc6255e8e903d5fcbc", size = 2612133 },
+]
+
+[[package]]
+name = "torch"
+version = "2.9.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "filelock" },
+ { name = "fsspec" },
+ { name = "jinja2" },
+ { name = "networkx" },
+ { name = "nvidia-cublas-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-cuda-cupti-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-cuda-nvrtc-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-cuda-runtime-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-cudnn-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-cufft-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-cufile-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-curand-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-cusolver-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-cusparse-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-cusparselt-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-nccl-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-nvjitlink-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-nvshmem-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "nvidia-nvtx-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "setuptools" },
+ { name = "sympy" },
+ { name = "triton", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" },
+ { name = "typing-extensions" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0f/27/07c645c7673e73e53ded71705045d6cb5bae94c4b021b03aa8d03eee90ab/torch-2.9.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:da5f6f4d7f4940a173e5572791af238cb0b9e21b1aab592bd8b26da4c99f1cd6", size = 104126592 },
+ { url = "https://files.pythonhosted.org/packages/19/17/e377a460603132b00760511299fceba4102bd95db1a0ee788da21298ccff/torch-2.9.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:27331cd902fb4322252657f3902adf1c4f6acad9dcad81d8df3ae14c7c4f07c4", size = 899742281 },
+ { url = "https://files.pythonhosted.org/packages/b1/1a/64f5769025db846a82567fa5b7d21dba4558a7234ee631712ee4771c436c/torch-2.9.1-cp312-cp312-win_amd64.whl", hash = "sha256:81a285002d7b8cfd3fdf1b98aa8df138d41f1a8334fd9ea37511517cedf43083", size = 110940568 },
+ { url = "https://files.pythonhosted.org/packages/6e/ab/07739fd776618e5882661d04c43f5b5586323e2f6a2d7d84aac20d8f20bd/torch-2.9.1-cp312-none-macosx_11_0_arm64.whl", hash = "sha256:c0d25d1d8e531b8343bea0ed811d5d528958f1dcbd37e7245bc686273177ad7e", size = 74479191 },
+]
+
+[[package]]
+name = "tqdm"
+version = "4.70.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/0d/ea/b2a5bd54b28a324dae8211928b2d730b6547500342c7e6c6dea08bd0a485/tqdm-4.70.1.tar.gz", hash = "sha256:cefd0eca11b2a37a3aee776544d4f4ae913f02688135b5556b8788dfa474afc4", size = 171846 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a7/03/921a3d3c75785aca9ebfbfcabfbc3a1be12e2ab5265deb026d55a5a3f83e/tqdm-4.70.1-py3-none-any.whl", hash = "sha256:c293e525e6fef9c20e8728fd4612df02a0aa31bb5fe91ecd93e123b1b7bffa73", size = 80199 },
+]
+
+[[package]]
+name = "transformers"
+version = "4.57.6"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "filelock" },
+ { name = "huggingface-hub" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "pyyaml" },
+ { name = "regex" },
+ { name = "requests" },
+ { name = "safetensors" },
+ { name = "tokenizers" },
+ { name = "tqdm" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/c4/35/67252acc1b929dc88b6602e8c4a982e64f31e733b804c14bc24b47da35e6/transformers-4.57.6.tar.gz", hash = "sha256:55e44126ece9dc0a291521b7e5492b572e6ef2766338a610b9ab5afbb70689d3", size = 10134912 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/03/b8/e484ef633af3887baeeb4b6ad12743363af7cce68ae51e938e00aaa0529d/transformers-4.57.6-py3-none-any.whl", hash = "sha256:4c9e9de11333ddfe5114bc872c9f370509198acf0b87a832a0ab9458e2bd0550", size = 11993498 },
+]
+
+[[package]]
+name = "triton"
+version = "3.5.1"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f2/50/9a8358d3ef58162c0a415d173cfb45b67de60176e1024f71fbc4d24c0b6d/triton-3.5.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d2c6b915a03888ab931a9fd3e55ba36785e1fe70cbea0b40c6ef93b20fc85232", size = 170470207 },
+]
+
+[[package]]
+name = "typing-extensions"
+version = "4.16.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571 },
+]
+
+[[package]]
+name = "urllib3"
+version = "2.8.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e3/05/b17359e1cefb4f909b5e40b1b90a496d987258916dbbf88e842c729f510e/urllib3-2.8.0.tar.gz", hash = "sha256:63bf2ead4c879426ebf22ef2a781eeb4aa3b4ae798a0435506f8687fd5bb9b63", size = 458972 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/92/9d/c4e665119135114480843e7ab388fa94d8480650450e6f8e26b70d323a4c/urllib3-2.8.0-py3-none-any.whl", hash = "sha256:0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3", size = 135717 },
+]
+
+[[package]]
+name = "websockets"
+version = "14.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/94/54/8359678c726243d19fae38ca14a334e740782336c9f19700858c4eb64a1e/websockets-14.2.tar.gz", hash = "sha256:5059ed9c54945efb321f097084b4c7e52c246f2c869815876a69d1efc4ad6eb5", size = 164394 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c1/81/04f7a397653dc8bec94ddc071f34833e8b99b13ef1a3804c149d59f92c18/websockets-14.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:1f20522e624d7ffbdbe259c6b6a65d73c895045f76a93719aa10cd93b3de100c", size = 163096 },
+ { url = "https://files.pythonhosted.org/packages/ec/c5/de30e88557e4d70988ed4d2eabd73fd3e1e52456b9f3a4e9564d86353b6d/websockets-14.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:647b573f7d3ada919fd60e64d533409a79dcf1ea21daeb4542d1d996519ca967", size = 160758 },
+ { url = "https://files.pythonhosted.org/packages/e5/8c/d130d668781f2c77d106c007b6c6c1d9db68239107c41ba109f09e6c218a/websockets-14.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6af99a38e49f66be5a64b1e890208ad026cda49355661549c507152113049990", size = 160995 },
+ { url = "https://files.pythonhosted.org/packages/a6/bc/f6678a0ff17246df4f06765e22fc9d98d1b11a258cc50c5968b33d6742a1/websockets-14.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:091ab63dfc8cea748cc22c1db2814eadb77ccbf82829bac6b2fbe3401d548eda", size = 170815 },
+ { url = "https://files.pythonhosted.org/packages/d8/b2/8070cb970c2e4122a6ef38bc5b203415fd46460e025652e1ee3f2f43a9a3/websockets-14.2-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b374e8953ad477d17e4851cdc66d83fdc2db88d9e73abf755c94510ebddceb95", size = 169759 },
+ { url = "https://files.pythonhosted.org/packages/81/da/72f7caabd94652e6eb7e92ed2d3da818626e70b4f2b15a854ef60bf501ec/websockets-14.2-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a39d7eceeea35db85b85e1169011bb4321c32e673920ae9c1b6e0978590012a3", size = 170178 },
+ { url = "https://files.pythonhosted.org/packages/31/e0/812725b6deca8afd3a08a2e81b3c4c120c17f68c9b84522a520b816cda58/websockets-14.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:0a6f3efd47ffd0d12080594f434faf1cd2549b31e54870b8470b28cc1d3817d9", size = 170453 },
+ { url = "https://files.pythonhosted.org/packages/66/d3/8275dbc231e5ba9bb0c4f93144394b4194402a7a0c8ffaca5307a58ab5e3/websockets-14.2-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:065ce275e7c4ffb42cb738dd6b20726ac26ac9ad0a2a48e33ca632351a737267", size = 169830 },
+ { url = "https://files.pythonhosted.org/packages/a3/ae/e7d1a56755ae15ad5a94e80dd490ad09e345365199600b2629b18ee37bc7/websockets-14.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e9d0e53530ba7b8b5e389c02282f9d2aa47581514bd6049d3a7cffe1385cf5fe", size = 169824 },
+ { url = "https://files.pythonhosted.org/packages/b6/32/88ccdd63cb261e77b882e706108d072e4f1c839ed723bf91a3e1f216bf60/websockets-14.2-cp312-cp312-win32.whl", hash = "sha256:20e6dd0984d7ca3037afcb4494e48c74ffb51e8013cac71cf607fffe11df7205", size = 163981 },
+ { url = "https://files.pythonhosted.org/packages/b3/7d/32cdb77990b3bdc34a306e0a0f73a1275221e9a66d869f6ff833c95b56ef/websockets-14.2-cp312-cp312-win_amd64.whl", hash = "sha256:44bba1a956c2c9d268bdcdf234d5e5ff4c9b6dc3e300545cbe99af59dda9dcce", size = 164421 },
+ { url = "https://files.pythonhosted.org/packages/7b/c8/d529f8a32ce40d98309f4470780631e971a5a842b60aec864833b3615786/websockets-14.2-py3-none-any.whl", hash = "sha256:7a6ceec4ea84469f15cf15807a747e9efe57e369c384fa86e022b3bea679b79b", size = 157416 },
+]
+
+[[package]]
+name = "yarl"
+version = "1.25.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "idna" },
+ { name = "multidict" },
+ { name = "propcache" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/75/16/e8be8e2fb175bbf41a0680381a319f1199fae256588241a2ac8677eafb49/yarl-1.25.1.tar.gz", hash = "sha256:03dd38de09bc213e9a8b29761eec33ee1d5318dac0e49d8af36e4d27830e23a7", size = 246245 }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/75/b3/cd32ac66ae622b854c2df0ac52106dda220d361b65a64fde7d5b3684aa3f/yarl-1.25.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:94d7aa6debf92a1dd14cb5280b083a764169a13cfb23a452111160274ed989f4", size = 144798 },
+ { url = "https://files.pythonhosted.org/packages/61/fb/a2c52a8007c2051ba74662afb112ecf3d00346af4c25e33df9d80fd14fb8/yarl-1.25.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:83d4a37e4b95da4d8bda930d6d35b75b4cdadbacbb4980cae290ea3100b5d51d", size = 104583 },
+ { url = "https://files.pythonhosted.org/packages/be/dd/ee38aec8e09fdf957e50d4085453fbe202f56c6c3b4cf07b81cdb4f09ee9/yarl-1.25.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:e029648f9c951db30e98a7d7ec90835db88ec4b32820efe2a9bdc2287e032eb6", size = 104325 },
+ { url = "https://files.pythonhosted.org/packages/1e/b3/058dbfb1857b484c9cf9cc135659f50b85ce66e03c99e44dc2f7b6161f55/yarl-1.25.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4d781294bb815ecb5ea57ff6bbf8038e0a31a95fdf3e1788f66e0dc100d64b58", size = 115358 },
+ { url = "https://files.pythonhosted.org/packages/db/39/29693446cf0cf6b15a0e2f75a5d40f93c56819b05b0622196f45e95b5cc0/yarl-1.25.1-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e12c538e00e7c1b286a07061046b90e8124e6a9793efae2c70db6a4aad07faad", size = 107658 },
+ { url = "https://files.pythonhosted.org/packages/86/b3/3c4dd7e1af43b931fba95e0a722737f2ea94a6d199c802585282831d7abd/yarl-1.25.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7e4de3ac4adbad3d0bc7c6f4360a7dbff5de2f15e3b723be3198074e17fd9c40", size = 122660 },
+ { url = "https://files.pythonhosted.org/packages/bd/b5/1b60dbc3cfc9c5712b15148c206748f2bc93953ffdbe25ea75b63dfc89c9/yarl-1.25.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:419f392a1da624877975709e3864dfe833af6cc7671b39318086d456e288380c", size = 126506 },
+ { url = "https://files.pythonhosted.org/packages/bc/7b/ca212cbe170ac8b96e45317ecbcf9c3c3ecf0cdec98d5b088a9c4088929b/yarl-1.25.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c6f117789d22dce188e5754e8bc65b7e6ebf8cb73963b9fa761f672a5883769d", size = 117050 },
+ { url = "https://files.pythonhosted.org/packages/cb/c3/72b4938cdbe619ad71ac156182faef4908846b84dc3ca4dbb4c4e6f84014/yarl-1.25.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:80e47012e730da131c9f059c80936783f9659aae22dc31c03c0595590d11ed54", size = 114174 },
+ { url = "https://files.pythonhosted.org/packages/e8/43/268717870f9ba0cc9701a95181587f6dc8c5f387aab4aeecc83158f38a79/yarl-1.25.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e80f557716fd765439577131e526b8942ffc2c07bdbc5e39fa62f660ba1e963f", size = 114944 },
+ { url = "https://files.pythonhosted.org/packages/da/84/baa5bf504d51fe062c4bcaf62936da97fffb43285978d0b39984824231fd/yarl-1.25.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:f61964f235a43738bfac50da46fc4254943a7eea3051aeb0b6fc7c992c29fadc", size = 108263 },
+ { url = "https://files.pythonhosted.org/packages/a4/28/779a2ed9e0152a601a27039bed9aead3f0b79797a67e2c44bfa444622dd8/yarl-1.25.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:e546fe1d4a93ebc2910f0d768baff19faa09843ab3f2036a67ed6e69fae4419d", size = 122184 },
+ { url = "https://files.pythonhosted.org/packages/f8/1f/118e9e5b8f07694d63fd3222e801d7782270003f1a222aa798df3f8d5933/yarl-1.25.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:cce0727fd5ac04d372fa9bbfde9febc2bcf209aadfcf0468e45dec72719895d1", size = 114001 },
+ { url = "https://files.pythonhosted.org/packages/0f/ae/a4cf1cf372313734b17996d4007f9f73596e7a178b9485802e5494ecf484/yarl-1.25.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:af4ea5b37403ef4e30f3927eaed540db942bde01d8d3ff083527c0704d1c9c68", size = 120565 },
+ { url = "https://files.pythonhosted.org/packages/05/79/ad94f93ca731bc9e44d321833ab96b82a4f9f5f63cf773f81a4aeea5ecc1/yarl-1.25.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:68782fdb4027b8d1eee25ec35e9a6db05e863b899eb0310b3a33b6c3fef55707", size = 117060 },
+ { url = "https://files.pythonhosted.org/packages/bb/cc/51a7b4abf4ac593b8e7eb3794b28e5a35ae26eed8bc04787628d215af82f/yarl-1.25.1-cp312-cp312-win_amd64.whl", hash = "sha256:7d575b54cb3863ef9bc290ea4b009999d55dc237326131e4853cf33e888fee03", size = 102593 },
+ { url = "https://files.pythonhosted.org/packages/9d/21/0941a6b93a58b59a1ec75e5333bf06929b671309c43c0cd201c172d9c39f/yarl-1.25.1-cp312-cp312-win_arm64.whl", hash = "sha256:bc3ac7bf569f6b64dad04dd7808c7872dae8a97df657856eac05e9b7e3614a85", size = 97697 },
+ { url = "https://files.pythonhosted.org/packages/54/22/318c7980066769c6bcd9221ed2248294f5698811da099013098c670565ed/yarl-1.25.1-py3-none-any.whl", hash = "sha256:681c758b0490f9e96b78e5fa8e8dc6e648e9185bb6eaebe73183c33ea0c445f3", size = 63617 },
+]