feat(examples): add reproducible router learning example

[svonava]

Add a self-contained example in examples/train-a-router/ for training, evaluating and serving a difficulty router. The walkthrough introduces the dataset and metrics first, then provides a numbered progression with accuracy in each heading and executable instructions for every stage.

The README defines training terms where readers first encounter them and explains what each command produces. Every command block starts with a purpose comment. Setup separates dependency installation, verified downloads and input preparation; Cloud/local server addresses are explicit.

Stage	Recorded test accuracy
Calibrated pretrained Laya	53.65%
Laya + LoRA and trained native head, 512 training examples	57.34% mean
Same LoRA recipe, 2,581 training examples	65.00% mean
Frozen Qwen3-Embedding-4B + ridge, 64 training examples	67.64%
Same encoder and classifier family, 512 training examples	72.03%

The main path calibrates, trains and evaluates native models on an Apple Silicon GPU or a local/VM CUDA GPU, then trains the embedding classifiers on CPU. Readers serve their trained Laya checkpoint through local SIE and use the learned embedding classifier with SIE Cloud or a local Qwen server. Cached vectors make classifier training independent of an API key; a separate collection flow supports fresh development embeddings, retraining, freezing and test evaluation. Cloud credits cover inference, not the training machine.

The target is human-rated difficulty within a named mathematics competition. Inputs exclude ratings, answers and solutions. Chronological partitions separate duplicate families and isolate validation from calibration. The benchmark has 247 validation, 246 calibration and 479 test problems, with nested training sets. All stages use the same public test, reused as an exposed regression benchmark. Different model sizes and training budgets make this a comparison of practical designs. Classification accuracy does not establish downstream solver quality or monetary savings.

The LoRA study screens six recipes across three declared seeds, then expands the validation-selected recipe. Rank-16 attention adapters plus the trained native head improve from 57.34% to 65.00% mean accuracy with more examples: 17.9% fewer errors, with a paired 95% gain interval of +3.20 to +12.04 points. All seeds improve; the unchanged epoch count also increases optimizer updates. Qwen plus ridge is the strongest measured classifier. Its 64-to-512 gain has an interval that includes zero. All candidates, learning curves, calibration, checkpoint hashes and export/runtime checks are retained.

Apple Silicon support uses FP32, micro-batch one with accumulation 16, and evaluation batches of four. Gradient checkpointing is disabled because a Torch 2.9.1 MPS probe reproduced identical forward outputs but different dropout gradients during recomputation. The CUDA profile and selection/export guards remain unchanged. Mac validation records a complete 512-example, eight-epoch run on an M3 Max with 128 GiB unified memory: 10.3 minutes, 9.1 GiB sampled peak MPS allocation, and 57.62% accuracy versus 53.65% for base Laya. This is a separate single-seed result; the full 2,581-example Mac run and smaller-memory Macs have not been measured.

A matched local serving check makes the accuracy/operations tradeoff visible: expanded Laya takes 48.77 ms median / 63.43 ms p95, while Qwen4B plus ridge takes 171.51 ms / 246.43 ms, on the same M3 Max with FP32 computation. Both use fresh sequential SIE 0.10.0 servers, 20 warmups, the same 32 requests and persistent SDK clients; both match all 32 frozen reference decisions. The embedding classifier's local work takes 0.87 ms median, including normalization and policy checks. Qwen is about 9.55 times larger and has different pretraining; this is not a size-controlled experiment. Timings include local HTTP, exclude loading/startup, and do not establish concurrent throughput, Cloud latency or downstream monetary savings. benchmark_serving.py reproduces the protocol and preserves incomplete receipts on failures.

Code, pinned dependencies and configuration live in the example. Larger data and evidence are immutable Hugging Face downloads:

• Benchmark inputs and baseline records.
• LoRA recipe and training-size study.
• Mac training, evaluation and serving records.
• Matched local serving measurements, environment and source.

Validation:

• 62 CPU tests pass, including progress receipt completion, partial failure preservation and early rejection in the new benchmark. The optional gradient regression previously passed on actual MPS and is skipped by CPU-only checks. Ruff lint and formatting pass. Independent documentation and ML reviews found no blocking issues. All 31 local links/anchors and 40 shell blocks pass static checks; every command block has a purpose comment.
• The complete Mac run passes selection, merge, FP32 export and calibration checks with zero changed validation decisions. Its exported checkpoint matches 32/32 fixed-prefix frozen decisions through public SIE 0.10.0 on MPS. All owned compute has stopped.
• The new Mac evidence downloads anonymously at its pinned revision; every manifest/file hash and reported accuracy was verified. Existing baseline and LoRA report replay remain exact. A clean public-data rehearsal has also reproduced classifier training, evaluation and figures.
• All 17 serving-evidence files download anonymously and match their published hashes. Final native/embedding samples use the same inputs and benchmark source; both servers are stopped and the shared Apple GPU reservation is released.
• The PR remains a draft. Repository CI for the latest commit is reported in the checks panel.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}