[JianhaoYu (JianhaoYu-AI)]

Summary

Add an offline checker for split manifests, saved per-item scores, and evaluation summaries, related to #158 and #326. It checks local artifacts without model calls or evaluator reruns.

The checker reports source/data revisions and file hashes, validates split IDs and result coverage, and reuses compute_score for hard/soft means including error rows. Comparisons require explicit aggregation and precision information. Recorded runs, reconstructed examples, and missing historical evidence remain distinct.

Available artifacts are validated independently: a missing manifest no longer hides digest mismatches in supplied files, and a missing or empty results file no longer skips summary parsing and field validation. Known failures take precedence over missing evidence; aggregate comparisons remain unavailable when their required inputs are incomplete.

Includes a small synthetic fixture, 96 checker regression cases, and usage documentation. No dependencies, released datasets, or evaluation defaults change.

Validation

Fix commit: 4de39e7975a6a17b84271641ba54b6f2659a70b8.

macOS 26.6 arm64, Python 3.12.14, pytest 9.1.1:

• Focused tests: 111 passed (test_reproducibility.py, test_scoring.py, test_data_manifests.py).
• Full suite: 1926 passed, 12 skipped, plus 368 passing subtests. Opt-in live backend tests were disabled.
• Ruff, git diff --check, and strict MkDocs build passed.
• Four regression cases for the two review findings fail against the previous PR HEAD 0014cf408a87 and pass with the fix.
• CLI regression tests cover FAIL/MISSING exit codes, JSON/text output, and preservation of input bytes.

The full suite used an ASCII checkout with the virtual environment on PATH. The first run in the original directory encountered three installer Unicode errors and one Python discovery failure; those were environment-dependent and required no source changes. The Python discovery failure also reproduced on the clean upstream baseline.

These results validate the new patch locally. The maintainer's earlier Linux validation applies to 0014cf408a87. CI for the new commit is awaiting maintainer approval; the CLA check passed.

Reproduction

python -m scripts.check_reproducibility tests/fixtures/reproducibility/split_manifest.json --results tests/fixtures/reproducibility/results.jsonl --summary tests/fixtures/reproducibility/eval_summary.json --evidence tests/fixtures/reproducibility/evidence.json
python -m pytest -q tests/test_reproducibility.py tests/test_scoring.py tests/test_data_manifests.py
SKILLOPT_TEST_REAL_OPENCODE=0 SKILLOPT_TEST_REAL_OPENCODE_SOURCE=0 python -m pytest -q
python -m ruff check skillopt/utils/reproducibility.py tests/test_reproducibility.py
python -m mkdocs build --strict

The synthetic fixture gives hard=0.5 and soft approximately 0.6. Its aggregate comparison passes; the overall result remains MISSING (exit 2) because it is explicitly RECONSTRUCTED. See docs/guide/offline-reproducibility.md for the schema and precision options.

Limitations

This checks saved-score aggregation, not model-answer correctness or historical authenticity. Hashes bind declarations to bytes. Multi-run, weighted, filtered, grouped, and subset protocols are not inferred. Original evaluation records, source/Skill snapshots, and effective configurations remain missing where unavailable. #158 and #326 remain open; #176's LiveMath metadata fix is unchanged.

[JianhaoYu (JianhaoYu-AI)]

@microsoft-github-policy-service agree

[Yifan Yang (Yif-Yang)]

Reviewed 0014cf408a87051fe9dfa2542b67a0050ba6b032. This is useful, well-scoped offline tooling: it preserves error rows, supports continuous hard scores, requires explicit aggregation/precision, and correctly distinguishes declarations from authentication. The synthetic example and new runs do not establish historical results. Linux validation passed all 75 focused tests and the full suite (1889 passed, 13 skipped); the review baseline 1a44050fc53a passed 1829 tests with the same 13 skips.

Two additional partial-input cases still need correction. If the manifest is missing, the early return skips the evidence and otherwise available files, hiding a known supplied digest mismatch as MISSING. If results are missing, an available, correctly hash-bound but malformed summary is never schema-checked and also remains MISSING. These are diagnostic omissions, not false PASS results, but they undermine the promised independent integrity checks and malformed-input reporting. Please validate each available artifact independently, preserve known failures, and withhold only the aggregate comparison when its inputs are incomplete.

The separate 26-case independent matrix passed 24 cases and reproduced those two failures. No supplied input bytes were modified. Please keep #158 and #326 open: this checker does not provide the original evaluation records, source/Skill snapshots, or effective runtime configurations.

Review baseline: 1a44050fc53a. The PR remains open; no merge was performed.

[JianhaoYu (JianhaoYu-AI)]

Thanks for the review. Fixed both cases in 4de39e7:

• Missing or malformed manifests no longer skip the other available artifacts or their digest checks.
• Summaries are parsed and their fields validated even when results are missing or empty. Only the aggregate comparison is withheld when required inputs are incomplete.

Added regressions for both findings and the related partial-input paths, including CLI exit codes and unchanged input bytes. The four core cases fail against the previous HEAD and pass with this fix.

On macOS / Python 3.12.14: 111 focused tests passed; the full suite passed 1926 tests with 12 skips. Ruff and the strict docs build passed as well. The full run used an ASCII checkout with the virtual environment on PATH to avoid local installer/path issues.

The updated CI workflow is waiting for maintainer approval. Could you approve the run when you have a chance?

#158 and #326 remain open; this does not supply the original run artifacts.