Repository navigation
test: make the default suite a 3.5-minute gate, split off a real-model tier - #24
Merged
Merged
Conversation
…l tier
The suite's cost was the denoise step budget, not fixture size. A denoise step
is ~1.3 s on an M3 Pro, so `max_denoising_steps` set the price of every
model-gated gate. The older gates inherited SamplerConfig's 48-step default or
asked for 24, while the residue gates written later ask for 8. None of these
assert reply quality, and both arms of every comparison share one cfg, so the
step count cancels out of the assertion.
So: step budgets down to 8, plus two token caps. The gates that stay expensive
at 8 steps, because they run two or more full generates or a lineage matrix of
multi-thousand-token prefills, move behind #[ignore] into a real-model tier
documented in AGENTS.md beside golden and smoketest.
Per-test, serial, model pack present, one process at a time:
before after
per_block_ops_match_monolithic_generate 273.9s 100.7s tier
ask_via_pipeline_matches_direct 254.7s 46.9s tier
kv_lineage_unaligned_delta_offsets 234.8s 234.8s tier
kv_lineage_paths_are_fingerprint_identical 218.5s 218.5s tier
pipeline_rewind_kv_byte_consistency 201.6s 68.4s tier
oplog_roundtrip_replays_bit_identically 132.6s 51.2s tier
wrap_crossing_rewind_restores_state 84.2s 57.1s
truncate_after_uncommitted_canvas_write 83.6s 37.4s
tool_compact_m1_m2_and_overlong_smoke 72.0s 72.0s tier
per_block_partial_commit_and_discard 69.3s 26.0s
forced_reroll_leaves_no_residue 61.9s 62.2s tier
synthetic_kv_extension_byte_consistency_100k 39.0s 45.2s
tool_smoke_call_shape_and_argument_types 13.9s 13.9s tier
begin_turn_after_abandoned_turn 12.1s 9.5s
cargo test --release, parallel - 211.3s
cargo test --release, serial ~1830s 244.3s
the tier - 894.2s
The parallel before was never taken: the first parallel run already had the
trims applied. The serial before is reconstructed from the rows above, so it
carries a tilde. Every other number is a stopwatch reading. Untouched rows
drift a few seconds between runs because the first test in a run pays the
session warm-up.
begin_turn_after_abandoned_turn only prefills, so its step change is inert. The
line matches its siblings and buys nothing.
synthetic_kv_extension_byte_consistency_100k is untouched. About 24 s of its
39 s is FNV-1a over ~24 GB of live KV, four full layers at 100k tokens by 8 KiB
per token per layer. A multi-accumulator hash would pay, but it changes every
fingerprint written to op-log JSONL and would make archived repro artifacts
replay as divergent, so it wants its own commit.
The two kv_lineage matrices went to the tier instead of being trimmed. Their
fork needs a wrapped ring, a misaligned resume and a multi-chunk delta together
at specific offsets, so a smaller fixture would stop reproducing it without
failing.
The tier invocation needs its --skip. Test filters match by substring, so
kv_lineage_paths_are_fingerprint_identical on its own also selects the
known-red q8 variant.
Threading is left alone. Before this split the default suite was slower in
parallel than serial, 987.7 s against 783.9 s, because the heavy gates
contended for the GPU. With them in the tier, parallel leads again, 211.3 s
against 244.3 s, so the existing AGENTS.md note holds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
cargo test --releasewith the model pack present goes from roughly 30 minutesto 3.5 minutes. The gates that were paying for that now run as a documented
real-model tier.
Why
The cost was the denoise step budget, not fixture size. A denoise step is about
1.3 s on an M3 Pro, so
max_denoising_stepsset the price of every model-gatedgate. The gates written first inherited
SamplerConfig's 48-step default orasked for 24, while the residue gates written later ask for 8. None of them
assert reply quality, and both arms of every comparison share one cfg, so the
step count cancels out of the assertion.
Changes
generates or a lineage matrix of multi-thousand-token prefills, moved behind
#[ignore = "real-model tier: ..."].goldenandsmoketest.Numbers
Per-test, serial, pack present, one process at a time.
cargo test --release, parallelThe parallel before was never taken, because the first parallel run already had
the trims applied. The serial before is reconstructed from the per-test rows, so
it carries a tilde. Everything else is a stopwatch reading with no membudget
stalls in any reported run.
For the reviewer
--skip. Test filters match by substring, sokv_lineage_paths_are_fingerprint_identicalon its own also selects theknown-red q8 variant and the tier comes back failed. Caught by running it.
fork needs a wrapped ring, a misaligned resume and a multi-chunk delta together
at specific offsets, so a smaller fixture would stop reproducing the bug
without failing. That seemed worse than a slow test.
begin_turn_after_abandoned_turnonly prefills, so its step change is inert.The line is there to match its siblings and it buys nothing. Happy to drop it.
synthetic_kv_extension_byte_consistency_100kis untouched. About 24 s of its39 s is FNV-1a over roughly 24 GB of live KV, four full layers at 100k tokens.
A multi-accumulator hash would pay, but it changes every fingerprint written to
op-log JSONL, which would make archived repro artifacts replay as divergent.
That wants its own commit.
serial, 987.7 s against 783.9 s, because the heavy gates contended for the GPU.
With them in the tier, parallel leads again and the existing AGENTS.md note
holds.
aa6bff6d.mainhas since moved tof4b43d28.🤖 Generated with Claude Code