Skip to content

test: make the default suite a 3.5-minute gate, split off a real-model tier - #24

Merged
mmastrac merged 1 commit into
mainfrom
claude/infallible-varahamihira-3eeb8f
Sep 16, 2026
Merged

mmastrac merged 1 commit into
mainfrom
claude/infallible-varahamihira-3eeb8f

Conversation

@mmastrac

Copy link
Copy Markdown
Owner

What

cargo test --release with the model pack present goes from roughly 30 minutes
to 3.5 minutes. The gates that were paying for that now run as a documented
real-model tier.

Why

The cost was the denoise step budget, not fixture size. A denoise step is about
1.3 s on an M3 Pro, so max_denoising_steps set the price of every model-gated
gate. The gates written first inherited SamplerConfig's 48-step default or
asked for 24, while the residue gates written later ask for 8. None of them
assert reply quality, and both arms of every comparison share one cfg, so the
step count cancels out of the assertion.

Changes

  1. Step budgets to 8 across six gates, plus two token caps. Assertions untouched.
  2. Nine gates that stay expensive at 8 steps, because they run two or more full
    generates or a lineage matrix of multi-thousand-token prefills, moved behind
    #[ignore = "real-model tier: ..."].
  3. AGENTS.md documents the tier beside golden and smoketest.

Numbers

Per-test, serial, pack present, one process at a time.

test before after
per_block_ops_match_monolithic_generate 273.9s 100.7s tier
ask_via_pipeline_matches_direct 254.7s 46.9s tier
kv_lineage_unaligned_delta_offsets 234.8s 234.8s tier
kv_lineage_paths_are_fingerprint_identical 218.5s 218.5s tier
pipeline_rewind_kv_byte_consistency 201.6s 68.4s tier
oplog_roundtrip_replays_bit_identically 132.6s 51.2s tier
wrap_crossing_rewind_restores_state 84.2s 57.1s
truncate_after_uncommitted_canvas_write 83.6s 37.4s
tool_compact_m1_m2_and_overlong_smoke 72.0s 72.0s tier
per_block_partial_commit_and_discard 69.3s 26.0s
forced_reroll_leaves_no_residue 61.9s 62.2s tier
synthetic_kv_extension_byte_consistency_100k 39.0s 45.2s
tool_smoke_call_shape_and_argument_types 13.9s 13.9s tier
begin_turn_after_abandoned_turn 12.1s 9.5s
gate before after
cargo test --release, parallel not measured 211.3s, 794 passed
same, serial ~1830s reconstructed 244.3s
the tier n/a 894.2s, 9 passed

The parallel before was never taken, because the first parallel run already had
the trims applied. The serial before is reconstructed from the per-test rows, so
it carries a tilde. Everything else is a stopwatch reading with no membudget
stalls in any reported run.

For the reviewer

  • The tier command needs its --skip. Test filters match by substring, so
    kv_lineage_paths_are_fingerprint_identical on its own also selects the
    known-red q8 variant and the tier comes back failed. Caught by running it.
  • The two kv_lineage matrices went to the tier rather than being trimmed. Their
    fork needs a wrapped ring, a misaligned resume and a multi-chunk delta together
    at specific offsets, so a smaller fixture would stop reproducing the bug
    without failing. That seemed worse than a slow test.
  • begin_turn_after_abandoned_turn only prefills, so its step change is inert.
    The line is there to match its siblings and it buys nothing. Happy to drop it.
  • synthetic_kv_extension_byte_consistency_100k is untouched. About 24 s of its
    39 s is FNV-1a over roughly 24 GB of live KV, four full layers at 100k tokens.
    A multi-accumulator hash would pay, but it changes every fingerprint written to
    op-log JSONL, which would make archived repro artifacts replay as divergent.
    That wants its own commit.
  • Threading is unchanged. Before this split the suite was slower in parallel than
    serial, 987.7 s against 783.9 s, because the heavy gates contended for the GPU.
    With them in the tier, parallel leads again and the existing AGENTS.md note
    holds.
  • Branched from aa6bff6d. main has since moved to f4b43d28.

🤖 Generated with Claude Code

…l tier

The suite's cost was the denoise step budget, not fixture size. A denoise step
is ~1.3 s on an M3 Pro, so `max_denoising_steps` set the price of every
model-gated gate. The older gates inherited SamplerConfig's 48-step default or
asked for 24, while the residue gates written later ask for 8. None of these
assert reply quality, and both arms of every comparison share one cfg, so the
step count cancels out of the assertion.

So: step budgets down to 8, plus two token caps. The gates that stay expensive
at 8 steps, because they run two or more full generates or a lineage matrix of
multi-thousand-token prefills, move behind #[ignore] into a real-model tier
documented in AGENTS.md beside golden and smoketest.

Per-test, serial, model pack present, one process at a time:

                                                 before     after
  per_block_ops_match_monolithic_generate        273.9s    100.7s  tier
  ask_via_pipeline_matches_direct                254.7s     46.9s  tier
  kv_lineage_unaligned_delta_offsets             234.8s    234.8s  tier
  kv_lineage_paths_are_fingerprint_identical     218.5s    218.5s  tier
  pipeline_rewind_kv_byte_consistency            201.6s     68.4s  tier
  oplog_roundtrip_replays_bit_identically        132.6s     51.2s  tier
  wrap_crossing_rewind_restores_state             84.2s     57.1s
  truncate_after_uncommitted_canvas_write         83.6s     37.4s
  tool_compact_m1_m2_and_overlong_smoke           72.0s     72.0s  tier
  per_block_partial_commit_and_discard            69.3s     26.0s
  forced_reroll_leaves_no_residue                 61.9s     62.2s  tier
  synthetic_kv_extension_byte_consistency_100k    39.0s     45.2s
  tool_smoke_call_shape_and_argument_types        13.9s     13.9s  tier
  begin_turn_after_abandoned_turn                 12.1s      9.5s

  cargo test --release, parallel                       -    211.3s
  cargo test --release, serial                  ~1830s      244.3s
  the tier                                             -    894.2s

The parallel before was never taken: the first parallel run already had the
trims applied. The serial before is reconstructed from the rows above, so it
carries a tilde. Every other number is a stopwatch reading. Untouched rows
drift a few seconds between runs because the first test in a run pays the
session warm-up.

begin_turn_after_abandoned_turn only prefills, so its step change is inert. The
line matches its siblings and buys nothing.

synthetic_kv_extension_byte_consistency_100k is untouched. About 24 s of its
39 s is FNV-1a over ~24 GB of live KV, four full layers at 100k tokens by 8 KiB
per token per layer. A multi-accumulator hash would pay, but it changes every
fingerprint written to op-log JSONL and would make archived repro artifacts
replay as divergent, so it wants its own commit.

The two kv_lineage matrices went to the tier instead of being trimmed. Their
fork needs a wrapped ring, a misaligned resume and a multi-chunk delta together
at specific offsets, so a smaller fixture would stop reproducing it without
failing.

The tier invocation needs its --skip. Test filters match by substring, so
kv_lineage_paths_are_fingerprint_identical on its own also selects the
known-red q8 variant.

Threading is left alone. Before this split the default suite was slower in
parallel than serial, 987.7 s against 783.9 s, because the heavy gates
contended for the GPU. With them in the tier, parallel leads again, 211.3 s
against 244.3 s, so the existing AGENTS.md note holds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mmastrac
mmastrac merged commit 026289d into main Sep 16, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant