[bitaccesscomau]

Problem and result

A prompt chunk of 1,024 tokens or more streams every non-resident expert of every layer through the pinned prompt ring
(src/prefill/prefill.cpp), and the issuer thread enqueues each copy as soon as the ring has room. The ring holds more
entries than a layer has, so a layer's copies are all on the copy stream before the layer's router has run: an expert
that no token of the chunk routes to is copied over the PCIe link anyway. On an RTX 3090 with the Swift IQ3_XXS pack,
the switch below skipped 6,379 of the 15,308 streamed experts of a 1,046-token read (42%), 3,347 of 15,481 on a
1,979-token follow-up turn (22%) and 2,054 of 67,372 at 31K tokens (3%).

With STRATA_PREFILL_ROUTED_STREAM=1 (off by default, read once):

• the issuer keeps at most STRATA_PREFILL_ROUTED_STREAM_DEPTH copies (default 16, clamped to 1 to 256) queued ahead of
  the link, so the copies are not committed far ahead of it; the ring's own rule is unchanged, so the stream still runs
  as far ahead of the compute as it does today;
• after each layer's host grouping the compute thread publishes which experts at least one token routes to;
• the issuer skips an entry it has not issued yet whose layer is published and whose expert no token routes to. Entries
  already issued stay, and an entry staged through the Stager's pinned buffers is never skipped (its job queue is in
  order).

Every routed entry is still copied, in the same order, into the same ring slot (k % ring), with the same slot wait and
the same copied event, and the consumer's walk is unchanged (it never reads an unrouted entry's slot), so the prompt's
products are the same bits. With the variable unset the engine runs today's code: no gate, no publication, and the
default issuer loop is not edited.

Measured on main on one RTX 3090 at the release defaults, the same binary with the switch off and on, 4 pairs: the
read of a follow-up turn (1,790 new tokens after 18,763 reused) took 17.9% less time, 3,214 to 2,640 ms (pair ratios
1.224, 1.212, 1.209, 1.225); fresh prompt reads ran 19.9% faster at 1,979 tokens, 8.2% at 3,963, 2.1% at 6,011 and 5.3%
at 32,105 (8,192-token chunks), and the conversation's 18,768-token first turn took 6.0% less time. Decode after the
counting prompts (the same greedy text in every arm) moved by -0.8% to +0.5%; the agent turns decoded 3.5% slower, on
texts that differ in every arm, with 60.6% of their drafts accepted against 62.3% (every row is in the table below).
In a deterministic lane the replies, the prompt and reuse counts, the engine's
session-state fingerprint and the prompt-path residuals are byte-identical to main from 1K to 32K tokens with the
switch unset and set. An earlier 4-pair run on the same machine with other settings and another base gave a follow-up
read 17.7% shorter.

Related discussion

Scope proposal: #4282. Issues and pull requests, open and closed, were searched again for duplicates on 2026-10-27,
including those filed since 2026-10-26 (STRATA_PREFILL_ROUTED_STREAM, ROUTED_STREAM, unrouted, routed-only,
stream_all, streamed experts, experts_streamed, route gate, no token routes, prompt ring, issuer,
copy queue depth, read-ahead experts, skip experts prefill, expert copies routing, PCIe expert copies prompt):
none skips unrouted entries of the streamed walk.

• perf(prefill): honor explicit routed-only staging for large chunks #225 (on main as e258e6ec): STRATA_PREFILL_RING=8 stages only the routed experts of a large chunk, but gives up
  the streamed ring's read-ahead (a ring of 8 slots). This keeps the full ring and its read-ahead and skips only what the
  routing shows is unused.
• [Engine] prefill: opt-in sparse copied events and deduplicated slot waits (Windows HIP prompt reads +36% at 4K, +15% at 32K) #4005 (open; sparse copied events and deduplicated slot waits, Windows HIP): the same issuer loop. Two textual
  conflicts in prefill.cpp (the start of the issuer thread and the trace after its join; keep both) and one semantic
  one: with [Engine] prefill: opt-in sparse copied events and deduplicated slot waits (Windows HIP prompt reads +36% at 4K, +15% at 32K) #4005's sparse events on, the consumer waits on counters (a_rec, rec_of) that only [Engine] prefill: opt-in sparse copied events and deduplicated slot waits (Windows HIP prompt reads +36% at 4K, +15% at 32K) #4005's loop writes,
  and the routed walk would not advance them. Whichever merges second needs one rule: the routed walk off while sparse
  events are on (one more condition in rs_on), or the walk recording through [Engine] prefill: opt-in sparse copied events and deduplicated slot waits (Windows HIP prompt reads +36% at 4K, +15% at 32K) #4005's record. I will rebase onto
  whichever lands first.
• [Engine] prefill: STRATA_PF_PEER_CPU=1 (opt-in) runs the CPU share beside a peer GPU (--peer-device) #2721 (open; STRATA_PF_PEER_CPU): both insert a few lines right after the host grouping's count loop; keep both.
  This switch is off when a peer prompt ring is in use.
• [Engine] Opt-in two-pass QSA prompt selection: 3xTF32 only near the top-k cut-off, the same cells (256K prompt read +5.2%) #4119 (open, mine): the test target in CMakeLists.txt and the paragraph in docs/DETAILS.md sit at the same place;
  keep both.
• [Engine] prompt path: experts staged from the resident arena keep the full streamed ring and 8 copy threads (unpinned 32K +5-19%); STRATA_ARENA_PIN_GIB=none #2968 (open; staged experts keep the full streamed ring and 8 copy threads) and [Engine] stage buffers: fence a queued copy per stream, not per buffer (#1237) #3483 (open; stage-buffer fences per
  stream): the Stager's entries, which this switch never skips.
• [Engine] qwen35: faster decode chain and prompt expert stream (two kernels always on, two opt-in flags) #3905 (open, the qwen35 prompt expert stream with event waits): another model's walk; nothing is skipped there.
• prefill: STRATA_PREFILL_CPU_SHARE=auto hands a small chunk's least-routed experts to the idle CPU pool (opt-in) #1050, prefill: STRATA_PREFILL_CPU_SHARE=auto hands a small chunk's least-routed experts to the idle CPU pool (opt-in) #1282, prefill: the CPU share (STRATA_PREFILL_CPU_SHARE) up to 3,072-token chunks, from mapped experts, on a layer split #1414 (the CPU share), prefill: overlap shared-expert work with routed expert uploads #789 (shared-expert work overlapped with routed uploads) and [Engine] Prefill pipe: STRATA_PREFILL_PIPE_FLOOR, the smallest pipelined prompt chunk (default 512, as before) #3821 (the pipe
  floor): other levers on the same reads.

Merge-tested against all 517 open pull requests on 2026-10-27, on this revision: 516 tested, and #4178's head could not
be fetched (its four files include none of this PR's). The conflicts this branch adds are the three above.

Approach

The change is in src/prefill/prefill.cpp (the chunk's stream setup, the issuer thread, the host grouping, the join)
plus a header-only helper, include/strata/prefill/routed_stream.hpp, which holds the route gate and the issuer walk so
the walk itself can be tested without a GPU.

• The gate. One byte per (layer, expert) per chunk. The compute thread fills a layer's row from the host grouping's
  per-expert counts (m.cnt) right after the count loop, then stores a per-layer atomic flag with release semantics; the
  issuer reads the flag with acquire before the row. One writer per layer per chunk, reset before the issuer thread
  starts, destroyed after it is joined.
• The walk. Entry k (0-based over the chunk's whole stream list) goes to ring slot k % ring, as today. An entry is
  copied only when both today's ring rule (k < a_consumed + ring) and the depth rule hold: the copy issued depth
  copies earlier has landed (cudaEventQuery on its slot's copied event). While the walk waits for either, it
  re-checks the skip, so an entry is skipped exactly when its layer's routing has landed before it was issued.
  a_issued advances past every entry, copied or skipped, so the consumer's wait_issued is unchanged.
• Deadlock and ordering. Every event the copy stream waits on is already recorded, and the compute stream waits only
  on copies already issued, so there is no cycle. A skipped entry touches no slot, event or stage_live flag, and the
  slot it would have used is still released by the consumer's own used event. The copy stream stays in order, so a
  group's single wait on its last copy still covers the group. a_stop is checked in the wait (early exits).
• The loan. After the read, the expert-cache slots the prompt borrowed for its ring are refilled from RAM, blob by
  blob, into the same slots, as today, whatever the ring held.
• Scope. Streamed chunks (ring > 8, 1,024 tokens or more) on one GPU with the issuer thread. The fused prompt path
  (STRATA_PF_FUSED) publishes no routing, so its layers skip nothing; the peer prompt ring (--peer-device) and the
  inline issuer (STRATA_PREFILL_ISSUER=0) keep the switch off. Host code only: both HIP builds compile the new calls
  through the existing compat headers; the Metal build compiles prefill.cpp, but its Prefill::run hands the prompt
  to the Metal window before run_impl, where the walk is; SYCL builds its own prefill.cpp.
• Diagnostics. STRATA_TRACE=1 prints, per chunk, the ring entries copied and skipped; one start-up line names the
  depth when the switch first acts.
• Test. tests/prefill/routed_stream_test.cpp (GPU-free, under STRATA_BUILD_TESTS) runs the same issue_walk
  template the engine uses against a simulated in-order link and a consumer that walks the layers as prefill.cpp does:
  8 fixed and 38 random simulated chunks plus 6 three-thread walks with real atomics. On every one: each routed entry is
  copied once into slot k % ring and read with its own bytes; only unrouted, arena-copied entries of published layers
  are skipped; never more than depth copies are queued; no copy goes past the ring; a_issued covers every entry.

Validation

Revision ab73738f: this branch, one commit on upstream/main 4b9e05a2 (v0.2.6, engine 0.1.51); base 4b9e05a2,
built the same way. The binaries below were built from 56294b33, which differs from ab73738f only in
docs/DETAILS.md (these numbers) and the commit message: CMakeLists.txt, prefill.cpp, the header and the test are
the same blobs. The change was first written and measured on a branch with two other opt-in commits under it; on main
its own lines are unchanged and only the placement of the test target and the DETAILS paragraph moved. Everything below
except the last block of details was built and measured on main.

System (one machine; no claim for others):

• GPU: NVIDIA GeForce RTX 3090 24 GB (sm_86), power limit 250 W (default 350 W), PCIe 3.0 x16 (the engine's probe: 11.8
  GB/s host to device), driver 580.178.04, CUDA 13.0 (nvcc 13.0.88). The card's tensor rate is unlocked by a kernel
  module (FP16 GEMM with FP32 accumulation measured 65.7 TFLOPS before and 100.5 TFLOPS after, at 250 W), active since
  2026-10-26 and the same in every run here.
• CPU: Intel Xeon E5-2698 v4 (Broadwell-EP, AVX2 and FMA, no AVX-512). The engine runs in a KVM guest with 32 vCPUs
  pinned to 16 physical cores and the GPU passed through; host RAM DDR4-2133 in four channels.
• Guest: 149 GiB RAM (free -g), no swap, Ubuntu 24.04.5 LTS, kernel 6.8.0-146-generic, gcc 13.3.0, CMake 4.4.3,
  Ninja 1.13.2; model files on a virtual (QEMU) disk.
• Build: Release, -DSTRATA_ENABLE_CUDA=ON -DSTRATA_BUILD_TESTS=OFF -DSTRATA_Q6K_EXPERTS=ON -DSTRATA_MMQ_KQUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_COMPILER=<cuda-13.0>/bin/nvcc -DCUDAToolkit_ROOT=<cuda-13.0> -DSTRATA_GGML_DIR=<llama.cpp at setup's pin 3cf03257> (the CMake cache equals the installed engine's apart from the
  STRATA_PLE_FIXTURE_DIR path).
• Model and settings: the public Swift Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS GGUF (two shards) and its pack, served by
  serve/server.py --engine strata --config <config> --port 8080 (v0.2.6, unchanged; run through a wrapper that only
  records the server's per-request log lines) with strata --serve --pack <pack> --native <gguf> --ple-gguf <gguf> --expert-profile <expert-profile.bin> --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <mtp dir> --max-context 262144 --kv int8 --kv-resident 32768. In the benchmark no STRATA_*
  variable but the switch; 10,320 expert slots and 23,738 MiB of VRAM in use at ready in every arm, switch on or off.

Passed:

• CUDA build, the line under System: cmake --build <build> --target strata strata_kq for base and head, rc 0 (165 steps
  each), 248 warning lines each with the same set (none in prefill.cpp); both report strata 0.1.51; the switch's
  name is in the head binary and not in the base one. The test build (-DSTRATA_BUILD_TESTS=ON) rc 0 with no warning,
  and ctest -R over the 12 GPU-free unit tests of CMakeLists.txt plus routed_stream_test: head 13 of 13, base 12
  of 12.

• Metal on macOS 15.7.7 (Apple M3, Apple clang 17.0.0), the steps of the Metal build workflow (metal-cpp 26 at its
  pinned SHA-256, llama.cpp 3cf03257, cmake -S metal -B build-metal -G Ninja -DCMAKE_BUILD_TYPE=Release -DMETAL_CPP_DIR=<metal-cpp> -DSTRATA_GGML_DIR=<llama.cpp> then cmake --build build-metal --target strata): base and
  head rc 0, the same 76 object files (prefill.cpp among them) and the same 10 warnings.

• HIP and SYCL, compile only (x86-64 Linux containers, no AMD or Intel GPU): base built in full, then this commit
  checked out in the same tree and rebuilt (ninja -k 0, every target).

  • HIP wave32, docs/AMD_HIP.md's line (-DSTRATA_ENABLE_HIP=ON -DSTRATA_ENABLE_CUDA=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1100) plus -DSTRATA_BUILD_TESTS=ON, in rocm/dev-ubuntu-24.04:7.14.1-full (HIP
    7.14.60850, AMD clang 23.0.0git, gcc 13.3.0, CMake 3.28.3): base rc 0 (428 steps), head rc 0 (15 steps rebuilt,
    prefill.cpp among them). ctest -R 'routed_stream_test|share_rules_test|host_limits_test|prefill_plan_test': 4 of
    4 pass.
  • HIP gfx906, docs/AMD_HIP.md's line (-DSTRATA_HIP_GFX906=ON -DCMAKE_HIP_ARCHITECTURES=gfx906, ROCm clang as the C
    and C++ compiler) plus -DSTRATA_BUILD_TESTS=ON, same image: prefill.cpp compiles and the same 4 tests pass. One
    target fails on base and head alike: src/core/dflash2.cpp (cudaDeviceGetStreamPriorityRange and
    cudaStreamCreateWithPriority have no mapping in the gfx906 compat headers), so strata does not link in this
    configuration on main either.
  • SYCL, the strata-sycl-dev recipe (docker build --pull -t strata-sycl-dev sycl/tools, then sycl/tools/build.sh
    without AOT; oneAPI DPC++ 2026.1.0, CMake 4.2.3), configured with STRATA_GGML_DIR at the pinned llama.cpp: base rc
    0 (232 steps), head rc 0 (50 relinks, nothing recompiled: the SYCL tree compiles none of the changed files).
  • New warnings against base, keyed by file and message: none on SYCL. On both HIP builds, 4 more of the hipError_t
    nodiscard warnings in prefill.cpp, at lines 3080 to 3089: the unchecked cudaStreamWaitEvent, two
    cudaMemcpyAsync and cudaEventRecord of the routed walk's copy, which repeats the default loop's copy (its same
    four calls already warn; base reports 235 (wave32) and 227 (gfx906) of these in prefill.cpp). No other new warning.

• routed_stream_test on macOS with Apple clang 17: c++ -std=c++20 -O2 -Wall -Wextra -Iinclude tests/prefill/routed_stream_test.cpp -o rst && ./rst prints routed_stream_test: OK with no warning, and the same with
  -O1 -g -fsanitize=thread gives no ThreadSanitizer report.

• Default path byte-identical, and the switch output-preserving, at 1K to 32K. One locked batch, a fresh server and
  engine per arm, four arms in this order: base (main), head with the switch unset, head with
  STRATA_PREFILL_ROUTED_STREAM=1, base again (the control). Deterministic lane in every arm (STRATA_PREFILL_CPU_SHARE=0,
  --pcie-frac 0.32, --adapt-every 0) plus STRATA_TRACE=1, STRATA_STATE_HASH=1 and STRATA_PREFILL_DUMP_R
  (the prompt-path residuals of every 64th position, fp32); temperature 0, thinking off, 64 tokens. Prompts: four
  consecutive slices of the 128K identity text (documentation, each with a label and a one-sentence question), a
  two-turn conversation (14,277 tokens, then a follow-up turn of 1,979 new tokens), and the 31,473- and 126,970-token
  identity prompts. Compared per step: the reply's SHA-256, the engine's prompt N tokens = R reused + K read and
  generated counts, its STATE_HASH line (GDN, PLE history, index tails and pools, K/V cells, the stale cells past the
  live length, MTP K/V) and the residual dump's SHA-256. Up to 32K every field is identical in all four arms; the head
  with the switch on equals the head with it unset on all eight steps, 128K included. No error line in any engine or
  server log; the switch's start-up line appears in its own arm only.

  Prompt	Tokens (reused + read)	Reply SHA-256: base	unset	on	base again	Experts streamed, unset / on	On, trace: copied + skipped
  Slice 1	1,046 (0 + 1,046)	f7f17435	same	same	same	15,308 / 8,929	8,929 + 6,379
  Slice 2	1,437 (0 + 1,437)	4f1175ca	same	same	same	15,365 / 10,228	10,228 + 5,137
  Slice 3	1,965 (0 + 1,965)	9eba857d	same	same	same	15,481 / 11,063	11,063 + 4,418
  Slice 4	4,867 (0 + 4,867)	d98257b1	same	same	same	16,110 / 13,877	13,877 + 2,233
  Conversation, turn 1	14,277 (0 + 14,277)	7311ac3a	same	same	same	33,686 / 31,790	31,790 + 1,896
  Conversation, turn 2	16,294 (14,315 + 1,979)	371f213d	same	same	same	15,481 / 12,134	12,134 + 3,347
  32K	31,473 (0 + 31,473)	e0ede8e7	same	same	same	67,372 / 65,318	65,318 + 2,054
  128K	126,970 (0 + 126,970)	47f66830	1fc92b9e	1fc92b9e	1fc92b9e	269,488 / 263,486	263,486 + 6,002

  The base streamed the same experts as the head with the switch unset at every step. The 128K row is under Failed.

• Speed on main, release defaults (above), the head binary with the switch unset and set, one locked batch in the
  order OFF ON ON OFF, then ON OFF OFF ON (pairs: arms 2 and 1, 3 and 4, 5 and 6, 8 and 7), a fresh server and engine per
  arm. Requests per arm, in this order: counting prompts of 6,011 and 1,979 tokens; six tiny chats (17 to 19 prompt
  tokens, 64 tokens); counting prompts of 955, 3,963 and 32,105 tokens; a 3,828-token natural-text prompt twice (384
  tokens); a 3,971-token prompt with thinking on, temperature 1.0 and seeds 4242, 4243, 4244 (256 tokens); an 18,768-token
  agent prompt with 33 tools and its second turn (1,790 new tokens), thinking on, the server's default sampling with a
  fixed seed (384 tokens). Counting prompts: a filler with a unique nonce in every repetition, 2 runs each (the second
  run's read is reported; each run is preceded by two 1-token calibration requests), 256 tokens, temperature 0, thinking
  off, 0 tokens reused. Engine-side numbers: prompt tokens over the engine's prompt ms, the engine's decode tok/s.

  Measure	Off	On	Pair ratios (off / on for ms, on / off for tok/s)	Median	Change
  Follow-up turn: 1,790 new tokens read after 18,763 reused, ms	3,214	2,640	1.224, 1.212, 1.209, 1.225	1.218	-17.9% (time)
  First turn of that conversation: 18,768 tokens read, ms	11,474	10,782	1.086, 1.057, 1.051, 1.062	1.060	-6.0% (time)
  Prompt read, 1,979 tokens (one chunk), tok/s	707.9	849.0	1.204, 1.194, 1.186, 1.214	1.199	+19.9%
  Prompt read, 3,963 tokens, tok/s	1,333.7	1,443.2	1.089, 1.081, 1.085, 1.073	1.083	+8.2%
  Prompt read, 6,011 tokens, tok/s	1,846.2	1,885.4	1.020, 1.029, 1.017, 1.019	1.019	+2.1%
  Prompt read, 32,105 tokens (8,192-token chunks), tok/s	2,278.3	2,399.1	1.044, 1.061, 1.050, 1.056	1.053	+5.3%
  Prompt read, 3,828-token natural text (2 runs pooled), tok/s	1,177.5	1,215.7	1.044, 1.039, 1.030, 1.017	1.035	+3.2%
  Prompt read, 955 tokens (below the 1,024-token stream floor), tok/s	665.6	664.2	0.998, 0.999, 0.987, 1.008	0.998	-0.2%
  Tiny chats, engine prompt ms (median of 6)	365	374	1.058, 0.982, 0.867, 1.010	0.996	+2.4% (time)
  Decode after 1,979 tokens (run 2), tok/s	171.6	170.6	0.984, 1.000, 0.984, 1.008	0.992	-0.6%
  Decode after 3,963 tokens (run 2), tok/s	170.4	169.8	0.989, 1.001, 0.998, 0.997	0.998	-0.4%
  Decode after 6,011 tokens (run 2), tok/s	166.2	164.9	0.983, 1.005, 0.980, 1.001	0.992	-0.8%
  Decode after 32,105 tokens (run 2), tok/s	162.4	163.2	1.015, 1.023, 0.994, 0.989	1.004	+0.5%
  Decode after 955 tokens (run 2), tok/s	171.7	171.4	0.994, 1.003, 0.997, 0.999	0.998	-0.2%
  Decode, natural text (2 pooled), tok/s	94.0	98.0	1.088, 1.084, 0.987, 1.022	1.053	+4.2%
  Decode, tiny chats (6 pooled), tok/s	86.3	84.1	0.946, 1.058, 0.890, 1.013	0.980	-2.6%
  Decode, sampled, thinking on (3 seeds pooled), tok/s	100.9	101.2	0.984, 1.069, 0.976, 0.989	0.986	+0.3%
  Decode, agent turns (2 pooled), tok/s	101.3	97.8	0.935, 0.959, 0.989, 0.975	0.967	-3.5%

  Off and On are the means of the four arms per side. Follow-up turn: median pair ratio 1.218, percentile bootstrap 95%
  interval 1.209 to 1.225 (20,000 resamples of the four ratios). The switch's code runs in every read above but the
  955-token one and the tiny chats; the streamed experts fell by 23% at 1,979 tokens, 16% at 3,963, 9% at 6,011, 4% at
  32,105 and 22% on the follow-up turn (table below). At these settings the replies differ between arms of the same
  binary and switch (the adaptive CPU share and PCIe split): five of the six tiny chats' replies differ among the four
  switch-off arms, and the natural-text, sampled and agent requests produce a different text in every arm, so those
  decode rows compare different texts.

Every run of the timing batch

Engine prompt ms per run (counting prompts: run 1 / run 2):

Arm	1,979	3,963	6,011	32,105	955	Natural 3,828, r1 / r2	Agent turn 1 / turn 2	Tiny chats (6)
1 OFF	2,832 / 2,822	2,977 / 2,985	3,222 / 3,251	14,009 / 13,967	1,449 / 1,435	3,850 / 2,816	11,807 / 3,262	460, 432, 319, 354, 366, 398
2 ON	2,351 / 2,343	2,763 / 2,741	3,191 / 3,188	13,296 / 13,373	1,463 / 1,438	3,702 / 2,684	10,874 / 2,666	403, 394, 359, 338, 363, 343
3 ON	2,347 / 2,338	2,748 / 2,759	3,183 / 3,181	13,287 / 13,348	1,471 / 1,452	3,574 / 2,699	10,743 / 2,655	441, 430, 334, 368, 399, 372
4 OFF	2,796 / 2,791	2,967 / 2,983	3,251 / 3,274	14,112 / 14,168	1,471 / 1,450	3,698 / 2,821	11,360 / 3,219	441, 421, 363, 367, 390, 344
5 ON	2,371 / 2,345	2,744 / 2,731	3,212 / 3,202	13,356 / 13,425	1,502 / 1,445	3,549 / 2,706	10,848 / 2,658	468, 456, 381, 355, 411, 376
6 OFF	2,799 / 2,780	2,956 / 2,963	3,252 / 3,256	14,053 / 14,102	1,460 / 1,426	3,623 / 2,820	11,402 / 3,214	416, 350, 333, 337, 374, 326
7 OFF	2,805 / 2,790	2,954 / 2,955	3,229 / 3,243	14,032 / 14,130	1,465 / 1,428	3,585 / 2,801	11,326 / 3,161	391, 364, 324, 349, 377, 305
8 ON	2,324 / 2,299	2,770 / 2,753	3,183 / 3,182	13,325 / 13,383	1,448 / 1,417	3,575 / 2,703	10,665 / 2,580	390, 364, 328, 342, 376, 306

Engine decode tok/s per run:

Arm	1,979	3,963	6,011	32,105	955	Natural, r1 / r2	Sampled, seeds 4242 / 4243 / 4244	Agent turn 1 / 2	Tiny chats (6)
1 OFF	172.2 / 173.5	170.9 / 171.3	125.3 / 168.7	163.4 / 161.8	161.2 / 172.8	80.9 / 97.5	98.3 / 94.2 / 100.2	94.8 / 103.5	69, 78, 85, 113, 112, 96
2 ON	170.2 / 170.6	169.3 / 169.6	123.0 / 165.7	163.4 / 164.2	159.7 / 171.8	86.7 / 108.1	89.8 / 97.9 / 100.6	95.6 / 89.7	67, 76, 78, 110, 105, 87
3 ON	168.8 / 169.8	168.0 / 169.0	119.2 / 164.9	163.2 / 162.0	157.9 / 170.4	85.9 / 103.5	92.3 / 106.5 / 106.7	91.4 / 95.2	69, 77, 76, 111, 101, 84
4 OFF	168.6 / 169.9	166.8 / 168.9	117.9 / 164.1	156.0 / 158.3	157.3 / 169.8	74.8 / 102.7	87.3 / 97.3 / 101.1	93.8 / 99.9	66, 65, 75, 107, 86, 93
5 ON	170.3 / 169.5	169.6 / 170.4	120.0 / 162.8	163.7 / 164.0	149.6 / 171.6	96.6 / 106.0	97.7 / 108.4 / 107.5	103.4 / 99.6	66, 64, 80, 106, 103, 87
6 OFF	169.6 / 172.1	169.8 / 170.6	123.3 / 166.1	164.8 / 165.0	159.9 / 172.1	94.3 / 112.2	100.4 / 115.6 / 105.7	101.1 / 104.6	70, 82, 83, 113, 116, 103
7 OFF	169.9 / 170.9	170.3 / 170.9	123.1 / 166.2	164.1 / 164.6	158.7 / 172.1	92.6 / 105.4	101.7 / 108.3 / 103.9	107.9 / 106.6	72, 79, 85, 110, 101, 92
8 ON	170.6 / 172.3	170.4 / 170.4	124.5 / 166.4	163.8 / 162.7	159.3 / 172.0	95.4 / 106.7	96.3 / 103.9 / 110.9	108.0 / 101.8	73, 85, 86, 107, 99, 93

Experts streamed per request (the server's per-request count), four arms per side:

Request	Off	On	Change of the median
1,979 tokens (run 2)	15,481 in all four	11,943, 11,854, 11,866, 11,875	-23.3%
3,963 tokens (run 2)	15,943 in all four	13,419, 13,436, 13,496, 13,401	-15.8%
6,011 tokens (run 2)	16,396 in all four	14,891, 14,883, 14,848, 14,889	-9.2%
32,105 tokens (run 2)	67,372 in all four	64,378, 64,482, 64,439, 64,332	-4.4%
955 tokens (run 2)	5,033, 5,030, 5,004, 5,057	5,037, 5,061, 5,013, 4,983	-0.1%
Natural text, run 1 / run 2 (3,828 tokens)	15,885 in all four	14,302 to 14,318 / 13,979 to 13,989	-9.9% / -12.0%
Sampled, seed 4242 (3,971 tokens)	15,943 in all four	13,326 to 13,342	-16.4%
Agent turn 1 (18,768 tokens)	51,846 to 51,980	48,805 to 49,037	-5.8%
Agent turn 2 (1,790 after 18,763 reused)	15,423 in all four	11,956 to 12,010	-22.3%

Unpaired prompt times from the identity arms (deterministic lane, one run each, not a benchmark), base / unset / on /
base again, ms: 1,046 tokens 2,948 / 2,933 / 2,030 / 2,963; 1,965 tokens 3,213 / 3,132 / 2,560 / 3,100; follow-up turn
3,049 / 3,049 / 2,486 / 2,957; 31,473 tokens 15,669 / 15,499 / 14,639 / 15,286; 126,970 tokens 60,480 / 60,615 / 57,034 /
60,734.

The earlier run: production settings and another base, 4 pairs

The same change before the rebase (identical lines in prefill.cpp, the header and the test), on v0.2.6 with two other
opt-in commits under it (#4119's STRATA_SELECT_TWO_PASS and a local asynchronous checkpoint, STRATA_CKPT_ASYNC, not
on main), measured on the same machine on 2026-10-26 with settings that are not the release defaults, in both arms:
STRATA_TOPK_SPLIT=1, STRATA_CKPT_POOL=1, STRATA_SELECT_TWO_PASS=1, STRATA_CKPT_ASYNC=1,
--conversation-cache-min-tokens 2048 and --conversation-cache-mib 8192. The same requests as above, two batches (OFF
ON ON OFF, then ON OFF OFF ON). Absolute numbers are not comparable with the table above; compare the ratios.

Measure	Off	On	Four pairs	Median	Change
Follow-up turn read, ms	3,336.5	2,747.5	1.234, 1.180, 1.269, 1.176	1.207	-17.7% (time)
First turn, 18,768 tokens, ms	11,053.8	10,435.0	1.044, 1.065, 1.065, 1.063	1.064	-5.6% (time)
Prompt read, 1,979 tokens, tok/s	779.8	937.4	1.229, 1.208, 1.191, 1.181	1.199	+20.2%
Prompt read, 3,963 tokens	1,451.7	1,572.2	1.090, 1.084, 1.078, 1.080	1.082	+8.3%
Prompt read, 6,011 tokens	2,010.0	2,042.5	1.013, 1.018, 1.014, 1.020	1.016	+1.6%
Prompt read, 32,105 tokens	2,327.1	2,443.9	1.042, 1.046, 1.063, 1.050	1.048	+5.0%
Prompt read, natural text	1,197.8	1,228.4	1.032, 1.019, 1.026, 1.025	1.026	+2.6%
Prompt read, 955 tokens	795.1	793.4	1.004, 1.008, 0.983, 0.998	1.001	-0.2%
Tiny chats, prompt ms (median of 6)	130.5	130.4	1.000, 0.954, 0.996, 1.058	0.998	-0.1% (time)
Decode after 1,979 / 3,963 / 6,011 / 32,105 / 955 tokens	173.1 / 171.6 / 168.1 / 165.2 / 173.3	172.6 / 171.2 / 168.0 / 164.6 / 173.0	medians 0.998 / 0.998 / 1.000 / 0.996 / 0.999		-0.4% to 0.0%
Decode, tiny chats	87.5	88.8	1.012, 1.043, 1.002, 1.003	1.008	+1.5%
Decode, natural text	100.5	100.5	1.027, 0.995, 1.021, 0.961	1.008	+0.1%
Decode, sampled / agent (texts differ)			medians 0.988 / 0.978

Pairs: arms 2 and 1, 3 and 4 of the first batch, 1 and 2, 4 and 3 of the second. Follow-up turn: percentile bootstrap
95% interval 1.176 to 1.269.

Failed:

• The 128K identity step, as a check of the default path: the control failed. The two base arms (main, the same
  binary) gave different replies to the 126,970-token prompt (47f66830 and 1fc92b9e) and different STATE_HASH
  lines after decode, with the same prompt-path residuals and counts; the first base arm logged another decode
  expert-cache hit count (33,733 against 32,948 in the other three arms). So byte identity at 128K is not established
  for the default path. The head with the switch unset and with it set both equal the second base arm byte for byte,
  and the prompt-path residuals are identical in all four arms.
• A rule I set before the timing batch (the follow-up turn's median pair ratio at least 1.02 with its bootstrap interval
  above 1.0, and no greedy decode or prompt-read tier with a median pair ratio below 0.98) held except for one tier: the
  tiny chats' decode, 0.9797. Those chats (17 to 19 prompt tokens) never reach the switch's code, their replies differ
  between arms of the same binary at these settings, and the same tier was 1.008 in the earlier run.

Not run:

• Running on AMD, Intel or Apple GPUs: the builds above only. On Metal the walk is compiled but never reached (the Metal
  window reads the prompt); on HIP it would run where the CUDA build runs it, and that is unverified on hardware.
• Windows (MSVC) builds; execution on any card but sm_86; a second machine.
• Timing beyond 32K on main (no 128K or 256K pairs; in the unpaired identity arms the switch read 126,970 tokens in
  5.9% less time, one run each), any depth other than 16, and the switch together with STRATA_PREFILL_STREAM_MIN.
• Output identity in the default lane (adaptive CPU share and PCIe split): replies differ between runs of the same binary
  there, so the identity runs pin both. Task-quality or long-context recall checks (the replies were byte-identical).
• The fused prompt path, the peer prompt ring and the inline issuer with the switch set (it skips nothing or is off there
  by construction).

Limits

• The skip reaches only entries not yet issued when their layer's routing lands, so the saving shrinks as prompts grow:
  on main 42% of the streamed experts at 1,046 tokens, 14% at 4,867, 6% at 14,277 (two chunks), 3% at 31,473 and 2% at
  126,970.
• Entries staged through the Stager's pinned buffers are never skipped. Here the experts come from the pinned arena; a
  host under memory pressure would skip less.
• The default depth (16) is not tuned, and the cudaEventQuery poll's host cost (one driver call per copy) was not
  measured on its own; a host whose CPUs are busy could see it.
• Identity rests on the refill rewriting the whole loan: on this pack every blob is a multiple of 256 bytes and the cache
  has no guard, so no slack is left inside the loan. Where a slot does have slack past its blob, those bytes hold
  prompt-dependent leftovers today as well, but this PR does not prove that no kernel reads them.
• The adaptive tier and the PCIe split learn from timing, so a faster read can change their later decisions; the identity
  runs pin both.
• One GPU and one model; a prompt-read claim, not a throughput claim. [Engine] prefill: opt-in sparse copied events and deduplicated slot waits (Windows HIP prompt reads +36% at 4K, +15% at 32K) #4005 and this PR need one rule when both are set
  (see Related discussion).