Repository navigation
Conversation
|
Turing data point, since the final commit hardcodes the 4070 tuning (50% / 46 CTAs / 16 MiB): on a 4 MB-L2 card it is a loss. RTX 2060 SUPER (sm_75, 4 MB L2, 34 SMs), stock clocks, Bonsai 2 27B PTQ1_0, q4_0 K/V, no MTP head, greedy, 2 prompts x 400 tokens per arm, server restarted per arm, three alternated runs per setting (first commit
Round-to-round spread with prefetch off was under 0.1 tok/s (~0.2%). Greedy output byte-identical in every arm, as expected for a prefetch. 2048 KB and 2048 KB with 34 CTAs landed between (+0.7..1.4%). So the byte cap wants to scale with the device: a quarter of |
|
@professorpalmer your 2060 numbers are right and it was my mistake: I tuned the 16 MiB cap and the 46 CTAs on the 4070 only and never tried a card with a small L2. Also, the description says "48 MB L2" for the 4070, it is 36 MiB. I repeated the tests on the 4070 (full kTrain stack, MTP n-max 2, depth 0, 12 pairs each against the old setting, outputs identical in every run). Byte cap with one prefetching CTA per SM:
So on this card 8 and 16 MiB are the same and smaller caps lose a little. On the PR branch alone, small caps looked slightly better (2 MiB +0.48 % against 16 MiB); I do not know why the sign flips on the full stack, so I trust the stack numbers. The best cap depends on the GPU (4070: 36 MiB L2, 2060 SUPER: 4 MiB), and I can only measure the 4070. So the new commit uses 8 MiB only on Ada (cc 8.9) and 2 MiB on every other GPU, with one prefetching CTA per SM (46 on the 4070, so no change there). The 2 MiB is your number and not measured by me. With this commit the 4070 is +0.02 % against the old setting (12 pairs, n.s.), The measurements and scripts were run with Claude Code (AI assistance); I can post the exact commands if useful. |
|
@sb32445 I ran the test again on the RTX 2060 SUPER with your new commit. The 2 MiB cap gives no gain on this card. Two sm_75 builds. The only difference between them is the prefetch (
Bonsai 2 27B PTQ1_0, q4_0 K/V, no MTP head,
The difference is in the run-to-run noise. Turning the prefetch off outside Ada is correct for this card. Thank you for the fix and for the 4070 data. |
e697877 to
dce8d99
Compare
…nto L2 from the last CTAs ktrain patch 0010 (PR PrismML-Eng#314), applied from pr/ptq1-l2-prefetch with the GGML_CUDA_L2_PREFETCH_* switches removed: the node loop sets a thread-local hint (next PTQ1_0 mat-vec, gate/up pairs counted once), the PTQ1_0 mat-vec launcher turns it into 50 % (max 16 MiB) of the next tensor and the last 46 CTAs issue prefetch.global.L2. The code sits in the dedicated PTQ1_0 kernel, separate from upstream's DGX Spark prefetch in the generic mmvq loop. Conflicts were two include/declaration lines only. mtp_bench (q4_0, 8192, n-max 2): all 8 output hashes identical to the commit before. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nto L2 from the last CTAs ktrain patch 0010 (PR PrismML-Eng#314), applied from pr/ptq1-l2-prefetch with the GGML_CUDA_L2_PREFETCH_* switches removed: the node loop sets a thread-local hint (next PTQ1_0 mat-vec, gate/up pairs counted once), the PTQ1_0 mat-vec launcher turns it into 50 % (max 16 MiB) of the next tensor and the last 46 CTAs issue prefetch.global.L2. The code sits in the dedicated PTQ1_0 kernel, separate from upstream's DGX Spark prefetch in the generic mmvq loop. Conflicts were two include/declaration lines only. mtp_bench (q4_0, 8192, n-max 2): all 8 output hashes identical to the commit before. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
bri-prism
left a comment
There was a problem hiding this comment.
Agent review: posted by the maintainer's coding agent at their request.
You asked about Hopper. Here's an H200 data point (sm_90, CUDA 12.8), this diff applied to prism 73bfe6c, Ternary-Bonsai-2-9B PTQ1_0:
- test-backend-ops: MUL_MAT ptq1_0, 180/180.
- llama-bench tg128: neutral. Base 342.0 / 346.2 vs PR 345.4 / 345.2 tok/s (A-B-A-B). pp512 is also unchanged apart from one outlier run.
- llama-server, 4 prompts × 128 greedy tokens: tokens identical to stock prism; decode -0.4% to +0.1%.
This is consistent with your guess: on sm_90 the launches already go through PDL and DRAM stays busy across kernel boundaries, so the prefetch doesn't add anything there, but it doesn't cost anything either. Gating it to cc 8.9 (or below sm_90) would keep the H200 path exactly as it is now, but leaving it on everywhere is also fine on this evidence.
…nto L2 from the last CTAs ktrain patch 0010 (PR PrismML-Eng#314), applied from pr/ptq1-l2-prefetch with the GGML_CUDA_L2_PREFETCH_* switches removed: the node loop sets a thread-local hint (next PTQ1_0 mat-vec, gate/up pairs counted once), the PTQ1_0 mat-vec launcher turns it into 50 % (max 16 MiB) of the next tensor and the last 46 CTAs issue prefetch.global.L2. The code sits in the dedicated PTQ1_0 kernel, separate from upstream's DGX Spark prefetch in the generic mmvq loop. Conflicts were two include/declaration lines only. mtp_bench (q4_0, 8192, n-max 2): all 8 output hashes identical to the commit before. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…mML-Eng#314 open Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pn9UL41f3M8tRbqcHXFuB8
…mML-Eng#314 open Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pn9UL41f3M8tRbqcHXFuB8
dce8d99 to
3dcfc88
Compare
|
Rebased on current |
…t CTAs Every PTQ1_0 mat-vec in the decode graph pays about 2.5 us of ramp-up and ramp-down in which the DRAM is not busy. The node loop now looks ahead for the next PTQ1_0 MUL_MAT (a gate/up pair feeding one GLU counts as one op) and hands its weight pointer to the launcher. The last 46 CTAs of the running kernel issue prefetch.global.L2 for the first 50 % (at most 16 MiB) of those weights right after their own loads. More CTAs or more bytes compete with the running kernel and get slower (184 CTAs: -2.3 %). Values are not changed. GGML_CUDA_L2_PREFETCH_PCT=0 turns it off; _CTAS and _MAX_KB tune it. Decode with MTP n-max 2, same binary, outputs identical: +2.00 % (greedy benchmark), +2.10 % (Hermes setup, ctx 114688), +2.2 / +2.1 / +1.7 % at depth 0 / 16k / 65k. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
GGML_CUDA_L2_PREFETCH_PCT, _MAX_KB and _CTAS of the previous commit were only there to tune and measure the change. The measured values (50 % of the next tensor, at most 16 MiB, 46 CTAs) are now constants. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The cap of 16 MiB and the fixed 46 CTAs were tuned on an RTX 4070 only. On an RTX 2060 SUPER (4 MiB L2, 34 SMs) that setting is 2.4 to 2.7 % slower than no prefetch (measured by a reviewer, see the PR). Prefetch with one CTA per SM, and cap the bytes at 8 MiB on Ada (cc 8.9) and at 2 MiB on every other GPU. RTX 4070 (36 MiB L2), MTP n-max 2, same outputs in all runs, 12 pairs against the previous setting: 8 MiB +0.02 % (n.s.), 16 MiB -0.03 % (n.s.), 4 MiB -0.17 %, 2 MiB -0.40 %. The 2 MiB default is not measured on other GPUs. The 4070 has 46 SMs, so one CTA per SM is the old count. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
3dcfc88 to
09cedb5
Compare
|
Rebased on current Re-measured on the RTX 4070 (Bonsai 2 27B PTQ1_0 + MTP head, n-max 2, q4_0 K/V, 8 interleaved pairs, outputs identical in all runs, KL divergence against
(My re-measurement on |
Overview
Every PTQ1_0 mat-vec in the decode graph pays about 2.5 us of ramp-up and ramp-down in which DRAM is idle (399 mat-vecs per verify step). The CUDA node loop now looks ahead for the next PTQ1_0
MUL_MAT(a gate/up pair that feeds one GLU counts as one op), stores its weight pointer and size in a thread-local hint (l2-hint.cuh), and the launcher ofmul_mat_vec_ptq1_0_ptpasses it to the kernel. The last 46 CTAs issueprefetch.global.L2for the first 50 % (at most 16 MiB) of those weights right after their own loads. Values are not changed.RTX 4070, Bonsai 2 27B PTQ1_0 + MTP head (n-max 2), q4_0 K/V, 8 interleaved A/B pairs, outputs identical in all runs:
prism6447906bf(two builds): decode 115.17 -> 116.56 tok/s, +1.21 % (95 % CI [+1.16, +1.25] %);GGML_CUDA_L2_PREFETCH_PCT=0, first commit of the branch) against the default: 114.73 -> 116.67 tok/s, +1.70 % (CI [+1.65, +1.76] %).The last measurement on the previous base (
prism2459f68b5, same switch method) gave +2.00 % (greedy benchmark) and +2.10 % (agent-like setup, context 114688); I have not repeated the agent-like setup and the depth series on the new base. The gain got smaller with the base; I did not find out why (other changes inprismsince then, or a different reference).Additional information
5628c7338to6447906bfwithout conflicts. Perplexity and KL againstprismare unchanged: KLD 0.000000, same top token 100 % (llama-perplexity -b 2048 -ub 512, 16 x 512 tokens).GGML_CUDA_L2_PREFETCH_PCT(=0turns it off),_CTASand_MAX_KB(defaults 50 / 46 / 16384) that were used for the tuning, the second fixes the measured values as constants and removes the switches, the third caps the prefetch at 8 MiB on Ada and 2 MiB elsewhere. To reproduce the switch measurement, build the first commit.prism(114.73 against 115.2 tok/s, different builds, I did not isolate whether the look-ahead code itself costs anything).ptxas:griddepcontrolrequires sm_90).ggml_cuda_kernel_launchalready opts into PDL there, so the idle window this PR fills may already be covered on those GPUs; I did not test whether the prefetch helps or hurts there.thread_localset by the node loop before each node and read by the launcher; the kernel receives the pointer, the number of lines per CTA and the number of prefetching CTAs as three extra arguments. The pointer and byte count come from the next PTQ1_0 weight tensor (never beyond the tensor). The "last CTAs" are assumed to be the ones that run last, which the hardware does in practice but does not guarantee; if not, only the speed-up shrinks.test-backend-ops test -b CUDA0run on this branch: 20726/20726 passed.Test results
-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89.prismat6447906bf.--spec-type draft-mtp --spec-draft-n-max 2, q4_0 K/V cache, one slot.2459f68b5/88c4bc60b, not repeated): prefetched fraction / prefetching CTAs 60 % / 92: +0.6 %; 60 % / 184: -2.27 %; 30 % / 46: +1.80 %; 50 % / 46: +2.02 %; 70 % / 46: +2.00 %; 100 % / 46: +1.78 %; depth 0 / 16384 / 65536: +2.2 / +2.1 / +1.7 % (one run each).data), several slots, models other than PTQ1_0 (only this kernel reads the hint).Requirements
🤖 Generated with Claude Code