Repository navigation
Conversation
979e2bd to
de1ed8b
Compare
|
This now conflicts with prism. Could you rebase it? I will review once it applies cleanly. |
… GQA ratio 6 With a GQA ratio of 6 (Bonsai 2 27B and Qwen3.5: 24 Q heads on 4 KV heads) the dispatcher picks ncols2 = 8, so two of the eight head slots of every MMA tile are padding, which is 25 % of the MMA work of the attention kernel. For many query tokens (Q->ne[1] >= 64, prompt processing) ncols2 = 2 computes without padding and the repeated K/V loads cost less than the saved MMA work. Decode and MTP verify (few tokens) keep ncols2 = 8; in a first trial without the token threshold decode lost 12 % at 32k depth and 26 % at 96k. RTX 4070, Ternary-Bonsai-2-27B PTQ1_0 + MTP head, q4_0 K/V, base prism c96baf4, this commit alone: llama-bench pp2048 (-ub 512, 2 runs each, A B B A) 1363.1 -> 1361.3 tok/s at depth 0 (-0.1 %), 960.7 -> 1027.7 at depth 32768 (+7.0 %), 576.8 -> 661.7 at depth 98304 (+14.7 %); llama-server prefill of 114688 tokens 157.4 -> 143.2 s (-9.0 %). Decode kernels are unchanged: MTP decode in 8 interleaved A/B pairs 115.97 -> 115.98 tok/s (+0.00 %, CI [-0.04, +0.03] %), outputs identical. Batches below 64 tokens are bit-identical; large batches change by float rounding: KL divergence against the previous kernel 0.00077 (16x512) and 0.00071 (4096 tokens per batch), perplexity 14.5263 against 14.5316. tests: flash attention cases for GQA ratio 6 with head size 256, 1 to 512 query tokens, f16/q8_0/q4_0 K/V (FLASH_ATTN_EXT 4401/4401 on CUDA0). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
de1ed8b to
3aed882
Compare
|
Rebased on current Re-measured against plain The description has the new numbers. |
Overview
With a GQA ratio of 6 (Bonsai 2 27B and Qwen3.5: 24 Q heads on 4 KV heads) the MMA flash attention dispatcher picks
ncols2 = 8, so two of the eight head slots of every MMA tile are padding (25 % of the MMA work of the attention kernel). For prompt batches (Q->ne[1] >= 64)ncols2 = 2computes without padding, and the repeated K/V loads cost less than the saved MMA work. Decode and MTP verify (few tokens) keepncols2 = 8; without the token threshold decode lost 12 % at 32k depth and 26 % at 96k, so the threshold matters.RTX 4070, Bonsai 2 27B PTQ1_0 + MTP head, q4_0 K/V, against plain
prism6447906bf:llama-benchpp2048 (-ub 512) +7.3 % at depth 32768 and +14.8 % at depth 98304. Depth 0 (+0.3 %) and decode (+0.00 %, identical outputs) are unchanged.Additional information
fattn.cu(oneifbefore the existinggqa_ratio > 4branch) plus 54 newFLASH_ATTN_EXTtest cases (head size 256, GQA ratio 6, 1 to 512 query tokens, f16/q8_0/q4_0 K/V).c96baf4abto6447906bf. Two conflicts, both resolved by keeping both sides: infattn.cuprismadded an AMD WMMA block before the place of thisif(the newifnow follows it; on AMD WMMA GPUs the new block returns first and already picksncols2 = 2for ratio 6 at every batch size, so this PR only changes the NVIDIA path there; I did not test AMD), and intest-backend-ops.cppthe new head-size test cases sit next toprism's new asymmetric head-dim cases. All numbers below are re-measured on the new base.Q->ne[1] >= 64are affected; every other model/shape takes the old path.-b 2048 -ub 512, 16 x 512 tokens), same top token 98.5 ± 0.19 %, 99th percentile KLD 0.004253; perplexity 14.5267 against 14.5423 (error 0.62).Test results
-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89.prismat6447906bf, this commit only (separate builds of base and patch).llama-bench -p 2048 -n 0 -ub 512 -d 0,32768,98304 -r 2, two runs per build in the order base, patch, patch, base):(tok/s, mean of two runs; run-to-run spread ≤ 0.3 %.) The server prefill of 114688 tokens (-9.0 % on the previous base) was not repeated.
--spec-draft-n-max 2, 4 greedy prompts x 256 tokens, 8 interleaved A/B pairs, 95 % bootstrap interval): 115.15 -> 115.15 tok/s, +0.00 % (CI [-0.05, +0.05] %), outputs identical, acceptance 73.5 % in both.test-backend-ops test -b CUDA0(full suite): 20780/20780 passed.Requirements
🤖 Generated with Claude Code