Skip to content

cuda: use ncols2 = 2 in the MMA flash attention for prompt batches at GQA ratio 6 - #410

Open
sb32445 wants to merge 1 commit into
PrismML-Eng:prismfrom
sb32445:pr/fattn-gqa6-prefill
Open

sb32445 wants to merge 1 commit into
PrismML-Eng:prismfrom
sb32445:pr/fattn-gqa6-prefill

Conversation

@sb32445

@sb32445 sb32445 commented Oct 27, 2026 •

Copy link
Copy Markdown

Overview

With a GQA ratio of 6 (Bonsai 2 27B and Qwen3.5: 24 Q heads on 4 KV heads) the MMA flash attention dispatcher picks ncols2 = 8, so two of the eight head slots of every MMA tile are padding (25 % of the MMA work of the attention kernel). For prompt batches (Q->ne[1] >= 64) ncols2 = 2 computes without padding, and the repeated K/V loads cost less than the saved MMA work. Decode and MTP verify (few tokens) keep ncols2 = 8; without the token threshold decode lost 12 % at 32k depth and 26 % at 96k, so the threshold matters.

RTX 4070, Bonsai 2 27B PTQ1_0 + MTP head, q4_0 K/V, against plain prism 6447906bf: llama-bench pp2048 (-ub 512) +7.3 % at depth 32768 and +14.8 % at depth 98304. Depth 0 (+0.3 %) and decode (+0.00 %, identical outputs) are unchanged.

Additional information

  • Seven added lines in fattn.cu (one if before the existing gqa_ratio > 4 branch) plus 54 new FLASH_ATTN_EXT test cases (head size 256, GQA ratio 6, 1 to 512 query tokens, f16/q8_0/q4_0 K/V).
  • Update since the last description: re-based from c96baf4ab to 6447906bf. Two conflicts, both resolved by keeping both sides: in fattn.cu prism added an AMD WMMA block before the place of this if (the new if now follows it; on AMD WMMA GPUs the new block returns first and already picks ncols2 = 2 for ratio 6 at every batch size, so this PR only changes the NVIDIA path there; I did not test AMD), and in test-backend-ops.cpp the new head-size test cases sit next to prism's new asymmetric head-dim cases. All numbers below are re-measured on the new base.
  • Only ratio 6 and only Q->ne[1] >= 64 are affected; every other model/shape takes the old path.
  • Not bit-identical for large batches (different accumulation split): batches below 64 tokens are identical, large batches change by float rounding. KL divergence against the unpatched kernel is 0.000755 ± 0.000014 (-b 2048 -ub 512, 16 x 512 tokens), same top token 98.5 ± 0.19 %, 99th percentile KLD 0.004253; perplexity 14.5267 against 14.5423 (error 0.62).
  • Decode kernels are not touched. In the MTP decode benchmark the outputs are identical and the speed is unchanged (+0.00 %, CI [-0.05, +0.05] %).
  • Tuned and measured on one GPU (RTX 4070, Ada, cc 8.9). The 64-token threshold was chosen from a decode/prefill trade-off on this GPU; other architectures (Hopper, Blackwell, RDNA) may want other values. I can restrict it to a compute capability if you prefer.
  • Together with cuda: scale PTQ1_0 MMQ results once per 128-weight block #411 (the PTQ1_0 MMQ scale): the two prefill gains add up roughly, measured on both branches merged: +4.8 % at depth 0, +11.0 % at 32768, +17.2 % at 98304 (the sum of the single gains is +4.4 / +10.2 / +16.5 %). The two PRs touch different files and apply independently.

Test results

  • Hardware / software: RTX 4070 12 GB (AD104, cc 8.9), Linux 6.18, NVIDIA driver 620.70, CUDA 13.4, GCC 16.2; Release build, -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89.
  • Base: prism at 6447906bf, this commit only (separate builds of base and patch).
  • Model: Ternary-Bonsai-2-27B (PTQ1_0) with a community MTP draft head (Q8_0), q4_0 K/V cache.
  • Prefill (llama-bench -p 2048 -n 0 -ub 512 -d 0,32768,98304 -r 2, two runs per build in the order base, patch, patch, base):
Depth base ↑ patch ↑ change
0 1354.9 1359.4 +0.3 %
32768 970.7 1041.3 +7.3 %
98304 593.3 680.8 +14.8 %

(tok/s, mean of two runs; run-to-run spread ≤ 0.3 %.) The server prefill of 114688 tokens (-9.0 % on the previous base) was not repeated.

  • Decode (MTP --spec-draft-n-max 2, 4 greedy prompts x 256 tokens, 8 interleaved A/B pairs, 95 % bootstrap interval): 115.15 -> 115.15 tok/s, +0.00 % (CI [-0.05, +0.05] %), outputs identical, acceptance 73.5 % in both.
  • Quality: perplexity (German text, 16 x 512): 14.5423 -> 14.5267; KL divergence against base 0.000755 ± 0.000014.
  • test-backend-ops test -b CUDA0 (full suite): 20780/20780 passed.
  • Not tested: other GPUs, other GQA ratios, other models, several slots, multi-GPU.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: The patches were developed with Claude Code (Anthropic's coding agent): it wrote the code, the measurement scripts and the first drafts of the commit messages and PR texts. I decided what to work on (which kernels to optimise, based on profiles of my own setup). The measurements and checks listed above were run in the Claude Code sessions; I did not re-run them independently. I will maintain the changes. Commits where Claude Code was used carry a Co-Authored-By trailer.

🤖 Generated with Claude Code

@bri-prism

Copy link
Copy Markdown
Collaborator

This now conflicts with prism. Could you rebase it? I will review once it applies cleanly.

… GQA ratio 6

With a GQA ratio of 6 (Bonsai 2 27B and Qwen3.5: 24 Q heads on 4 KV heads) the dispatcher picks ncols2 = 8, so
two of the eight head slots of every MMA tile are padding, which is 25 % of the MMA work of the attention kernel.
For many query tokens (Q->ne[1] >= 64, prompt processing) ncols2 = 2 computes without padding and the repeated
K/V loads cost less than the saved MMA work. Decode and MTP verify (few tokens) keep ncols2 = 8; in a first trial
without the token threshold decode lost 12 % at 32k depth and 26 % at 96k.

RTX 4070, Ternary-Bonsai-2-27B PTQ1_0 + MTP head, q4_0 K/V, base prism c96baf4, this commit alone:
llama-bench pp2048 (-ub 512, 2 runs each, A B B A) 1363.1 -> 1361.3 tok/s at depth 0 (-0.1 %), 960.7 -> 1027.7
at depth 32768 (+7.0 %), 576.8 -> 661.7 at depth 98304 (+14.7 %); llama-server prefill of 114688 tokens
157.4 -> 143.2 s (-9.0 %). Decode kernels are unchanged: MTP decode in 8 interleaved A/B pairs 115.97 -> 115.98
tok/s (+0.00 %, CI [-0.04, +0.03] %), outputs identical.
Batches below 64 tokens are bit-identical; large batches change by float rounding: KL divergence against the
previous kernel 0.00077 (16x512) and 0.00071 (4096 tokens per batch), perplexity 14.5263 against 14.5316.
tests: flash attention cases for GQA ratio 6 with head size 256, 1 to 512 query tokens, f16/q8_0/q4_0 K/V
(FLASH_ATTN_EXT 4401/4401 on CUDA0).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@sb32445

sb32445 commented Nov 16, 2026

Copy link
Copy Markdown
Author

Rebased on current prism (6447906bf, was c96baf4ab), as requested. Two conflicts, both resolved by keeping both sides: in fattn.cu prism now has an AMD WMMA block in front of the place of this if (on AMD WMMA GPUs it returns first and already picks ncols2 = 2 for ratio 6, so this PR only changes the NVIDIA path there; I did not test AMD), and in test-backend-ops.cpp the new test cases sit next to the new asymmetric head-dim cases.

Re-measured against plain prism 6447906bf (RTX 4070, Bonsai 2 27B PTQ1_0, q4_0 K/V): llama-bench pp2048 -ub 512 at depth 0 / 32768 / 98304: +0.3 / +7.3 / +14.8 % (was +7.0 / +14.7 % at 32768 / 98304 on the older base); decode with MTP +0.00 % (identical outputs); KL divergence 0.000755 ± 0.000014, same top token 98.5 %; full test-backend-ops test -b CUDA0: 20780/20780. Together with #411 (both branches merged) the prefill gain is +4.8 / +11.0 / +17.2 %. The server prefill of 114688 tokens was not repeated.

The description has the new numbers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants