Skip to content

cuda: scale PTQ1_0 MMQ results once per 128-weight block - #411

Open
sb32445 wants to merge 1 commit into
PrismML-Eng:prismfrom
sb32445:pr/ptq1-mmq-block-scale
Open

sb32445 wants to merge 1 commit into
PrismML-Eng:prismfrom
sb32445:pr/ptq1-mmq-block-scale

Conversation

@sb32445

@sb32445 sb32445 commented Oct 27, 2026 •

Copy link
Copy Markdown

Overview

The PTQ1_0 weight scale is one value per 128 weights, so it is the same for the four 32-wide MMA steps of one vec_dot call (the tile loader writes the block scale into all four x_df entries). mul_mat_q for PTQ1_0 uses the generic Q8_0 routine, which scales every MMA result with dA and dB separately (I2F, FMUL, FFMA per element and step); that was 62 % of the executed instructions of the kernel (profile on an earlier base). A PTQ1_0 routine accumulates the four steps with dB and applies dA once. Prefill (batches that go through MMQ) gets faster; decode uses other kernels and is not touched.

RTX 4070, Bonsai 2 27B PTQ1_0, against plain prism 6447906bf: llama-bench pp2048 (-ub 512) +4.1 % at depth 0, +2.9 % at 32768, +1.8 % at 98304.

Additional information

  • One new function ggml_cuda_mmq_vec_dot_ptq1_0_q8_1_mma in mmq-vec-dot.cuh (+92 lines) and a one-line change in mmq.cuh that selects it for GGML_TYPE_PTQ1_0.
  • Update since the last description: re-based from c96baf4ab to 6447906bf. prism changed the MMQ helpers in between: ggml_cuda_mmq_get_sram_stride and ggml_cuda_mmq_get_rows_per_warp take the source precision now, and ggml_cuda_mmq_write_back_mma takes prec_src1 and a fusion flag. The new routine calls them with GGML_PREC_Q8 as the generic Q8_0 routine does, and the mmq.cuh line passes prec_src1, false like the neighbouring cases. All numbers below are re-measured on the new base.
  • Not bit-identical: the summation order changes (scale applied once per 128 weights instead of per 32), so results differ by float rounding. KL divergence against the previous kernel is 0.000924 ± 0.000065 (-b 2048 -ub 512, 16 x 512 tokens), same top token 98.31 ± 0.20 %, 99th percentile KLD 0.005035; perplexity 14.5356 against 14.5423 (error 0.62).
  • Decode with MTP: the decode kernels are unchanged, but the rounding of the prefill changes the KV values and with that sometimes the MTP drafts. In my 4-prompt benchmark acceptance went 73.5 -> 72.1 % and decode 115.20 -> 114.12 tok/s (-0.88 %, CI [-0.94, -0.83] %); the greedy outputs of 2 of 4 prompts differ from the base. I believe this is draft noise and not a kernel effect, but I cannot prove it; the sign depends on the prompt (on the previous base: +0.9 % at 2k, 0.0 % at 32k, -1.0 % at 114k depth).
  • The gain is moderate; if the changed rounding and the decode noise outweigh it for you, I understand if you do not want it.
  • Together with cuda: use ncols2 = 2 in the MMA flash attention for prompt batches at GQA ratio 6 #410 (the GQA 6 prefill dispatch): the two prefill gains add up roughly, measured on both branches merged: +4.8 % at depth 0, +11.0 % at 32768, +17.2 % at 98304 (sum of the single gains +4.4 / +10.2 / +16.5 %); KL divergence of the merged branch against prism 0.000938 ± 0.000051, same top token 98.06 %. The two PRs touch different files and apply independently.
  • Tuned and measured on one GPU (RTX 4070, Ada, cc 8.9); I did not test other architectures.

Test results

  • Hardware / software: RTX 4070 12 GB (AD104, cc 8.9), Linux 6.18, NVIDIA driver 620.70, CUDA 13.4, GCC 16.2; Release build, -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89.
  • Base: prism at 6447906bf, this commit only (separate builds of base and patch).
  • Model: Ternary-Bonsai-2-27B (PTQ1_0) with a community MTP draft head (Q8_0), q4_0 K/V cache.
  • Prefill (llama-bench -p 2048 -n 0 -ub 512 -d 0,32768,98304 -r 2, two runs per build in the order base, patch, patch, base):
Depth base ↑ patch ↑ change
0 1354.9 1410.2 +4.1 %
32768 970.7 998.9 +2.9 %
98304 593.3 603.9 +1.8 %

(tok/s, mean of two runs; run-to-run spread ≤ 0.2 %.) The server prefill of 114688 tokens (-2.1 % on the previous base) was not repeated.

  • Decode (MTP --spec-draft-n-max 2, 4 greedy prompts x 256 tokens, 8 interleaved A/B pairs, 95 % bootstrap interval): 115.20 -> 114.12 tok/s, -0.88 % (CI [-0.94, -0.83] %), acceptance 73.5 -> 72.1 %, outputs differ (2 of 4 prompts).
  • Quality: perplexity (German text, 16 x 512): 14.5423 -> 14.5356; KL divergence against base 0.000924 ± 0.000065.
  • test-backend-ops test -b CUDA0 (full suite): 20725/20726 passed. The one failure is MUL_MAT pq2_0, m=1, n=1, k=1024 (ERR 0.00063 against 0.0005), which also fails now and then on plain prism (4 of 25 repeated runs, random inputs) and does not involve this change.
  • Not tested: other GPUs, multi-GPU, other PTQ1_0 models.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: The patches were developed with Claude Code (Anthropic's coding agent): it wrote the code, the measurement scripts and the first drafts of the commit messages and PR texts. I decided what to work on (which kernels to optimise, based on profiles of my own setup). The measurements and checks listed above were run in the Claude Code sessions; I did not re-run them independently. I will maintain the changes. Commits where Claude Code was used carry a Co-Authored-By trailer.

🤖 Generated with Claude Code

@bri-prism

Copy link
Copy Markdown
Collaborator

This now conflicts with prism. Could you rebase it? I will review once it applies cleanly.

The PTQ1_0 weight scale is one value per 128 weights, so it is the same for the four 32-wide MMA steps of one
vec_dot call (the tile loader writes the block scale into all four x_df entries). The generic Q8_0 routine scaled
every MMA result with dA and dB separately (I2F, FMUL, FFMA per element and step), which was 62 % of the executed
instructions of mul_mat_q. The new PTQ1_0 routine accumulates the four steps with dB and applies dA once.

RTX 4070, Ternary-Bonsai-2-27B PTQ1_0 + MTP head, q4_0 K/V, base prism c96baf4, this commit alone:
llama-bench pp2048 (-ub 512, 2 runs each, A B B A) 1363.1 -> 1412.1 tok/s at depth 0 (+3.6 %), 960.7 -> 988.7 at
depth 32768 (+2.9 %), 576.8 -> 586.9 at depth 98304 (+1.8 %); llama-server prefill of 114688 tokens
157.4 -> 154.1 s (-2.1 %). An earlier measurement of this change on another base: mul_mat_q instructions -15 %,
registers 254 -> 220. Decode kernels are not touched.
The summation order changes, so results are not bit-identical to the previous kernel: KL divergence against it
0.00090 (16x512) and 0.00082 (4096 tokens per batch), perplexity 14.5351 against 14.5316. Through the prompt's
KV values this can change MTP drafts: in the 4-prompt decode benchmark acceptance went 74.0 -> 72.0 % and decode
115.98 -> 115.05 tok/s (-0.80 %, CI [-0.82, -0.77] %, 8 pairs), sign depends on the prompt.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@sb32445

sb32445 commented Nov 16, 2026

Copy link
Copy Markdown
Author

Rebased on current prism (6447906bf, was c96baf4ab), as requested. Besides the conflict in mmq.cuh, prism changed the MMQ helpers in between, which broke the build of the old patch: ggml_cuda_mmq_get_sram_stride and ggml_cuda_mmq_get_rows_per_warp take the source precision now, and ggml_cuda_mmq_write_back_mma takes prec_src1 and a fusion flag. The new PTQ1_0 routine calls them like the generic Q8_0 routine (GGML_PREC_Q8, and prec_src1, false in the mmq.cuh line).

Re-measured against plain prism 6447906bf (RTX 4070, Bonsai 2 27B PTQ1_0, q4_0 K/V): llama-bench pp2048 -ub 512 at depth 0 / 32768 / 98304: +4.1 / +2.9 / +1.8 % (was +3.6 / +2.9 / +1.8 %); decode with MTP -0.88 % (was -0.80 %; draft acceptance 73.5 -> 72.1 %, the same cause as before, the greedy outputs of 2 of 4 prompts differ); KL divergence 0.000924 ± 0.000065, same top token 98.3 %; full test-backend-ops test -b CUDA0: 20725/20726 (the one failure is a pq2_0 MUL_MAT case that also fails now and then on plain prism). Together with #410 the prefill gain is +4.8 / +11.0 / +17.2 %.

The description has the new numbers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants