Skip to content

cuda: fuse SwiGLU + sign flip + Hadamard (and the PTQ1_0 MMQ q8_1 quantization) for prompt batches - #445

Open
sb32445 wants to merge 1 commit into
PrismML-Eng:prismfrom
sb32445:pr/swiglu-hadamard-prefill
Open

sb32445 wants to merge 1 commit into
PrismML-Eng:prismfrom
sb32445:pr/swiglu-hadamard-prefill

Conversation

@sb32445

@sb32445 sb32445 commented Nov 16, 2026

Copy link
Copy Markdown

Overview

The FFN down input of a Hadamard-folded model (Bonsai) is SWIGLU(gate, up) -> MUL(signs) -> RESHAPE -> MUL_MAT(Hadamard hint). For prompt batches (more than 8 tokens; decode and verify batches keep ggml_cuda_try_fwht_q8) the CUDA backend ran the GLU kernel, then the transform, then (for a PTQ1_0 weight) the q8_1 MMQ quantization: three launches and three passes over the activation.

  • fwht_cuda_block_swiglu forms silu(gate) * up with the expression of the GLU kernel, applies the scale, the signs and the same butterfly, and writes the transformed rows: one launch and one pass less. The output may sit exactly on top of the gate buffer (the allocator reuses it); the kernel reads a whole transform before it writes it, any other overlap refuses the fusion. GGML_CUDA_GLU_FWHT_FUSION=0 disables it.
  • When the only reader of the transformed rows is a PTQ1_0 mul_mat_q, the same launch also quantizes the rows to block_q8_1_mmq (D4) and the F32 rows are never written (swiglu_fwht_quantize_mmq reproduces the GLU kernel, the signed transform and quantize_mmq_q8_1 in order, including the 127/amax form, so the rows are bit-identical to the three-launch version). The rows go into a pool block held for the graph and are registered in ctx.fwht_q8() with a new mmq flag; ggml_cuda_mul_mat hands them to ggml_cuda_mul_mat_q as external_q8. The fusion is refused if the transform output has another reader, is a graph output, or the consumer would not take the MMQ path. GGML_CUDA_GLU_FWHT_Q8=0 keeps only the F32-output version.

RTX 4070, Bonsai 2 27B PTQ1_0, q4_0 K/V, llama-bench pp2048 against plain prism 6447906bf (mean of two runs): -ub 512 1354.8 -> 1411.6 tok/s (+4.2 %), -ub 2048 1279.2 -> 1343.0 (+5.0 %); at depth 32768 +3.1 % and +3.5 %. Decode is unchanged (MTP, 8 pairs: +0.01 %, CI [-0.10, +0.10] %). Results are bit-identical: perplexity 14.5423 in both, KL divergence 0.000000, same top token 100 %.

Additional information

Test results

  • Hardware / software: RTX 4070 12 GB (AD104, cc 8.9), Linux 6.18, NVIDIA driver 620.70, CUDA 13.4, GCC 16.2; Release build, -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89.
  • Base: prism at 6447906bf (separate builds of base and patch).
  • Model: Ternary-Bonsai-2-27B (PTQ1_0) with a community MTP draft head (Q8_0), q4_0 K/V cache.
  • Prefill (llama-bench -p 2048 -n 128 -ub 512,2048 -d 0,32768 -r 2, order base, patch, patch, base; tok/s ↑):
-ub Depth base patch change
512 0 1354.8 1411.6 +4.2 %
512 32768 969.2 998.8 +3.1 %
2048 0 1279.2 1343.0 +5.0 %
2048 32768 934.8 967.9 +3.5 %

(run-to-run spread ≤ 0.2 %.) tg128 is unchanged (65.8 -> 65.7 tok/s at depth 0, 59.1 -> 59.1 at 32768).

  • Decode with MTP (--spec-draft-n-max 2, 4 greedy prompts x 256 tokens, 8 interleaved A/B pairs): 115.25 -> 115.24 tok/s, +0.01 %, outputs identical.
  • Quality: llama-perplexity (German text, 16 x 512, -b 2048 -ub 512): PPL 14.5423 against 14.5423, mean KLD 0.000000, same top token 100 %.
  • Not tested: other GPUs, multi-GPU, other PTQ1_0 models, several slots.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: The patches were developed with Claude Code (Anthropic's coding agent): it wrote the code, the measurement scripts and the first drafts of the commit messages and PR texts. I decided what to work on (which kernels and host paths to optimise, based on profiles of my own decode setup). The measurements and checks listed in the PR texts were run in the Claude Code sessions; I did not re-run them independently. I will maintain the changes. Commits where Claude Code was used carry a Co-Authored-By trailer.

🤖 Generated with Claude Code

…batches, and quantize in the same launch for the MMQ

The FFN down input of a Hadamard-folded model is SWIGLU(gate, up) -> MUL(signs) -> RESHAPE -> MUL_MAT(Hadamard hint).
For more than 8 tokens (decode and verify batches keep ggml_cuda_try_fwht_q8) the CUDA backend ran this as the GLU
kernel, then fwht_cuda_block. fwht_cuda_block_swiglu reads gate and up, forms silu(gate)*up with the expression of the
GLU kernel, applies scale, signs and the same butterfly, and writes the transformed rows: one launch less and one
full pass over the activation less. The output may sit exactly on top of the gate buffer (the allocator reuses it);
the kernel reads a whole transform before it writes it, any other overlap refuses the fusion.
GGML_CUDA_GLU_FWHT_FUSION=0 disables it.

When the only reader of the transformed rows is a PTQ1_0 mul_mat_q (prompt batches), the same launch also quantizes
the rows to block_q8_1_mmq (D4) and the F32 rows are never written (swiglu_fwht_quantize_mmq reproduces
unary_gated_op_kernel<op_silu>, the signed fwht_cuda_block and quantize_mmq_q8_1 in order, including its 127/amax form,
so the rows are bit-identical to the three-launch version). The rows go into a pool block held for the graph and are
registered in ctx.fwht_q8() with the new mmq flag; ggml_cuda_mul_mat hands them to ggml_cuda_mul_mat_q as external_q8
without quantizing. The fusion is refused if the transform output has any other reader, is a graph output, or the
consumer would not take the MMQ path. GGML_CUDA_GLU_FWHT_Q8=0 keeps only the F32-output version.

Ternary-Bonsai-2-27B PTQ1_0, RTX 4070, llama-bench pp2048, q4_0 K/V, against prism 6447906 (mean of two runs):
-ub 512 1354.8 -> 1411.6 tok/s (+4.2 %), -ub 2048 1279.2 -> 1343.0 (+5.0 %); at depth 32768 +3.1 % and +3.5 %.
Decode (MTP, 8 interleaved pairs) is unchanged (+0.01 %, outputs identical). Perplexity and the KL against the
unfused build are unchanged (mean KLD 0.000000, same top token 100 %).
tests: MUL_MAT_HADAMARD cases with swiglu (9, 33, 64 and 17 tokens) and with a PTQ1_0 ffn_down consumer (9, 33, 512,
64 and 17 tokens).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant