Repository navigation
cuda: fuse SwiGLU + sign flip + Hadamard (and the PTQ1_0 MMQ q8_1 quantization) for prompt batches - #445
Open
sb32445 wants to merge 1 commit into
Open
cuda: fuse SwiGLU + sign flip + Hadamard (and the PTQ1_0 MMQ q8_1 quantization) for prompt batches#445sb32445 wants to merge 1 commit into
sb32445 wants to merge 1 commit into
Conversation
…batches, and quantize in the same launch for the MMQ The FFN down input of a Hadamard-folded model is SWIGLU(gate, up) -> MUL(signs) -> RESHAPE -> MUL_MAT(Hadamard hint). For more than 8 tokens (decode and verify batches keep ggml_cuda_try_fwht_q8) the CUDA backend ran this as the GLU kernel, then fwht_cuda_block. fwht_cuda_block_swiglu reads gate and up, forms silu(gate)*up with the expression of the GLU kernel, applies scale, signs and the same butterfly, and writes the transformed rows: one launch less and one full pass over the activation less. The output may sit exactly on top of the gate buffer (the allocator reuses it); the kernel reads a whole transform before it writes it, any other overlap refuses the fusion. GGML_CUDA_GLU_FWHT_FUSION=0 disables it. When the only reader of the transformed rows is a PTQ1_0 mul_mat_q (prompt batches), the same launch also quantizes the rows to block_q8_1_mmq (D4) and the F32 rows are never written (swiglu_fwht_quantize_mmq reproduces unary_gated_op_kernel<op_silu>, the signed fwht_cuda_block and quantize_mmq_q8_1 in order, including its 127/amax form, so the rows are bit-identical to the three-launch version). The rows go into a pool block held for the graph and are registered in ctx.fwht_q8() with the new mmq flag; ggml_cuda_mul_mat hands them to ggml_cuda_mul_mat_q as external_q8 without quantizing. The fusion is refused if the transform output has any other reader, is a graph output, or the consumer would not take the MMQ path. GGML_CUDA_GLU_FWHT_Q8=0 keeps only the F32-output version. Ternary-Bonsai-2-27B PTQ1_0, RTX 4070, llama-bench pp2048, q4_0 K/V, against prism 6447906 (mean of two runs): -ub 512 1354.8 -> 1411.6 tok/s (+4.2 %), -ub 2048 1279.2 -> 1343.0 (+5.0 %); at depth 32768 +3.1 % and +3.5 %. Decode (MTP, 8 interleaved pairs) is unchanged (+0.01 %, outputs identical). Perplexity and the KL against the unfused build are unchanged (mean KLD 0.000000, same top token 100 %). tests: MUL_MAT_HADAMARD cases with swiglu (9, 33, 64 and 17 tokens) and with a PTQ1_0 ffn_down consumer (9, 33, 512, 64 and 17 tokens). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
The FFN down input of a Hadamard-folded model (Bonsai) is
SWIGLU(gate, up) -> MUL(signs) -> RESHAPE -> MUL_MAT(Hadamard hint). For prompt batches (more than 8 tokens; decode and verify batches keepggml_cuda_try_fwht_q8) the CUDA backend ran the GLU kernel, then the transform, then (for a PTQ1_0 weight) the q8_1 MMQ quantization: three launches and three passes over the activation.fwht_cuda_block_swigluformssilu(gate) * upwith the expression of the GLU kernel, applies the scale, the signs and the same butterfly, and writes the transformed rows: one launch and one pass less. The output may sit exactly on top of the gate buffer (the allocator reuses it); the kernel reads a whole transform before it writes it, any other overlap refuses the fusion.GGML_CUDA_GLU_FWHT_FUSION=0disables it.mul_mat_q, the same launch also quantizes the rows toblock_q8_1_mmq(D4) and the F32 rows are never written (swiglu_fwht_quantize_mmqreproduces the GLU kernel, the signed transform andquantize_mmq_q8_1in order, including the 127/amax form, so the rows are bit-identical to the three-launch version). The rows go into a pool block held for the graph and are registered inctx.fwht_q8()with a newmmqflag;ggml_cuda_mul_mathands them toggml_cuda_mul_mat_qasexternal_q8. The fusion is refused if the transform output has another reader, is a graph output, or the consumer would not take the MMQ path.GGML_CUDA_GLU_FWHT_Q8=0keeps only the F32-output version.RTX 4070, Bonsai 2 27B PTQ1_0, q4_0 K/V,
llama-benchpp2048 against plainprism6447906bf(mean of two runs):-ub 5121354.8 -> 1411.6 tok/s (+4.2 %),-ub 20481279.2 -> 1343.0 (+5.0 %); at depth 32768 +3.1 % and +3.5 %. Decode is unchanged (MTP, 8 pairs: +0.01 %, CI [-0.10, +0.10] %). Results are bit-identical: perplexity 14.5423 in both, KL divergence 0.000000, same top token 100 %.Additional information
MUL_MAT_HADAMARDcases withswiglufor 9, 33, 64 and 17 tokens, and with a PTQ1_0ffn_downconsumer for 9, 33, 512, 64 and 17 tokens (9 new cases in total).test-backend-ops test -b CUDA0(full suite) on this branch: 20735/20735 passed.ktrain4c0c11d7e) were +2.8 % for the F32-output version alone and +5.8 / +6.3 % with the quantization; those were measured against a different reference, the numbers above are againstprism.Test results
-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89.prismat6447906bf(separate builds of base and patch).llama-bench -p 2048 -n 128 -ub 512,2048 -d 0,32768 -r 2, order base, patch, patch, base; tok/s ↑):-ub(run-to-run spread ≤ 0.2 %.)
tg128is unchanged (65.8 -> 65.7 tok/s at depth 0, 59.1 -> 59.1 at 32768).--spec-draft-n-max 2, 4 greedy prompts x 256 tokens, 8 interleaved A/B pairs): 115.25 -> 115.24 tok/s, +0.01 %, outputs identical.llama-perplexity(German text, 16 x 512,-b 2048 -ub 512): PPL 14.5423 against 14.5423, mean KLD 0.000000, same top token 100 %.Requirements
🤖 Generated with Claude Code