Repository navigation
Conversation
b494c5e to
1b84f18
Compare
|
This now conflicts with prism. Could you rebase it? I will review once it applies cleanly. |
The PTQ1_0 weight scale is one value per 128 weights, so it is the same for the four 32-wide MMA steps of one vec_dot call (the tile loader writes the block scale into all four x_df entries). The generic Q8_0 routine scaled every MMA result with dA and dB separately (I2F, FMUL, FFMA per element and step), which was 62 % of the executed instructions of mul_mat_q. The new PTQ1_0 routine accumulates the four steps with dB and applies dA once. RTX 4070, Ternary-Bonsai-2-27B PTQ1_0 + MTP head, q4_0 K/V, base prism c96baf4, this commit alone: llama-bench pp2048 (-ub 512, 2 runs each, A B B A) 1363.1 -> 1412.1 tok/s at depth 0 (+3.6 %), 960.7 -> 988.7 at depth 32768 (+2.9 %), 576.8 -> 586.9 at depth 98304 (+1.8 %); llama-server prefill of 114688 tokens 157.4 -> 154.1 s (-2.1 %). An earlier measurement of this change on another base: mul_mat_q instructions -15 %, registers 254 -> 220. Decode kernels are not touched. The summation order changes, so results are not bit-identical to the previous kernel: KL divergence against it 0.00090 (16x512) and 0.00082 (4096 tokens per batch), perplexity 14.5351 against 14.5316. Through the prompt's KV values this can change MTP drafts: in the 4-prompt decode benchmark acceptance went 74.0 -> 72.0 % and decode 115.98 -> 115.05 tok/s (-0.80 %, CI [-0.82, -0.77] %, 8 pairs), sign depends on the prompt. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
1b84f18 to
872dab5
Compare
|
Rebased on current Re-measured against plain The description has the new numbers. |
Overview
The PTQ1_0 weight scale is one value per 128 weights, so it is the same for the four 32-wide MMA steps of one
vec_dotcall (the tile loader writes the block scale into all fourx_dfentries).mul_mat_qfor PTQ1_0 uses the generic Q8_0 routine, which scales every MMA result withdAanddBseparately (I2F, FMUL, FFMA per element and step); that was 62 % of the executed instructions of the kernel (profile on an earlier base). A PTQ1_0 routine accumulates the four steps withdBand appliesdAonce. Prefill (batches that go through MMQ) gets faster; decode uses other kernels and is not touched.RTX 4070, Bonsai 2 27B PTQ1_0, against plain
prism6447906bf:llama-benchpp2048 (-ub 512) +4.1 % at depth 0, +2.9 % at 32768, +1.8 % at 98304.Additional information
ggml_cuda_mmq_vec_dot_ptq1_0_q8_1_mmainmmq-vec-dot.cuh(+92 lines) and a one-line change inmmq.cuhthat selects it forGGML_TYPE_PTQ1_0.c96baf4abto6447906bf.prismchanged the MMQ helpers in between:ggml_cuda_mmq_get_sram_strideandggml_cuda_mmq_get_rows_per_warptake the source precision now, andggml_cuda_mmq_write_back_mmatakesprec_src1and a fusion flag. The new routine calls them withGGML_PREC_Q8as the generic Q8_0 routine does, and themmq.cuhline passesprec_src1, falselike the neighbouring cases. All numbers below are re-measured on the new base.-b 2048 -ub 512, 16 x 512 tokens), same top token 98.31 ± 0.20 %, 99th percentile KLD 0.005035; perplexity 14.5356 against 14.5423 (error 0.62).prism0.000938 ± 0.000051, same top token 98.06 %. The two PRs touch different files and apply independently.Test results
-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89.prismat6447906bf, this commit only (separate builds of base and patch).llama-bench -p 2048 -n 0 -ub 512 -d 0,32768,98304 -r 2, two runs per build in the order base, patch, patch, base):(tok/s, mean of two runs; run-to-run spread ≤ 0.2 %.) The server prefill of 114688 tokens (-2.1 % on the previous base) was not repeated.
--spec-draft-n-max 2, 4 greedy prompts x 256 tokens, 8 interleaved A/B pairs, 95 % bootstrap interval): 115.20 -> 114.12 tok/s, -0.88 % (CI [-0.94, -0.83] %), acceptance 73.5 -> 72.1 %, outputs differ (2 of 4 prompts).test-backend-ops test -b CUDA0(full suite): 20725/20726 passed. The one failure isMUL_MATpq2_0, m=1, n=1, k=1024 (ERR 0.00063 against 0.0005), which also fails now and then on plainprism(4 of 25 repeated runs, random inputs) and does not involve this change.Requirements
🤖 Generated with Claude Code