Skip to content

common : keep the fast top-k path when penalties only lower logits - #444

Open
sb32445 wants to merge 1 commit into
PrismML-Eng:prismfrom
sb32445:pr/sampler-topk-penalties
Open

sb32445 wants to merge 1 commit into
PrismML-Eng:prismfrom
sb32445:pr/sampler-topk-penalties

Conversation

@sb32445

@sb32445 sb32445 commented Nov 16, 2026

Copy link
Copy Markdown

Overview

#313 (merged) lets the sampler take the top-k straight from the logits when top-k starts the sampler chain. With a presence, frequency or repeat penalty the chain starts with the penalties sampler, so that fast path was switched off (common_sampler_fast_topk returned 0) and the full-vocabulary array (248k tokens for Qwen3.5 / Bonsai) was built and sorted for every token.

Penalties that only lower logits (penalty_repeat >= 1, penalty_freq >= 0, penalty_present >= 0) change at most penalty_last_n distinct tokens, and only downwards. So the top k after the penalties lies inside the top k + penalty_last_n of the raw logits. The fast path now uses k + penalty_last_n when the penalties sampler comes first. Everything else keeps the old path: penalties that can raise a logit (negative values), penalty_last_n < 0 or > 128 (and top_k > 128 as before).

RTX 4070, Bonsai 2 27B PTQ1_0 + MTP head (n-max 2), q4_0 K/V, llama-server, presence_penalty 0.5, temperature 0.6, top_k 20, top_p 0.95, penalty_last_n 64, against plain prism 6447906bf, 4 interleaved pairs (4 prompts x 2 seeds, 256 tokens each, fixed seeds): 83.95 -> 92.18 tok/s, +9.81 % (95 % CI [+9.74, +9.87] %). The generated texts are identical to the full path in all runs. Without a penalty the speed is unchanged (94.12 -> 93.99 tok/s, -0.15 %, 2 pairs only).

Additional information

  • One commit, one file (+9/-2). No new options, no change for requests without penalties or with penalty values that can raise logits.
  • Speed with a penalty is now about the speed without one (92.2 against 94.0 tok/s); before it was 11 % lower.
  • The result does not depend on the model; it removes a per-token sort of the whole vocabulary, so the gain is larger the faster the model decodes and the larger the vocabulary is. On a Qwen3.5 9B (Q8_0) the same change gave +3 to +4 % (earlier measurement on an older base); on a 35B-A3B MoE +10.7 % (separate fork).
  • Test: the generated text of 8 requests per run (fixed seeds) is identical with and without the change, in all 8 runs. There is no sampler unit test for this path in the repo; I did not add one. I can add a test that compares the chain output with and without LLAMA_SAMPLER_FAST_TOPK=0, if you want one.
  • test-backend-ops full suite on this branch: 20724/20726; the two failures are MUL_MAT cases that fail now and then on plain prism too (pq2_0, m=1, n=1, k=1024, random inputs; and one q4_1 case, ERR 0.00053 against 0.0005, not reproduced in 40 repeated runs on prism or on this branch). test-backend-ops does not use common/sampling.cpp, so the change cannot cause them.
  • Not tested: other models, backend_sampling, several slots, penalty_last_n between 64 and 128 with large top_k.

Test results

  • Hardware / software: RTX 4070 12 GB (AD104, cc 8.9), Linux 6.18, NVIDIA driver 620.70, CUDA 13.4, GCC 16.2; Release build, -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89.
  • Base: prism at 6447906bf.
  • Model: Ternary-Bonsai-2-27B (PTQ1_0) with a community MTP draft head (Q8_0), --spec-type draft-mtp --spec-draft-n-max 2, q4_0 K/V cache, one slot, --cache-ram 0, thinking off.
  • Method: separate builds of prism and the branch, one server start per run and arm, alternating A B B A, one warm-up request, speed = predicted tokens / predicted time of the server timings; paired differences with a 95 % bootstrap interval (script tools/sampler_ab.py of my notes repository, available on request).

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: The patches were developed with Claude Code (Anthropic's coding agent): it wrote the code, the measurement scripts and the first drafts of the commit messages and PR texts. I decided what to work on (which kernels and host paths to optimise, based on profiles of my own decode setup). The measurements and checks listed in the PR texts were run in the Claude Code sessions; I did not re-run them independently. I will maintain the changes. Commits where Claude Code was used carry a Co-Authored-By trailer.

🤖 Generated with Claude Code

With presence/frequency/repeat penalties the chain started with the
penalties sampler, so the fast top-k path was off and the full-vocabulary
array was built and sorted per token. Penalties that only lower logits
(repeat >= 1, frequency >= 0, presence >= 0) touch at most penalty_last_n
distinct tokens, so the top k after them is inside the top k + penalty_last_n
of the raw logits. Use that as k. Other penalty values, penalty_last_n < 0 or
> 128 keep the old path.

Ternary-Bonsai-2-27B PTQ1_0 + MTP head, RTX 4070, llama-server, presence_penalty 0.5,
temperature 0.6, top_k 20, top_p 0.95, penalty_last_n 64, against prism 6447906
(4 interleaved pairs, 8 generations per run): 83.95 -> 92.18 tok/s (+9.8 %, 95 % CI
[+9.7, +9.9] %). The texts with a fixed seed are identical to the full path in all runs.
Without penalties the speed is unchanged (-0.15 %, 2 pairs).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant