Repository navigation
Conversation
With presence/frequency/repeat penalties the chain started with the penalties sampler, so the fast top-k path was off and the full-vocabulary array was built and sorted per token. Penalties that only lower logits (repeat >= 1, frequency >= 0, presence >= 0) touch at most penalty_last_n distinct tokens, so the top k after them is inside the top k + penalty_last_n of the raw logits. Use that as k. Other penalty values, penalty_last_n < 0 or > 128 keep the old path. Ternary-Bonsai-2-27B PTQ1_0 + MTP head, RTX 4070, llama-server, presence_penalty 0.5, temperature 0.6, top_k 20, top_p 0.95, penalty_last_n 64, against prism 6447906 (4 interleaved pairs, 8 generations per run): 83.95 -> 92.18 tok/s (+9.8 %, 95 % CI [+9.7, +9.9] %). The texts with a fixed seed are identical to the full path in all runs. Without penalties the speed is unchanged (-0.15 %, 2 pairs). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
#313 (merged) lets the sampler take the top-k straight from the logits when top-k starts the sampler chain. With a presence, frequency or repeat penalty the chain starts with the penalties sampler, so that fast path was switched off (
common_sampler_fast_topkreturned 0) and the full-vocabulary array (248k tokens for Qwen3.5 / Bonsai) was built and sorted for every token.Penalties that only lower logits (
penalty_repeat >= 1,penalty_freq >= 0,penalty_present >= 0) change at mostpenalty_last_ndistinct tokens, and only downwards. So the top k after the penalties lies inside the topk + penalty_last_nof the raw logits. The fast path now usesk + penalty_last_nwhen the penalties sampler comes first. Everything else keeps the old path: penalties that can raise a logit (negative values),penalty_last_n < 0or> 128(andtop_k > 128as before).RTX 4070, Bonsai 2 27B PTQ1_0 + MTP head (n-max 2), q4_0 K/V,
llama-server,presence_penalty 0.5, temperature 0.6, top_k 20, top_p 0.95,penalty_last_n64, against plainprism6447906bf, 4 interleaved pairs (4 prompts x 2 seeds, 256 tokens each, fixed seeds): 83.95 -> 92.18 tok/s, +9.81 % (95 % CI [+9.74, +9.87] %). The generated texts are identical to the full path in all runs. Without a penalty the speed is unchanged (94.12 -> 93.99 tok/s, -0.15 %, 2 pairs only).Additional information
LLAMA_SAMPLER_FAST_TOPK=0, if you want one.test-backend-opsfull suite on this branch: 20724/20726; the two failures areMUL_MATcases that fail now and then on plainprismtoo (pq2_0, m=1, n=1, k=1024, random inputs; and oneq4_1case, ERR 0.00053 against 0.0005, not reproduced in 40 repeated runs onprismor on this branch).test-backend-opsdoes not usecommon/sampling.cpp, so the change cannot cause them.backend_sampling, several slots,penalty_last_nbetween 64 and 128 with largetop_k.Test results
-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89.prismat6447906bf.--spec-type draft-mtp --spec-draft-n-max 2, q4_0 K/V cache, one slot,--cache-ram 0, thinking off.prismand the branch, one server start per run and arm, alternating A B B A, one warm-up request, speed = predicted tokens / predicted time of the servertimings; paired differences with a 95 % bootstrap interval (scripttools/sampler_ab.pyof my notes repository, available on request).Requirements
🤖 Generated with Claude Code