Repository navigation
Npu e2b kvonly - #432
Open
anikulk wants to merge 4 commits into
Open
Npu e2b kvonly#432anikulk wants to merge 4 commits into
anikulk wants to merge 4 commits into
Conversation
…ep for the outputs On the static path (NPU, or any device with GGML_OPENVINO_FORCE_STATIC=1) the prefill model of a graph with KV-shared attention layers is compiled with only the cache writes as outputs. The new conversion pass KeepCacheResults (openvino/pass/keep_cache_results.cpp, registered in pass_registry/common.cpp from GraphTraits::kv_only_outputs) removes the other Results and the Parameters nothing reads any more, so OpenVINO drops everything that only feeds the logits: for models whose tail layers read an earlier layer's KV (Gemma 3n/4 E-series) every KV-shared layer and the LM head. When the batch needs logits, the decode model is run once on the output token: it rewrites that token's KV rows with the same values and runs the dropped layers against the complete cache. The decoder keeps its full output list and only reports which outputs are caches; an imported cached model takes the cache outputs as its output names. It is on by default and applies only where it pays off and is valid: graphs with KV-shared attention, at most one output token per batch, and no HFA prefill or decode KV windows. GGML_OPENVINO_KV_ONLY_PREFILL=0 turns it off; the earlier name GGML_OPENVINO_NPU_KV_ONLY_PREFILL is still accepted. Gemma 4 E2B Q4_0 on NPU (KV_SLICE=0, NPU_TURBO=YES): llama-bench pp512 1320-1375 -> 2349-2454 t/s, pp2048 1146-1202 -> 2268-2513 t/s; E4B pp512 587 -> 773, pp2048 591 -> 847 t/s. llama-server, E2B, 3016-token prompt, prefill chunk 512, context checkpoints off: 1097-1108 -> 2136-2448 t/s. Greedy outputs match the full prefill on 3 of 4 needle prompts; the fourth differs by one near-tie token that also flips between baseline runs.
GGML_OPENVINO_NPU_HOST_GATHER=1 gathers the rows of large host-resident embedding tables (at least 64 MiB, raw GGUF data, ids from a graph input) on the host. The static graph gets the rows through a ggml_host_rows_<node> input instead of an OpenVINO Gather over the table. With NPUW function calls the table becomes a function input, and gathering from an input that large costs about as much as streaming the whole table every call: 21.6 ms of the ~62 ms Gemma 4 E2B decode step, against 0.45 ms for the same partition with the table baked in. A table used only for such lookups (the Gemma 3n/4 per-layer embedding table) is no longer converted, requantized or bound to the NPU. Gemma 4 on NPU (NPU_TURBO=YES, KV_SLICE=0, 868-token prompt), together with KV-only prefill: E2B decode 65.3 -> 40.3 ms/token, prefill 925 -> 392 ms, system commit 8.5 -> 4.4 GB, load 109 -> 61 s; E4B decode 150 -> 72 ms/token, prefill 1931 -> 1022 ms, commit 14.6 -> 9.1 GB. Greedy outputs match the default on 3 of 4 needle prompts; the fourth differs by one near-tie token that also flips between baseline runs.
…s blob cache Make the two NPU options work with GGML_OPENVINO_NPU_WEIGHTLESS_BANK / SELF_CONTAINED_MMAP: - host gather also accepts tables in OpenVINO buffers when a raw-weight cache mode keeps (or, with SELF_CONTAINED_MMAP, wraps) the raw GGUF bytes there; - a warm import binds only the decoder inputs present in the compiled blob, since KV-only prefill (out_ids) and host gather (token ids) can leave inputs unused, as HFA prefill already did; - a weights file that cannot be written (e.g. a full disk) no longer fails the graph: the pair is compiled and run without being cached. Gemma 4 on NPU (NPU_TURBO=YES, KV_SLICE=0, 868-token prompt), weightless cache + MMAP warm start with host gather and KV-only prefill: E2B load 1.8 s, system commit 2.5 GB, prefill 385-414 ms, decode 39.5-41 ms/token, cache 1.5 GB; E4B load 3.3-3.6 s, commit 4.7-4.8 GB, prefill ~1.0 s, decode 68.5-69.8 ms/token, cache 3.3 GB. Greedy outputs are byte-identical to the default path on export and import.
…5_K tables when set Q4_K and Q5_K token_embd tables default to the channel-wise int8 of ggml_openvino_get_requant_type() on NPU and ignored GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT. They now follow it when it is set explicitly; the Q6_K behaviour and all defaults are unchanged. Gemma 4 E2B ties a Q4_K table to the output layer: int8 stores it in 384 MiB, q4_0_128 in 198 MiB (llama-server working set -0.18 GB; accuracy not yet measured).
Collaborator
|
How it identifies MLA models with no V cache? |
cavusmustafa
requested changes
Nov 17, 2026
|
|
||
| // Input of the decode model for token |row| of the current prefill graph, used to compute that token's outputs after a | ||
| // KV-only prefill. The decoder must be in decode mode so that the KV cache and the masks bind at full context. | ||
| static ov::Tensor get_ov_input_tensor_static_decode_row(const std::shared_ptr<GgmlOvDecoder> & ggml_decoder, |
Collaborator
There was a problem hiding this comment.
I think we should combine all get_ov_input_tensor_static_* functions into one.
| input_tensor = get_ov_input_tensor(ggml_decoder, param_name); | ||
| } | ||
| if (input_tensor.get_shape() != port_shape) { | ||
| throw std::runtime_error("KV-only prefill: decode input " + param_name + " has shape " + |
Collaborator
There was a problem hiding this comment.
Is it possible to fallback to normal prefill if it fails?
| if (n_outputs > 1) { | ||
| throw std::runtime_error( | ||
| "GGML_OPENVINO_KV_ONLY_PREFILL supports one output token per batch; got " + | ||
| std::to_string(n_outputs)); |
Collaborator
There was a problem hiding this comment.
Is it possible to fallback to normal prefill if it fails?
| config; | ||
| // Decided per compiled pair from the graph that triggers the compile; later calls follow entry->kv_only_prefill. | ||
| // The output step binds the full KV cache to the pair's decode model, so it needs a full-context decode window. | ||
| const bool kv_only_prefill = !no_kv_cache && ggml_openvino_kv_only_prefill_enabled() && |
Collaborator
There was a problem hiding this comment.
Does this check for recurrent-state ?
| } | ||
| } | ||
| // Every layer with its own cache writes K and V. | ||
| return n_attn > 0 && 2 * n_attn > n_kv_writes; |
Collaborator
There was a problem hiding this comment.
Can we do this by checking if one cache tensor is read by more than one layers? This would only work with the assumption that every layers writes to K and V cache tensors which may fail with different architectures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
KV-only prefill on the static path (
GGML_OPENVINO_KV_ONLY_PREFILL, on by default). In these models the lastlayers reuse the KV cache of earlier layers, so a prompt token only needs the earlier layers to fill the cache. The
prefill model is compiled with only the KV-cache writes as outputs (new pass
pass::KeepCacheResults), which dropsthe KV-shared layers and the output layer from it (E2B: layers 15-34 and the LM head). When a batch needs its output
token, the decode model runs once on that token against the full cache. It takes effect only with full-context
decode (
GGML_OPENVINO_NPU_BLOB_PREBUILD=0), without HFA prefill, and with at most one output token per batch.GGML_OPENVINO_NPU_HOST_GATHER=1, opt-in). Embedding tables of 64 MiB or more are looked up onthe host from the GGUF data and passed to the NPU graph as an input, instead of binding the whole table to the NPU,
where NPUW turns it into a function input that is streamed on every call. A table used only for lookups (the
per-layer embedding table of E2B, 262,144 x 8,960) is not converted for the NPU at all.
GGML_OPENVINO_NPU_WEIGHTLESS_BANK): theexport and import paths handle the inputs these two features remove or add.
GGML_OPENVINO_NPU_TOKEN_EMBD_QUANTfor Q4_K/Q5_K tables when set. E2B ties a Q4_K table to the output layer;q4_0_128stores it in 198 MiB instead of 384 MiB. Unset, the behaviour is unchanged.Docs: rows for the two new variables and the token_embd change in
docs/backend/OPENVINO.md, and a section on eachfeature in
ggml/src/ggml-openvino/README.md.Results
Gemma 4 E2B Q4_0,
llama-server -c 8192 -b 3072 -ub 1024 -fa on --ctx-checkpoints 0 --cache-ram 0, a two-turn chat(3016-token first prompt, 46-token follow-up), greedy, 512 output tokens. Environment for all rows:
GGML_OPENVINO_DEVICE=NPU PREFILL_CHUNK_SIZE=512 NPU_HOST_GATHER=1 LAZY_HOST_THRESHOLD_MB=128 REDUCE_COMPILE_MEM=1 NPU_WEIGHTLESS_BANK=1 COMPILED_MODEL_CACHE_DIR=<dir>,SELF_CONTAINED_MMAP=1on the measured (import) runs.Intel NPU 5010 (Panther Lake), driver 32.0.100.5540, OpenVINO 2026.5.0-23436-1cf039c02df. Values: two import runs.
GGML_OPENVINO_KV_SLICE=0 GGML_OPENVINO_NPU_BLOB_PREBUILD=0(one full-context pair, KV-only prefill on)GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT=q4_0_128KV windows inside the request (174 s); with the second row's settings the export compiles one pair at load.
decode as real.
correctly. NPU compilation is not bit-identical between exports, so the text differs slightly between
configurations.