Skip to content

Npu e2b kvonly - #432

Open
anikulk wants to merge 4 commits into
ravi9:dev_backend_openvinofrom
anikulk:npu-e2b-kvonly
Open

anikulk wants to merge 4 commits into
ravi9:dev_backend_openvinofrom
anikulk:npu-e2b-kvonly

Conversation

@anikulk

@anikulk anikulk commented Nov 17, 2026

Copy link
Copy Markdown

KV-only prefill on the static path (GGML_OPENVINO_KV_ONLY_PREFILL, on by default). In these models the last
layers reuse the KV cache of earlier layers, so a prompt token only needs the earlier layers to fill the cache. The
prefill model is compiled with only the KV-cache writes as outputs (new pass pass::KeepCacheResults), which drops
the KV-shared layers and the output layer from it (E2B: layers 15-34 and the LM head). When a batch needs its output
token, the decode model runs once on that token against the full cache. It takes effect only with full-context
decode (GGML_OPENVINO_NPU_BLOB_PREBUILD=0), without HFA prefill, and with at most one output token per batch.

  • NPU host gather (GGML_OPENVINO_NPU_HOST_GATHER=1, opt-in). Embedding tables of 64 MiB or more are looked up on
    the host from the GGUF data and passed to the NPU graph as an input, instead of binding the whole table to the NPU,
    where NPUW turns it into a function input that is streamed on every call. A table used only for lookups (the
    per-layer embedding table of E2B, 262,144 x 8,960) is not converted for the NPU at all.
  • Host gather and KV-only prefill with the weightless blob cache (GGML_OPENVINO_NPU_WEIGHTLESS_BANK): the
    export and import paths handle the inputs these two features remove or add.
  • GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT for Q4_K/Q5_K tables when set. E2B ties a Q4_K table to the output layer;
    q4_0_128 stores it in 198 MiB instead of 384 MiB. Unset, the behaviour is unchanged.

Docs: rows for the two new variables and the token_embd change in docs/backend/OPENVINO.md, and a section on each
feature in ggml/src/ggml-openvino/README.md.

Results

Gemma 4 E2B Q4_0, llama-server -c 8192 -b 3072 -ub 1024 -fa on --ctx-checkpoints 0 --cache-ram 0, a two-turn chat
(3016-token first prompt, 46-token follow-up), greedy, 512 output tokens. Environment for all rows:
GGML_OPENVINO_DEVICE=NPU PREFILL_CHUNK_SIZE=512 NPU_HOST_GATHER=1 LAZY_HOST_THRESHOLD_MB=128 REDUCE_COMPILE_MEM=1 NPU_WEIGHTLESS_BANK=1 COMPILED_MODEL_CACHE_DIR=<dir>, SELF_CONTAINED_MMAP=1 on the measured (import) runs.
Intel NPU 5010 (Panther Lake), driver 32.0.100.5540, OpenVINO 2026.5.0-23436-1cf039c02df. Values: two import runs.

Configuration First prompt Follow-up Decode Peak working set
Defaults of this branch (KV slicing and decode KV windows on, so KV-only prefill is off) 3.22-3.52 s 0.36 s 43.6-44.3 ms/token 2.75 GiB
+ GGML_OPENVINO_KV_SLICE=0 GGML_OPENVINO_NPU_BLOB_PREBUILD=0 (one full-context pair, KV-only prefill on) 1.27-1.36 s (2,210-2,370 tok/s) 0.20 s 48.8 ms/token 2.31 GiB
+ GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT=q4_0_128 1.35-1.40 s 0.19 s 45.2-45.5 ms/token 2.11 GiB
  • With the defaults, the export run's first prompt also compiled the prefill models of four KV lengths and the decode
    KV windows inside the request (174 s); with the second row's settings the export compiles one pair at load.
  • Decode varied between 44 and 49 ms/token across these runs without NPU turbo; I don't read the row differences in
    decode as real.
  • Every export and import run of a configuration produced the same text, and all answer the prompt's question
    correctly. NPU compilation is not bit-identical between exports, so the text differs slightly between
    configurations.

…ep for the outputs

On the static path (NPU, or any device with GGML_OPENVINO_FORCE_STATIC=1) the prefill model of a graph with
KV-shared attention layers is compiled with only the cache writes as outputs. The new conversion pass
KeepCacheResults (openvino/pass/keep_cache_results.cpp, registered in pass_registry/common.cpp from
GraphTraits::kv_only_outputs) removes the other Results and the Parameters nothing reads any more, so OpenVINO
drops everything that only feeds the logits: for models whose tail layers read an earlier layer's KV (Gemma 3n/4
E-series) every KV-shared layer and the LM head. When the batch needs logits, the decode model is run once on the
output token: it rewrites that token's KV rows with the same values and runs the dropped layers against the
complete cache. The decoder keeps its full output list and only reports which outputs are caches; an imported
cached model takes the cache outputs as its output names.

It is on by default and applies only where it pays off and is valid: graphs with KV-shared attention, at most one
output token per batch, and no HFA prefill or decode KV windows. GGML_OPENVINO_KV_ONLY_PREFILL=0 turns it off; the
earlier name GGML_OPENVINO_NPU_KV_ONLY_PREFILL is still accepted.

Gemma 4 E2B Q4_0 on NPU (KV_SLICE=0, NPU_TURBO=YES): llama-bench pp512 1320-1375 -> 2349-2454 t/s,
pp2048 1146-1202 -> 2268-2513 t/s; E4B pp512 587 -> 773, pp2048 591 -> 847 t/s. llama-server, E2B, 3016-token
prompt, prefill chunk 512, context checkpoints off: 1097-1108 -> 2136-2448 t/s. Greedy outputs match the full
prefill on 3 of 4 needle prompts; the fourth differs by one near-tie token that also flips between baseline runs.
GGML_OPENVINO_NPU_HOST_GATHER=1 gathers the rows of large host-resident embedding tables (at least
64 MiB, raw GGUF data, ids from a graph input) on the host. The static graph gets the rows through a
ggml_host_rows_<node> input instead of an OpenVINO Gather over the table. With NPUW function calls the
table becomes a function input, and gathering from an input that large costs about as much as
streaming the whole table every call: 21.6 ms of the ~62 ms Gemma 4 E2B decode step, against 0.45 ms
for the same partition with the table baked in. A table used only for such lookups (the Gemma 3n/4
per-layer embedding table) is no longer converted, requantized or bound to the NPU.

Gemma 4 on NPU (NPU_TURBO=YES, KV_SLICE=0, 868-token prompt), together with KV-only prefill: E2B decode
65.3 -> 40.3 ms/token, prefill 925 -> 392 ms, system commit 8.5 -> 4.4 GB, load 109 -> 61 s; E4B decode
150 -> 72 ms/token, prefill 1931 -> 1022 ms, commit 14.6 -> 9.1 GB. Greedy outputs match the default on
3 of 4 needle prompts; the fourth differs by one near-tie token that also flips between baseline runs.
…s blob cache

Make the two NPU options work with GGML_OPENVINO_NPU_WEIGHTLESS_BANK / SELF_CONTAINED_MMAP:
- host gather also accepts tables in OpenVINO buffers when a raw-weight cache mode keeps (or, with
  SELF_CONTAINED_MMAP, wraps) the raw GGUF bytes there;
- a warm import binds only the decoder inputs present in the compiled blob, since KV-only prefill
  (out_ids) and host gather (token ids) can leave inputs unused, as HFA prefill already did;
- a weights file that cannot be written (e.g. a full disk) no longer fails the graph: the pair is
  compiled and run without being cached.

Gemma 4 on NPU (NPU_TURBO=YES, KV_SLICE=0, 868-token prompt), weightless cache + MMAP warm start with
host gather and KV-only prefill: E2B load 1.8 s, system commit 2.5 GB, prefill 385-414 ms, decode
39.5-41 ms/token, cache 1.5 GB; E4B load 3.3-3.6 s, commit 4.7-4.8 GB, prefill ~1.0 s, decode 68.5-69.8
ms/token, cache 3.3 GB. Greedy outputs are byte-identical to the default path on export and import.
…5_K tables when set

Q4_K and Q5_K token_embd tables default to the channel-wise int8 of ggml_openvino_get_requant_type() on NPU and
ignored GGML_OPENVINO_NPU_TOKEN_EMBD_QUANT. They now follow it when it is set explicitly; the Q6_K behaviour and
all defaults are unchanged. Gemma 4 E2B ties a Q4_K table to the output layer: int8 stores it in 384 MiB,
q4_0_128 in 198 MiB (llama-server working set -0.18 GB; accuracy not yet measured).
@cavusmustafa

Copy link
Copy Markdown
Collaborator

How it identifies MLA models with no V cache?


// Input of the decode model for token |row| of the current prefill graph, used to compute that token's outputs after a
// KV-only prefill. The decoder must be in decode mode so that the KV cache and the masks bind at full context.
static ov::Tensor get_ov_input_tensor_static_decode_row(const std::shared_ptr<GgmlOvDecoder> & ggml_decoder,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should combine all get_ov_input_tensor_static_* functions into one.

input_tensor = get_ov_input_tensor(ggml_decoder, param_name);
}
if (input_tensor.get_shape() != port_shape) {
throw std::runtime_error("KV-only prefill: decode input " + param_name + " has shape " +

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to fallback to normal prefill if it fails?

if (n_outputs > 1) {
throw std::runtime_error(
"GGML_OPENVINO_KV_ONLY_PREFILL supports one output token per batch; got " +
std::to_string(n_outputs));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to fallback to normal prefill if it fails?

config;
// Decided per compiled pair from the graph that triggers the compile; later calls follow entry->kv_only_prefill.
// The output step binds the full KV cache to the pair's decode model, so it needs a full-context decode window.
const bool kv_only_prefill = !no_kv_cache && ggml_openvino_kv_only_prefill_enabled() &&

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this check for recurrent-state ?

}
}
// Every layer with its own cache writes K and V.
return n_attn > 0 && 2 * n_attn > n_kv_writes;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do this by checking if one cache tensor is read by more than one layers? This would only work with the assumption that every layers writes to K and V cache tensors which may fail with different architectures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants