Skip to content

ggml-openvino: Fix LFM2 context create on GPU and DSpark speculative decoding - #387

Open
swathiananth wants to merge 243 commits into
ravi9:dev_backend_openvinofrom
swathiananth:LFM2_5-VL-3B-DSpark_Context_Create_issue
Open

swathiananth wants to merge 243 commits into
ravi9:dev_backend_openvinofrom
swathiananth:LFM2_5-VL-3B-DSpark_Context_Create_issue

Conversation

@swathiananth

@swathiananth swathiananth commented Nov 6, 2026 •

Copy link
Copy Markdown

Overview

LFM2.5-VL-3B with the DSpark drafter (LiquidAI/LFM2.5-VL-3B-DSpark-GGUF) did not work on the OpenVINO backend. Three backend bugs, fixed in this PR:

  1. LFM2 models fail context create on GPU (0-byte tensor in a USM buffer).
  2. Raw F32 embedding inputs get a static token dim, so the DSpark / DFlash draft graph fails when the token count changes.
  3. Layer inputs flagged as outputs are not exported, so the drafter gets stale target features and draft acceptance is ~0%.

Additional information

Changes

1. Skip 0-byte tensors in init_tensor (ggml-openvino.cpp)

LFM2 has n_embd_s() == 0, so every cache_s_l* recurrent tensor is 0 bytes. Tensors named cache_* go to a GPU USM buffer, and init_tensor built an ov::Tensor extra for each of them. A 0-byte tensor at the end of the buffer points at the end of the USM allocation, and the GPU plugin rejects it:

[GPU] shared USM buffer has smaller size (0) than specified layout (4)

Now init_tensor does not create an extra for a 0-byte tensor. All consumers already handle extra == nullptr.

2. Dynamic token dim for raw F32 embedding inputs (ggml-decoder.h, ggml-decoder.cpp)

The DSpark KV-injection graph takes inp_target_features [n_embd_inp_enc, n_tokens] (5 x 2048 for LFM2.5-VL-3B). It is an F32 graph input that feeds MUL_MAT(fc, ...). The decoder did not see it as a token input, so it got a static shape. graph_key ignores shapes, so a model compiled for one chunk (455 tokens) was reused for the next (512):

[GPU] ... model input (shape=[1,1,455,10240]) and tensor (shape=[1,1,512,10240]) are incompatible

New helper is_inp_embd_leaf(tensor, op). The tensor must be an F32 graph-input leaf (GGML_OP_NONE, GGML_TENSOR_FLAG_INPUT), and one of these must hold:

  • it is src[1] of a MUL_MAT whose src[0] is a weight leaf (src[0]->op == GGML_OP_NONE). The weight-leaf check keeps inp_mean out: mean pooling is MUL_MAT(cont(transpose(inp)), inp_mean), so src[0] is computed.
  • it is src[0] of an RMS_NORM.

New member has_inp_embd_leaf_use(tensor): true if any node of the current graph matches. get_graph_input_shape() is called once per consumer and the last call wins, so the shape check must look at all consumers, not only the op it gets. The graph is scanned each time (no cached set), because update_io() rebinds a cached decoder to a new cgraph without recomputing per-tensor state.

It is used in:

  • get_graph_input_shape(): token dim dynamic (only when !m_is_static, so NPU is unchanged)
  • compute_node_dynamic_dims(): dynamic-dim root, dim 1, so the dim reaches the ops below (without it, Reshape of Vcur_injected-* keeps the old token count)

Coverage:

Input First consumer Covered
DFlash / DSpark inp_target_features MUL_MAT(fc) yes
EAGLE3 encoder inp_embd MUL_MAT(fc), or RMS_NORM with norm_before_fc yes
EAGLE3 decoder inp_g_embeddings RMS_NORM yes
Raw embd batch inp_embd (ubatch.embd, e.g. mtmd) RMS_NORM (attn norm) in most models yes
Raw embd batch, first op is not RMS_NORM (e.g. scale) other no (unchanged)
inp_mean MUL_MAT with computed src[0] no, on purpose

3. Export layer inputs flagged as outputs (ggml-decoder.cpp)

DSpark reads the inputs of 5 target layers after each target decode:

  • llama_set_embeddings_layer_inp() turns this on; the tensor is t_layer_inp[il], marked with ggml_set_output().
  • extract_layer_inputs() reads it with ggml_backend_tensor_get_async().

compute_model_outputs() did not check the output flag: a node became a Result only if it had no consumer in the graph. t_layer_inp[il] is always consumed by the layer norm, so it was never written back, and the drafter read stale memory.

Now a node flagged GGML_TENSOR_FLAG_OUTPUT stays a Result even when it is used inside the graph, except result_norm. llama flags result_norm (t_embd) in every graph but reads it only when embeddings are on.

Test results

Host: Intel Arc Pro B70 (32 GB), Windows 11, OpenVINO 2026.4, commit c63b77e39. base = same commit without this PR.

LFM2.5-VL-3B target alone (GPU)

llama-completion -m LFM2.5-VL-3B-F16.gguf -f prompt_1024.txt -c 1382 -n 128 -st -ngl -1 -fa 1 -no-cnv --ignore-eos

Build Result Prompt Generation
base context create fails (USM error) - -
this PR OK, coherent output 2352 t/s (963 tokens) 59.5 t/s

LFM2.5-VL-3B + DSpark drafter

llama-cli -m LFM2.5-VL-3B-F16.gguf -md LFM2.5-VL-3B-DSpark-F16.gguf --spec-type draft-dspark --spec-draft-n-max 8 --spec-draft-n-min 0 -f prompt_1024.txt -c 1382 -n 128 -st -ngl -1 -ngld -1 -fa on --ignore-eos

Build / device Result Draft acceptance
ggml-CPU reference (-dev none -devd none) OK 78 / 381 = 20.5%
base, GPU context create fails (change 1) -
changes 1 + 2 only, GPU OK, output correct 1 / 320 = 0.3%
changes 1 + 2 only, OpenVINO CPU OK, output correct 10 / 879 = 1.1%
this PR, GPU OK, output correct, 128 tokens 72 / 422 = 17.1%
  • test-backend-ops: 8547 / 8547 passed.

Impact on other models:

  • Greedy completion: same text on 6 other models.

  • KLD vs CPU (ctx 1024): no change.

    Model KLD this PR / base Same top this PR / base
    llama32_1b 0.053887 / 0.053894 88.540% / 88.527%
    smollm2_1p7b 0.093410 / 0.093418 85.335% / 85.335%
    phi4_mini 0.047983 / 0.047983 88.552% / 88.552%
    qwen35_4b 0.029749 / 0.029755 92.319% / 92.368%
    gemma4_e2b 0.367304 / 0.367304 77.838% / 77.838%
    phi35_mini 0.043021 / 0.043021 91.720% / 91.720%
    lfm25_vl_3b 0.000002 / fail 99.951% / fail
  • Decode and prefill: differences are inside the run-to-run spread (+-5-20%), with mixed signs.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES. AI used for issue analysis and test execution.

osabnis and others added 30 commits October 26, 2026 11:16
When the SWA cache size matches the full cache size, identify SWA
layers by the number of layers using each mask instead of cache size;
the larger group is SWA, and a tie leaves the layers unclassified.
Give SWA KV cache index tensors unique input names (a _swa suffix, like
the masks) to prevent them from being merged with the full-attention
cache indices.
Two attention masks do not imply SWA: DeepSeek sparse attention uses two
full-attention masks with the same name. When cache extents match,
classify the larger mask group as SWA only once its mask has fewer keys
per row than the other mask. Keep the classification per KV cache buffer
so new sequences reuse the same graph, and include swa_layers when
checking graph reuse.
Waiting for the window to appear in the mask caused a mid-sequence graph
recompile (~9 s for Gemma-4 at -c 1024 after position 512).

SWA classification already gives each mask group its own mask, KV write
indices, and attention size, which is correct whether or not the mask is
a sliding window. Stateful mask rebuild recovers the window from the
mask, or the causal mask for full attention. Classify the larger of two
mask groups from the first graph and drop the per-KV-buffer bookkeeping.
Add a TODO above the equal-extent fallback to document that the larger
mask group is currently assumed to be the SWA group. Since stateless
graphs only require the two groups to remain distinct, this heuristic
may only be necessary for stateful execution and could potentially be
restricted to that path.
Dynamic CPU/GPU graphs previously passed n_seq_active and
attention_size(_swa) as separate inputs. Derive these values directly
from the KQ mask shape using ShapeOf + Gather inside the model. This
allows the GPU plugin to infer the Q reshape and KV slice shapes from
shape inference rather than runtime input values. Q's head size also
becomes static, enabling the fused SDPA kernel.

GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK=1 restores the previous
separate-input behavior.

The fused SDPA path also requires two supporting changes:

- Clamp the GPU mask to -30000. A key block that is entirely -inf for
  a row produces NaNs in the fused kernel.
- Restate the head count and head size after the KV slice. The plugin
  loses all slice dimensions when the end index is dynamic, causing
  SDPA to fall back to the reference kernel.

Additional graph and quantization improvements:

- Skip the identity reshape of rows in SET_ROWS.
- Share a single Split for one-element view slices in dynamic GPU
  graphs (e.g. Gemma-4 per-layer embeddings), replacing one
  StridedSlice per layer.
- Add GGML_OPENVINO_REQUANT_EMBD=native (off by default) to preserve
  the file's native quantization for token_embd/output instead of
  requantizing to Q8_0_C.
- Include both environment settings in the graph and model cache keys
  and document them.
The dynamic GPU path reused a Split output for any single-element view
by deriving the chunk index from offset / stride. This produced
incorrect results for views that were not chunk-aligned or whose
remaining dimensions did not preserve the source strides, as seen in
test-backend-ops CONT with use_view_slice.

Restrict Split reuse to cases where the view exactly matches a split
chunk, meaning the offset is aligned to the split stride and the
remaining non-unit dimensions preserve the original source strides.
Gemma-4 per-layer embedding slices continue to use the shared Split
path.
Initialize the newly added `shape_source` and `shape_axis` fields in
the `ModelExtraInputInfo` brace initializer used by
`add_extra_inputs()`.

Without initializers for these fields, GCC emits
`-Wmissing-field-initializers`, causing CI to fail with
`LLAMA_FATAL_WARNINGS`.
The option identifies embedding and output weights by their tensor
names, which are defined by core llama.cpp. Remove it from this PR;
preserving the file quantization of these weights will be addressed in
a separate change that identifies them by their roles in the graph.
The manual GQA attention path reshaped Q, K, and V to 5-D so that
query heads sharing a KV head could broadcast against it. The GPU
plugin cannot fuse this graph: it materializes the attention scores
and copies the transposed K and V caches for every layer.

Instead, stack the query heads for each KV head along the sequence
axis (Q [B, H, S, D] -> [B, H_kv, factor * S, D]) and tile the mask
accordingly. This keeps the graph 4-D from the MatMuls through the
Softmax, allowing the GPU plugin to fuse the attention into its SDPA
kernel and fold the K and V transposes into the fused operation.

Clamp the mask to -30000, matching the SDPA path, because the fused
kernel can return NaN when an entire block is masked with -inf.

Models where the K or V head size differs from the Q head size now
use the SDPA path, since the manual path assumes one head size.
In stateless GPU mode, bind the entire KV cache as a
`[1, ctx, n_heads_kv, head_size]` tensor, with one token per row,
instead of binding the first `n_kv` rows as a
`[1, 1, ctx, n_heads_kv * head_size]` tensor.

Write cache updates with `ScatterUpdate` along the token axis, and
read the cache through `VariadicSplit` and `Transpose`. This allows
the GPU plugin's `StatelessKVFusion` to replace cache reads and writes
with in-place `StatelessKV` nodes, avoiding full-cache copies on every
decode step and repeated memory pool allocations as the bound cache
size grows.

Set `GGML_OPENVINO_DISABLE_KV_4D=1` to retain the previous layout.
With -fa off, llama.cpp stores the V cache transposed, with value
elements as rows and tokens along each row. The KV slicing code assumes
one token per row and binds the wrong data when n_kv < kv_size. Detect
transposed caches with GgmlOvDecoder::is_kvcache_token_rows() and skip
slicing for them.

For grouped-query attention, avoid repeating K and V for each query
head. Stack the query heads for each KV head along the row dimension
and let SOFT_MAX apply the scale, mask, and softmax so the GPU plugin
can fuse the operation with the MatMuls.

Read only the current layer's V-cache rows when layers use different
cache sizes. Reject -fa off in stateful mode with a clear error and
document that it requires stateless mode.
With `-fa off`, the graph writes the V cache in transposed form, which is
not supported by the stateful KV cache. Detect this pattern in
`compute_llm_params()` as a `SET_ROWS` into a KV cache through a `[1, n]`
reshape.

When stateful execution is enabled, log a warning and fall back to
stateless execution instead of failing graph conversion.

Update the `-fa off` note in `OPENVINO.md` to document this behavior.

Co-authored-by: Suhas Somashekar <suhas.somashekar@intel.com>
With -fa off, the 4-D KV cache layout introduced in ravi9#344 breaks MUL_MAT
attention for Phi-3.5 mini and Phi-4 mini, causing oneDNN to fail with
"could not construct a memory descriptor using strides".

Preserve the [1, 1, ctx, n_heads_kv * head_size] layout when the graph
writes a transposed V cache.
Q5_0 weights are requantized to Q4_0, but test-backend-ops builds naive
graphs, where requantization is skipped. Q5_0 then reached the normal
weight extraction, which had no Q5_0 case, so every Q5_0 MUL_MAT and
GET_ROWS op test failed with "Unsupported quantized type: q5_0".

Extract Q5_0 as symmetric i8 weights (q - 16) with one f16 scale per 32
weights, the same layout as Q8_0. Models are unchanged: they still
requantize Q5_0 to Q4_0.
The NPU GatedDeltaNet kernel writes its output in FP16 and flushes values below 6.1e-5 to zero.
1. Add envs desc into doc
2. Refine the code with removing duplicated env
Reduce host overhead outside `infer()` from 0.5–1.1 ms to 0.14–0.38 ms
per token on Panther Lake.

- Reuse KV cache tensors and skip redundant port bindings.
- Use persistent USM host tensors for small GPU inputs and outputs.
- Defer graph-wide consumer lookup until after shape checks.
- Return model and compute params by const reference.
- Preserve tensor extras when memory, shape and type are unchanged.
Add GGML_OPENVINO_NPU_KV_SLICE as an opt-in NPU prefill optimization. When enabled, static prefill keeps the graph's attention dimension at the active prompt/KV length instead of expanding it to the full context capacity, and the KV-cache input/output tensors are bound with a shortened context axis over the same backing storage.

This reduces the amount of KV cache and attention-mask data imported by the NPU plugin and lets the prefill graph plan smaller attention buffers for early prompt chunks. For example, with a 4096-token context and a 256-token prefill chunk, the first chunk can expose KV/mask dimensions at 256 instead of 4096, avoiding work over padded cache rows.

The trade-off is graph stability: NPU static shapes now depend on the aligned active attention length for prefill, so a cached prefill model is reused only when the attention sizes match. Decode keeps a fixed full-context graph to avoid recompiling on every generated token, and SWA/ring-cache cases still fall back to full-size binding when the live prefix is not a simple leading slice.
Add GGML_OPENVINO_NPU_L0_HOST_TENSORS as an opt-in NPU buffer allocation mode. When enabled, non-remote OpenVINO backend buffers on NPU are allocated through the NPU default context with create_host_tensor(), and ggml tensor data points directly at that OpenVINO-owned Level Zero host allocation.

The optimization targets the data path rather than the graph math: KV-cache and compute tensors can be passed back to the NPU plugin as OpenVINO tensors backed by importable Level Zero host memory. That avoids an extra ordinary-host-memory-to-plugin-staging copy on each infer boundary, which is especially useful for prefill where large KV/cache buffers dominate input and output traffic.

If Level Zero host allocation fails, the backend logs a warning and falls back to the existing allocation path. The destructor tracks OpenVINO-owned storage so it does not free memory owned by the host tensor. The trade-off is that this is NPU-specific, consumes Level Zero host allocation resources, and improves transfer/import overhead without changing attention compute complexity or tensor shapes.
Add GGML_OPENVINO_NPU_FAST_MASK as an opt-in NPU static prefill input-packing optimization. When enabled, the prefill attention mask is built directly in the OpenVINO input tensor instead of first materializing a padded std::vector and then copying that whole buffer into the tensor.

The new fill_prefill_mask() helper writes each row once: it copies the valid mask prefix, fills padding columns with -inf, handles padded rows, and restores the causal diagonal to zero. This keeps the final mask numerically identical to the existing pad_input() plus set_zero_diagonal() path while removing one intermediate allocation and one chunk_size-by-context_size host copy per prefill chunk.

The optimization targets CPU-side staging overhead, not attention math. It composes with GGML_OPENVINO_NPU_KV_SLICE: KV slice reduces the mask dimensions, while FAST_MASK reduces the work needed to construct whatever mask shape remains. The trade-off is a second mask construction path to maintain, with limited benefit for very small prompts or contexts.
Add GGML_OPENVINO_NPU_REQUANT_POLICY as the NPU-side selector for the default weight requantization layout. The existing NPU behavior remains the default: unset or group-128 maps quantized weights to Q4_0_128, while channel-wise selects the experimental Q4_0_C layout. Unknown policy values fail fast during model loading instead of silently compiling a graph with unintended weight packing.

The optimization target is NPU weight bandwidth and compressed-weight code generation. Q4_0_128 keeps weights in a 4-bit symmetric layout with one scale per 128 values, reducing both payload bytes and scale metadata versus smaller groups while preserving a more conservative quantization granularity than channel-wise. Q4_0_C uses one scale over the innermost dimension, which can further shrink metadata and simplify dequant broadcast, but may cost accuracy because the quantization block is much larger.

Include the selected NPU requant policy in the OpenVINO model cache discriminator. The same cgraph compiled with group-128 and channel-wise has different weight constants and dequant shapes, so reusing a cached decoder across policies would be incorrect. This keeps cache reuse stable within a policy while forcing recompilation when the layout policy changes.

Trade-off: policy selection gives a performance/accuracy knob for NPU experiments, but all options still pay the load/compile-time requantization cost. The default remains group-128 for a safer balance of bandwidth, metadata, and accuracy; channel-wise is intentionally documented as experimental.
Add GGML_OPENVINO_NPU_CONFIG as the recommended catch-all override for OpenVINO NPU plugin and compiler properties. The value is parsed as comma-separated KEY=VALUE pairs and applied after the built-in defaults and named NPU aliases, so advanced users can override any exposed property without adding a new llama.cpp environment variable for every plugin tuning knob.

Keep the common knobs as named aliases for discoverability and compatibility: GGML_OPENVINO_NPU_COMPILER_TYPE maps to NPU_COMPILER_TYPE, GGML_OPENVINO_NPUW_FUNCALL_FOR_ALL maps to NPUW_FUNCALL_FOR_ALL, GGML_OPENVINO_NPUW_UNFOLD_IREQS maps to NPUW_UNFOLD_IREQS, and GGML_OPENVINO_COMPILATION_NUM_THREADS maps to COMPILATION_NUM_THREADS. GGML_OPENVINO_NPU_COMPILE_CONFIG is retained as a legacy alias for NPU_COMPILATION_MODE_PARAMS, while the documentation now recommends expressing it through GGML_OPENVINO_NPU_CONFIG.

This keeps the public tuning surface compact while preserving high-signal aliases for settings with known performance or stability implications. NPU_COMPILER_TYPE can select the driver compiler, which currently generates faster prefill kernels on the tested NPU stack. NPUW_FUNCALL_FOR_ALL and NPUW_UNFOLD_IREQS expose NPUW graph lowering trade-offs around dispatch overhead, memory use, and long-context stability. COMPILATION_NUM_THREADS gives a way to cap compiler worker parallelism to reduce peak host memory during large graph compilation.

Trade-off: a free-form KEY=VALUE string is less type-safe than one env var per option and invalid plugin properties will only be diagnosed by the OpenVINO stack. Applying it last is intentional: it gives one escape hatch for bisecting driver/compiler behavior and for testing new plugin options without rebuilding llama.cpp, while the named aliases continue to document the most important performance knobs.
@ravi9
ravi9 force-pushed the dev_backend_openvino branch from 2545829 to 68abd50 Compare November 6, 2026 19:27
@swathiananth
swathiananth marked this pull request as ready for review November 6, 2026 20:40
@swathiananth

Copy link
Copy Markdown
Author

The test is failing after the rebase. Root caused to the upstream commit 040ce19, "graph: store the MTP conv snapshots of every recurrent model in one set_rows" (ggml-org#31660), It changed how the conv state is saved when rollback slots are on (n_rs_seq > 0, i.e. speculative / MTP decoding).

@swathiananth
swathiananth marked this pull request as draft November 7, 2026 00:02
@ravi9
ravi9 force-pushed the dev_backend_openvino branch 5 times, most recently from a74e40b to d97e07f Compare November 12, 2026 20:43
@swathiananth
swathiananth marked this pull request as ready for review November 12, 2026 23:01
@swathiananth
swathiananth marked this pull request as draft November 12, 2026 23:02
@ravi9
ravi9 force-pushed the dev_backend_openvino branch 5 times, most recently from d1990e9 to 2910fcd Compare November 16, 2026 17:35
@swathiananth
swathiananth marked this pull request as ready for review November 18, 2026 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants