Repository navigation
Qwen MTP and multi-seq optimization with new OV ops - #370
Merged
Merged
Conversation
When the SWA cache size matches the full cache size, identify SWA layers by the number of layers using each mask instead of cache size; the larger group is SWA, and a tie leaves the layers unclassified.
Give SWA KV cache index tensors unique input names (a _swa suffix, like the masks) to prevent them from being merged with the full-attention cache indices.
Two attention masks do not imply SWA: DeepSeek sparse attention uses two full-attention masks with the same name. When cache extents match, classify the larger mask group as SWA only once its mask has fewer keys per row than the other mask. Keep the classification per KV cache buffer so new sequences reuse the same graph, and include swa_layers when checking graph reuse.
Waiting for the window to appear in the mask caused a mid-sequence graph recompile (~9 s for Gemma-4 at -c 1024 after position 512). SWA classification already gives each mask group its own mask, KV write indices, and attention size, which is correct whether or not the mask is a sliding window. Stateful mask rebuild recovers the window from the mask, or the causal mask for full attention. Classify the larger of two mask groups from the first graph and drop the per-KV-buffer bookkeeping.
Upstream ggml-org#29622 adds a mixed token/embd branch to every input embedding graph through ggml_build_forward_select(). Its nodes are not flagged for compute, but the backend translated them anyway, and the DUP in that branch was unsupported, so the scheduler split the graph and passed the embeddings across the split with a fixed token count. The first single-token decode then failed (test-thread-safety on CPU and GPU). Build the OV model from the compute nodes only, and translate a same-type contiguous DUP like CONT so the graph stays on one backend.
ggml-org#29622 also moves the per-token embedding scale (gemma3, gemma3n, gemma4) into a new [1, n_tokens] input. Give it a dynamic token dim and pad it per chunk on the static (NPU) path.
Op tests build Q4_1/Q4_K weights as u4 with an f16 zero point. The GPU plugin fails to compile that form for some row counts with "clFinish, error code: -5 CL_OUT_OF_RESOURCES", which aborts test-backend-ops on the MUL_MAT cases added in ggml-org#29869 (e.g. m=1000, n=2, k=1024). Model weights use a u4 zero point and are not affected. Report these cases as unsupported on GPU until the plugin is fixed. Op tests check support before allocating, so the check matches unbound weights only; model loading probes with a dummy buffer and keeps its weights on the GPU.
translate_fill always built an f32 constant, so an f16 FILL produced f32 data and the copy back overran the f16 output buffer. Use the output type for the constant.
Quantized inputs are dequantized when translated, so the backend cannot write a quantized CONCAT output. Report it as unsupported, as for CPY to a quantized type.
ggml-org#29856 changed build_rs to gather all recurrent states with one GET_ROWS on the s_copy leaf and take the ubatch and extra states as views of it. The stateful path matched only the previous form, a GET_ROWS per view of s_copy, so Qwen3.5 failed with stateful execution on CPU and GPU ("is_axis_valid(axis, r)" in a Concat). For a single-slot cache, treat the GET_ROWS on the s_copy leaf as the active-state gather, keep the rank-4 layout of reshapes that read a view of it, and map the copy of the empty extra-state view to the single-slot remainder writeback. Do not warn about the dynamic dim of empty views.
…randRanks The pass unsqueezes the lower-rank operand of an Add/Multiply/Subtract whose operand ranks differ. In gemma-3 the lower-rank operand of the post-attention residual add is the norm output, and unsqueezing it makes the GPU plugin compute the layer wrongly: gemma-3 returns empty answers on GPU with stateful execution. Skip the rewrite when the lower-rank operand is an RMS norm output.
Dynamic CPU/GPU graphs previously passed n_seq_active and attention_size(_swa) as separate inputs. Derive these values directly from the KQ mask shape using ShapeOf + Gather inside the model. This allows the GPU plugin to infer the Q reshape and KV slice shapes from shape inference rather than runtime input values. Q's head size also becomes static, enabling the fused SDPA kernel. GGML_OPENVINO_DISABLE_SHAPE_FROM_MASK=1 restores the previous separate-input behavior. The fused SDPA path also requires two supporting changes: - Clamp the GPU mask to -30000. A key block that is entirely -inf for a row produces NaNs in the fused kernel. - Restate the head count and head size after the KV slice. The plugin loses all slice dimensions when the end index is dynamic, causing SDPA to fall back to the reference kernel. Additional graph and quantization improvements: - Skip the identity reshape of rows in SET_ROWS. - Share a single Split for one-element view slices in dynamic GPU graphs (e.g. Gemma-4 per-layer embeddings), replacing one StridedSlice per layer. - Add GGML_OPENVINO_REQUANT_EMBD=native (off by default) to preserve the file's native quantization for token_embd/output instead of requantizing to Q8_0_C. - Include both environment settings in the graph and model cache keys and document them.
The dynamic GPU path reused a Split output for any single-element view by deriving the chunk index from offset / stride. This produced incorrect results for views that were not chunk-aligned or whose remaining dimensions did not preserve the source strides, as seen in test-backend-ops CONT with use_view_slice. Restrict Split reuse to cases where the view exactly matches a split chunk, meaning the offset is aligned to the split stride and the remaining non-unit dimensions preserve the original source strides. Gemma-4 per-layer embedding slices continue to use the shared Split path.
Add a TODO above the equal-extent fallback to document that the larger mask group is currently assumed to be the SWA group. Since stateless graphs only require the two groups to remain distinct, this heuristic may only be necessary for stateful execution and could potentially be restricted to that path.
ggml-openvino: Fix SWA layer detection and KV write indices (Gemma-4-E2B past 512 tokens)
Initialize the newly added `shape_source` and `shape_axis` fields in the `ModelExtraInputInfo` brace initializer used by `add_extra_inputs()`. Without initializers for these fields, GCC emits `-Wmissing-field-initializers`, causing CI to fail with `LLAMA_FATAL_WARNINGS`.
The option identifies embedding and output weights by their tensor names, which are defined by core llama.cpp. Remove it from this PR; preserving the file quantization of these weights will be addressed in a separate change that identifies them by their roles in the graph.
ggml-openvino: derive attention sizes from the KQ mask shape
Assisted-by: Codex
Use runtime reset, reorder and partial writeback paths at every stateless slot count. Preserve initially empty cache writes across graph reuse, select active state rows dynamically, and fix reset byte offsets. Add a four-slot reset/copy/rollback regression and document the corrected performance baseline and native cache-fusion opportunities. Validate the 35B model in four-slot serving with and without MTP, plus CPU/GPU operator checks and the existing hidden-state rollback test. Assisted-by: Codex
Translate recurrent reset, reorder and writeback through native GPU cache kernels. Derive active dimensions from row-map shapes, reuse USM host controls and fold GDN state layouts into gather/update. Keep primitive fallbacks, update cache compatibility, move development reports into dev_doc and record Qwen3.6 UD-Q4_K_M as the new target. Validated clean build, recurrent regressions, native tests and four-slot serving with and without MTP. Requires native OpenVINO commit 9e08e52ad5 for the optimized GPU path.
Convert dense Q8_0/Q5_1 under q4_asym64_all while preserving channel-wise INT8 embedding/output overrides and backup for incompatible row widths. Invalidate compiled-model caches from the previous precision policy. Document converted layouts and the five-repeat UD/yamini performance retest: UD tg512 improves from 27.41 to 33.17 tokens/s and matches yamini at 33.21 tokens/s. Release build and CPU/GPU packed-weight conversion checks pass.
Assisted-by: Codex
Assisted-by: Codex
Keep pure top-k branches and their output names stable, pack runtime sampling rows into one input, and bind inactive results to private buffers. Add four-slot cache-only reuse coverage and document UD server validation. Assisted-by: Codex
Assisted-by: Codex
Keep current in-place and stable-sampling guidance, measured benchmark matrices and a concise decision history. Preserve historical Vulkan rows when regenerating the compact report. Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Keep the released ordinary GDN constructor and use the independent recurrent update op for standalone snapshots. Skip full-model recurrence setup in naive graphs and warn when an unpatched package cannot execute snapshot/rollback graphs. Bump the compiled-model cache revision to 14.
Assisted-by: Codex
Assisted-by: Codex
Revert the edits outside ggml/src/ggml-openvino (llama-graph, llama-quant, test-backend-ops, test-backend-sampler, server-bench scripts) and remove the benchmark scripts and result documents from dev_doc. Keep the documentation that describes the current implementation. Tested Qwen3.5-0.8B F16 on GPU with the fused-op OpenVINO build and with the released 2026.4.1 package. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ravi9
pushed a commit
that referenced
this pull request
Nov 5, 2026
All changes of #370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder
ravi9
pushed a commit
that referenced
this pull request
Nov 6, 2026
All changes of #370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder
ravi9
pushed a commit
that referenced
this pull request
Nov 6, 2026
All changes of #370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder
ravi9
pushed a commit
that referenced
this pull request
Nov 10, 2026
All changes of #370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder
ravi9
pushed a commit
that referenced
this pull request
Nov 11, 2026
All changes of #370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder
ravi9
pushed a commit
that referenced
this pull request
Nov 12, 2026
All changes of #370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder
ravi9
pushed a commit
that referenced
this pull request
Nov 12, 2026
All changes of #370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder
ravi9
pushed a commit
that referenced
this pull request
Nov 12, 2026
All changes of #370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder
ravi9
pushed a commit
to nazanin-beheshti/llama.cpp
that referenced
this pull request
Nov 12, 2026
All changes of ravi9#370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder
ravi9
pushed a commit
that referenced
this pull request
Nov 14, 2026
All changes of #370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder
ravi9
pushed a commit
that referenced
this pull request
Nov 15, 2026
All changes of #370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder
ravi9
pushed a commit
that referenced
this pull request
Nov 16, 2026
All changes of #370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder
ravi9
pushed a commit
that referenced
this pull request
Nov 16, 2026
All changes of #370 as one commit. The PR branch had dev_backend_openvino merged into it, so its commits could not be replayed one by one when dev_backend_openvino was rebased onto upstream master. Commits of the PR: - openvino : support Qwen MTP decoding - openvino: Optimize TopK - openvino: remove single-slot recurrent cache shortcuts - openvino: fuse generic recurrent caches and remove shape waits - openvino: include dense Q8 weights in all INT4 requantization - openvino: use two GDN outputs for snapshots - docs: explain native OpenVINO recurrent cache contracts - docs: clarify Qwen3.5 purpose of native OpenVINO cache ops - llama: skip unallocated hybrid graph inputs - quantize: preserve Qwen hybrid layer metadata when pruning - openvino: emit fused recurrent computation and rotate device caches - openvino: limit KV slicing to attention caches and share compiled blobs - tools: add natural-text server benchmark and OpenVINO profile summary - docs: record fused-cache branch split and UD validation - server-bench: preserve native token counts and save request timings - openvino: document MTP regression and host USM alias recovery - openvino: record large-context allocation diagnosis and default-batch validation - server-bench: count buffered native tokens and reject stream errors - openvino: document UD split and unified server benchmarks - openvino: record Vulkan long-context baseline and GPU-fit controls - openvino: finalize UD benchmark report and MTP comparison - openvino: use stable in-place recurrent caches and optional separate convolution - openvino: accept matching empty axes in DIV repeat translation - openvino: remove the obsolete cache rotation environment setting - openvino: validate imported KV results in their four-dimensional layout - docs: record the bounded in-place KV and convolution benchmark rerun - docs: complete default-batch OV unified-KV 32K benchmarks - openvino: remove the separate recurrent convolution path - openvino: handle zero-logits batches and changing sampler sequences - docs: compare default and large Qwen3.6 benchmark batches - docs: use the historical Vulkan default-batch reference consistently - server-bench: use an integer histogram bin count - benchmarks: summarize current OV unified MMLU timings - docs: record cache-only unified KV MMLU results - openvino: preserve the TopK index port in GET_ROWS - openvino: reuse the MTP sampling model across active slots - openvino: expose selection flags and offsets in graph dumps - docs: consolidate Qwen OpenVINO defaults and performance evidence - docs: retain continuous-request MTP benchmark results - docs: compare stable MTP with the non-MTP serving baseline - openvino: isolate GDN snapshots and skip naive cache fusion - ggml: document Qwen3.6 GenAI benchmarks - ggml-openvino: keep PR changes inside the backend folder
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enable Qwen3.5/Qwen3.6 multi-sequence recurrent-cache updates and MTP in the OpenVINO backend. The new recurrent ops combine reset/reorder, recurrent computation, and direct cache writeback; the sampling graph is reused across changing active slots.
Requires the matching OpenVINO ops in openvino#39355, stacked on the dynamic-output binding fix. Ordinary GDN semantics remain unchanged, and naive graphs skip full-model fusion setup. Snapshot/rollback with an unpatched OV package uses scheduler fallback.
Batched benchmark
Qwen3.6-35B-A3B-UD-Q4_K_M, Intel Arc B390 iGPU, context 524288, default batch/microbatch 2048/512, 512 outputs per sequence. Each cell is prefill seconds @ aggregate decode tokens/s, one active / four active sequences.
-npl 1,4 --kv-unified-npl 1,4 --kv-unified-npl 1,4 --kv-unifiedThe OV rows use the same binaries with recurrent fusion enabled/disabled; decode throughput improves by 67-101% at 1K/8K with the fused in-place ops. OV uses
q4_asym64_alland native softplus. Vulkan uses stored GGUF weights and full GPU offload with the preserveddd06588b1build. Its 32K/four-active result was measured on 2026-11-03; the other Vulkan references are from 2026-10-23. OV measurements are from 2026-10-30 (604e22e24, OV8e4210a33b). Missing cases remain explicit.Continuous MMLU scheduling
Eight question texts, 512 outputs per request, four server slots, unified KV, context 524288, and immediate request replacement. Values are native decode tokens/s per request, excluding prompt evaluation.
OV MMLU measurements are from 2026-10-30; Vulkan references are full-GPU runs from 2026-10-23. Both use the same questions and output length. These are single-pass measurements.
See benchmark details and reproduction.
Validation
Merged
dev_backend_openvinoatc63b77e39; conflict resolution and the completed Vulkan benchmark document are inb502f715b.