Skip to content

ggml-openvino: correct the RESHAPE dynamic-dim comment for size-1 dims - #382

Merged
ravi9 merged 1 commit into
ravi9:dev_backend_openvinofrom
osabnis:ov-reshape-size1-dim
Nov 5, 2026
Merged

ravi9 merged 1 commit into
ravi9:dev_backend_openvinofrom
osabnis:ov-reshape-size1-dim

Conversation

@osabnis

@osabnis osabnis commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Update: the fix itself is now in dev_backend_openvino through #384, which has the same logic. This PR is rebased onto dev and now carries only the comment correction: the old comment said the RESHAPE stride match is unambiguous, which a size-1 dim breaks. The testing below also covers the merged logic.

The original description follows.


Problem

Models with a single KV head fail on the current dev_backend_openvino tip in all configurations:

Check 'backward_compatible_check || in_out_elements_equal' failed
While validating node 'opset1::Reshape reshape:Kcur-0 (view)'
Shape inference input shapes {[1,512,1,256],[4]}
Requested output shape [1,1,2,256] is incompatible with input shape

Affected models (all 6 configurations: GGML_OPENVINO_STATEFUL_EXECUTION=0/1 × -fa on/off/auto):

  • Gemma-3 1B IT (ggml-org Q4_K_M)
  • Gemma-3 270M (QuantFactory Q4_K_M)
  • Gemma-4-E2B IT (Q4_K_M)

All three models run on e602f31e9 (the commit before #370), so #370 exposed the issue. The underlying dynamic-dim
rule predates #370.

Cause

For ggml_reshape_3d(Kcur, head_dim, 1, n_tokens), the size-1 head dimension and the token dimension have the same
stride.

The RESHAPE rule in compute_node_dynamic_dims() selected the first dimension with the source stride. This marked the
head dimension as dynamic and left the token dimension static at the warm-up size of 2.

Fix

When several dimensions match the source stride, prefer the one that also matches the source dimension's size. If none
does, fall back to the first stride match, as before.

This only changes RESHAPEs where the previous match had a different size from the source dimension. #375 (open) applies
the same approach to VIEW nodes.

Testing

Environment: B70 (GPU.0), OpenVINO 2026.4.1, dev_backend_openvino at 9c238a1e5 (the two later merges, #379 and
#381, do not touch ggml-decoder.cpp).

Command: llama-completion -f <1024-token prompt> -c 1382 -n 128 -st -ngl -1 -no-cnv --ignore-eos --temp 0

Each model was run in all 6 configurations (GGML_OPENVINO_STATEFUL_EXECUTION=0/1 × -fa on/off/auto).

Affected models

Model Before After
Gemma-3 1B IT Fails 6/6 Passes 6/6; same output in all 6
Gemma-3 270M Fails 6/6 Passes 6/6
Gemma-4-E2B IT Fails 6/6 Passes 6/6

With the fix, Gemma-3 1B IT and Gemma-4-E2B IT give the same output as e602f31e9 (before #370), stateless with
-fa on.

Regression testing

Tip vs tip + this PR, same 6 configurations:

  • Llama-3.2-1B
  • SmolLM2-1.7B
  • Phi-3.5-mini
  • Phi-4-mini
  • Qwen3.5-4B
  • Gemma-2-2B
  • SmolLM3-3B
  • Qwen1.5-MoE

All give the same output, except for run-to-run GPU variation in Phi-3.5-mini and Gemma-2-2B, which shows on both
trees. A debug build that logs every RESHAPE whose dynamic dimension changes found none in those two models.

Among the tested models, the fix changes the dynamic dimension only for RESHAPEs in Gemma-3/4 and Qwen1.5-MoE. In
Qwen1.5-MoE, the ffn_norm RESHAPE now selects the token dimension; its output is unchanged.

Panther Lake (Arc B390 iGPU), OpenVINO 2026.4.1: same command and configurations, tip vs tip + this PR.

  • Gemma-3 1B IT, Gemma-3 270M, Gemma-4-E2B IT: fail 6/6 on the tip, pass 6/6 with this PR.
  • Gemma-3 4B IT (4 KV heads), Llama-3.2-1B, SmolLM2-1.7B, Phi-3.5-mini, Phi-4-mini, Qwen3.5-4B, Gemma-2-2B,
    SmolLM3-3B, Qwen1.5-MoE: same output on both trees in all 6 configurations.
  • No ops fall back to CPU in any run.

The comment above the RESHAPE stride match in compute_node_dynamic_dims()
still said the match is unambiguous because result strides are strictly
monotone. A size-1 dim has the same stride as the dim after it
([head_dim, T] -> [head_dim, 1, T] with one KV head), so it is not.

Explain the size-1 case in the block comment, drop the duplicate comment
inside the loop, and name the source dim's size once.
@ravi9
ravi9 force-pushed the ov-reshape-size1-dim branch from 2d8ffb6 to 5b0149d Compare November 5, 2026 00:55
@ravi9 ravi9 changed the title ggml-openvino: fix dynamic token dim detection for single-KV-head models ggml-openvino: correct the RESHAPE dynamic-dim comment for size-1 dims Nov 5, 2026
@ZackyLake

Copy link
Copy Markdown

actually, why not simply enforce ne and nb both matching with dynamic_dim?

@ravi9
ravi9 merged commit 8b96e9f into ravi9:dev_backend_openvino Nov 5, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants