Repository navigation
ggml-openvino: correct the RESHAPE dynamic-dim comment for size-1 dims - #382
Merged
Merged
Conversation
The comment above the RESHAPE stride match in compute_node_dynamic_dims() still said the match is unambiguous because result strides are strictly monotone. A size-1 dim has the same stride as the dim after it ([head_dim, T] -> [head_dim, 1, T] with one KV head), so it is not. Explain the size-1 case in the block comment, drop the duplicate comment inside the loop, and name the source dim's size once.
ravi9
force-pushed
the
ov-reshape-size1-dim
branch
from
November 5, 2026 00:55
2d8ffb6 to
5b0149d
Compare
|
actually, why not simply enforce ne and nb both matching with dynamic_dim? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update: the fix itself is now in dev_backend_openvino through #384, which has the same logic. This PR is rebased onto dev and now carries only the comment correction: the old comment said the RESHAPE stride match is unambiguous, which a size-1 dim breaks. The testing below also covers the merged logic.
The original description follows.
Problem
Models with a single KV head fail on the current
dev_backend_openvinotip in all configurations:Affected models (all 6 configurations:
GGML_OPENVINO_STATEFUL_EXECUTION=0/1×-fa on/off/auto):All three models run on
e602f31e9(the commit before #370), so #370 exposed the issue. The underlying dynamic-dimrule predates #370.
Cause
For
ggml_reshape_3d(Kcur, head_dim, 1, n_tokens), the size-1 head dimension and the token dimension have the samestride.
The RESHAPE rule in
compute_node_dynamic_dims()selected the first dimension with the source stride. This marked thehead dimension as dynamic and left the token dimension static at the warm-up size of 2.
Fix
When several dimensions match the source stride, prefer the one that also matches the source dimension's size. If none
does, fall back to the first stride match, as before.
This only changes RESHAPEs where the previous match had a different size from the source dimension. #375 (open) applies
the same approach to VIEW nodes.
Testing
Environment: B70 (GPU.0), OpenVINO 2026.4.1,
dev_backend_openvinoat9c238a1e5(the two later merges, #379 and#381, do not touch
ggml-decoder.cpp).Command:
llama-completion -f <1024-token prompt> -c 1382 -n 128 -st -ngl -1 -no-cnv --ignore-eos --temp 0Each model was run in all 6 configurations (
GGML_OPENVINO_STATEFUL_EXECUTION=0/1×-fa on/off/auto).Affected models
With the fix, Gemma-3 1B IT and Gemma-4-E2B IT give the same output as
e602f31e9(before #370), stateless with-fa on.Regression testing
Tip vs tip + this PR, same 6 configurations:
All give the same output, except for run-to-run GPU variation in Phi-3.5-mini and Gemma-2-2B, which shows on both
trees. A debug build that logs every RESHAPE whose dynamic dimension changes found none in those two models.
Among the tested models, the fix changes the dynamic dimension only for RESHAPEs in Gemma-3/4 and Qwen1.5-MoE. In
Qwen1.5-MoE, the
ffn_normRESHAPE now selects the token dimension; its output is unchanged.Panther Lake (Arc B390 iGPU), OpenVINO 2026.4.1: same command and configurations, tip vs tip + this PR.
SmolLM3-3B, Qwen1.5-MoE: same output on both trees in all 6 configurations.