Skip to content

ggml-openvino: fix Qwen3.5 perplexity on NPU and Gemma 4 E2B - #384

Merged
ravi9 merged 2 commits into
dev_backend_openvinofrom
fix-ov-output-idx-reshape-dim
Nov 4, 2026
Merged

ravi9 merged 2 commits into
dev_backend_openvinofrom
fix-ov-output-idx-reshape-dim

Conversation

@ravi9

@ravi9 ravi9 commented Nov 4, 2026

Copy link
Copy Markdown
Owner

Two regressions that came in with #370.

Qwen3.5 perplexity on NPU

The output_len scan now runs for every model and takes the first GET_ROWS whose index is a graph input. On Qwen3.5 that is the recurrent state gather of build_rs, which indexes the inp->s_copy leaf directly, so output_len became 1. In static (NPU) mode the prompt model then returned one logits row per chunk, and llama-perplexity read stale rows (PPL about 240000 on Qwen3.5-2B). Generation needs only the last row, so it was not affected.

  • is_output_idx skips GET_ROWS that read a recurrent state cache.
  • The old predicate stays as is_get_rows_input_idx for the dynamic-dim bookkeeping, so dynamic CPU/GPU graphs do not change.

Gemma 4 E2B on all devices

The RESHAPE dynamic-dim rule matches the source's dynamic stride against the result's strides. A size-1 dim repeats the next dim's stride: Gemma 4 E2B has one KV head, so [head_size, n_tokens] -> [head_size, 1, n_tokens] has the same stride on dims 1 and 2. The first match picked the size-1 head dim, so the K/V views lost their dynamic token dim and became static reshapes with the first graph's token count. The first decode step after the prompt then failed with a Reshape error on Kcur_pos-0 (view).

  • Prefer the stride match that keeps the dynamic size, and fall back to the first stride match for reshapes that change it.

ravi9 added 2 commits November 5, 2026 03:48
The output_len scan now runs for every model and takes the first GET_ROWS
whose index is a graph input. On Qwen3.5 that is the recurrent state gather
of build_rs, which indexes the inp->s_copy leaf directly, so output_len
became 1. In static (NPU) mode the prompt model then returned one logits
row per chunk, and llama-perplexity read stale rows (PPL about 240000 on
Qwen3.5-2B). The same predicate also gave the s_copy input the out_ids
shape and data in static mode.

is_output_idx now skips GET_ROWS that read a recurrent state cache. The old
predicate stays as is_get_rows_input_idx for the dynamic-dim bookkeeping,
so the s_copy leaf keeps its dynamic length there.
…a stride

The RESHAPE dynamic-dim rule matches the source's dynamic stride against
the result's strides. A size-1 dim repeats the next dim's stride: Gemma 4
E2B has one KV head, so [head_size, n_tokens] -> [head_size, 1, n_tokens]
has the same stride on dims 1 and 2. The first match picked the size-1 head
dim, so the K/V views lost their dynamic token dim and became static
reshapes with the first graph's token count. Every later graph with another
token count failed.

Prefer the stride match that keeps the dynamic size, and fall back to the
first stride match for reshapes that change it.
@ravi9
ravi9 merged commit d7f796c into dev_backend_openvino Nov 4, 2026
3 checks passed
@ravi9
ravi9 deleted the fix-ov-output-idx-reshape-dim branch November 11, 2026 01:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant