Repository navigation
ggml-openvino: fix Qwen3.5 perplexity on NPU and Gemma 4 E2B - #384
Merged
Merged
Conversation
The output_len scan now runs for every model and takes the first GET_ROWS whose index is a graph input. On Qwen3.5 that is the recurrent state gather of build_rs, which indexes the inp->s_copy leaf directly, so output_len became 1. In static (NPU) mode the prompt model then returned one logits row per chunk, and llama-perplexity read stale rows (PPL about 240000 on Qwen3.5-2B). The same predicate also gave the s_copy input the out_ids shape and data in static mode. is_output_idx now skips GET_ROWS that read a recurrent state cache. The old predicate stays as is_get_rows_input_idx for the dynamic-dim bookkeeping, so the s_copy leaf keeps its dynamic length there.
…a stride The RESHAPE dynamic-dim rule matches the source's dynamic stride against the result's strides. A size-1 dim repeats the next dim's stride: Gemma 4 E2B has one KV head, so [head_size, n_tokens] -> [head_size, 1, n_tokens] has the same stride on dims 1 and 2. The first match picked the size-1 head dim, so the K/V views lost their dynamic token dim and became static reshapes with the first graph's token count. Every later graph with another token count failed. Prefer the stride match that keeps the dynamic size, and fall back to the first stride match for reshapes that change it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two regressions that came in with #370.
Qwen3.5 perplexity on NPU
The output_len scan now runs for every model and takes the first
GET_ROWSwhose index is a graph input. On Qwen3.5 that is the recurrent state gather ofbuild_rs, which indexes theinp->s_copyleaf directly, so output_len became 1. In static (NPU) mode the prompt model then returned one logits row per chunk, and llama-perplexity read stale rows (PPL about 240000 on Qwen3.5-2B). Generation needs only the last row, so it was not affected.is_output_idxskipsGET_ROWSthat read a recurrent state cache.is_get_rows_input_idxfor the dynamic-dim bookkeeping, so dynamic CPU/GPU graphs do not change.Gemma 4 E2B on all devices
The RESHAPE dynamic-dim rule matches the source's dynamic stride against the result's strides. A size-1 dim repeats the next dim's stride: Gemma 4 E2B has one KV head, so
[head_size, n_tokens] -> [head_size, 1, n_tokens]has the same stride on dims 1 and 2. The first match picked the size-1 head dim, so the K/V views lost their dynamic token dim and became static reshapes with the first graph's token count. The first decode step after the prompt then failed with a Reshape error onKcur_pos-0 (view).