Skip to content

fix(oga-hipep): no vision warm-up and no shared image budget on Qwen3.5 (WP-23) - #1064

Open
Allegro-ma-non-troppo wants to merge 1 commit into
mainfrom
wp-23-oga-image-turns
Open

Allegro-ma-non-troppo wants to merge 1 commit into
mainfrom
wp-23-oga-image-turns

Conversation

@Allegro-ma-non-troppo

Copy link
Copy Markdown
Collaborator

Why

On the Qwen3.5/3.6/3.8 -oga-hipep models:

  • The first image request after a load stalled. FLM first ran the vision model on a blank 8100-patch image, which took 5.5-7 s inside the user's request: TTFT 6.7 s, against 0.9 s for a text request of the same length.
  • The images of one prompt shared a budget of 8100 patches. At the default --img-pre-resize 2, a third 1280x720 screenshot shrank all of them: 3 screenshots gave 1969 tokens instead of 2671. Earlier images changed their size and their tokens when another image arrived.

Both were workarounds for hip-ep 0.5.2 (onnxruntime-genai 0.17), which gave wrong image features without them. FLM now requires onnxruntime-genai 0.19.

What changed

  • No vision warm-up.
    • OgaEngine::WarmUpVision and the per-prompt patch limit are gone (kMaxImagePatches, OgaVisionConfig::max_patches).
    • On 0.19 the warm-up does not change the answers. The run reproduced the old trigger, a small first image and then a large one, in two sizes: 256 then 7056 patches, and 256 then 32400 patches.
  • Each image is sized on its own, as on the NPU and in the Hugging Face Qwen2-VL processor that vLLM and SGLang use.
    • Each image is resized to at most the height --img-pre-resize sets, within the package's max_pixels.
    • Another image no longer makes it smaller, so it keeps its tokens.
  • Early refusal. On OGA, a prompt whose images alone reach the context gets HTTP 400 at that image. The remaining images are not preprocessed first. The message names the images and their tokens.
  • Docs. cli.md "Images" describes the per-image sizing, the measured times and the 400.

What it costs

The owner chose per-image sizing with no total (run log Q1 and Q2). Images are larger than before, so they take longer. All times are TTFT on a Radeon 8060S with qwen3.5-oga-hipep:9b.

1280x720 screenshots in one prompt main fc13a07 this PR prompt tokens, main / PR
3 3.65 s 5.12 s 1992 / 2694
4 3.54 s 7.92 s 1976 / 3576
6 3.55 s 9.90 s 1932 / 5340
8 3.53 s 14.35 s 1912 / 7104

Medians of 5 runs, 3 for the PR's 6-screenshot row. Each run is a new conversation with new images, and every answer names the colors in order.

  • A conversation pays this again each time it gets a new image, because a new image replays the conversation (see Follow-ups).
  • One image can now have 65536 patches, where main stopped at 8100. A 23300x720 panorama at the default setting takes 126 s (main 12 s, 6 of them its warm-up). A 4096x4096 image at --img-pre-resize 0 takes 128 s (main 10 s, 5.5 of them its warm-up). Both answer correctly and the server stays up. One run each.
  • Images can now fill the context. A screenshot is 882 tokens, so the default 32768 tokens hold at most 37.

What the time buys is legible small text. Each page below is a 1280x720 order page with one 6-digit code; the model is asked for every code. Codes read right out of 32 per cell (two runs at 8 screenshots):

Screenshots, font main fc13a07 this PR
1, 10 to 20 px 32 32
4, 10 px 28 (4 misread) 32
8, 10 px 5 and 5 (24-25 misread) 24 and 24 (0 misread)
8, 14 px 25 and 26 (2-3 misread) 29 and 29 (0 misread)
8, 20 px 31 and 29 31 and 29

On both sides the remaining misses at 8 screenshots are mostly answers that stop before the last image or two; the model ends them, not the token limit. With text of 20 px or more, the shared budget read as well as this PR.

Benchmark

Behavior HF / vLLM / SGLang FLM NPU FLM oga-hipep before after
Several images in one prompt smart_resize caps each image at max_pixels, no total; vLLM --limit-mm-per-prompt limits the items per prompt (default 999) each image by --img-pre-resize and max_pixels the images share 8100 patches each image on its own
First image after a load none in a user's request; SGLang sends a warm-up request with a 32x32 PNG before it reports ready none a blank 8100-patch pass (5.5-7 s) none

Tests run

Check Before (main fc13a07) After
T1 src/test/oga_hipep the new and updated tests fail: 4 engine tests (an extra SetInputs; 8192 patches refused) pass, 0 warnings
T1 src/test/oga_hipep/frontend OgaImagesReachTheEngineAsFloat16 fails (each image got 4050x256 max_pixels) 2/2 pass; new cases for the early 400 and the image at which it comes
L2 first image after a load TTFT 6.67 s (text 0.88 s), 1 run 1.96-2.16 s in 3 runs (text 0.96-1.11 s); a later image 1.1-1.3 s
L2 1 to 4 screenshots at 1280x720 907, 1789, 1969, 1953 prompt tokens 907, 1789, 2671, 3553 (882 each)
L2 a 256-patch image, then a 2880x2880 grid of nine colors (32400 patches), -r 0 not run the same answer as with the large image first; first-token logprob -0.39 against -0.43
L2 images past --ctx-len 2048 not run 400 in 0.27 s: "the first 3 image(s) alone have 2640 image tokens"; the next request answers
L2 a third screenshot joins in turn 2 not run 0 cached tokens (full replay); the text turn after it reuses 2713 of 2736

With and without the warm-up at 256 then 7056 patches, the greedy answers match except for one trailing period at a logprob tie, and the first-token logprobs differ by up to 0.15. Two identical runs differ by up to 0.02.

Acceptance

  • No request runs a vision warm-up; first-image TTFT measured (2.0-2.2 s, was 6.7 s)
  • Earlier images keep their token runs when an image is added (T1 and L2)
  • Image mid-generator (WP note 3): not in this PR. WP-23 stays open.

The WP's target, a first image within 2x of a text request of the same length, is at the limit: 2.01, 1.94 and 2.04 in the three runs.

Unverified

  • Packages other than qwen3.5-oga-hipep:9b. They share the same frontend and engine path.
  • GPU memory of a large vision pass. 65536 patches answer; the memory was not recorded.
  • onnxruntime-genai 0.17. FLM no longer loads it.
  • The early 400 for images given to flm run on the command line. It is the same check; only the request path has a test.

Follow-ups

  • About 0.9 s of first-use cost remains on the first image request. A small warm-up request during load, as SGLang does, would absorb it.
  • An image added mid-conversation still replays it: OGA takes images only in a generator's first call (WP note 3). Until that changes, stable image tokens do not save a replay.

🤖 Generated with Claude Code

….5 (WP-23)

The vision warm-up and the shared 8100-patch budget worked around wrong
image features in hip-ep 0.5.2 (onnxruntime-genai 0.17). FLM requires 0.19,
where neither is needed.

- Remove OgaEngine::WarmUpVision and the per-prompt patch limit
  (kMaxImagePatches, OgaVisionConfig::max_patches).
- Size each image on its own, by --img-pre-resize and the package's
  max_pixels, so an image keeps its tokens when another one joins.
- On OGA, return HTTP 400 at the image that makes the images alone reach
  the context, before the remaining images are preprocessed.
- Document the per-image sizing, its measured times and the 400 in cli.md.
@github-actions

Copy link
Copy Markdown
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

This branch was successfully deployed

1 active deployment
npu-approval — 159de845 Deployed Nov 17, 2026 by Allegro-ma-non-troppo via npu-gate #450
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant