Repository navigation
fix(oga-hipep): no vision warm-up and no shared image budget on Qwen3.5 (WP-23) - #1064
Open
Allegro-ma-non-troppo wants to merge 1 commit into
Open
Allegro-ma-non-troppo wants to merge 1 commit into
Allegro-ma-non-troppo wants to merge 1 commit into
Conversation
….5 (WP-23) The vision warm-up and the shared 8100-patch budget worked around wrong image features in hip-ep 0.5.2 (onnxruntime-genai 0.17). FLM requires 0.19, where neither is needed. - Remove OgaEngine::WarmUpVision and the per-prompt patch limit (kMaxImagePatches, OgaVisionConfig::max_patches). - Size each image on its own, by --img-pre-resize and the package's max_pixels, so an image keeps its tokens when another one joins. - On OGA, return HTTP 400 at the image that makes the images alone reach the context, before the remaining images are preprocessed. - Document the per-image sizing, its measured times and the 400 in cli.md.
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On the Qwen3.5/3.6/3.8
-oga-hipepmodels:--img-pre-resize 2, a third 1280x720 screenshot shrank all of them: 3 screenshots gave 1969 tokens instead of 2671. Earlier images changed their size and their tokens when another image arrived.Both were workarounds for hip-ep 0.5.2 (onnxruntime-genai 0.17), which gave wrong image features without them. FLM now requires onnxruntime-genai 0.19.
What changed
OgaEngine::WarmUpVisionand the per-prompt patch limit are gone (kMaxImagePatches,OgaVisionConfig::max_patches).--img-pre-resizesets, within the package'smax_pixels.cli.md"Images" describes the per-image sizing, the measured times and the 400.What it costs
The owner chose per-image sizing with no total (run log Q1 and Q2). Images are larger than before, so they take longer. All times are TTFT on a Radeon 8060S with
qwen3.5-oga-hipep:9b.Medians of 5 runs, 3 for the PR's 6-screenshot row. Each run is a new conversation with new images, and every answer names the colors in order.
--img-pre-resize 0takes 128 s (main 10 s, 5.5 of them its warm-up). Both answer correctly and the server stays up. One run each.What the time buys is legible small text. Each page below is a 1280x720 order page with one 6-digit code; the model is asked for every code. Codes read right out of 32 per cell (two runs at 8 screenshots):
On both sides the remaining misses at 8 screenshots are mostly answers that stop before the last image or two; the model ends them, not the token limit. With text of 20 px or more, the shared budget read as well as this PR.
Benchmark
smart_resizecaps each image atmax_pixels, no total; vLLM--limit-mm-per-promptlimits the items per prompt (default 999)--img-pre-resizeandmax_pixelsTests run
src/test/oga_hipepsrc/test/oga_hipep/frontendOgaImagesReachTheEngineAsFloat16fails (each image got 4050x256 max_pixels)-r 0--ctx-len 2048With and without the warm-up at 256 then 7056 patches, the greedy answers match except for one trailing period at a logprob tie, and the first-token logprobs differ by up to 0.15. Two identical runs differ by up to 0.02.
Acceptance
The WP's target, a first image within 2x of a text request of the same length, is at the limit: 2.01, 1.94 and 2.04 in the three runs.
Unverified
qwen3.5-oga-hipep:9b. They share the same frontend and engine path.flm runon the command line. It is the same check; only the request path has a test.Follow-ups
🤖 Generated with Claude Code