Repository navigation
feat(oga): serve gemma4-12b on oga-hipep with shared KV and images - #1052
Conversation
Register gemma4-12b on the oga-hipep backend and publish the public Gemma 4 12B GPU package as gemma4-12b-oga-hipep:12b. gemma4_unified sessions reuse the preallocated present buffer and leave graph capture off. JPEG and PNG go through the package image processor. Co-authored-by: Cursor <cursoragent@cursor.com>
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
… stale staged prompt gemma4-12b-oga-hipep:12b ships the same package as gemma4-it-oga-hipep:12b, so it names that model_info.json record with model_info_key; without it flm pull found no file records for the tag. An image request stages the processor inputs for the next prefill. A request that stopped before its prefill (cancelled, or refused for its length) left the stage behind, and the next request's prefill was checked against it and threw. The image branch now drops the stage after its insert, prefill() takes the stage before refusing it, and StagePromptImages starts from no stage. The image branch also clears a held conversation first, so the staged sequence is prefilled from token 0 and never as a suffix after a cached prefix. Co-Authored-By: Claude <noreply@anthropic.com>
|
Thanks a lot for this PR, @amd-jileil. Serving gemma4-12b on oga-hipep with images is a great addition. While reviewing, I merged current main into the branch (it merged cleanly, including #1053) and pushed one small follow-up commit (f6e2383) for two things I found. Please let me know if you'd rather handle either one differently. 1. 2. A staged image prompt could be left for the next request
I kept the throw for a mismatch inside the same request, because a silent fallback would run the image soft tokens as plain text. One trade-off to check: after an image turn, the next text turn re-prefills its history instead of reusing the cache. If you think that's too costly, I'm happy to look at another approach. I added |
…lots) OgaEngine keeps #1042's generator slots in place of the single live generator, and the staged image prompt now runs in one of them: its prefill builds a new generator in a free slot or in place of the least recently used one, gives it the processor inputs, and makes it the active slot. An OGA error in that prefill drops only that slot and leaves the logical history without the image tokens. The gemma4_unified overlay is on the model config, so every slot's generator gets it. Tests: a staged image prompt next to held slots, a repeated one, and a failed one. Co-Authored-By: Claude <noreply@anthropic.com>
|
Thank you for this PR. I merged the latest main (500ef91) (which now includes #1042's generator slots) into the branch and resolved the two conflicts so both features stay:
The staged image prefill now goes through a small I also added three engine tests: a staged image prompt while another conversation holds a slot, the same staged prompt twice, and a failed staged prefill. |
|
Since #1052 (fc13a07), the first prefill on CauseBoth tags open the same package, With the flag on,
ReproSTX Halo (Radeon 8060S, driver 32.0.31041.1004), Windows 11 Enterprise 10.0.26100, flm v1.0.14 from main's CI build of fc13a07, hip-ep 0.5.3 (OGA 0.19). flm pull gemma4-it-oga-hipep:12b
flm serve
# in another shell; the model loads on the first request
Invoke-RestMethod http://127.0.0.1:52625/v1/chat/completions -Method Post -ContentType application/json `
-Body '{"model":"gemma4-it-oga-hipep:12b","messages":[{"role":"user","content":"Say hi"}],"max_tokens":16}'From my run, a first chat turn with a 26-token prompt:
The bound shape is {1,8,32768,256} every time. npu_test (chat, api, shutdown) on Before #1052, #996 measured Suggested fix
I have not built or run this change. One package, two tagsThe two tags share the package and its folder, and differ only in family ( |
Motivation
Serve the public Gemma 4 12B GPU package on oga-hipep.
Reason
That package is model type gemma4_unified. Its present buffer is allocated to max_length once past_present_share_buffer is on, and the package ships the flag as false. Treating that capacity as already-used tokens and adding the new tokens asks for a longer present than the one already bound, so a long prefill is rejected. Graph capture is recorded for one sequence length and does not match the shared present. The package has a vision encoder and an image token, and no vision-start token, so the patch-grid vision path does not apply.
Changes
src/common/AutoModel/builtin_backends.cpp
Register gemma4-12b on oga-hipep. The NPU registration of the same family stays. The tag gemma4-12b-oga-hipep selects this backend.
src/model_list.json
Add gemma4-12b-oga-hipep:12b for amd/Gemma-4-12B-it_gpuep_1.0_gpu_128. details.family stays gemma4-12b and vlm is true. gemma4-it-oga-hipep is unchanged.
src/common/oga/oga_hipep_backend.cpp
For gemma4_unified only, set past_present_share_buffer and turn graph capture off. Other model types keep the previous session overlay.
src/common/oga/oga_engine.cpp, src/common/AutoModel/modeling_gemma4_12b.cpp
JPEG and PNG are staged through the package image processor and prefilled as that processor's whole token sequence. The empty thought channel is trimmed from the staged text when thinking is off, because generate() feeds it. Audio is rejected on this backend. The NPU image path and the NPU text trim are unchanged.
Compatibility
A request that does not use gemma4-12b-oga-hipep stays on its previous backend. NPU gemma4-12b still uses the NPU engine.
Verification
src/test/oga_hipep/test_oga_hipep.cpp covers the gemma4_unified overlay, the absence of that overlay on another model type, a staged image prefill, a missing processor entry point, and text generation for a gemma4_unified package that has no vision-start token. These host tests were not run on this machine because XRT and Boost are not installed.