Skip to content

feat(oga): serve gemma4-12b on oga-hipep with shared KV and images - #1052

Merged
zaneni6 merged 5 commits into
mainfrom
gemma4-12b-oga-hipep
Nov 17, 2026
Merged

zaneni6 merged 5 commits into
mainfrom
gemma4-12b-oga-hipep

Conversation

@amd-jileil

Copy link
Copy Markdown
Collaborator

Motivation

Serve the public Gemma 4 12B GPU package on oga-hipep.

Reason

That package is model type gemma4_unified. Its present buffer is allocated to max_length once past_present_share_buffer is on, and the package ships the flag as false. Treating that capacity as already-used tokens and adding the new tokens asks for a longer present than the one already bound, so a long prefill is rejected. Graph capture is recorded for one sequence length and does not match the shared present. The package has a vision encoder and an image token, and no vision-start token, so the patch-grid vision path does not apply.

Changes

  1. src/common/AutoModel/builtin_backends.cpp
    Register gemma4-12b on oga-hipep. The NPU registration of the same family stays. The tag gemma4-12b-oga-hipep selects this backend.

  2. src/model_list.json
    Add gemma4-12b-oga-hipep:12b for amd/Gemma-4-12B-it_gpuep_1.0_gpu_128. details.family stays gemma4-12b and vlm is true. gemma4-it-oga-hipep is unchanged.

  3. src/common/oga/oga_hipep_backend.cpp
    For gemma4_unified only, set past_present_share_buffer and turn graph capture off. Other model types keep the previous session overlay.

  4. src/common/oga/oga_engine.cpp, src/common/AutoModel/modeling_gemma4_12b.cpp
    JPEG and PNG are staged through the package image processor and prefilled as that processor's whole token sequence. The empty thought channel is trimmed from the staged text when thinking is off, because generate() feeds it. Audio is rejected on this backend. The NPU image path and the NPU text trim are unchanged.

Compatibility

A request that does not use gemma4-12b-oga-hipep stays on its previous backend. NPU gemma4-12b still uses the NPU engine.

Verification

src/test/oga_hipep/test_oga_hipep.cpp covers the gemma4_unified overlay, the absence of that overlay on another model type, a staged image prefill, a missing processor entry point, and text generation for a gemma4_unified package that has no vision-start token. These host tests were not run on this machine because XRT and Boost are not installed.

Register gemma4-12b on the oga-hipep backend and publish the public Gemma 4 12B GPU package as gemma4-12b-oga-hipep:12b. gemma4_unified sessions reuse the preallocated present buffer and leave graph capture off. JPEG and PNG go through the package image processor.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

zaneni6 and others added 2 commits November 16, 2026 13:07
… stale staged prompt

gemma4-12b-oga-hipep:12b ships the same package as gemma4-it-oga-hipep:12b,
so it names that model_info.json record with model_info_key; without it
flm pull found no file records for the tag.

An image request stages the processor inputs for the next prefill. A request
that stopped before its prefill (cancelled, or refused for its length) left
the stage behind, and the next request's prefill was checked against it and
threw. The image branch now drops the stage after its insert, prefill() takes
the stage before refusing it, and StagePromptImages starts from no stage. The
image branch also clears a held conversation first, so the staged sequence is
prefilled from token 0 and never as a suffix after a cached prefix.

Co-Authored-By: Claude <noreply@anthropic.com>
@zaneni6

zaneni6 commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Thanks a lot for this PR, @amd-jileil. Serving gemma4-12b on oga-hipep with images is a great addition. While reviewing, I merged current main into the branch (it merged cleanly, including #1053) and pushed one small follow-up commit (f6e2383) for two things I found. Please let me know if you'd rather handle either one differently.

1. flm pull gemma4-12b-oga-hipep:12b
The new catalog entry had no record in model_info.json, so the downloader couldn't find file records for the tag. It ships the same package as gemma4-it-oga-hipep:12b (same repo, revision, files and file_sources), so I added "model_info_key": "gemma4-it-oga-hipep:12b" to the entry, the same way other entries point at a shared record. Every catalog entry now resolves to a model_info.json record.

2. A staged image prompt could be left for the next request
StagePromptImages holds the processor inputs until the next prefill(). If the request stopped before its prefill (a client cancel in _chunked_insert, or a 400 from the capacity check), the stage stayed behind and the next request's prefill threw. Also, if prompt reuse skipped a cached prefix, prefill() would get only a suffix. The commit:

  • adds OgaEngine::DropStagedPrompt(), which the gemma4-12b image branch calls after _shared_insert, both on return and on exception;
  • makes prefill() take the stage before it can refuse it, and makes StagePromptImages() start with no stage;
  • clears any conversation the model still holds before an image insert, so the staged sequence is always prefilled from token 0.

I kept the throw for a mismatch inside the same request, because a silent fallback would run the image soft tokens as plain text. One trade-off to check: after an image turn, the next text turn re-prefills its history instead of reusing the cache. If you think that's too costly, I'm happy to look at another approach.

I added TestADroppedOrRefusedStageDoesNotReachTheNextPrefill to test_oga_hipep. test_oga_hipep and test_model_list_platform pass, and flm builds with -DFLM_ENABLE_OGA_HIPEP=ON. I haven't tried it on a real GPU, so a quick image chat on your setup would be really appreciated if you have time. Thanks again!

…lots)

OgaEngine keeps #1042's generator slots in place of the single live
generator, and the staged image prompt now runs in one of them: its
prefill builds a new generator in a free slot or in place of the least
recently used one, gives it the processor inputs, and makes it the active
slot. An OGA error in that prefill drops only that slot and leaves the
logical history without the image tokens. The gemma4_unified overlay is
on the model config, so every slot's generator gets it.

Tests: a staged image prompt next to held slots, a repeated one, and a
failed one.

Co-Authored-By: Claude <noreply@anthropic.com>
@zaneni6

zaneni6 commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Thank you for this PR. I merged the latest main (500ef91) (which now includes #1042's generator slots) into the branch and resolved the two conflicts so both features stay:

The staged image prefill now goes through a small PrefillStaged helper. It builds a new generator in a free slot, or in place of the least recently used one, gives it the processor inputs, and makes it the active slot. It only takes the image tokens into the history after the logits are read, so an OGA error drops just that slot and leaves no image tokens to replay. The gemma4_unified overlay is set on the model config, so every slot's generator picks it up.

I also added three engine tests: a staged image prompt while another conversation holds a slot, the same staged prompt twice, and a failed staged prefill. test_oga_hipep (107 tests), both frontend tests and the full flm build pass locally. Could you confirm whether the gemma4-12b package's genai_config.json states the decoder shape? Without it the engine keeps a single slot for this model.

@zaneni6
zaneni6 merged commit fc13a07 into main Nov 17, 2026
115 of 116 checks passed
@andrew180917

Copy link
Copy Markdown
Collaborator

Since #1052 (fc13a07), the first prefill on gemma4-it-oga-hipep:12b or gemma4-12b-oga-hipep:12b hits an ONNX Runtime fatal error and flm serve exits with 0xC0000409, so every chat request fails and takes the whole server down. gemma4-it-oga-hipep:12b shipped in v1.0.13, which does not have the change; main, and any release cut from it, crashes.

Cause

Both tags open the same package, amd/Gemma-4-12B-it_gpuep_1.0_gpu_128 at adc217f3, whose genai_config.json has model.type gemma4_unified. OpenModel() keys the #1052 overlay on that type rather than on the tag, so both get kGemmaSearchOverlay, which turns on search.past_present_share_buffer; the package ships it as false. #1052 says gemma4-it-oga-hipep is unchanged: its catalog entry is, but its session is not.

With the flag on, present.0.key is bound at max_length, 32768 here (the context length). The computed present is always 32768 plus the prompt: this export grows its KV by concatenation, as the comment in OpenModel() says, so it appends the new tokens to the whole bound length. That is the mismatch #1052's description sets out to avoid, and it happens on every prefill, not only long ones.

gemma4-it-oga-hipep:26b-a4b is gemma4, not gemma4_unified (its genai_config.json at 66d30d3b), so the overlay does not apply to it. I did not run it.

Repro

STX Halo (Radeon 8060S, driver 32.0.31041.1004), Windows 11 Enterprise 10.0.26100, flm v1.0.14 from main's CI build of fc13a07, hip-ep 0.5.3 (OGA 0.19).

flm pull gemma4-it-oga-hipep:12b
flm serve
# in another shell; the model loads on the first request
Invoke-RestMethod http://127.0.0.1:52625/v1/chat/completions -Method Post -ContentType application/json `
  -Body '{"model":"gemma4-it-oga-hipep:12b","messages":[{"role":"user","content":"Say hi"}],"max_tokens":16}'

From my run, a first chat turn with a 26-token prompt:

[FLM]  Prefill chunk 1/1 with 26 tokens
F20261117 03:50:30.222006  9232 MlirCustomOp.cpp:664] output allocator callback failed for out_idx 1: ... [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : The output OrtValue provided for output 'present.0.key' of node 'AMDGPUExecutionProvider_AMDGPUExecutionProvider_3050254323306677413_1_0' (AMDGPUExecutionProvider_3050254323306677413_1) has shape {1,8,32768,256} but the computed output shape for this run is {1,8,32794,256}. When calling Run() with pre-allocated output OrtValues on a model with dynamic output shapes, either supply unallocated output OrtValues or ensure the pre-allocated shapes match the expected output shapes for each run.
Tag Request Prompt Computed present.0.key
gemma4-it-oga-hipep:12b first chat turn 26 tokens {1,8,32794,256}
gemma4-12b-oga-hipep:12b npu_test chat/answer 21 tokens {1,8,32789,256}
gemma4-12b-oga-hipep:12b npu_test api/load, max_tokens 1 10 tokens {1,8,32778,256}

The bound shape is {1,8,32768,256} every time. npu_test (chat, api, shutdown) on gemma4-12b-oga-hipep:12b passed 4 and failed 5: the passes are pull, a script check and the two flm run exits, none of which prefills.

Before #1052, #996 measured gemma4-it-oga-hipep:12b on the same hip-ep 0.5.3 at 98/1/1 in chat and api, the one failure being api/test_tool_calls.round_trip. Main's HIPEP CI tests only qwen3.5-oga-hipep:9b, so no check ran either 12b tag.

Suggested fix

  • Drop "search":{"past_present_share_buffer":true} from kGemmaSearchOverlay, so the package keeps its own false, and keep the AMDGPU options with graph capture off.
  • Update the two tests that assert the flag, at L2508 and L2537.
  • On a GPU, run chat, api and shutdown on both 12b tags, plus a long prompt and an image on gemma4-12b-oga-hipep:12b, since a long prefill is what feat(oga): serve gemma4-12b on oga-hipep with shared KV and images #1052 set out to fix.

I have not built or run this change.

One package, two tags

The two tags share the package and its folder, and differ only in family (gemma4e vs gemma4-12b), vlm and labels, flm_min_version and model_info_key. The NPU names this model gemma4-it:12b, in family gemma4-12b. Once the crash is fixed, gemma4-it-oga-hipep:12b could move to family gemma4-12b with vlm true, the image labels and flm_min_version 1.0.14, and gemma4-12b-oga-hipep:12b, which is not in a release yet, could go.

cc @amd-jileil @zaneni6

This branch was previously deployed

1 inactive deployment
npu-approval — 500ef911 Deployed Nov 17, 2026 by zaneni6 via npu-gate #445
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants