Skip to content

ggml-openvino: Optimize model recompiles of prefill stage - #428

Open
yzhou51 wants to merge 3 commits into
ravi9:dev_backend_openvinofrom
yzhou51:recompile-models
Open

yzhou51 wants to merge 3 commits into
ravi9:dev_backend_openvinofrom
yzhou51:recompile-models

Conversation

@yzhou51

@yzhou51 yzhou51 commented Nov 16, 2026

Copy link
Copy Markdown

Optimize the model recompiles of prefill stage when attended size changed between prompts. This can save model recompile time for llama-bench and llama-server.

Overview

Problem

We fixed the out of memeory issue in the Patch 00d689441: ggml-openvino: fix long NPU prompts failing with KV slicing, which will erase the model when attended size changed at 256, 512, 1024, etc. This will cause the model re-compiled in the llama-bench testing and chat changed in the llama-server.
From the llama-bench.cpp:2390-2448, for -p 1024 -n 0 with the defaults -r 5, -b 2048, -ub 512:

llama_memory_clear(KV)                 // line 2390
warmup: test_prompt(1024 tokens)       // line 2425, not timed
repeat 5 times:                        // -r 5
    llama_memory_clear(KV)             // line 2448, every repetition starts at position 0
    test_prompt(1024 tokens)           // timed; t/s is the mean of these

The context is n_prompt + n_gen = 1024. Each test_prompt is split by -ub 512 into 2 backend calls. The NPU runs each call as 2 chunks of 256 tokens on one compiled static model:

Call Positions Attended KV size (power of 2)
1 0–511 512
2 512–1023 1024

Because the KV cache is cleared before each repetition, every repetition goes 512 → 1024. The size goes back down from 1024 to 512 at the start of every repetition after the warmup.
Compiles happen inside the graph compute of a llama_decode call. So compiles in the warmup are not counted in t/s, but compiles in a timed repetition are.

 code LNL (NPU4000) PTL
Before 335.29 ± 1.61 510.68
patch (00d6894) 1.72 ± 0.01 ~5

Proposed solution

  • Before PR ggml-openvino: fix long NPU prompts failing with KV slicing.
  • Patch 00d6894 ggml-openvino: fix long NPU prompts failing with KV slicing: the exact-size cache check plus erasing the previous pair.
  • Optimization A: Reuse of a larger cached prefill model.
  • Optimization C: With a switch GGML_OPENVINO_NPU_KEEP_PREFILL_MODELS=1: keep the compiled prefill model of every size.

Models loaded after the attended size grows 512 → 4096:

 Code Loaded
Before 4 × (prefill + decode + decode KV windows)
patch (00d6894) A 1 prefill + 1 decode set
A 1 prefill (largest) + 1 decode set
C 4 prefill + 1 shared decode set

llama-server / chat

Scenario Before patch (00d6894) A C
First chat grows through 512 → 1024 → 2048 1 compile per new size same same same
New chat after a long chat (KV cleared, size back to 512) 0 recompiles each size again 0 (runs at the largest size, masked) 0
Short new prompt after a long chat 0 1+ compile 0 (runs at the largest size) 0 (runs at its own size)
Follow-up in the same chat, growing past the largest size so far 1 per new size 1 per new size 1 per new size 1 per new size
Decode steps 0 0 0 0

NPU memory with C (LNL Windows, -c 4096, 4 prefill models of 512, 1024, 2048 and 4096 kept loaded):

Test Result
Qwen3.5-4B, llama-bench -p 4096 pass
smollm2-1.7B, llama-completion -c 4096 -ub 512, 2370-token prompt (ravi's failing case) pass, coherent output

Choosing between A and C

A C
Recompiles none after each size's first compile
NPU memory 1 prefill model (same as ravi's patch)
Prefill cost runs at the largest size seen. Free on PTL, ~33% per doubling on LNL
Best on PTL (+4% at pp1024)

In the current code A is the default and C is opt-in with switch GGML_OPENVINO_NPU_KEEP_PREFILL_MODELS.

Additional information

Requirements

@zhaixuejun1993

Copy link
Copy Markdown
Collaborator

for solution C, which is used for warm setup (export & import). u mean when new KV size model recompile the old compiled model (small KV size) will be erase?

@ravi9
ravi9 force-pushed the dev_backend_openvino branch from d1990e9 to 2910fcd Compare November 16, 2026 17:35
@yzhou51

yzhou51 commented Nov 17, 2026

Copy link
Copy Markdown
Author

In C, the old and smaller-size model is not erased. C is also not about export/import. It works on the in-memory compiled cache (shared_cache->graphs), not on blobs on disk.
When a later prompt needs the smaller size again, C finds that pair in shared_cache->graphs. It only creates new infer requests, with no compile and no import.

@yzhou51

yzhou51 commented Nov 17, 2026 •

Copy link
Copy Markdown
Author
Before patch patch (00d6894) solution C
Prefill sizes The exact attended length of each ubatch: 512, 1024, 1536, 2048, 2304, ... Power of 2: 512, 1024, 2048, 4096 Power of 2 (from patch (00d6894))
Decode models Each size compiles its own decode model and decode KV windows One decode set, taken over by each new pair One decode set, shared (from ravi's patch)
A new size compiles prefill + decode + decode windows prefill only prefill only
Old pairs kept erased (when one backend uses them) kept
Old pair's infer requests released on a size switch released on a size switch released on a size switch
Cache check for a prefill size exact match exact match exact match
A size seen before, later found in the cache, only new requests erased, so recompile found in the cache, only new requests
Base decode model of a new pair built at its own window the window covering the next decode step same as patch (00d6894)
Self-contained blobs (export/import) keep every pair, each with its own decode same, since this path is unchanged same
Default always on always on opt-in (KEEP_PREFILL_MODELS=1)

Example: The failing case (-c 4096 -ub 512, 2200-token prompt, decode windows 1024, 2048 and 4096):

Prefill models loaded at the end Decode models loaded Compiles during the first prompt Compiles for the next prompt of the same length
Before patch 5 (512, 1024, 1536, 2048, ~2304) up to 15 (5 × 3) 5 full pairs, each with its windows 0
patch (00d6894) 1 (4096) 3 1 full pair + 3 prefill-only 4 (512, 1024, 2048, 4096 again)
solution C 4 (512, 1024, 2048, 4096) 3 1 full pair + 3 prefill-only 0

For the PR, solution A sits between patch (00d6894)and solution C. It is based on patch(00d6894) with the cache check changed to "cached size ≥ needed size":

  • 1 prefill model loaded, like ravi's patch;
  • 0 compiles for the next prompt, like solution C;
  • the cost is that later prompts run at the largest size seen.

Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.
@yzhou51 yzhou51 reopened this Nov 17, 2026
Optimize the model recompiles of prefill stage when attended size
changed between prompts. This can save model recompile time for
llama-bench and llama-server.
@yzhou51

yzhou51 commented Nov 18, 2026

Copy link
Copy Markdown
Author

I remove the Solution C and keep Solution A as default.

Why C was added

A reuses the larger compiled prefill model, so a short prompt attends to masked KV rows it doesn't need. C was meant to avoid that cost by keeping one compiled model per size, so each call runs at its own size.

The first pp1024 results (Qwen3.5-4B Q4_0, -fa 1 -n 0, t/s) showed a gain on LNL:

LNL PTL
Before (00d6894) 335.29 510.68
00d6894(exact size, recompiles) 1.72 ~5
A 273.41 / 282.74 495.50 / 494.87
C 311.33 / 304.94 477.47 / 477.66
GGML_OPENVINO_KV_SLICE=0 272.45 491.24
  • LNL: C was 10–14% faster than A and closer to the old build's 335.
  • LNL pp4096: C gave 157.36, against about 150–153 with KV_SLICE=0. A wasn't measured at this size.
  • PTL: C was already about 4% slower at pp1024. That's why C stayed opt-in (GGML_OPENVINO_NPU_KEEP_PREFILL_MODELS=1) instead of becoming the default.

Why C was removed

More PTL data shows no real gain.

Qwen3.5-4B Q4_0, t/s:

||pp1024 | pp2048 | pp4096 | pp8192|
|-- | -- | -- | --|
|A | 493.63 | 482.27 | 468.42 | 432.32|
|C | 475.68 | 472.91 | 468.78 | 450.57|

C is 2–4% slower at 1k–2k, the same at 4k, and 4% faster only at 8k. Default pp512 and tg128 are the same with or without C.

C runs out of memory with larger models

With Qwen3.5-9B Q4_K_M (32 GB system memory, 18 GB of it shared with the NPU):

  • PTL fails at tg512 @ d8192 with ZE_RESULT_ERROR_OUT_OF_HOST_MEMORY in create_infer_request. A passes at 10.55 t/s.
  • LNL fails at d4096, d8192 and pp8192.
    C keeps a compiled prefill model for every size it has seen. At 8192 that's 5 models (512 to 8192) instead of 1, and each one holds NPU and host memory.

C doesn't help decode.

It changes prefill only. 9B decode on PTL with C vs A: d1024 11.64 / 11.66, d2048 11.76 / 11.56, d4096 11.33 / 11.30.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants