Repository navigation
Conversation
|
for solution C, which is used for warm setup (export & import). u mean when new KV size model recompile the old compiled model (small KV size) will be erase? |
d1990e9 to
2910fcd
Compare
|
In C, the old and smaller-size model is not erased. C is also not about export/import. It works on the in-memory compiled cache (shared_cache->graphs), not on blobs on disk. |
Example: The failing case (-c 4096 -ub 512, 2200-token prompt, decode windows 1024, 2048 and 4096):
For the PR, solution A sits between patch (00d6894)and solution C. It is based on patch(00d6894) with the cache check changed to "cached size ≥ needed size":
|
fa03bdc to
2910fcd
Compare
Optimize the model recompiles of prefill stage when attended size changed between prompts. This can save model recompile time for llama-bench and llama-server.
Optimize the model recompiles of prefill stage when attended size changed between prompts. This can save model recompile time for llama-bench and llama-server.
|
I remove the Solution C and keep Solution A as default. Why C was addedA reuses the larger compiled prefill model, so a short prompt attends to masked KV rows it doesn't need. C was meant to avoid that cost by keeping one compiled model per size, so each call runs at its own size. The first pp1024 results (Qwen3.5-4B Q4_0, -fa 1 -n 0, t/s) showed a gain on LNL:
Why C was removedMore PTL data shows no real gain.Qwen3.5-4B Q4_0, t/s: ||pp1024 | pp2048 | pp4096 | pp8192| C is 2–4% slower at 1k–2k, the same at 4k, and 4% faster only at 8k. Default pp512 and tg128 are the same with or without C. C runs out of memory with larger modelsWith Qwen3.5-9B Q4_K_M (32 GB system memory, 18 GB of it shared with the NPU):
C doesn't help decode.It changes prefill only. 9B decode on PTL with C vs A: d1024 11.64 / 11.66, d2048 11.76 / 11.56, d4096 11.33 / 11.30. |
Optimize the model recompiles of prefill stage when attended size changed between prompts. This can save model recompile time for llama-bench and llama-server.
Overview
Problem
We fixed the out of memeory issue in the Patch
00d689441: ggml-openvino: fix long NPU prompts failing with KV slicing, which will erase the model when attended size changed at 256, 512, 1024, etc. This will cause the model re-compiled in the llama-bench testing and chat changed in the llama-server.From the llama-bench.cpp:2390-2448, for -p 1024 -n 0 with the defaults -r 5, -b 2048, -ub 512:
llama_memory_clear(KV) // line 2390 warmup: test_prompt(1024 tokens) // line 2425, not timed repeat 5 times: // -r 5 llama_memory_clear(KV) // line 2448, every repetition starts at position 0 test_prompt(1024 tokens) // timed; t/s is the mean of theseThe context is n_prompt + n_gen = 1024. Each test_prompt is split by -ub 512 into 2 backend calls. The NPU runs each call as 2 chunks of 256 tokens on one compiled static model:
Because the KV cache is cleared before each repetition, every repetition goes 512 → 1024. The size goes back down from 1024 to 512 at the start of every repetition after the warmup.
Compiles happen inside the graph compute of a llama_decode call. So compiles in the warmup are not counted in t/s, but compiles in a timed repetition are.
Proposed solution
ggml-openvino: fix long NPU prompts failing with KV slicing.ggml-openvino: fix long NPU prompts failing with KV slicing: the exact-size cache check plus erasing the previous pair.Models loaded after the attended size grows 512 → 4096:
llama-server / chat
NPU memory with C (LNL Windows, -c 4096, 4 prefill models of 512, 1024, 2048 and 4096 kept loaded):
Choosing between A and C
In the current code A is the default and C is opt-in with switch
GGML_OPENVINO_NPU_KEEP_PREFILL_MODELS.Additional information
Requirements