Repository navigation
Conversation
…martcomputer-ai#190) Chat templates of self-hosted models such as Qwen reject any system message that is not the first message. The Compatible Chat Completions dialect now merges consecutive leading instruction and catalog entries into one system message and renders later instruction or catalog entries (for example a refreshed catalog) as user messages with an update header. A later entry that arrives during a tool exchange waits until the tool results are in, so it never splits an assistant's tool calls from their results. Compaction requests share the same lowering. OpenAI, OpenRouter, and DeepSeek are unchanged.
…martcomputer-ai#190) Builds on the previous commit and widens it from the Compatible dialect to every provider, as discussed on smartcomputer-ai#190. - system_prompt::collated_instructions joins all instruction entries in planned order (each trimmed, empty ones skipped, blank-line separated). The Anthropic system block, the Responses `instructions` field and now one leading Chat Completions system/developer message are built from it, for every Chat Completions dialect. - Catalogs render as user text at their position, as on Anthropic, for every dialect. A refreshed catalog lands at the tail, so no system message follows the conversation. A catalog that arrives during a tool exchange still waits for the results. The existing update header marks a refreshed catalog, so the extra "[Catalog update]" header is dropped. Tests: - Byte-level prefix tests for every dialect: request N is a prefix of N+1 across a new run, a tool round trip, a catalog refresh and a catalog arriving mid tool exchange. - Chat-template conformance: real templates of Qwen 3.5, Qwen 3, Qwen 2.5, gpt-oss and Mistral v0.3 (fixtures with source revision and license) render the requests without error, and each rendered request extends the previous one. A negative control shows the old shape is rejected by Qwen 3.5 and Mistral. minijinja 3.0.5 is a dev-dependency. - The Completions caching live suite can target a self-hosted server (OPENAI_COMPLETIONS_PROVIDER_ID, _OUTPUT_LIMIT, _EXTRA_BODY) and gains a catalog-refresh case under layered instructions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1d1dbff to
b5ff801
Compare
|
I've added a follow-up commit to this branch, on top of @marvinvr's, and rebased both onto What the follow-up addsProblemEach instruction layer and each catalog went out as its own ChangeOne system prompt, built the same way for every provider. Catalogs are user text at their position, as on Anthropic. A refreshed catalog lands at the tail, so no system message ever follows the conversation. Kept from Marvin's commit:
The existing "Updated catalog" header already marks a refreshed catalog, so the extra Behaviour changes to review
Tests
Live checkQwen3.5-9B (revision
To rerun against a vLLM server: OPENAI_COMPLETIONS_BASE_URL=http://127.0.0.1:8000/v1 OPENAI_COMPLETIONS_API_KEY=unused \
OPENAI_COMPLETIONS_MODEL=Qwen/Qwen3.5-9B OPENAI_COMPLETIONS_PROVIDER_ID=vllm \
OPENAI_COMPLETIONS_OUTPUT_LIMIT=1024 \
OPENAI_COMPLETIONS_EXTRA_BODY='{"chat_template_kwargs":{"enable_thinking":false}}' \
cargo test -p llm-runtime --test openai_completions_caching_live -- --ignored --test-threads=1Thinking is off in that run because Qwen 3.5 can spend the whole output budget reasoning. With thinking on, the cache hit was the same (99 %), but the answer was cut off at the length limit. EvidenceRaw output, not summaries, recorded on this PR's head 1. Unit and chat-template tests pass on
|
| # | HTTP | Messages sent (role ×count, in order) | prompt_tokens | cached_tokens | Error | sha256 of request body |
|---|---|---|---|---|---|---|
| 1 | 400 | system×4 → user×1 | System message must be at the beginning. | ea7593760dd32f1c |
||
| 2 | 200 | system×1 → user×1 | 6403 | 6336 | f447714f79a4b4e6 |
|
| 3 | 200 | system×1 → user×1 → assistant×1 → user×1 | 6470 | 6336 | a5a4f2eb05f9dc71 |
|
| 4 | 200 | system×1 → user×1 | 6657 | 6336 | 7d0ef437fcc7558e |
|
| 5 | 200 | system×1 → user×1 → assistant×1 → tool×1 | 6707 | 6336 | abffbfeacffe9fea |
after (PR b5ff801)
| # | HTTP | Messages sent (role ×count, in order) | prompt_tokens | cached_tokens | Error | sha256 of request body |
|---|---|---|---|---|---|---|
| 1 | 200 | system×1 → user×1 | 6441 | 0 | d0cdba1e9c0d7b71 |
|
| 2 | 200 | system×1 → user×1 → assistant×1 → user×1 | 6520 | 6336 | 5afe84f03ec114c2 |
|
| 3 | 200 | system×1 → user×1 | 6403 | 6336 | ae0e26ade1e2ee0c |
|
| 4 | 200 | system×1 → user×1 → assistant×1 → user×1 | 6458 | 6336 | ca01a2a7afea73bf |
|
| 5 | 200 | system×1 → user×1 | 6657 | 0 | a72c291a669a8d35 |
|
| 6 | 200 | system×1 → user×1 → assistant×1 → tool×1 | 6707 | 6336 | 4757139ecff3e253 |
How to read the table:
- Cold start: each run's first request was cold (0 cached).
- Shared cache: the cases run one after another against one server, so a case's first request can arrive warm from the case before. On
main, request 2 shows 6336 cached for that reason. - Cache granularity: the cached amount is 6336 tokens in every warm request. That's the long shared system prompt as Qwen 3.5's hybrid cache stores it, in blocks.
- Tool case: its first request is cold on both sides, because Qwen's template puts tool definitions ahead of the instructions.
- What the refresh case shows: after a catalog refresh, the next request still reads that prefix from cache. On
main, the same case is rejected before anything is cached.
First request and response of the catalog-refresh case, both sides
before: first request of the catalog-refresh case, HTTP 400 (long contents shortened)
{
"model": "Qwen/Qwen3.5-9B",
"max_tokens": 1024,
"store": false,
"stream": false,
"prompt_cache_key": "session-live-openai-completions-caching",
"chat_template_kwargs": {
"enable_thinking": false
},
"messages": [
{
"role": "system",
"content": "You are a warehouse operations assistant. Use the stock tabl… [23804 chars]"
},
{
"role": "system",
"content": "You are the release assistant of a small team."
},
{
"role": "system",
"content": "Commands run in /data/workspace."
},
{
"role": "system",
"content": "Sub-agent catalog:\n\n- reviewer: reviews a diff\n- tester: runs the test suite"
},
{
"role": "user",
"content": "Item 17 needs a status line. Reply in one sentence."
}
]
}Response:
{
"error": {
"message": "System message must be at the beginning.",
"type": "BadRequestError",
"param": null,
"code": 400
}
}after: first request of the catalog-refresh case, HTTP 200 (long contents shortened)
{
"model": "Qwen/Qwen3.5-9B",
"max_tokens": 1024,
"store": false,
"stream": false,
"prompt_cache_key": "session-live-openai-completions-caching",
"chat_template_kwargs": {
"enable_thinking": false
},
"messages": [
{
"role": "system",
"content": "You are a warehouse operations assistant. Use the stock tabl… [23885 chars]"
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "Sub-agent catalog:\n\n- reviewer: reviews a diff\n- tester: runs the test suite"
},
{
"type": "text",
"text": "Item 17 needs a status line. Reply in one sentence."
}
]
}
]
}Response:
{
"choices[0].message.content": "Item 17 currently needs a status line added to its record.",
"usage": {
"prompt_tokens": 6441,
"total_tokens": 6456,
"completion_tokens": 15,
"prompt_tokens_details": {
"cached_tokens": 0,
"created_cache_tokens": 6336,
"multimodal_tokens": null
},
"completion_tokens_details": {
"reasoning_tokens": 0
}
}
}vLLM server log for these runs: arguments, every request with status, prefix-cache hit rate
non-default args: {'model_tag': 'Qwen/Qwen3.5-9B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'enable_prompt_tokens_details': True, 'host': '127.0.0.1', 'model': 'Qwen/Qwen3.5-9B', 'revision': 'c202236235762e1c871ad0ccb60c8ee5ba337b9a', 'max_model_len': 16384, 'reasoning_parser': 'qwen3', 'enable_prefix_caching': True, 'limit_mm_per_prompt': {'image': 0, 'video': 0}, 'max_num_seqs': 4}
INFO: 127.0.0.1:33456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:33462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:33478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:33488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:33500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:33516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:39808 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
INFO: 127.0.0.1:39812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 11-18 23:46:43 [loggers.py:335] Engine 000: Avg prompt throughput: 1390.9 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 69.5%
INFO: 127.0.0.1:39824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:39838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:39840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 11-18 23:46:53 [loggers.py:335] Engine 000: Avg prompt throughput: 82.6 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 77.5%
The full request and response bodies (280 KB) don't fit in a comment. The sha256 prefixes in the tables identify each request body (JSON with sorted keys, no whitespace).
🤖 Generated with Claude Code
|
Implemented in #194. Thanks for the contribution. I discussed the approach the details with Marvin. |
Problem
With a custom OpenAI-compatible provider, every instruction and catalog context entry became its own
systemmessage, and a refreshed catalog showed up as asystemmessage mid-conversation. Qwen-style chat templates reject this ("System message must be at the beginning.").Change
For the Compatible Chat Completions dialect only:
OpenAI, OpenRouter (
developer) and DeepSeek are unchanged.Tests
Added unit tests in
openai_completions.rs: leading entries become one system message and a later catalog becomes a user message (with positions); a catalog arriving during a tool exchange waits until after the tool results; OpenAI output is unchanged.cargo test -p llm-runtimeand clippy pass.Fixes #190