Skip to content

Send a single leading system message to OpenAI-compatible providers - #191

Closed
marvinvr wants to merge 2 commits into
smartcomputer-ai:mainfrom
marvinvr:fix-compatible-system-messages
Closed

marvinvr wants to merge 2 commits into
smartcomputer-ai:mainfrom
marvinvr:fix-compatible-system-messages

Conversation

@marvinvr

@marvinvr marvinvr commented Nov 18, 2026 •

Copy link
Copy Markdown

Problem

With a custom OpenAI-compatible provider, every instruction and catalog context entry became its own system message, and a refreshed catalog showed up as a system message mid-conversation. Qwen-style chat templates reject this ("System message must be at the beginning.").

Change

For the Compatible Chat Completions dialect only:

  • Instruction and catalog entries at the start merge into one system message, and their request positions map to index 0.
  • Later instruction or catalog entries become user messages with an update header and fold into neighbouring user messages. One that arrives during a tool exchange waits until the tool results are in.
  • Compaction requests use the same code, so they get this too.

OpenAI, OpenRouter (developer) and DeepSeek are unchanged.

Tests

Added unit tests in openai_completions.rs: leading entries become one system message and a later catalog becomes a user message (with positions); a catalog arriving during a tool exchange waits until after the tool results; OpenAI output is unchanged. cargo test -p llm-runtime and clippy pass.

Fixes #190

@marvinvr
marvinvr marked this pull request as ready for review November 18, 2026 08:02
@dfilimon dfilimon changed the title Send a single leading system message to OpenAI-compatible providers Send one system prompt to every provider and catalogs as user text Nov 18, 2026
marvinvr and others added 2 commits November 18, 2026 23:36
…martcomputer-ai#190)

Chat templates of self-hosted models such as Qwen reject any system message
that is not the first message. The Compatible Chat Completions dialect now
merges consecutive leading instruction and catalog entries into one system
message and renders later instruction or catalog entries (for example a
refreshed catalog) as user messages with an update header. A later entry that
arrives during a tool exchange waits until the tool results are in, so it never
splits an assistant's tool calls from their results. Compaction requests share
the same lowering. OpenAI, OpenRouter, and DeepSeek are unchanged.
…martcomputer-ai#190)

Builds on the previous commit and widens it from the Compatible dialect to
every provider, as discussed on smartcomputer-ai#190.

- system_prompt::collated_instructions joins all instruction entries in
  planned order (each trimmed, empty ones skipped, blank-line separated).
  The Anthropic system block, the Responses `instructions` field and now one
  leading Chat Completions system/developer message are built from it, for
  every Chat Completions dialect.
- Catalogs render as user text at their position, as on Anthropic, for
  every dialect. A refreshed catalog lands at the tail, so no system message
  follows the conversation. A catalog that arrives during a tool exchange
  still waits for the results. The existing update header marks a refreshed
  catalog, so the extra "[Catalog update]" header is dropped.

Tests:
- Byte-level prefix tests for every dialect: request N is a prefix of N+1
  across a new run, a tool round trip, a catalog refresh and a catalog
  arriving mid tool exchange.
- Chat-template conformance: real templates of Qwen 3.5, Qwen 3, Qwen 2.5,
  gpt-oss and Mistral v0.3 (fixtures with source revision and license)
  render the requests without error, and each rendered request extends the
  previous one. A negative control shows the old shape is rejected by
  Qwen 3.5 and Mistral. minijinja 3.0.5 is a dev-dependency.
- The Completions caching live suite can target a self-hosted server
  (OPENAI_COMPLETIONS_PROVIDER_ID, _OUTPUT_LIMIT, _EXTRA_BODY) and gains a
  catalog-refresh case under layered instructions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dfilimon
dfilimon force-pushed the fix-compatible-system-messages branch from 1d1dbff to b5ff801 Compare November 18, 2026 22:39
@dfilimon dfilimon changed the title Send one system prompt to every provider and catalogs as user text Send a single leading system message to OpenAI-compatible providers Nov 18, 2026
@dfilimon

dfilimon commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

I've added a follow-up commit to this branch, on top of @marvinvr's, and rebased both onto main. It applies one rule to every provider, as proposed on #190. The description above covers Marvin's commit; this covers the follow-up. The analysis behind it, including what it means for prompt caching, is on #190.

What the follow-up adds

Problem

Each instruction layer and each catalog went out as its own system message, and a refreshed catalog became a system message in the middle of the conversation. Chat templates of self-hosted models reject that. Qwen 3.5 answers 400 System message must be at the beginning., and older Mistral templates require strict user/assistant alternation.

Change

One system prompt, built the same way for every provider. system_prompt::collated_instructions joins all instruction entries in planned order: each trimmed, empty ones skipped, separated by a blank line. The Anthropic system block and the Responses instructions field already did exactly this with their own copies, and now share the function. Chat Completions sends the result as one leading system message (developer for OpenAI and OpenRouter), for all four dialects.

Catalogs are user text at their position, as on Anthropic. A refreshed catalog lands at the tail, so no system message ever follows the conversation. Kept from Marvin's commit:

  • A catalog that arrives during a tool exchange still waits until the results are in.
  • Request positions still map each entry to its message.

The existing "Updated catalog" header already marks a refreshed catalog, so the extra [Catalog update] header is dropped.

Behaviour changes to review

  • OpenAI and OpenRouter: catalogs move from developer messages to user text. Catalogs are menus rather than instructions, and they have been user text on Anthropic all along. Keeping developer for these two dialects would be a small change if preferred.
  • One-time cache miss after deploy: the request shape changes, so each existing session's next request reads its prompt uncached once.

Tests

Test What it shows
Prefix tests (all four dialects) Request N is a byte-level prefix of request N+1 across a new run, a tool round trip, a catalog refresh, and a catalog arriving mid tool exchange
Chat-template conformance Real templates of Qwen 3.5, Qwen 3, Qwen 2.5, gpt-oss and Mistral v0.3 render every Compatible and DeepSeek request without error, and each rendered request extends the previous one
Negative control The old shape is rejected by Qwen 3.5 and by Mistral, so the conformance test would catch a regression
Live caching suite Can point at a self-hosted server; new case: layered instructions plus a catalog refresh
  • Template fixtures are committed unchanged, with source revision and license, in tests/fixtures/chat_templates/ (all Apache-2.0). Rendering uses minijinja 3.0.5, a dev-dependency only.
  • Template exceptions:
    • Mistral v0.3 is checked for acceptance only: its template moves the system prompt next to the newest user message, so no client can keep its prefix.
    • Qwen 3.5 re-renders earlier assistant turns once a new user message arrives, so across runs it is checked up to the previous user message.
  • Live-suite settings: OPENAI_COMPLETIONS_PROVIDER_ID, OPENAI_COMPLETIONS_OUTPUT_LIMIT and OPENAI_COMPLETIONS_EXTRA_BODY.

cargo test -p llm-runtime (251 pass) and clippy pass.

Live check

Qwen3.5-9B (revision c202236) on vLLM 0.33.0, on one RTX 4090, started with --enable-prefix-caching --enable-prompt-tokens-details. The caching live suite ran from main and from this branch against the same server:

Case main This PR
Layered instructions + catalog, then a catalog refresh 400 System message must be at the beginning. passes, 98 % of the previous prompt cached
Second turn 99 % cached 99 % cached
Tool round trip 95 % cached 95 % cached

To rerun against a vLLM server:

OPENAI_COMPLETIONS_BASE_URL=http://127.0.0.1:8000/v1 OPENAI_COMPLETIONS_API_KEY=unused \
OPENAI_COMPLETIONS_MODEL=Qwen/Qwen3.5-9B OPENAI_COMPLETIONS_PROVIDER_ID=vllm \
OPENAI_COMPLETIONS_OUTPUT_LIMIT=1024 \
OPENAI_COMPLETIONS_EXTRA_BODY='{"chat_template_kwargs":{"enable_thinking":false}}' \
cargo test -p llm-runtime --test openai_completions_caching_live -- --ignored --test-threads=1

Thinking is off in that run because Qwen 3.5 can spend the whole output budget reasoning. With thinking on, the cache hit was the same (99 %), but the answer was cut off at the length limit.

Evidence

pr191-evidence.zip

Raw output, not summaries, recorded on this PR's head b5ff8012 and on main b304ae87 (2026-11-18). Each block below is the command's output as printed.

1. Unit and chat-template tests pass on b5ff8012

cargo test -p llm-runtime: openai_completions::tests, plus the per-binary summary for the whole crate
commit: b5ff80127632c00ebd7467f629c2a06559456de1
rustc 1.99.0 (b940084d7 2026-09-28)
cargo 1.99.0 (5f94df478 2026-08-27)
host: Darwin arm64
date: 2026-11-18T22:42:16Z
$ cargo test -p llm-runtime --lib openai_completions::tests
running 33 tests
test openai_completions::tests::compatible_failure_finish_reenters_the_retry_path ... ok
test openai_completions::tests::every_adapter_collates_the_same_system_prompt ... ok
test openai_completions::tests::deepseek_thinking_omits_auto_tool_choice_and_rejects_forced_choices ... ok
test openai_completions::tests::deepseek_uses_system_role_and_legacy_token_budget_for_generation_and_compaction ... ok
test openai_completions::tests::deepseek_none_reasoning_disables_thinking_and_gpt_5_5_rejects_unsupported_tiers ... ok
test openai_completions::tests::a_catalog_arriving_mid_tool_exchange_waits_for_the_results ... ok
test openai_completions::tests::deepseek_rejects_nonportable_multimodal_schema_and_strict_features_before_io ... ok
test openai_completions::tests::folds_assistant_tool_calls_and_materializes_tool_results ... ok
test openai_completions::tests::content_filter_finish_fails_the_turn_with_the_refusal_text ... ok
test openai_completions::tests::every_dialect_sends_one_leading_instruction_message_and_catalogs_as_user ... ok
test openai_completions::tests::compaction_uses_target_budget_and_returns_recognized_summary ... ok
test openai_completions::tests::no_system_message_follows_the_conversation ... ok
test openai_completions::tests::materializes_developer_user_params_and_neutral_fields ... ok
test openai_completions::tests::rejects_foreign_provider_native_context ... ok
test openai_completions::tests::maps_refusal_tool_call_usage_and_finish_facts ... ok
test openai_completions::tests::length_finish_fails_the_turn_but_keeps_partial_text ... ok
test openai_completions::tests::generate_passes_stored_oauth_bearer_to_client ... ok
test openai_completions::tests::generate_passes_stored_api_key_to_client ... ok
test openai_completions::tests::runtime_mcp_is_available_on_openai_compatible_completions ... ok
test openai_completions::tests::rejects_stop_for_openai_reasoning_models_before_io ... ok
test openai_completions::tests::code_mode_describes_injected_mcp_structured_content_without_changing_inputs ... ok
test openai_completions::tests::tracked_lowering_maps_each_entry_to_its_message ... ok
test openai_completions::tests::preserves_annotations_and_compatible_cache_accounting_without_replaying_annotations ... ok
test openai_completions::tests::assistant_content_projects_and_replays_exact_text_with_bounded_previews ... ok
test openai_completions::tests::custom_provider_passes_endpoint_and_explicit_anonymous_auth_to_client ... ok
test openai_completions::tests::tool_media_entries_follow_the_tool_message_as_user_parts ... ok
test openai_completions::tests::folds_images_and_pdfs_into_one_user_message ... ok
test openai_completions::tests::deepseek_drops_tool_media_with_a_note_but_still_rejects_run_input_media ... ok
test openai_completions::tests::the_prefix_holds_across_a_new_turn_a_tool_round_trip_and_a_catalog_refresh ... ok
test openai_completions::tests::chat_templates::the_old_shape_is_rejected_where_it_broke ... ok
test openai_completions::tests::preserves_and_exactly_replays_reasoning_extensions_with_tool_calls ... ok
test openai_completions::tests::chat_templates::every_rendered_request_extends_the_previous_one ... ok
test openai_completions::tests::chat_templates::every_template_accepts_every_dialects_requests ... ok
test result: ok. 33 passed; 0 failed; 0 ignored; 0 measured; 194 filtered out; finished in 0.15s

$ cargo test -p llm-runtime  (whole crate, summary per test binary)
     Running unittests src/lib.rs (target/debug/deps/llm_runtime-15996c71884733e5)
test result::tests::request_positions_group_entries_by_position ... ok
test result: ok. 227 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 24.96s
     Running tests/anthropic_messages_caching_live.rs (target/debug/deps/anthropic_messages_caching_live-686c9f1292a277de)
test result: ok. 0 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/anthropic_messages_compaction_live.rs (target/debug/deps/anthropic_messages_compaction_live-2c53445624c91efb)
test result: ok. 0 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/anthropic_messages_live.rs (target/debug/deps/anthropic_messages_live-36576b90292e2c94)
test result: ok. 0 passed; 0 failed; 27 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/anthropic_messages_mcp_live.rs (target/debug/deps/anthropic_messages_mcp_live-3044b56dece103b7)
test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/anthropic_messages_prompts_live.rs (target/debug/deps/anthropic_messages_prompts_live-5013aba9625d1173)
test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/anthropic_messages_skills_live.rs (target/debug/deps/anthropic_messages_skills_live-1d641ed25e8e2697)
test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/builtin_catalog_parity.rs (target/debug/deps/builtin_catalog_parity-7f3a4f1b63a8d0a0)
test result: ok. 3 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.04s
     Running tests/code_mode_schemas.rs (target/debug/deps/code_mode_schemas-9de4d70839f82c6a)
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
     Running tests/content_materialization.rs (target/debug/deps/content_materialization-daaa48ff1dbd4b8a)
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
     Running tests/live_config.rs (target/debug/deps/live_config-748fe56f2b621ea2)
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/openai_completions_caching_live.rs (target/debug/deps/openai_completions_caching_live-07c66f8d59c9575d)
test result: ok. 0 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/openai_completions_compaction_live.rs (target/debug/deps/openai_completions_compaction_live-d5ae5afd22380ab1)
test result: ok. 0 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/openai_completions_live.rs (target/debug/deps/openai_completions_live-5e4d6ca45daf74ca)
test result: ok. 0 passed; 0 failed; 13 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/openai_completions_prompts_live.rs (target/debug/deps/openai_completions_prompts_live-c1fe586e95e03658)
test result: ok. 0 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/openai_completions_replay.rs (target/debug/deps/openai_completions_replay-35fcc8960ffe33dd)
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s
     Running tests/openai_completions_skills_live.rs (target/debug/deps/openai_completions_skills_live-b32cbb95776c58d4)
test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/openai_responses_caching_live.rs (target/debug/deps/openai_responses_caching_live-88bb0432ec0f9d1f)
test result: ok. 0 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/openai_responses_compaction_live.rs (target/debug/deps/openai_responses_compaction_live-a9d7bd65447ee1a6)
test result: ok. 0 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/openai_responses_live.rs (target/debug/deps/openai_responses_live-7d93b72acb5f0692)
test result: ok. 0 passed; 0 failed; 9 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/openai_responses_mcp_live.rs (target/debug/deps/openai_responses_mcp_live-ea83b838f0e28adc)
test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/openai_responses_prompts_live.rs (target/debug/deps/openai_responses_prompts_live-eb11ee3e15955602)
test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/openai_responses_skills_live.rs (target/debug/deps/openai_responses_skills_live-ce7c5e97541e74e3)
test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
     Running tests/provider_failures.rs (target/debug/deps/provider_failures-ac79d722272026e4)
test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

2. The new tests catch the bug

openai_completions.rs was temporarily switched back to main's rendering (one message per instruction layer, catalogs in the instruction role), with the tests unchanged. Six of the new tests fail, and Qwen 3.5's real template raises the issue's error. The byte-level prefix test passes either way, because the old shape was also prefix-stable: it guards the cache, not this bug.

Mutation run output
Mutation check: crates/llm-runtime/src/openai_completions.rs temporarily changed back to main's rendering
(one message per instruction layer; catalogs in the instruction role), tests unchanged.
$ cargo test -p llm-runtime --lib openai_completions::tests
test openai_completions::tests::a_catalog_arriving_mid_tool_exchange_waits_for_the_results ... FAILED
test openai_completions::tests::every_dialect_sends_one_leading_instruction_message_and_catalogs_as_user ... FAILED
test openai_completions::tests::every_adapter_collates_the_same_system_prompt ... FAILED
test openai_completions::tests::no_system_message_follows_the_conversation ... FAILED
test openai_completions::tests::chat_templates::every_rendered_request_extends_the_previous_one ... FAILED
test openai_completions::tests::chat_templates::every_template_accepts_every_dialects_requests ... FAILED
test result: FAILED. 27 passed; 6 failed; 0 ignored; 0 measured; 194 filtered out; finished in 0.03s

---- openai_completions::tests::every_dialect_sends_one_leading_instruction_message_and_catalogs_as_user stdout ----
thread 'openai_completions::tests::every_dialect_sends_one_leading_instruction_message_and_catalogs_as_user' (69610674) panicked at crates/llm-runtime/src/openai_completions.rs:3204:13:
assertion `left == right` failed: OpenAi
---- openai_completions::tests::no_system_message_follows_the_conversation stdout ----
thread 'openai_completions::tests::no_system_message_follows_the_conversation' (69610683) panicked at crates/llm-runtime/src/openai_completions.rs:3252:13:
assertion `left == right` failed: OpenAi
---- openai_completions::tests::chat_templates::every_template_accepts_every_dialects_requests stdout ----
thread 'openai_completions::tests::chat_templates::every_template_accepts_every_dialects_requests' (69610660) panicked at crates/llm-runtime/src/openai_completions.rs:3536:29:
qwen3.5 rejects the Compatible request: invalid operation: System message must be at the beginning. (in qwen3.5:85)

3. Live runs against vLLM

Qwen/Qwen3.5-9B (revision c202236) on vLLM 0.33.0, on one RTX 4090, with prefix caching and cached-token reporting on. The suite ran through a pass-through proxy that logged every request and response in full. main ran with only the live-test files from b5ff8012, so both sides send the same cases.

Live suite output on main (before)
commit: b304ae873fd02e6e60195eb0bfb85daa93b41db1  (live test files from b5ff8012)
$ cargo test -p llm-runtime --test openai_completions_caching_live -- --ignored --test-threads=1 --nocapture

running 3 tests

thread 'openai_completions_caching_live_catalog_refresh_keeps_the_prefix_warm' (69657193) panicked at crates/llm-runtime/tests/openai_completions_caching_live.rs:237:10:
generate: Provider { source: HttpStatus(ProviderHttpError { api_kind: "openai:completions", status: 400, kind: InvalidRequest, message: "System message must be at the beginning.", error_code: None, error_type: Some("BadRequestError"), retryable: false, retry_after: None, raw_json: Some(Object {"error": Object {"code": Number(400), "message": String("System message must be at the beginning."), "param": Null, "type": String("BadRequestError")}}), raw_text: Some("{\"error\":{\"message\":\"System message must be at the beginning.\",\"type\":\"BadRequestError\",\"param\":null,\"code\":400}}"), headers: HeaderSnapshot { entries: [("content-length", "113"), ("content-type", "application/json"), ("date", "Wed, 18 Nov 2026 22:46:42 GMT"), ("server", "BaseHTTP/0.6 Python/3.11.8")] } }) }
openai_completions_caching_live_catalog_refresh_keeps_the_prefix_warm --- FAILED
cached share [second turn]: 6336 of the previous 6403 prompt tokens read from cache (99%)
.cached share [after the tool round trip]: 6336 of the previous 6657 prompt tokens read from cache (95%)
.
failures:

failures:
    openai_completions_caching_live_catalog_refresh_keeps_the_prefix_warm

test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.70s

error: test failed, to rerun pass `-p llm-runtime --test openai_completions_caching_live`
Live suite output on b5ff801 (after)
commit: b5ff80127632c00ebd7467f629c2a06559456de1  (live test files from b5ff8012)
$ cargo test -p llm-runtime --test openai_completions_caching_live -- --ignored --test-threads=1 --nocapture

running 3 tests
cached share [after a catalog refresh]: 6336 of the previous 6441 prompt tokens read from cache (98%)
.cached share [second turn]: 6336 of the previous 6403 prompt tokens read from cache (99%)
.cached share [after the tool round trip]: 6336 of the previous 6657 prompt tokens read from cache (95%)
.
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.11s

Every request the suite sent, from the proxy log:

before (main b304ae8)

# HTTP Messages sent (role ×count, in order) prompt_tokens cached_tokens Error sha256 of request body
1 400 system×4 → user×1 System message must be at the beginning. ea7593760dd32f1c
2 200 system×1 → user×1 6403 6336 f447714f79a4b4e6
3 200 system×1 → user×1 → assistant×1 → user×1 6470 6336 a5a4f2eb05f9dc71
4 200 system×1 → user×1 6657 6336 7d0ef437fcc7558e
5 200 system×1 → user×1 → assistant×1 → tool×1 6707 6336 abffbfeacffe9fea

after (PR b5ff801)

# HTTP Messages sent (role ×count, in order) prompt_tokens cached_tokens Error sha256 of request body
1 200 system×1 → user×1 6441 0 d0cdba1e9c0d7b71
2 200 system×1 → user×1 → assistant×1 → user×1 6520 6336 5afe84f03ec114c2
3 200 system×1 → user×1 6403 6336 ae0e26ade1e2ee0c
4 200 system×1 → user×1 → assistant×1 → user×1 6458 6336 ca01a2a7afea73bf
5 200 system×1 → user×1 6657 0 a72c291a669a8d35
6 200 system×1 → user×1 → assistant×1 → tool×1 6707 6336 4757139ecff3e253

How to read the table:

  • Cold start: each run's first request was cold (0 cached).
  • Shared cache: the cases run one after another against one server, so a case's first request can arrive warm from the case before. On main, request 2 shows 6336 cached for that reason.
  • Cache granularity: the cached amount is 6336 tokens in every warm request. That's the long shared system prompt as Qwen 3.5's hybrid cache stores it, in blocks.
  • Tool case: its first request is cold on both sides, because Qwen's template puts tool definitions ahead of the instructions.
  • What the refresh case shows: after a catalog refresh, the next request still reads that prefix from cache. On main, the same case is rejected before anything is cached.
First request and response of the catalog-refresh case, both sides

before: first request of the catalog-refresh case, HTTP 400 (long contents shortened)

{
 "model": "Qwen/Qwen3.5-9B",
 "max_tokens": 1024,
 "store": false,
 "stream": false,
 "prompt_cache_key": "session-live-openai-completions-caching",
 "chat_template_kwargs": {
  "enable_thinking": false
 },
 "messages": [
  {
   "role": "system",
   "content": "You are a warehouse operations assistant. Use the stock tabl… [23804 chars]"
  },
  {
   "role": "system",
   "content": "You are the release assistant of a small team."
  },
  {
   "role": "system",
   "content": "Commands run in /data/workspace."
  },
  {
   "role": "system",
   "content": "Sub-agent catalog:\n\n- reviewer: reviews a diff\n- tester: runs the test suite"
  },
  {
   "role": "user",
   "content": "Item 17 needs a status line. Reply in one sentence."
  }
 ]
}

Response:

{
 "error": {
  "message": "System message must be at the beginning.",
  "type": "BadRequestError",
  "param": null,
  "code": 400
 }
}

after: first request of the catalog-refresh case, HTTP 200 (long contents shortened)

{
 "model": "Qwen/Qwen3.5-9B",
 "max_tokens": 1024,
 "store": false,
 "stream": false,
 "prompt_cache_key": "session-live-openai-completions-caching",
 "chat_template_kwargs": {
  "enable_thinking": false
 },
 "messages": [
  {
   "role": "system",
   "content": "You are a warehouse operations assistant. Use the stock tabl… [23885 chars]"
  },
  {
   "role": "user",
   "content": [
    {
     "type": "text",
     "text": "Sub-agent catalog:\n\n- reviewer: reviews a diff\n- tester: runs the test suite"
    },
    {
     "type": "text",
     "text": "Item 17 needs a status line. Reply in one sentence."
    }
   ]
  }
 ]
}

Response:

{
 "choices[0].message.content": "Item 17 currently needs a status line added to its record.",
 "usage": {
  "prompt_tokens": 6441,
  "total_tokens": 6456,
  "completion_tokens": 15,
  "prompt_tokens_details": {
   "cached_tokens": 0,
   "created_cache_tokens": 6336,
   "multimodal_tokens": null
  },
  "completion_tokens_details": {
   "reasoning_tokens": 0
  }
 }
}
vLLM server log for these runs: arguments, every request with status, prefix-cache hit rate
non-default args: {'model_tag': 'Qwen/Qwen3.5-9B', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'enable_prompt_tokens_details': True, 'host': '127.0.0.1', 'model': 'Qwen/Qwen3.5-9B', 'revision': 'c202236235762e1c871ad0ccb60c8ee5ba337b9a', 'max_model_len': 16384, 'reasoning_parser': 'qwen3', 'enable_prefix_caching': True, 'limit_mm_per_prompt': {'image': 0, 'video': 0}, 'max_num_seqs': 4}
INFO:     127.0.0.1:33456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:33462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:33478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:33488 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:33500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:33516 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:39808 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
INFO:     127.0.0.1:39812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 11-18 23:46:43 [loggers.py:335] Engine 000: Avg prompt throughput: 1390.9 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 69.5%
INFO:     127.0.0.1:39824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:39838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:39840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 11-18 23:46:53 [loggers.py:335] Engine 000: Avg prompt throughput: 82.6 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 77.5%

The full request and response bodies (280 KB) don't fit in a comment. The sha256 prefixes in the tables identify each request body (JSON with sorted keys, no whitespace).

🤖 Generated with Claude Code

@lukebuehler

Copy link
Copy Markdown
Contributor

Implemented in #194. Thanks for the contribution. I discussed the approach the details with Marvin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chat Completions: merge system messages for OpenAI-compatible providers

3 participants