Skip to content

Chat Completions: keep one leading instruction message and render catalogs as user context - #194

Merged
lukebuehler merged 4 commits into
mainfrom
fix/chat-completions-context-roles
Nov 19, 2026
Merged

lukebuehler merged 4 commits into
mainfrom
fix/chat-completions-context-roles

Conversation

@lukebuehler

@lukebuehler lukebuehler commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #190. Related implementation and investigation: #191.

Why this fails

A session with profile instructions and a skill catalog currently sends system, system, user through a custom Chat Completions endpoint. Two instruction layers alone produce the same sequence. Qwen3.5-9B’s published chat template rejects the second system message. We reproduced the reported HTTP 400 on Parasail through OpenRouter using Lightspeed’s custom-provider dialect and a pinned route. DeepInfra accepted the same payload on the same model.

How the adapter now renders context

  • Join the active instruction layers in their planned order into one leading message when instructions exist. Keep the dialect’s existing role: developer for OpenAI and OpenRouter, system for DeepSeek and custom endpoints such as LiteLLM. Skip blank layers and omit the leading message when all layers are blank. Only instruction entries that contribute text map to message 0 in rejection diagnostics.
  • Render every catalog as user content at its harness-planned position, including the initial catalog. Adjacent user content shares a message. The harness already keeps tool calls adjacent to their results and marks a refreshed catalog as superseding its predecessor.
  • Generation and standalone compaction use the same context renderer.

Why catalogs stay out of the leading message

Suppose a run starts with skills A and B, and the agent writes skill C. On the next run the harness can append an updated catalog listing A, B and C. Rebuilding the leading instruction message with that new inventory would change the prompt near the beginning and shorten the prefix eligible for cache reuse. Appending the successor preserves the earlier conversation. A genuine change to profile or environment instructions intentionally rebuilds the leading message.

Catalog removal is a separate concern. The regression test checks that the earlier messages remain a prefix across a refresh. Actual cache hits depend on the serving stack.

Relation to #191

#191 identified the template constraint and contributed the self-hosted Qwen investigation on #190. Its later work also proposes a single instruction prompt and user-role catalogs. This PR was developed independently from main. Its production change is confined to Chat Completions, with live reproductions that can target pinned OpenRouter backends or a LiteLLM deployment.

Verification

  • All 10 Qwen live cases passed on each of Parasail, SiliconFlow, and DeepInfra. Both previously failing Parasail requests now succeed. The suite covers catalog refresh, parallel tool results, steering, tool media, and compaction.
  • The two original failing payloads also rendered with Qwen’s published template. The on-prem LiteLLM deployment itself remains unverified.
  • cargo test --locked -p llm-runtime and cargo clippy --workspace --all-targets --locked -- -D warnings passed.

To reproduce with OPENROUTER_API_KEY in the environment or root .env:

cargo test --locked -p llm-runtime --test openai_completions_live qwen:: -- --ignored --nocapture --test-threads=1

For LiteLLM, set QWEN_LIVE_BASE_URL, QWEN_LIVE_MODEL, and, if needed, QWEN_LIVE_API_KEY.

Platform fixes

  • The OpenRouter provider preset and both endpoint placeholders now use https://openrouter.ai/api/v1. The clients append /chat/completions or /responses for the chosen API kind.
  • The runtime HTTP proxy closes an established event stream cleanly at its scheduled 60-second reauthorization boundary. Unexpected upstream failures still propagate to the client.
  • Verified the provider form and runtime HTTP test files (32 tests total), the Platform web typecheck, and the root TypeScript typecheck.

@lukebuehler
lukebuehler merged commit 41f79ee into main Nov 19, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chat Completions: merge system messages for OpenAI-compatible providers

1 participant