perf(llm): preserve prompt prefixes and apply provider-specific caching

docs/prompt-cache-policy.md

@@ -0,0 +1,46 @@
+# Provider prompt cache contract
+
+The client optimizes reusable input prefixes. It does not cache model outputs or skip model calls. Cache availability, minimum lengths, expiry and routing remain properties of the selected service and model. A token-weighted hit rate is not a request success rate; unique input and cold requests remain in the denominator when reported.
+
+## Request boundaries
+
+Fixed style, translation and question-answering instructions precede changing application and cursor context. Existing context escaping, text envelopes, trailing injection defenses and output constraints remain intact. Custom style templates and vocabulary order are not rewritten. Ordinary dictation does not acquire unrelated conversation history to inflate the shared prefix.
+
+Provider-specific parameters are selected from the actual HTTPS origin, API path, request format and known model capabilities, rather than the channel's display name or template. Unknown gateways retain the shared prefix improvement without unverified vendor parameters. User-supplied routing headers and cache settings take precedence.
+
+Routing keys are bounded opaque hashes of service, model and reusable system prefix. They contain no credentials, raw prompt, generated application context or per-request UUID. Cache breakpoints divide an existing system string into contiguous slices: concatenating the slices must reproduce the original text exactly. Only the static slice receives a system cache marker. For growing Anthropic conversations, a second marker may cover the last completed assistant turn; the current user input is not marked. Multi-turn OpenAI requests retain implicit history caching alongside the static explicit breakpoint.
+
+## Service strategies
+
+| Service and format | Strategy |
+| --- | --- |
+| DeepSeek official APIs | Stable prefixes; provider-managed implicit caching. No OpenAI cache parameters are forwarded. |
+| OpenAI Chat Completions / Responses | Stable routing key for related prefixes. Known GPT-5.6 and GPT-6 Responses models additionally mark static developer input with an explicit breakpoint. Older models retain implicit caching. |
+| Anthropic Messages | Mark the stable system content with the default ephemeral cache policy for Claude models. |
+| OpenRouter Chat Completions / Messages | Stable `session_id` for cache-aware routing; explicit stable-system markers for supported Anthropic models. Existing route selection is preserved. |
+| Ark Chat / Responses | Stable prefixes for all models. Only documented Seed 2 series receive `X-Prompt-Cache-Id`; DeepSeek models do not receive this Seed-specific option. |
+| DashScope | Stable prefixes; explicit markers only for verified standard API/model combinations. Coding-plan and unsupported model endpoints retain implicit behavior. |
+| Native Gemini, Codex and other gateways | Shared stable prompt composition. No generic endpoint's cache flags are assumed compatible with their transport. |
+
+The policy does not create separate remote cache objects, extend retention, enable response storage, send paid warm-up requests or change provider/model selection. An explicit marker can incur a provider's normal cache-write charge; its benefit depends on reuse before expiry.
+
+## Accounting and validation
+
+Usage records retain the service and protocol snapshot and the raw returned quantities. Canonical Chat Completions, Responses and Gemini input totals include cached input; Messages input is additive with cache reads and writes. Ambiguous or missing fields retain unknown semantics. Old records are not retroactively assigned the current service or a guessed denominator.
+
+The displayed rate is `sum(cache reads) / sum(comparable total input)` for the same reported records. The UI exposes the number of comparable reports and groups by service, channel, model and protocol. This remains a partial ledger: native Gemini/Codex recording, every request attempt and unreported usage require separate work. Neither a missing report nor a short uncached request proves a client failure.
+
+Acceptance checks cover origin/model/format boundaries, exact text preservation, stable keys across application changes, caller overrides, inclusive/additive usage, unknown values, overflow and streaming report merging. Service-side improvement must be measured from subsequent real responses; local request tests cannot establish a universal 97–98% cache rate.
+
+## Official references
+
+Verified on 2026-11-09; capability tables must be checked when adding a model or service:
+
+- [DeepSeek context caching](https://api-docs.deepseek.com/guides/kv_cache/)
+- [OpenAI prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching)
+- [Anthropic prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
+- [Ark implicit and explicit caching](https://docs.volcengine.com/docs/ark/context-cache?lang=zh)
+- [Ark routing key and supported models](https://docs.volcengine.com/docs/ark/prompt-cache-key?lang=zh)
+- [DashScope context caching](https://help.aliyun.com/zh/model-studio/context-cache)
+- [OpenRouter cache-aware routing](https://openrouter.ai/blog/tutorials/prompt-caching-sticky-routing/)
+- [Gemini context caching](https://ai.google.dev/gemini-api/docs/caching)

docs/usage-accounting-follow-up.md

@@ -4,7 +4,9 @@ The initial desktop page reads retained dictation history, which is capped and c
 
 An initial bounded, text-free local ledger and background writer now record explicitly reported tokens from desktop OpenAI-compatible text-polisher responses. The desktop page presents those reports separately from history-derived counts and labels them incomplete. Gemini, Codex, SIWC, multimodal, ASR, TTS, retries and all other call paths are not yet fully covered. The writer's error count is process-local; durable loss reporting and an explicit shutdown drain remain required. No historical token values can be recovered from the existing history.
 
-Official OpenAI Chat Completions streaming requests explicitly ask for usage only when using the official endpoint and an observer is attached. An interrupted stream may not return its final usage report. DeepSeek documents its own final-stream usage and cache fields. Do not apply an OpenAI-specific request option to arbitrary compatible endpoints: [OpenAI API reference](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create), [DeepSeek API reference](https://api-docs.deepseek.com/api/create-chat-completion/).
+Known official Chat Completions services request streaming usage when their endpoint/model supports it and an observer is attached. An interrupted stream may not return its final usage report. Unknown gateways receive no unverified usage option. See the [provider cache contract](prompt-cache-policy.md) and its official references.
+
+New reports retain a service/protocol snapshot and explicit inclusive or additive cache accounting derived from canonical response fields. The cache dashboard groups comparable reports by service, channel, model and protocol; it shows the count of comparable saved reports, not coverage of all actual attempts. Legacy and ambiguous reports keep unknown cache semantics. Existing raw quantities are preserved, and no current endpoint or model configuration is assigned retroactively.
 
 To satisfy complete usage accounting, add a separately persisted, bounded, local ledger at the service-call boundary. Record a unique attempt identifier, timestamp, operation (including dictation post-processing, translation, selection, QA and other call sites), provider and model, outcome, and provider-reported quantities. Preserve unknown quantities as unknown, not zero; never infer token counts or price from characters. Capture ASR audio units and TTS units where the underlying provider reports them. Cover streaming, non-streaming, retries, cancellation, errors, multimodal calls and supported provider protocols, with tests proving one record per actual attempt and no secret or text content in the ledger. Distinguish missing legacy data from a zero-usage interval. Do not silently attach current model prices to historical requests.
 

openless-all/app/crates/openless-core/src/cloud_providers.rs

@@ -1137,6 +1137,7 @@ impl TextPolisher for SharedCloudTextPolisher {
                     (usage_recorder, usage_generation, usage_label)
                 {
                     let purpose = usage_purpose(&context);
+                    let (service, protocol) = provider.usage_metadata();
                     let channel = crate::usage_ledger::sanitized_identifier(&context.llm.provider_id);
                     provider.set_usage_observer(Arc::new(move |id, usage| {
                         let record = crate::usage_ledger::UsageRecord {
@@ -1146,6 +1147,9 @@ impl TextPolisher for SharedCloudTextPolisher {
                             purpose,
                             outcome: crate::usage_ledger::ServiceOutcome::Unknown,
                             provider: crate::usage_ledger::sanitized_identifier(&label.provider),
+                            service: crate::usage_ledger::sanitized_identifier(&service),
+                            protocol: crate::usage_ledger::sanitized_identifier(&protocol),
+                            input_accounting: usage.input_accounting,
                             channel: channel.clone(),
                             model: crate::usage_ledger::sanitized_identifier(&label.model),
                             audio_ms: None,
@@ -3759,6 +3763,10 @@ mod tests {
         assert_eq!(records[0].input_tokens, Some(30));
         assert_eq!(records[0].output_tokens, Some(4));
         assert_eq!(records[0].channel.as_deref(), Some("llm-channel"));
+        assert_eq!(records[0].protocol.as_deref(), Some("chat_completions"));
+        assert_eq!(records[0].input_accounting, Some(crate::usage_ledger::CacheInputAccounting::Inclusive));
+        assert!(records[0].service.is_some());
+        assert!(!records[0].service.as_deref().unwrap().contains(&address.to_string()));
         assert_eq!(records[0].purpose, crate::usage_ledger::ServicePurpose::DictationPostprocess);
         let stored = std::fs::read_to_string(directory.join("usage.json")).unwrap();
         assert!(!stored.contains("private text"));

openless-all/app/crates/openless-core/src/lib.rs

@@ -57,6 +57,7 @@ mod persistence;
 pub mod polish;
 pub mod ports;
 pub mod preferences;
+mod prompt_cache;
 pub mod prompt_compose;
 pub mod prompts;
 pub mod provider_registry;

openless-all/app/crates/openless-core/src/polish.rs

@@ -571,6 +571,18 @@ impl OpenAICompatibleLLMProvider {
         self.usage_observer = Some(observer);
     }
 
+    pub(crate) fn usage_metadata(&self) -> (String, String) {
+        let protocol = match self.config.protocol.format {
+            LlmRequestFormat::ChatCompletions => "chat_completions",
+            LlmRequestFormat::Responses => "responses",
+            LlmRequestFormat::Messages => "messages",
+        };
+        (
+            crate::prompt_cache::service_id(&self.config.base_url).into(),
+            protocol.into(),
+        )
+    }
+
     fn report_usage(&self, request_id: uuid::Uuid, usage: crate::usage_ledger::ReportedTokens) {
         if usage != crate::usage_ledger::ReportedTokens::default() {
             if let Some(observer) = &self.usage_observer {
@@ -855,7 +867,9 @@ impl OpenAICompatibleLLMProvider {
 
     fn chat_body(&self, stream: bool, messages: Vec<Value>) -> Value {
         if self.config.protocol.format != LlmRequestFormat::ChatCompletions {
-            return crate::llm_protocol::request_body(&self.config, stream, messages);
+            let mut body = crate::llm_protocol::request_body(&self.config, stream, messages);
+            crate::prompt_cache::apply(&self.config, &mut body);
+            return body;
         }
         let mut body = json!({
             "model": self.config.model,
@@ -864,12 +878,7 @@ impl OpenAICompatibleLLMProvider {
         });
         if stream
             && self.usage_observer.is_some()
-            && self.config.provider_id == "openai"
-            && url::Url::parse(&self.config.base_url)
-                .ok()
-                .and_then(|url| url.host_str().map(str::to_owned))
-                .as_deref()
-                == Some("api.openai.com")
+            && crate::prompt_cache::supports_usage(&self.config)
         {
             body["stream_options"] = json!({"include_usage": true});
         }
@@ -893,6 +902,7 @@ impl OpenAICompatibleLLMProvider {
         self.config
             .protocol
             .apply_service_tier(&self.config.provider_id, &mut body);
+        crate::prompt_cache::apply(&self.config, &mut body);
         body
     }
 
@@ -959,7 +969,7 @@ impl OpenAICompatibleLLMProvider {
             })
             .map_err(|error| LLMError::ParseError(error.message))?;
         let request = self
-            .authorize(self.polish_client.post(url))
+            .authorize(self.polish_client.post(url), body)
             .header("Content-Type", "application/json")
             .json(body);
 
@@ -1091,7 +1101,7 @@ impl OpenAICompatibleLLMProvider {
             self.config.protocol.format
         );
         let request = self
-            .authorize(self.polish_client.post(&url))
+            .authorize(self.polish_client.post(&url), &body)
             .header("Content-Type", "application/json")
             .header("Accept", "text/event-stream")
             .json(&body);
@@ -1199,13 +1209,27 @@ impl OpenAICompatibleLLMProvider {
         Ok(full_text)
     }
 
-    fn authorize(&self, mut request: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
+    fn authorize(
+        &self,
+        mut request: reqwest::RequestBuilder,
+        body: &Value,
+    ) -> reqwest::RequestBuilder {
         for (name, value) in self.config.protocol.format.headers(&self.config.api_key) {
             request = request.header(name, value);
         }
         for (name, value) in &self.config.extra_headers {
             request = request.header(name, value);
         }
+        if let Some((name, value)) = crate::prompt_cache::routing_header(&self.config, body) {
+            if !self
+                .config
+                .extra_headers
+                .keys()
+                .any(|configured| configured.eq_ignore_ascii_case(name))
+            {
+                request = request.header(name, value);
+            }
+        }
         request
     }
 }
@@ -1401,15 +1425,12 @@ impl CodexOAuthLLMProvider {
         F: Fn(&str) + Send + Sync,
         C: Fn() -> bool + Send + Sync,
     {
-        let mut system_prompt = prompts::qa_system_prompt();
-        if let Some(premise) = context_premise(
+        let system_prompt = compose_qa_system_prompt(
             working_languages,
             chinese_script_preference,
             output_language_preference,
             front_app,
-        ) {
-            system_prompt = format!("{}\n\n{}", premise, system_prompt);
-        }
+        );
 
         let mut request_messages = Vec::with_capacity(messages.len() + 1);
         request_messages.push(json!({ "role": "system", "content": system_prompt }));
@@ -2670,7 +2691,7 @@ mod tests {
     }
 
     #[test]
-    fn usage_stream_option_is_limited_to_official_openai_chat_requests() {
+    fn usage_stream_option_is_limited_to_verified_chat_services() {
         let observer: LlmUsageObserver = std::sync::Arc::new(|_, _| {});
         let official = OpenAICompatibleLLMProvider::new(OpenAICompatibleConfig::new(
             "openai",
@@ -2702,6 +2723,73 @@ mod tests {
             .is_none());
     }
 
+    #[test]
+    fn cache_policy_survives_protocol_conversion_and_header_authorization() {
+        let messages = vec![
+            json!({"role": "system", "content": "fixed rules\n\n# 上下文\nApp A"}),
+            json!({"role": "user", "content": "unique input"}),
+        ];
+        let mut config = OpenAICompatibleConfig::new(
+            "custom",
+            "Official",
+            "https://api.anthropic.com/v1",
+            "test",
+            "claude-sonnet-4-6",
+        );
+        config.protocol.format = LlmRequestFormat::Messages;
+        let provider = OpenAICompatibleLLMProvider::new(config);
+        let body = provider.chat_body(true, messages.clone());
+        assert_eq!(body["system"][0]["cache_control"]["type"], "ephemeral");
+        assert_eq!(body["system"][1]["text"], "\n\n# 上下文\nApp A");
+        assert_eq!(body["messages"][0]["content"], "unique input");
+
+        let mut config = OpenAICompatibleConfig::new(
+            "custom",
+            "Official",
+            "https://api.openai.com/v1",
+            "test",
+            "gpt-6.1",
+        );
+        config.protocol.format = LlmRequestFormat::Responses;
+        let provider = OpenAICompatibleLLMProvider::new(config);
+        let body = provider.chat_body(true, messages.clone());
+        assert_eq!(body["prompt_cache_options"]["mode"], "explicit");
+        assert_eq!(body["input"][0]["role"], "developer");
+        assert_eq!(body["store"], false);
+        assert_eq!(body["input"][1]["content"], "unique input");
+
+        let mut config = OpenAICompatibleConfig::new(
+            "ark",
+            "Ark",
+            "https://ark.cn-beijing.volces.com/api/v3",
+            "test",
+            "doubao-seed-2-1-pro-260628",
+        );
+        config
+            .extra_headers
+            .insert("x-prompt-cache-id".into(), "user-affinity".into());
+        let provider = OpenAICompatibleLLMProvider::new(config);
+        let body = provider.chat_body(true, messages);
+        let request = provider
+            .authorize(
+                provider
+                    .polish_client
+                    .post("https://ark.cn-beijing.volces.com/api/v3/chat/completions"),
+                &body,
+            )
+            .build()
+            .unwrap();
+        assert_eq!(request.headers()["x-prompt-cache-id"], "user-affinity");
+        assert_eq!(
+            request
+                .headers()
+                .get_all("x-prompt-cache-id")
+                .iter()
+                .count(),
+            1
+        );
+    }
+
     #[tokio::test]
     async fn polish_streaming_handles_multibyte_split_in_http_chunk() {
         let listener = TcpListener::bind("127.0.0.1:0").unwrap();
@@ -4712,14 +4800,14 @@ mod tests {
         let mut expected = compose_system_prompt(&prompts::system_prompt(PolishMode::Light), &[]);
         expected = format!(
             "{}\n\n{}",
+            expected,
             context_premise(
                 &["中文".to_string()],
                 ChineseScriptPreference::Auto,
                 OutputLanguagePreference::Auto,
                 Some("Notes (com.apple.Notes)"),
             )
-            .unwrap(),
-            expected
+            .unwrap()
         );
         expected = format!("{}\n\n{}", expected, prompts::polish_injection_defense());
         assert_eq!(without, expected);