diff --git a/docs/awareness-follow-up.md b/docs/awareness-follow-up.md new file mode 100644 index 000000000..0abd9c671 --- /dev/null +++ b/docs/awareness-follow-up.md @@ -0,0 +1,17 @@ +# Awareness follow-up + +The desktop Awareness page groups cursor context, opt-in correction suggestions, and an initial opt-in expression-learning implementation. The initial implementation learns only a small set of global formatting directions, stores abstract evidence locally, and applies stable directions to future dictation prompts. It does not yet implement application-awareness controls, per-app/category profiles, a user blacklist, or the complete correction system requested by the user. + +## Reference and evidence + +A local Type4Me source checkout at upstream revision `f21339e214d65e9bd4d191ff5136513902e7473b` was reviewed. The installed app identifies itself as 3.2.0; equivalence to that checkout has not been established. The checkout is MIT-licensed: retain its copyright and license notice if source or substantial portions are ported. Relevant upstream files are `Sources/Type4Me/Services/IntelliSenseContext.swift`, `UserEditClassifier.swift`, `ExpressionProfileStore.swift`, `BatchCorrectionInference.swift`, and `Sources/Type4MeIntelliSenseCore/IntelliSensePrompt.swift` in https://github.com/joewongjc/type4me. + +The reviewed implementation separates app/control context, cursor context, correction detection, and expression learning. It classifies observed edits and rejects sensitive content and changes to facts or negation before treating edits as style evidence. Style profiles hold abstract features and use sample, day-span, consistency and decay thresholds before emitting limited form-only directives. Its batched correction candidates use independent sessions, multiple days and directional confidence. These are observations about that source revision, not claims about an inaccessible server or later app version. + +## Required implementation + +- Define independently controlled, opt-in learning and capture boundaries; keep secure fields, sensitive apps, uncertain target changes and blacklist entries excluded. Describe when contextual text may be sent to the configured model. +- The initial implementation accepts only uniquely anchored formatting edits, rejects sensitive or content-changing observations, and stores bounded text-free global evidence. Opt-out stops observation and new-session application; users can clear the profile. Mixed edits and richer classification remain unsupported. +- Extend the profile to app/category scopes with explicit precedence and decay, plus a user-maintained exclusion list. Validate reset and opt-out races across all hosts, and ensure form directives cannot override facts, explicit corrections or user instructions. +- Correction suggestions now reject changes to numeric or negation signatures and sensitive contexts, while retaining explicit user confirmation. Add a richer lexical affinity model, conflicting-candidate handling and repeated-session evidence without treating one edit as a stable habit. +- Integrate the controls into the single Awareness page and exercise opt-out, clearing, app exclusion, restart and prompt behavior across supported hosts. Add deterministic classification, persistence, safety, stale-observer and native integration tests before enabling the feature. diff --git a/docs/usage-accounting-follow-up.md b/docs/usage-accounting-follow-up.md new file mode 100644 index 000000000..57dacac6e --- /dev/null +++ b/docs/usage-accounting-follow-up.md @@ -0,0 +1,11 @@ +# Service usage accounting follow-up + +The initial desktop page reads retained dictation history, which is capped and can be cleared. It is not a service usage ledger. Provider labels in those records do not represent retries or complete billing data. + +An initial bounded, text-free local ledger and background writer now record explicitly reported tokens from desktop OpenAI-compatible text-polisher responses. The desktop page presents those reports separately from history-derived counts and labels them incomplete. Gemini, Codex, SIWC, multimodal, ASR, TTS, retries and all other call paths are not yet fully covered. The writer's error count is process-local; durable loss reporting and an explicit shutdown drain remain required. No historical token values can be recovered from the existing history. + +Official OpenAI Chat Completions streaming requests explicitly ask for usage only when using the official endpoint and an observer is attached. An interrupted stream may not return its final usage report. DeepSeek documents its own final-stream usage and cache fields. Do not apply an OpenAI-specific request option to arbitrary compatible endpoints: [OpenAI API reference](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create), [DeepSeek API reference](https://api-docs.deepseek.com/api/create-chat-completion/). + +To satisfy complete usage accounting, add a separately persisted, bounded, local ledger at the service-call boundary. Record a unique attempt identifier, timestamp, operation (including dictation post-processing, translation, selection, QA and other call sites), provider and model, outcome, and provider-reported quantities. Preserve unknown quantities as unknown, not zero; never infer token counts or price from characters. Capture ASR audio units and TTS units where the underlying provider reports them. Cover streaming, non-streaming, retries, cancellation, errors, multimodal calls and supported provider protocols, with tests proving one record per actual attempt and no secret or text content in the ledger. Distinguish missing legacy data from a zero-usage interval. Do not silently attach current model prices to historical requests. + +Expose aggregated date ranges and paginated, filterable records through the existing host contract, independent of history deletion. Add migration, retention and explicit deletion controls, deterministic aggregation tests, and UI tests for partial data, empty states and failed reads. The current history-derived view must be replaced or clearly separated when the ledger exists. diff --git a/openless-all/app/crates/openless-core/src/api.rs b/openless-all/app/crates/openless-core/src/api.rs index 14d05219a..cbbded797 100644 --- a/openless-all/app/crates/openless-core/src/api.rs +++ b/openless-all/app/crates/openless-core/src/api.rs @@ -1746,6 +1746,7 @@ pub struct BackendRepositories { pub preferences: Arc, pub history: Arc, pub activity: Arc, + pub expression_profile: crate::expression_observation::SharedExpressionProfile, pub vocabulary: Arc, pub correction_rules: Arc, pub style_packs: Arc, @@ -1771,6 +1772,9 @@ impl BackendRepositories { activity: Arc::new( ActivityStore::at_data_dir(data_dir).unwrap_or_else(|_| ActivityStore::in_memory()), ), + expression_profile: crate::expression_observation::open_shared_expression_profile( + data_dir.join("expression-profile.json"), + ), vocabulary: Arc::new(DictionaryStore::at_data_dir(data_dir)), correction_rules: Arc::new(CorrectionRuleStore::at_data_dir(data_dir)), style_packs, @@ -2614,6 +2618,7 @@ pub struct OpenLessBackend { history: Arc, history_revision: Arc, activity: Arc, + expression_profile: crate::expression_observation::SharedExpressionProfile, style_packs: Arc, style_pack_revision: Arc, preferences: Arc, @@ -2675,6 +2680,57 @@ struct CoreEditObservationSink { events: Arc, } +struct CombinedEditObservationSink { + vocabulary: Option, + expression_profile: Option>, + expression_generation: Option, + typed_text: String, + session_id: SessionId, + clock: Arc, + generation: Arc, + expected_generation: u64, + settings_gate: Arc>, + duration: std::time::Duration, +} + +impl EditObservationSink for CombinedEditObservationSink { + fn observation_duration(&self) -> std::time::Duration { + self.duration + } + + fn publish(&self, edit: crate::host_document::EditPair) -> bool { + use chrono::Datelike; + if self.generation.load(Ordering::Acquire) != self.expected_generation { + return false; + } + let signal = self.expression_profile.as_ref().and_then(|_| { + crate::expression_observation::signal_from_observed_edit(&self.typed_text, &edit).ok() + }); + let mut recorded = false; + if let (Some(profile), Some(profile_generation), Some(signal)) = + (&self.expression_profile, self.expression_generation, signal) + { + if let Ok(_guard) = self.settings_gate.try_lock() { + if self.generation.load(Ordering::Acquire) == self.expected_generation { + match profile.record_if_generation( + self.session_id.as_uuid(), + self.clock.today_local().num_days_from_ce(), + signal, + profile_generation, + ) { + Ok(saved) => recorded = saved, + Err(error) => log::warn!("failed to save expression evidence: {error}"), + } + } + } + } + self.vocabulary + .as_ref() + .is_some_and(|sink| sink.publish(edit)) + || recorded + } +} + impl EditObservationSink for CoreEditObservationSink { fn observation_duration(&self) -> std::time::Duration { std::time::Duration::from_secs(self.settings.observation_seconds as u64) @@ -2687,6 +2743,9 @@ impl EditObservationSink for CoreEditObservationSink { if self.generation.load(Ordering::Acquire) != self.expected_generation { return false; } + if !crate::correction_observation::permits_vocabulary_suggestion(&self.typed_text, &edit) { + return false; + } let rules = crate::host_document::learned_rules_with_max_chars( &edit, self.settings.max_phrase_chars as usize, @@ -2923,6 +2982,7 @@ impl OpenLessBackend { preferences: Arc::new(PreferencesStore::in_memory()), history: Arc::new(HistoryStore::at_path(std::path::PathBuf::new())), activity: Arc::new(ActivityStore::in_memory()), + expression_profile: Arc::new(Mutex::new(None)), vocabulary: Arc::new(DictionaryStore::at_path(std::path::PathBuf::new())), correction_rules: Arc::new(CorrectionRuleStore::at_path(std::path::PathBuf::new())), style_packs: Arc::new(StylePackStore::in_memory()), @@ -3365,6 +3425,7 @@ impl OpenLessBackend { history: repositories.history, history_revision, activity: repositories.activity, + expression_profile: repositories.expression_profile, style_packs: repositories.style_packs, style_pack_revision, preferences: repositories.preferences, @@ -4061,6 +4122,7 @@ impl OpenLessBackend { preferences: Arc::clone(&self.preferences), history: Arc::clone(&self.history), activity: Arc::clone(&self.activity), + expression_profile: Arc::clone(&self.expression_profile), vocabulary: Arc::clone(&self.vocabulary), correction_rules: Arc::clone(&self.correction_rules), style_packs: Arc::clone(&self.style_packs), @@ -6815,6 +6877,16 @@ impl OpenLessBackend { .reset(); } + if previous.expression_learning_enabled && !preferences.expression_learning_enabled { + if let Ok(profile) = self.expression_profile.lock() { + if let Some(profile) = profile.as_ref() { + if let Err(error) = profile.invalidate_observers() { + log::warn!("failed to invalidate expression observers: {error}"); + } + } + } + self.disarm_edit_observation(); + } if (previous.vocabulary_learning_enabled && !preferences.vocabulary_learning_enabled) || previous.vocabulary_learning_settings != preferences.vocabulary_learning_settings { @@ -7283,6 +7355,38 @@ impl OpenLessBackend { pending.clone() } + pub fn expression_profile_directions( + &self, + ) -> Result, BackendError> { + let profile = self.expression_profile.lock().map_err(|_| { + BackendError::new(BackendErrorCode::Internal, "expression profile lock unavailable") + })?.clone(); + profile + .ok_or_else(|| BackendError::new(BackendErrorCode::Persistence, "expression profile is unavailable"))? + .stable_directions() + } + + pub fn clear_expression_profile(&self) -> Result<(), BackendError> { + self.ensure_runtime_ready()?; + let _guard = self.settings_write_gate.lock().map_err(|_| { + BackendError::new(BackendErrorCode::Internal, "settings lock unavailable") + })?; + self.disarm_edit_observation(); + let mut profile = self.expression_profile.lock().map_err(|_| { + BackendError::new(BackendErrorCode::Internal, "expression profile lock unavailable") + })?; + if let Some(existing) = profile.as_ref() { + if existing.clear().is_ok() { + return Ok(()); + } + } + let store = crate::expression_observation::ExpressionProfileStore::reset( + self.config.data_dir.join("expression-profile.json"), + )?; + *profile = Some(Arc::new(store)); + Ok(()) + } + /// Queue one observed manual correction. Duplicate pairs are ignored and /// the oldest item is dropped when the bounded card capacity is reached. pub fn queue_pending_correction( @@ -7334,24 +7438,48 @@ impl OpenLessBackend { } } self.disarm_edit_observation(); + let preferences = self.get_preferences(); if !enabled - || !self.get_preferences().vocabulary_learning_enabled + || (!preferences.vocabulary_learning_enabled && !preferences.expression_learning_enabled) || !matches!(insert_outcome, Some(InsertOutcome::Inserted)) || typed_text.trim().is_empty() { return; } let expected_generation = self.edit_observation_generation.load(Ordering::Acquire); - let sink = Arc::new(CoreEditObservationSink { - settings: self - .get_preferences() - .vocabulary_learning_settings - .normalized(), - expected_generation, - generation: Arc::clone(&self.edit_observation_generation), + let settings = preferences.vocabulary_learning_settings.normalized(); + let vocabulary = preferences + .vocabulary_learning_enabled + .then(|| CoreEditObservationSink { + settings, + expected_generation, + generation: Arc::clone(&self.edit_observation_generation), + typed_text: typed_text.to_string(), + pending: Arc::clone(&self.pending_corrections), + events: Arc::clone(&self.events), + }); + let expression_profile = if preferences.expression_learning_enabled { + self.expression_profile + .lock() + .ok() + .and_then(|profile| profile.clone()) + } else { + None + }; + if vocabulary.is_none() && expression_profile.is_none() { + return; + } + let sink = Arc::new(CombinedEditObservationSink { + vocabulary, + expression_generation: expression_profile.as_ref().map(|profile| profile.generation()), + expression_profile, typed_text: typed_text.to_string(), - pending: Arc::clone(&self.pending_corrections), - events: Arc::clone(&self.events), + session_id, + clock: Arc::clone(&self.clock), + generation: Arc::clone(&self.edit_observation_generation), + expected_generation, + settings_gate: Arc::clone(&self.settings_write_gate), + duration: std::time::Duration::from_secs(settings.observation_seconds as u64), }); if let Err(error) = self .deps @@ -9320,6 +9448,32 @@ impl OpenLessBackend { options, ); context.pipeline_mode = pipeline_mode; + if purpose == DictationContextPurpose::Dictation + && !context.polish.translation_active + && preferences.expression_learning_enabled + { + if let Ok(_guard) = self.settings_write_gate.try_lock() { + if self.get_preferences().expression_learning_enabled { + if let Some(profile) = self + .expression_profile + .lock() + .ok() + .and_then(|profile| profile.clone()) + { + match profile.stable_directions() { + Ok(directions) => { + if let Some(instructions) = + crate::expression_observation::prompt_instructions(&directions) + { + context.polish.style_system_prompt.push_str(&instructions); + } + } + Err(error) => log::warn!("expression profile unavailable: {error}"), + } + } + } + } + } // Issue #1032: when the LLM channel is not configured, mark the // session as eligible for the Raw fallback instead of dispatching a // polish request that cannot succeed. The context keeps its original @@ -9508,6 +9662,7 @@ mod tests { fn vocabulary_learning_settings_migrate_normalize_and_roundtrip() { let old: UserPreferences = serde_json::from_str("{}").unwrap(); assert!(!old.vocabulary_learning_enabled); + assert!(!old.expression_learning_enabled); assert_eq!(old.vocabulary_learning_settings, Default::default()); let prefs: UserPreferences = serde_json::from_value(serde_json::json!({ "vocabularyLearningSettings": { @@ -9526,6 +9681,151 @@ mod tests { ); } + #[test] + fn expression_observation_obeys_generation_and_never_persists_text() { + let directory = TestDataDir::new("expression-observation"); + let path = directory.path().join("profile.json"); + let profile = Arc::new(crate::expression_observation::ExpressionProfileStore::open(path.clone()).unwrap()); + let generation = Arc::new(AtomicU64::new(4)); + let gate = Arc::new(Mutex::new(())); + let sink = CombinedEditObservationSink { + vocabulary: None, + expression_generation: Some(profile.generation()), + expression_profile: Some(profile), + typed_text: "今天我们先完成这个工作然后再讨论".into(), + session_id: SessionId::new(), + clock: Arc::new(SystemClock), + generation: Arc::clone(&generation), + expected_generation: 4, + settings_gate: Arc::clone(&gate), + duration: std::time::Duration::from_secs(60), + }; + let edit = crate::host_document::EditPair { + source: String::new(), + target: "\n".into(), + before: "今天我们先完成这个工作".into(), + after: "然后再讨论".into(), + }; + { + let _guard = gate.lock().unwrap(); + assert!(!sink.publish(edit.clone())); + assert!(!path.exists()); + } + assert!(sink.publish(edit.clone())); + let saved = std::fs::read_to_string(&path).unwrap(); + assert!(!saved.contains("今天")); + generation.fetch_add(1, Ordering::AcqRel); + assert!(!sink.publish(edit)); + assert_eq!(std::fs::read_to_string(&path).unwrap(), saved); + } + + #[tokio::test] + async fn expression_prompt_requires_consent_and_clear_removes_stable_directions() { + let (backend, _) = backend(); + assert!(Arc::ptr_eq(&backend.expression_profile, &backend.repositories().expression_profile)); + backend.set_credential( + crate::CredentialKey::new(crate::CredentialNamespace::Llm, Some("ark".into()), crate::credentials::LLM_API_KEY_ACCOUNT).unwrap(), + crate::SecretValue::new("fixture-key"), + ).await.unwrap(); + let profile = backend.expression_profile.lock().unwrap().as_ref().unwrap().clone(); + let signal = crate::expression_observation::ExpressionSignal { + line_breaks: 1, + exclamations: 0, + terminal_punctuation: 0, + }; + for index in 0..10 { + profile.record(uuid::Uuid::new_v4(), 100 + index % 4, signal).unwrap(); + } + assert_eq!(profile.stable_directions().unwrap().len(), 1); + let options = DictationStartOptions::default(); + let capture = || backend.capture_dictation_context(&options, DictationContextPurpose::Dictation); + assert!(!capture().await.unwrap().polish.style_system_prompt.contains("用户经多次编辑")); + let mut prefs = backend.get_preferences(); + prefs.expression_learning_enabled = true; + backend.set_preferences(prefs).unwrap(); + assert!(backend.get_preferences().expression_learning_enabled); + let context = capture().await.unwrap(); + assert!(context.polish.style_system_prompt.contains("用户经多次编辑"), "translation={} prompt={:?}", context.polish.translation_active, context.polish.style_system_prompt); + backend.clear_expression_profile().unwrap(); + assert!(backend.expression_profile_directions().unwrap().is_empty()); + assert!(!capture().await.unwrap().polish.style_system_prompt.contains("用户经多次编辑")); + } + + #[tokio::test] + async fn expression_capture_does_not_wait_for_busy_settings_gate() { + let (backend, _) = backend(); + let mut prefs = backend.get_preferences(); + prefs.expression_learning_enabled = true; + backend.set_preferences(prefs).unwrap(); + let gate = Arc::clone(&backend.settings_write_gate); + let (locked_tx, locked_rx) = std::sync::mpsc::sync_channel(1); + let (release_tx, release_rx) = std::sync::mpsc::channel(); + let holder = std::thread::spawn(move || { + let _guard = gate.lock().unwrap(); + locked_tx.send(()).unwrap(); + let _ = release_rx.recv(); + }); + locked_rx.recv_timeout(std::time::Duration::from_secs(2)).unwrap(); + let options = DictationStartOptions::default(); + let context = tokio::time::timeout( + std::time::Duration::from_secs(2), + backend.capture_dictation_context(&options, DictationContextPurpose::Dictation), + ).await.expect("busy settings must not block capture").unwrap(); + release_tx.send(()).unwrap(); + holder.join().unwrap(); + assert!(!context.polish.style_system_prompt.contains("用户经多次编辑")); + } + + #[test] + fn expression_profile_corruption_requires_explicit_clear() { + let directory = TestDataDir::new("expression-corruption"); + let path = directory.path().join("expression-profile.json"); + std::fs::create_dir_all(directory.path()).unwrap(); + std::fs::write(&path, b"unknown profile").unwrap(); + let backend = OpenLessBackend::new( + BackendConfig { data_dir: directory.path().to_path_buf(), ..BackendConfig::default() }, + BackendDependencies::unsupported(), + ).unwrap(); + assert!(backend.expression_profile_directions().is_err()); + assert_eq!(std::fs::read(&path).unwrap(), b"unknown profile"); + backend.clear_expression_profile().unwrap(); + assert!(backend.expression_profile_directions().unwrap().is_empty()); + } + + #[test] + fn expression_opt_out_preserves_pending_vocabulary_suggestions() { + let (backend, _) = backend(); + let mut prefs = backend.get_preferences(); + prefs.expression_learning_enabled = true; + prefs.vocabulary_learning_enabled = true; + backend.set_preferences(prefs.clone()).unwrap(); + backend.queue_pending_correction("原词".into(), "新词".into()).unwrap(); + prefs.expression_learning_enabled = false; + backend.update_settings(prefs, crate::SettingsUpdateOptions::STRICT, &crate::NoopSettingsRuntime).unwrap(); + assert_eq!(backend.pending_corrections().len(), 1); + } + + #[test] + fn vocabulary_observer_does_not_offer_fact_changes_as_words() { + let (backend, _) = backend(); + let sink = CoreEditObservationSink { + settings: crate::shared_types::VocabularyLearningSettings::default(), + expected_generation: backend.edit_observation_generation.load(Ordering::Acquire), + generation: Arc::clone(&backend.edit_observation_generation), + typed_text: "今天买了十个苹果".into(), + pending: Arc::clone(&backend.pending_corrections), + events: Arc::clone(&backend.events), + }; + let edit = crate::host_document::EditPair { + before: "今天买了".into(), + source: "十".into(), + target: "九".into(), + after: "个苹果".into(), + }; + assert!(!sink.publish(edit)); + assert!(backend.pending_corrections().is_empty()); + } + #[test] fn vocabulary_learning_config_change_invalidates_pending_and_stale_sink() { let (backend, _) = backend(); @@ -13615,6 +13915,81 @@ mod tests { assert!(backend.pending_corrections().is_empty()); } + #[tokio::test] + async fn expression_learning_can_observe_without_vocabulary_and_stops_on_opt_out() { + let data_dir = TestDataDir::new("expression-consent"); + let observation = Arc::new(FakeEditObservation::default()); + let mut services = crate::domains::BackendServices::unsupported(); + services.edit_observation = observation.clone(); + let text = "今天我们先完成这个工作然后再讨论"; + let backend = OpenLessBackend::new( + BackendConfig { data_dir: data_dir.path().to_path_buf(), ..BackendConfig::default() }, + BackendDependencies { + host_actions: Arc::new(FakeHost::default()), + text_inserter: Arc::new(FakeInserter), + dictation_engine: Arc::new(crate::testing::FixtureDictationEngine::successful(text, text)), + credential_store: Arc::new(crate::credentials::InMemoryCredentialStore::default()), + services, + ..BackendDependencies::unsupported() + }, + ).unwrap(); + let mut prefs = backend.get_preferences(); + assert!(!prefs.expression_learning_enabled); + prefs.expression_learning_enabled = true; + prefs.vocabulary_learning_enabled = false; + backend.set_preferences(prefs).unwrap(); + backend.start().await.unwrap(); + backend.start_dictation().await.unwrap(); + backend.stop_dictation().await.unwrap(); + assert_eq!(observation.typed_texts.lock().unwrap().as_slice(), [text]); + let edit = crate::host_document::EditPair { + source: String::new(), target: "\n".into(), + before: "今天我们先完成这个工作".into(), after: "然后再讨论".into(), + }; + observation.publish(0, edit.clone()); + let path = data_dir.path().join("expression-profile.json"); + let saved = std::fs::read_to_string(&path).unwrap(); + assert!(!saved.contains(text)); + assert!(backend.pending_corrections().is_empty()); + let mut prefs = backend.get_preferences(); + prefs.expression_learning_enabled = false; + backend.update_settings(prefs, crate::SettingsUpdateOptions::STRICT, &crate::NoopSettingsRuntime).unwrap(); + observation.publish(0, edit); + assert_eq!(std::fs::read_to_string(&path).unwrap(), saved); + backend.shutdown().await.unwrap(); + } + + #[tokio::test] + async fn expression_opt_out_during_dictation_does_not_arm_an_observer() { + let data_dir = TestDataDir::new("expression-opt-out-during-dictation"); + let observation = Arc::new(FakeEditObservation::default()); + let mut services = crate::domains::BackendServices::unsupported(); + services.edit_observation = observation.clone(); + let backend = OpenLessBackend::new( + BackendConfig { data_dir: data_dir.path().to_path_buf(), ..BackendConfig::default() }, + BackendDependencies { + host_actions: Arc::new(FakeHost::default()), + text_inserter: Arc::new(FakeInserter), + dictation_engine: Arc::new(FakeEngine), + credential_store: Arc::new(crate::credentials::InMemoryCredentialStore::default()), + services, + ..BackendDependencies::unsupported() + }, + ).unwrap(); + let mut prefs = backend.get_preferences(); + prefs.expression_learning_enabled = true; + backend.set_preferences(prefs).unwrap(); + backend.start().await.unwrap(); + backend.start_dictation().await.unwrap(); + let mut prefs = backend.get_preferences(); + prefs.expression_learning_enabled = false; + backend.update_settings(prefs, crate::SettingsUpdateOptions::STRICT, &crate::NoopSettingsRuntime).unwrap(); + backend.stop_dictation().await.unwrap(); + assert!(observation.typed_texts.lock().unwrap().is_empty()); + assert!(!data_dir.path().join("expression-profile.json").exists()); + backend.shutdown().await.unwrap(); + } + #[tokio::test] async fn disabled_vocabulary_learning_is_not_rearmed_by_an_older_dictation() { for streaming in [false, true] { diff --git a/openless-all/app/crates/openless-core/src/cloud_providers.rs b/openless-all/app/crates/openless-core/src/cloud_providers.rs index 40b8eafef..05c6421c7 100644 --- a/openless-all/app/crates/openless-core/src/cloud_providers.rs +++ b/openless-all/app/crates/openless-core/src/cloud_providers.rs @@ -995,6 +995,7 @@ fn map_tencent_asr_error(error: TencentCloudASRError) -> BackendError { pub struct SharedCloudTextPolisher { credentials: Arc, active: Arc>>>, + usage_recorder: Option>, } impl SharedCloudTextPolisher { @@ -1002,8 +1003,14 @@ impl SharedCloudTextPolisher { Self { credentials, active: Arc::new(Mutex::new(HashMap::new())), + usage_recorder: None, } } + + pub fn with_usage_recorder(mut self, recorder: Arc) -> Self { + self.usage_recorder = Some(recorder); + self + } } #[derive(Default)] @@ -1056,6 +1063,24 @@ enum CloudPolisherProvider { Gemini(crate::llm_gemini::GeminiProvider), } +fn usage_purpose(context: &DictationContext) -> crate::usage_ledger::ServicePurpose { + use crate::dictation_context::DictationOutputTarget; + use crate::usage_ledger::ServicePurpose; + if context.polish.selection_input { + ServicePurpose::Selection + } else if context.polish.translation_active { + ServicePurpose::Translation + } else { + match context.output_target { + DictationOutputTarget::Qa => ServicePurpose::Question, + DictationOutputTarget::QuickNote => ServicePurpose::QuickNote, + DictationOutputTarget::VoiceEdit => ServicePurpose::VoiceEdit, + DictationOutputTarget::CloudNote => ServicePurpose::CloudNote, + _ => ServicePurpose::DictationPostprocess, + } + } +} + impl TextPolisher for SharedCloudTextPolisher { fn polish( &self, @@ -1087,6 +1112,8 @@ impl TextPolisher for SharedCloudTextPolisher { active: Arc::clone(&self.active), }; let credentials = Arc::clone(&self.credentials); + let usage_recorder = self.usage_recorder.clone(); + let usage_generation = usage_recorder.as_ref().map(|recorder| recorder.generation()); Box::pin(async move { let _registration = registration; if raw_text.trim().is_empty() { @@ -1097,11 +1124,41 @@ impl TextPolisher for SharedCloudTextPolisher { } let mut provider = build_cloud_polisher_provider(credentials.as_ref(), context.as_ref()).await?; + let usage_label = match &provider { + CloudPolisherProvider::OpenAi(provider) => Some(provider.call_label()), + CloudPolisherProvider::Gemini(_) => None, + }; if let CloudPolisherProvider::OpenAi(crate::polish::ActiveLLMProvider::OpenAI( provider, )) = &mut provider { provider.set_dictation_session(session_id); + if let (Some(recorder), Some(generation), Some(label)) = + (usage_recorder, usage_generation, usage_label) + { + let purpose = usage_purpose(&context); + let channel = crate::usage_ledger::sanitized_identifier(&context.llm.provider_id); + provider.set_usage_observer(Arc::new(move |id, usage| { + let record = crate::usage_ledger::UsageRecord { + id, + occurred_at_ms: chrono::Utc::now().timestamp_millis(), + kind: crate::usage_ledger::ServiceKind::Llm, + purpose, + outcome: crate::usage_ledger::ServiceOutcome::Unknown, + provider: crate::usage_ledger::sanitized_identifier(&label.provider), + channel: channel.clone(), + model: crate::usage_ledger::sanitized_identifier(&label.model), + audio_ms: None, + input_tokens: usage.input, + output_tokens: usage.output, + cached_input_tokens: usage.cached_input, + cache_write_input_tokens: usage.cache_write_input, + reasoning_tokens: usage.reasoning, + synthesized_characters: None, + }; + let _ = recorder.submit(generation, record); + })); + } } tokio::select! { _ = cancellation.cancelled() => Err(cancelled_provider_error()), @@ -3660,6 +3717,74 @@ mod tests { assert_eq!(error.message, "LLM API key is not configured"); } + #[tokio::test] + async fn cloud_polisher_records_only_provider_reported_tokens() { + use std::io::{Read, Write}; + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let address = listener.local_addr().unwrap(); + let server = std::thread::spawn(move || { + let (mut socket, _) = listener.accept().unwrap(); + let mut bytes = Vec::new(); + let mut chunk = [0u8; 4096]; + loop { + let count = socket.read(&mut chunk).unwrap(); + assert!(count > 0); + bytes.extend_from_slice(&chunk[..count]); + if let Some(header_end) = bytes.windows(4).position(|window| window == b"\r\n\r\n") { + let header_end = header_end + 4; + let headers = String::from_utf8_lossy(&bytes[..header_end]); + let length = headers.lines().find_map(|line| line.to_ascii_lowercase().strip_prefix("content-length:").and_then(|value| value.trim().parse::().ok())).unwrap_or(0); + if bytes.len() >= header_end + length { break; } + } + } + let events = b"data: {\"choices\":[{\"delta\":{\"content\":\"hello\"}}]}\n\ndata: {\"choices\":[],\"usage\":{\"prompt_tokens\":30,\"completion_tokens\":4}}\n\ndata: [DONE]\n\n"; + write!(socket, "HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", events.len()).unwrap(); + socket.write_all(events).unwrap(); + }); + let directory = std::env::temp_dir().join(format!("openless-polish-usage-{}", uuid::Uuid::new_v4())); + let ledger = Arc::new(crate::usage_ledger::UsageLedger::open(directory.join("usage.json")).unwrap()); + let recorder = Arc::new(crate::usage_ledger::UsageRecorder::start(ledger).unwrap()); + let credentials = Arc::new(InMemoryCredentialStore::default()); + write_channel_secret(&credentials, CredentialNamespace::Llm, "llm-channel", LLM_API_KEY_ACCOUNT, "fixture-key").await; + write_channel_secret(&credentials, CredentialNamespace::Llm, "llm-channel", LLM_ENDPOINT_ACCOUNT, &format!("http://{address}")).await; + let mut llm = ProviderInvocation::new("llm-channel", "deepseek"); + llm.model = Some("fixture-model".into()); + let context = Arc::new(DictationContext { llm, ..DictationContext::default() }); + let polisher = SharedCloudTextPolisher::new(credentials).with_usage_recorder(recorder.clone()); + let result = polisher.polish(SessionId::new(), context, "private text".into(), Arc::new(IgnoreTextStreamSink)).await.unwrap(); + assert_eq!(result.text, "hello"); + let (records, failures) = recorder.snapshot().unwrap(); + assert_eq!(failures, 0); + assert_eq!(records.len(), 1); + assert_eq!(records[0].input_tokens, Some(30)); + assert_eq!(records[0].output_tokens, Some(4)); + assert_eq!(records[0].channel.as_deref(), Some("llm-channel")); + assert_eq!(records[0].purpose, crate::usage_ledger::ServicePurpose::DictationPostprocess); + let stored = std::fs::read_to_string(directory.join("usage.json")).unwrap(); + assert!(!stored.contains("private text")); + assert!(!stored.contains("fixture-key")); + server.join().unwrap(); + drop(polisher); + drop(recorder); + std::fs::remove_dir_all(directory).unwrap(); + } + + #[test] + fn usage_purpose_distinguishes_selection_translation_and_tools() { + use crate::dictation_context::DictationOutputTarget; + use crate::usage_ledger::ServicePurpose; + let mut context = DictationContext::default(); + assert_eq!(usage_purpose(&context), ServicePurpose::DictationPostprocess); + context.polish.translation_active = true; + assert_eq!(usage_purpose(&context), ServicePurpose::Translation); + context.polish.selection_input = true; + assert_eq!(usage_purpose(&context), ServicePurpose::Selection); + context.polish.selection_input = false; + context.polish.translation_active = false; + context.output_target = DictationOutputTarget::QuickNote; + assert_eq!(usage_purpose(&context), ServicePurpose::QuickNote); + } + #[tokio::test] async fn duplicate_polish_session_keeps_the_original_cancellation_route() { let credentials: Arc = Arc::new(InMemoryCredentialStore::default()); diff --git a/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/registry.rs b/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/registry.rs index 4502a8d25..6ea485c1c 100644 --- a/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/registry.rs +++ b/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/registry.rs @@ -809,6 +809,13 @@ pub const PREFERENCE_FIELDS: &[PreferenceField] = &[ shape: PreferenceShape::Boolean, reason: "local_consent_or_secret", }, + PreferenceField { + rust_name: "expression_learning_enabled", + key: "expressionLearningEnabled", + class: PreferenceClass::Excluded, + shape: PreferenceShape::Boolean, + reason: "local_consent_or_secret", + }, PreferenceField { rust_name: "vocabulary_learning_settings", key: "vocabularyLearningSettings", diff --git a/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/tests.rs b/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/tests.rs index 369ccf8ea..6916ca4f1 100644 --- a/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/tests.rs +++ b/openless-all/app/crates/openless-core/src/cloud_sync_e2ee_documents/tests.rs @@ -156,11 +156,26 @@ fn testing_features_opt_in_is_never_exported_or_accepted_from_another_device() { ); } +#[test] +fn expression_learning_consent_is_never_accepted_from_another_device() { + let mut invalid = sample().documents().clone(); + invalid.documents.push(LogicalDocument { + id: "expressionLearningEnabled".into(), + kind: DocumentKind::Preferences, + schema_version: 1, + value: json!(true), + }); + assert_eq!( + validate_sync_documents(invalid, Revision::new(3)).unwrap_err(), + DocumentError::ExcludedField + ); +} + #[test] fn controlled_export_keeps_real_service_keys_but_excludes_local_secrets_and_grants() { let mut data = channel_snapshot(); data.preferences = SecretJson::new( - json!({"themeMode":"dark","remoteInputPin":"fixture-pin-never-export","codingAgentEnabled":true,"cursorContextEnabled":true,"vocabularyLearningEnabled":true,"marketplaceDevLogin":"fixture-login","splashSeenVersion":"2","activeLlmProvider":"llm-one","codingAgentExe":"/fixture/never-run","futureAppearance":{"accent":"violet"}}), + json!({"themeMode":"dark","remoteInputPin":"fixture-pin-never-export","codingAgentEnabled":true,"cursorContextEnabled":true,"vocabularyLearningEnabled":true,"expressionLearningEnabled":true,"marketplaceDevLogin":"fixture-login","splashSeenVersion":"2","activeLlmProvider":"llm-one","codingAgentExe":"/fixture/never-run","futureAppearance":{"accent":"violet"}}), ); let exported = export_snapshot(data).unwrap(); let raw = serde_json::to_string(exported.documents.documents()).unwrap(); @@ -172,6 +187,7 @@ fn controlled_export_keeps_real_service_keys_but_excludes_local_secrets_and_gran assert!(!raw.contains("codingAgentEnabled")); assert!(raw.contains("cursorContextEnabled")); assert!(!raw.contains("vocabularyLearningEnabled")); + assert!(!raw.contains("expressionLearningEnabled")); assert!(!raw.contains("fixture-login")); assert!(!raw.contains("activeLlmProvider")); assert!(raw.contains("futureAppearance")); diff --git a/openless-all/app/crates/openless-core/src/correction_observation.rs b/openless-all/app/crates/openless-core/src/correction_observation.rs new file mode 100644 index 000000000..d68ba3ba5 --- /dev/null +++ b/openless-all/app/crates/openless-core/src/correction_observation.rs @@ -0,0 +1,122 @@ +//! Conservative safeguards for locally observed vocabulary suggestions. + +use crate::host_document::EditPair; +use regex::Regex; +use std::sync::OnceLock; +use unicode_normalization::UnicodeNormalization; + +fn normalized(value: &str) -> String { + value.nfkc().flat_map(char::to_lowercase).collect() +} + +fn number_signature(value: &str) -> String { + value + .chars() + .filter(|character| { + character.is_numeric() + || "零〇一二两兩三四五六七八九十百千万萬亿億壹贰貳叁參肆伍陆陸柒捌玖拾佰仟" + .contains(*character) + }) + .collect() +} + +fn negation_pattern() -> &'static Regex { + static PATTERN: OnceLock = OnceLock::new(); + PATTERN.get_or_init(|| { + Regex::new(r"(?-u:\b)(?:no|not|never|neither|nor)(?-u:\b)|n['’]t|[不没沒無无非别別]") + .expect("fixed negation expression is valid") + }) +} + +fn negation_signature(value: &str) -> Vec<&str> { + negation_pattern() + .find_iter(value) + .map(|found| found.as_str()) + .collect() +} + +/// Refuse suggestions when the observed edit could alter a number, negation, +/// or sensitive content. The caller still requires explicit user confirmation. +pub(crate) fn permits_vocabulary_suggestion(inserted: &str, edit: &EditPair) -> bool { + if crate::expression_observation::contains_sensitive_text(inserted) { + return false; + } + let before = normalized(&format!("{}{}{}", edit.before, edit.source, edit.after)); + let after = normalized(&format!("{}{}{}", edit.before, edit.target, edit.after)); + if crate::expression_observation::contains_sensitive_text(&before) + || crate::expression_observation::contains_sensitive_text(&after) + { + return false; + } + let source = normalized(&edit.source); + let target = normalized(&edit.target); + if [&source, &target].iter().any(|changed| { + !number_signature(changed).is_empty() || !negation_signature(changed).is_empty() + }) { + return false; + } + number_signature(&before) == number_signature(&after) + && negation_signature(&before) == negation_signature(&after) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn edit(before: &str, source: &str, target: &str, after: &str) -> EditPair { + EditPair { + before: before.into(), + source: source.into(), + target: target.into(), + after: after.into(), + } + } + + #[test] + fn permits_ordinary_technical_word_corrections() { + assert!(permits_vocabulary_suggestion( + "请导出 ZIP 文件", + &edit("请导出 ", "ZIP", "VIP", " 文件") + )); + assert!(permits_vocabulary_suggestion( + "使用 codx 完成任务", + &edit("使用 ", "codx", "codex", " 完成任务") + )); + } + + #[test] + fn rejects_fact_and_negation_changes() { + assert!(!permits_vocabulary_suggestion( + "今天买了十个苹果", + &edit("今天买了", "十", "九", "个苹果") + )); + assert!(!permits_vocabulary_suggestion( + "订单编号是 10", + &edit("订单编号是 ", "10", "11", "") + )); + assert!(!permits_vocabulary_suggestion( + "这件事情不能做", + &edit("这件事情", "不", "", "能做") + )); + assert!(!permits_vocabulary_suggestion( + "I can't send this", + &edit("I can", "'t", "", " send this") + )); + assert!(!permits_vocabulary_suggestion( + "回复no即可", + &edit("回复", "no", "yes", "即可") + )); + } + + #[test] + fn rejects_sensitive_context_without_persisting_it() { + assert!(!permits_vocabulary_suggestion( + "联系 user@example.com 处理 ZIP", + &edit("处理 ", "ZIP", "VIP", "") + )); + assert!(!permits_vocabulary_suggestion( + "无敏感文本", + &edit("账户 ", "12345678", "87654321", "") + )); + } +} diff --git a/openless-all/app/crates/openless-core/src/dictation_context.rs b/openless-all/app/crates/openless-core/src/dictation_context.rs index 2e870af7f..cfe5a7720 100644 --- a/openless-all/app/crates/openless-core/src/dictation_context.rs +++ b/openless-all/app/crates/openless-core/src/dictation_context.rs @@ -402,7 +402,8 @@ impl DictationContext { | DictationOutputTarget::Qa | DictationOutputTarget::VoiceEdit ), - observe_edits: preferences.vocabulary_learning_enabled, + observe_edits: preferences.vocabulary_learning_enabled + || preferences.expression_learning_enabled, // An undecided Android capture must not stream text into the // foreground app before the terminal gesture classifies it. streaming: !matches!( @@ -413,10 +414,9 @@ impl DictationContext { | DictationOutputTarget::VoiceEdit ) && preferences.streaming_insert, save_streamed_text_to_clipboard: preferences.streaming_insert_save_clipboard, - // Composition display is a TSF text-service feature; other insertion - // modes have no underlined preedit channel. - composition_input: (cfg!(target_os = "macos") && preferences.macos_composition_input) - || cfg!(target_os = "windows") + // macOS AX composition is retired. Ignore legacy enabled values so saved + // settings cannot bypass the regular final/streaming insertion protocol. + composition_input: cfg!(target_os = "windows") && preferences.windows_composition_input && preferences.windows_insertion_mode == WindowsInsertionMode::Tsf, restore_clipboard_after_paste: preferences.restore_clipboard_after_paste, @@ -887,6 +887,33 @@ mod tests { assert!(preferences.stable_transcription_enabled); } + #[test] + fn legacy_macos_live_input_does_not_enable_composition_in_either_pipeline() { + for pipeline_mode in [PipelineMode::Traditional, PipelineMode::Multimodal] { + let preferences = UserPreferences { + macos_composition_input: true, + windows_composition_input: false, + multimodal_pipeline_enabled: true, + pipeline_mode, + ..UserPreferences::default() + }; + let context = DictationContext::capture( + &preferences, + &builtin_style_pack_for_mode(PolishMode::Structured), + DictationProviderInvocations::new( + ProviderInvocation::for_provider("asr"), + ProviderInvocation::for_provider("llm"), + ProviderInvocation::for_provider("omni"), + ), + Vec::new(), + Vec::new(), + &DictationStartOptions::default(), + ); + assert_eq!(context.pipeline_mode, pipeline_mode); + assert!(!context.insertion.composition_input); + } + } + #[test] fn context_is_an_owned_snapshot_of_session_preferences() { let mut preferences = UserPreferences { diff --git a/openless-all/app/crates/openless-core/src/domains.rs b/openless-all/app/crates/openless-core/src/domains.rs index b5783229e..ce70f23ea 100644 --- a/openless-all/app/crates/openless-core/src/domains.rs +++ b/openless-all/app/crates/openless-core/src/domains.rs @@ -2139,6 +2139,7 @@ pub struct BackendServices { pub platform: Arc, pub host_context: Arc, pub edit_observation: Arc, + pub usage_recorder: Option>, coding_agent_process: Option>, auxiliary_polisher: Option>, auxiliary_transcription: Option>, @@ -2168,6 +2169,7 @@ impl BackendServices { platform: Arc::new(UnsupportedDomainServices), host_context: Arc::new(crate::ports::NoopHostContextAdapter), edit_observation: Arc::new(crate::ports::NoopEditObservationAdapter), + usage_recorder: None, coding_agent_process: None, auxiliary_polisher: None, auxiliary_transcription: None, diff --git a/openless-all/app/crates/openless-core/src/expression_observation.rs b/openless-all/app/crates/openless-core/src/expression_observation.rs new file mode 100644 index 000000000..42a75d7e1 --- /dev/null +++ b/openless-all/app/crates/openless-core/src/expression_observation.rs @@ -0,0 +1,661 @@ +//! Local, text-free signals for conservative expression learning. +//! +//! The feature and safety model is informed by Type4Me's MIT-licensed +//! ExpressionProfileStore and UserEditClassifier; see type4me-LICENSE.txt. + +use regex::{Regex, RegexSet}; +use serde::{Deserialize, Serialize}; +use std::io::Read; +use std::path::PathBuf; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, Mutex, OnceLock}; +use unicode_normalization::UnicodeNormalization; + +const MIN_CHARS: usize = 12; +const MAX_CHARS: usize = 16_384; + +/// Only counts and directions leave this module; neither input text is retained. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct ExpressionSignal { + pub line_breaks: i8, + pub exclamations: i8, + pub terminal_punctuation: i8, +} + +const MAX_EVIDENCE: usize = 1024; +const MIN_DIRECTIONAL_SAMPLES: usize = 10; +const MIN_DAY_SPAN: i32 = 3; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum ExpressionFeature { + LineBreaks, + Exclamations, + TerminalPunctuation, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct StableExpressionDirection { + pub feature: ExpressionFeature, + pub direction: i8, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +struct ExpressionEvidence { + session_id: uuid::Uuid, + day: i32, + signal: ExpressionSignal, +} + +/// Bounded aggregate inputs contain no recognized or edited text. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ExpressionProfile { + evidence: Vec, +} + +const MAX_PROFILE_BYTES: u64 = 512 * 1024; + +#[derive(Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +struct ProfileDocument { + version: u32, + evidence: Vec, +} + +/// The store never retains recognized or edited text and rejects unknown files. +pub struct ExpressionProfileStore { + path: PathBuf, + profile: Mutex>, + generation: AtomicU64, +} + +pub type SharedExpressionProfile = Arc>>>; + +pub fn open_shared_expression_profile(path: PathBuf) -> SharedExpressionProfile { + let profile = match ExpressionProfileStore::open(path) { + Ok(store) => Some(Arc::new(store)), + Err(error) => { + log::warn!("expression profile unavailable: {error}"); + None + } + }; + Arc::new(Mutex::new(profile)) +} + +impl ExpressionProfileStore { + /// Explicit user reset can recover an unreadable local profile. + pub fn reset(path: PathBuf) -> Result { + crate::persistence::atomic_write(&path, br#"{"version":1,"evidence":[]}"#)?; + Self::open(path) + } + + pub fn open(path: PathBuf) -> Result { + let mut profile = ExpressionProfile::default(); + match std::fs::File::open(&path) { + Ok(file) => { + let mut data = Vec::new(); + file.take(MAX_PROFILE_BYTES + 1) + .read_to_end(&mut data) + .map_err(|_| { + crate::persistence::persistence_error("read expression profile") + })?; + if data.len() as u64 > MAX_PROFILE_BYTES { + return Err(crate::persistence::persistence_error( + "expression profile too large", + )); + } + let document: ProfileDocument = serde_json::from_slice(&data).map_err(|_| { + crate::persistence::persistence_error("decode expression profile") + })?; + if document.version != 1 || document.evidence.len() > MAX_EVIDENCE { + return Err(crate::persistence::persistence_error( + "invalid expression profile", + )); + } + for evidence in document.evidence { + if !profile.observe(evidence.session_id, evidence.day, evidence.signal) { + return Err(crate::persistence::persistence_error( + "invalid expression evidence", + )); + } + } + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(_) => { + return Err(crate::persistence::persistence_error( + "read expression profile", + )) + } + } + Ok(Self { + path, + profile: Mutex::new(Some(profile)), + generation: AtomicU64::new(0), + }) + } + + pub fn generation(&self) -> u64 { + self.generation.load(Ordering::Acquire) + } + + pub fn invalidate_observers(&self) -> Result<(), crate::errors::BackendError> { + let _profile = self + .profile + .lock() + .map_err(|_| crate::persistence::persistence_error("lock expression profile"))?; + self.generation.fetch_add(1, Ordering::AcqRel); + Ok(()) + } + + pub fn record( + &self, + session_id: uuid::Uuid, + day: i32, + signal: ExpressionSignal, + ) -> Result { + self.record_if_generation(session_id, day, signal, self.generation()) + } + + pub fn record_if_generation( + &self, + session_id: uuid::Uuid, + day: i32, + signal: ExpressionSignal, + expected_generation: u64, + ) -> Result { + let mut profile = self + .profile + .lock() + .map_err(|_| crate::persistence::persistence_error("lock expression profile"))?; + if self.generation.load(Ordering::Acquire) != expected_generation { + return Ok(false); + } + let mut next = profile + .as_ref() + .ok_or_else(|| { + crate::persistence::persistence_error("expression profile needs reload") + })? + .clone(); + if !next.observe(session_id, day, signal) { + return Ok(false); + } + if let Err(error) = self.persist(&next) { + *profile = None; + return Err(error); + } + *profile = Some(next); + Ok(true) + } + + pub fn stable_directions( + &self, + ) -> Result, crate::errors::BackendError> { + let profile = self + .profile + .lock() + .map_err(|_| crate::persistence::persistence_error("lock expression profile"))?; + Ok(profile + .as_ref() + .ok_or_else(|| { + crate::persistence::persistence_error("expression profile needs reload") + })? + .stable_directions()) + } + + pub fn clear(&self) -> Result<(), crate::errors::BackendError> { + let mut profile = self + .profile + .lock() + .map_err(|_| crate::persistence::persistence_error("lock expression profile"))?; + self.generation.fetch_add(1, Ordering::AcqRel); + if profile.is_none() { + return Err(crate::persistence::persistence_error( + "expression profile needs reload", + )); + } + let next = ExpressionProfile::default(); + if let Err(error) = self.persist(&next) { + *profile = None; + return Err(error); + } + *profile = Some(next); + Ok(()) + } + + fn persist(&self, profile: &ExpressionProfile) -> Result<(), crate::errors::BackendError> { + let data = serde_json::to_vec(&ProfileDocument { + version: 1, + evidence: profile.evidence.clone(), + }) + .map_err(|_| crate::persistence::persistence_error("encode expression profile"))?; + crate::persistence::atomic_write(&self.path, &data) + } +} + +impl ExpressionProfile { + pub fn observe(&mut self, session_id: uuid::Uuid, day: i32, signal: ExpressionSignal) -> bool { + if self + .evidence + .iter() + .any(|item| item.session_id == session_id) + || [ + signal.line_breaks, + signal.exclamations, + signal.terminal_punctuation, + ] + .iter() + .any(|direction| !(-1..=1).contains(direction)) + || signal + == (ExpressionSignal { + line_breaks: 0, + exclamations: 0, + terminal_punctuation: 0, + }) + { + return false; + } + self.evidence.push(ExpressionEvidence { + session_id, + day, + signal, + }); + self.evidence.sort_unstable_by_key(|item| item.day); + if self.evidence.len() > MAX_EVIDENCE { + self.evidence.drain(..self.evidence.len() - MAX_EVIDENCE); + } + true + } + + pub fn stable_directions(&self) -> Vec { + type FeatureSelector = (ExpressionFeature, fn(ExpressionSignal) -> i8); + let selectors: [FeatureSelector; 3] = [ + (ExpressionFeature::LineBreaks, |signal| signal.line_breaks), + (ExpressionFeature::Exclamations, |signal| { + signal.exclamations + }), + (ExpressionFeature::TerminalPunctuation, |signal| { + signal.terminal_punctuation + }), + ]; + selectors + .into_iter() + .filter_map(|(feature, select)| { + let directional: Vec<_> = self + .evidence + .iter() + .filter_map(|item| { + let value = select(item.signal); + (value != 0).then_some((item.day, value)) + }) + .collect(); + let first = directional.first()?.0; + let last = directional.last()?.0; + if directional.len() < MIN_DIRECTIONAL_SAMPLES + || last.saturating_sub(first) < MIN_DAY_SPAN + { + return None; + } + let positive = directional + .iter() + .filter(|(_, direction)| *direction > 0) + .count(); + let negative = directional.len() - positive; + let dominant = positive.max(negative); + (dominant * 10 >= directional.len() * 7).then_some(StableExpressionDirection { + feature, + direction: if positive > negative { 1 } else { -1 }, + }) + }) + .collect() + } +} + +pub fn prompt_instructions(directions: &[StableExpressionDirection]) -> Option { + let statements: Vec<&str> = directions + .iter() + .filter_map(|item| match (item.feature, item.direction) { + (ExpressionFeature::LineBreaks, 1) => Some("用户通常偏好更清晰的分行。"), + (ExpressionFeature::LineBreaks, -1) => Some("用户通常偏好较少的分行。"), + (ExpressionFeature::Exclamations, 1) => Some("用户通常偏好使用感叹号。"), + (ExpressionFeature::Exclamations, -1) => Some("用户通常偏好少用感叹号。"), + (ExpressionFeature::TerminalPunctuation, 1) => Some("用户通常偏好句末标点。"), + (ExpressionFeature::TerminalPunctuation, -1) => { + Some("在不影响语义时，用户通常偏好省略陈述句末标点；保留问号和必要标点。") + } + _ => None, + }) + .collect(); + if statements.is_empty() { + return None; + } + Some(format!("\n\n用户经多次编辑形成的形式偏好（仅供参考；不得改变事实、否定、原意、语气、明确指令或必要结构）：\n{}", statements.join("\n"))) +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ObservationRejection { + Unchanged, + InsufficientOrOversized, + Sensitive, + ContentOrAmbiguous, +} + +fn sensitive_patterns() -> &'static RegexSet { + static PATTERNS: OnceLock = OnceLock::new(); + PATTERNS.get_or_init(|| { + RegexSet::new([ + r"(?i)\b[\w.%+-]+@[\w.-]+\.[a-z]{2,}\b", + r"(?i)(?:https?://|www\.)\S+", + r"(?:\d[ -]?){7,}", + r"(?:^|\s)(?:/[^\s]+|[a-zA-Z]:\\[^\s]+)", + r"\b[A-Za-z0-9_-]{20,}\b", + ]) + .expect("fixed sensitivity expressions are valid") + }) +} + +fn ascii_words() -> &'static Regex { + static WORDS: OnceLock = OnceLock::new(); + WORDS.get_or_init(|| Regex::new(r"[a-z0-9]+").expect("fixed word expression is valid")) +} + +fn normalized(text: &str) -> String { + text.nfkc().flat_map(char::to_lowercase).collect() +} + +pub(crate) fn contains_sensitive_text(text: &str) -> bool { + text.chars().take(MAX_CHARS + 1).count() > MAX_CHARS + || sensitive_patterns().is_match(&normalized(text)) +} + +fn skeleton(text: &str) -> String { + text.chars() + .filter(|character| !character.is_whitespace() && !",.;:!，。；：！、".contains(*character)) + .collect() +} + +fn direction(before: usize, after: usize) -> i8 { + match after.cmp(&before) { + std::cmp::Ordering::Less => -1, + std::cmp::Ordering::Equal => 0, + std::cmp::Ordering::Greater => 1, + } +} + +fn count(text: &str, chars: &[char]) -> usize { + text.chars() + .filter(|character| chars.contains(character)) + .count() +} + +fn ends_in_terminal_punctuation(text: &str) -> usize { + usize::from( + text.trim_end() + .chars() + .last() + .is_some_and(|c| ".!?。！？".contains(c)), + ) +} + +/// Accept only a narrow set of formatting edits while preserving symbols, +/// question marks, content and Latin word boundaries. +pub fn expression_signal( + original: &str, + edited: &str, +) -> Result { + if original == edited { + return Err(ObservationRejection::Unchanged); + } + let original_length = original.chars().count(); + let edited_length = edited.chars().count(); + if !(MIN_CHARS..=MAX_CHARS).contains(&original_length) + || !(MIN_CHARS..=MAX_CHARS).contains(&edited_length) + { + return Err(ObservationRejection::InsufficientOrOversized); + } + let original_normalized = normalized(original); + let edited_normalized = normalized(edited); + if sensitive_patterns().is_match(&original_normalized) + || sensitive_patterns().is_match(&edited_normalized) + { + return Err(ObservationRejection::Sensitive); + } + if skeleton(&original_normalized) != skeleton(&edited_normalized) + || !ascii_words() + .find_iter(&original_normalized) + .map(|word| word.as_str()) + .eq(ascii_words() + .find_iter(&edited_normalized) + .map(|word| word.as_str())) + { + return Err(ObservationRejection::ContentOrAmbiguous); + } + Ok(ExpressionSignal { + line_breaks: direction(count(original, &['\n']), count(edited, &['\n'])), + exclamations: direction(count(original, &['!', '！']), count(edited, &['!', '！'])), + terminal_punctuation: direction( + ends_in_terminal_punctuation(original), + ends_in_terminal_punctuation(edited), + ), + }) +} + +/// Rebuild one edited insertion only when the source and surrounding context +/// identify a unique location within the text that OpenLess actually inserted. +pub fn signal_from_observed_edit( + inserted: &str, + edit: &crate::host_document::EditPair, +) -> Result { + if inserted.chars().count() > MAX_CHARS { + return Err(ObservationRejection::InsufficientOrOversized); + } + let before_anchor: String = edit + .before + .chars() + .rev() + .take(16) + .collect::() + .chars() + .rev() + .collect(); + let after_anchor: String = edit.after.chars().take(16).collect(); + let mut candidate = None; + for (offset, _) in inserted + .char_indices() + .chain(std::iter::once((inserted.len(), '\0'))) + { + let Some(remaining) = inserted[offset..].strip_prefix(&edit.source) else { + continue; + }; + if !inserted[..offset].ends_with(&before_anchor) || !remaining.starts_with(&after_anchor) { + continue; + } + if candidate.is_some() { + return Err(ObservationRejection::ContentOrAmbiguous); + } + candidate = Some((offset, offset + edit.source.len())); + } + let (start, end) = candidate.ok_or(ObservationRejection::ContentOrAmbiguous)?; + let edited = format!("{}{}{}", &inserted[..start], edit.target, &inserted[end..]); + expression_signal(inserted, &edited) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn accepts_formatting_only_and_emits_no_text() { + let signal = expression_signal( + "今天我们先完成这个工作然后再讨论", + "今天我们先完成这个工作，\n然后再讨论！", + ) + .unwrap(); + assert_eq!(signal.line_breaks, 1); + assert_eq!(signal.exclamations, 1); + assert_eq!(signal.terminal_punctuation, 1); + let serialized = serde_json::to_string(&signal).unwrap(); + assert!(!serialized.contains("今天")); + } + + #[test] + fn rejects_changed_facts_negation_and_word_boundaries() { + for (before, after) in [ + ("今天我们需要购买十二个苹果", "今天我们需要购买十三个苹果"), + ("这件事情现在我们不能这样做", "这件事情现在我们能这样做"), + ( + "This is a notable result today", + "This is a not able result today", + ), + ("请把这个结果标记为通过 ✅", "请把这个结果标记为通过 ❌"), + ("这段程序应该使用 C++ 版本", "这段程序应该使用 C# 版本"), + ] { + assert_eq!( + expression_signal(before, after), + Err(ObservationRejection::ContentOrAmbiguous) + ); + } + } + + #[test] + fn only_unambiguous_edits_inside_the_inserted_text_are_observed() { + let inserted = "今天我们先完成这个工作然后再讨论"; + let edit = crate::host_document::EditPair { + source: String::new(), + target: "\n".into(), + before: "今天我们先完成这个工作".into(), + after: "然后再讨论".into(), + }; + assert_eq!( + signal_from_observed_edit(inserted, &edit) + .unwrap() + .line_breaks, + 1 + ); + let ambiguous = crate::host_document::EditPair { + source: "，".into(), + target: "。".into(), + before: String::new(), + after: String::new(), + }; + assert_eq!( + signal_from_observed_edit("这是一个足够长的句子，里面还有另一个，分隔符", &ambiguous), + Err(ObservationRejection::ContentOrAmbiguous) + ); + assert_eq!( + signal_from_observed_edit(inserted, &ambiguous), + Err(ObservationRejection::ContentOrAmbiguous) + ); + } + + #[test] + fn rejects_sensitive_and_unbounded_observations() { + assert_eq!( + expression_signal( + "Email me at user@example.com please", + "Email me at user@example.com, please!" + ), + Err(ObservationRejection::Sensitive) + ); + assert_eq!( + expression_signal("短句", "短句！"), + Err(ObservationRejection::InsufficientOrOversized) + ); + let long = "测".repeat(MAX_CHARS + 1); + assert_eq!( + expression_signal(&long, &(long.clone() + "！")), + Err(ObservationRejection::InsufficientOrOversized) + ); + } + + #[test] + fn stable_directions_require_independent_consistent_samples_across_days() { + let mut profile = ExpressionProfile::default(); + let id = uuid::Uuid::new_v4(); + let positive = ExpressionSignal { + line_breaks: 1, + exclamations: 0, + terminal_punctuation: 0, + }; + assert!(profile.observe(id, 100, positive)); + assert!(!profile.observe(id, 103, positive)); + for index in 0..9 { + assert!(profile.observe(uuid::Uuid::new_v4(), 100 + (index % 3), positive)); + } + assert!(profile.stable_directions().is_empty()); + assert!(profile.observe(uuid::Uuid::new_v4(), 103, positive)); + assert_eq!( + profile.stable_directions(), + vec![StableExpressionDirection { + feature: ExpressionFeature::LineBreaks, + direction: 1, + }] + ); + let negative = ExpressionSignal { + line_breaks: -1, + ..positive + }; + for _ in 0..8 { + assert!(profile.observe(uuid::Uuid::new_v4(), 104, negative)); + } + assert!(profile.stable_directions().is_empty()); + } + + #[test] + fn profile_store_roundtrips_abstract_evidence_and_clears() { + let directory = + std::env::temp_dir().join(format!("openless-expression-{}", uuid::Uuid::new_v4())); + let path = directory.join("profile.json"); + let store = ExpressionProfileStore::open(path.clone()).unwrap(); + let signal = ExpressionSignal { + line_breaks: 1, + exclamations: 0, + terminal_punctuation: 0, + }; + for index in 0..10 { + assert!(store + .record(uuid::Uuid::new_v4(), 100 + (index % 4), signal) + .unwrap()); + } + let saved = std::fs::read_to_string(&path).unwrap(); + assert!(!saved.contains("今天")); + drop(store); + let reloaded = ExpressionProfileStore::open(path.clone()).unwrap(); + assert_eq!(reloaded.stable_directions().unwrap().len(), 1); + let previous_generation = reloaded.generation(); + reloaded.clear().unwrap(); + assert!(!reloaded + .record_if_generation(uuid::Uuid::new_v4(), 104, signal, previous_generation) + .unwrap()); + let opted_in_generation = reloaded.generation(); + reloaded.invalidate_observers().unwrap(); + assert!(!reloaded + .record_if_generation(uuid::Uuid::new_v4(), 105, signal, opted_in_generation) + .unwrap()); + drop(reloaded); + assert!(ExpressionProfileStore::open(path) + .unwrap() + .stable_directions() + .unwrap() + .is_empty()); + std::fs::remove_dir_all(directory).unwrap(); + } + + #[test] + fn corrupt_and_unknown_profile_is_not_overwritten() { + let directory = + std::env::temp_dir().join(format!("openless-expression-{}", uuid::Uuid::new_v4())); + std::fs::create_dir_all(&directory).unwrap(); + let path = directory.join("profile.json"); + let content = br#"{"version":1,"evidence":[],"newField":"keep"}"#; + std::fs::write(&path, content).unwrap(); + assert!(ExpressionProfileStore::open(path.clone()).is_err()); + assert_eq!(std::fs::read(&path).unwrap(), content); + let reset = ExpressionProfileStore::reset(path).unwrap(); + assert!(reset.stable_directions().unwrap().is_empty()); + std::fs::remove_dir_all(directory).unwrap(); + } +} diff --git a/openless-all/app/crates/openless-core/src/lib.rs b/openless-all/app/crates/openless-core/src/lib.rs index 1ee73c76a..44b455fe8 100644 --- a/openless-all/app/crates/openless-core/src/lib.rs +++ b/openless-all/app/crates/openless-core/src/lib.rs @@ -24,6 +24,7 @@ mod cloud_sync_transaction; mod cloud_sync_types; mod cloud_sync_validation; pub mod coding_agent; +mod correction_observation; pub mod coding_agent_guard; pub mod config; pub mod correction; @@ -36,6 +37,7 @@ pub mod edit_plan; pub mod endpoint_security; pub mod errors; pub mod events; +pub mod expression_observation; pub mod experimental_asr; pub mod external_audio; pub mod history; @@ -78,6 +80,7 @@ pub mod style_pack_store; pub mod style_packs; pub mod testing; pub mod types; +pub mod usage_ledger; pub mod vocabulary; pub mod voice_activation; pub mod voice_edit_session; diff --git a/openless-all/app/crates/openless-core/src/llm_protocol.rs b/openless-all/app/crates/openless-core/src/llm_protocol.rs index 41d6ad20a..987f05781 100644 --- a/openless-all/app/crates/openless-core/src/llm_protocol.rs +++ b/openless-all/app/crates/openless-core/src/llm_protocol.rs @@ -410,6 +410,7 @@ pub(crate) struct TextEventStream { pub done: bool, siwc: bool, request_id: Option, + reported_tokens: crate::usage_ledger::ReportedTokens, } impl TextEventStream { @@ -422,6 +423,7 @@ impl TextEventStream { done: false, siwc: false, request_id: None, + reported_tokens: crate::usage_ledger::ReportedTokens::default(), } } @@ -435,6 +437,10 @@ impl TextEventStream { crate::polish::append_utf8_sse_chunk(&mut self.buffer, &mut self.pending, chunk) } + pub fn reported_tokens(&self) -> crate::usage_ledger::ReportedTokens { + self.reported_tokens + } + pub fn next(&mut self) -> Result, LLMError> { if self.done { return Ok(None); @@ -463,6 +469,9 @@ impl TextEventStream { } Err(_) => return Err(response_error("invalid LLM SSE JSON")), }; + if let Some(reported) = crate::usage_ledger::ReportedTokens::from_response(&value) { + self.reported_tokens.merge_report(reported); + } let kind = value["type"] .as_str() .or_else(|| { @@ -502,8 +511,10 @@ impl TextEventStream { None } "response.failed" | "response.incomplete" => { - if self.siwc { return Err(LLMError::Siwc(crate::siwc::error("response_incomplete"))); } - return Err(response_error("llmResponseIncomplete")) + if self.siwc { + return Err(LLMError::Siwc(crate::siwc::error("response_incomplete"))); + } + return Err(response_error("llmResponseIncomplete")); } _ => None, }, @@ -555,6 +566,30 @@ mod tests { use super::*; use crate::credentials::{InMemoryCredentialStore, SecretValue}; + #[test] + fn streaming_usage_is_collected_without_discarding_text_or_guessing_counts() { + let mut stream = TextEventStream::new(LlmRequestFormat::ChatCompletions); + stream.push(b"data: {\"choices\":[{\"delta\":{\"content\":\"hello\"}}]}\n\ndata: {\"choices\":[],\"usage\":{\"prompt_tokens\":42,\"completion_tokens\":5}}\n\ndata: [DONE]\n\n").unwrap(); + let mut text = String::new(); + while let Some(event) = stream.next().unwrap() { + if let StreamEvent::Text(delta) = event { + text.push_str(&delta); + } + } + assert_eq!(text, "hello"); + assert_eq!(stream.reported_tokens.input, Some(42)); + assert_eq!(stream.reported_tokens.output, Some(5)); + assert_eq!(stream.reported_tokens.cached_input, None); + + let mut no_usage = TextEventStream::new(LlmRequestFormat::ChatCompletions); + no_usage.push(b"data: [DONE]\n\n").unwrap(); + assert!(matches!(no_usage.next().unwrap(), Some(StreamEvent::Done))); + assert_eq!( + no_usage.reported_tokens, + crate::usage_ledger::ReportedTokens::default() + ); + } + #[tokio::test] async fn protocol_defaults_overrides_and_channels_are_independent() { let store = InMemoryCredentialStore::default(); diff --git a/openless-all/app/crates/openless-core/src/polish.rs b/openless-all/app/crates/openless-core/src/polish.rs index 4ff06a1f5..a717716df 100644 --- a/openless-all/app/crates/openless-core/src/polish.rs +++ b/openless-all/app/crates/openless-core/src/polish.rs @@ -518,6 +518,9 @@ impl ActiveLLMProvider { } } +pub type LlmUsageObserver = + std::sync::Arc; + pub struct OpenAICompatibleLLMProvider { config: OpenAICompatibleConfig, siwc_access: Option>, @@ -531,6 +534,7 @@ pub struct OpenAICompatibleLLMProvider { /// connection pool — a fresh TLS handshake on every polish, exactly the cost that /// cache exists to avoid. A constant cap keeps the cache key single. polish_client: reqwest::Client, + usage_observer: Option, } impl OpenAICompatibleLLMProvider { @@ -553,6 +557,31 @@ impl OpenAICompatibleLLMProvider { dictation_session_id: None, siwc_access: None, polish_client, + usage_observer: None, + } + } + + /// The observer must return promptly and receives no request text. + pub fn with_usage_observer(mut self, observer: LlmUsageObserver) -> Self { + self.usage_observer = Some(observer); + self + } + + pub(crate) fn set_usage_observer(&mut self, observer: LlmUsageObserver) { + self.usage_observer = Some(observer); + } + + fn report_usage(&self, request_id: uuid::Uuid, usage: crate::usage_ledger::ReportedTokens) { + if usage != crate::usage_ledger::ReportedTokens::default() { + if let Some(observer) = &self.usage_observer { + if std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + observer(request_id, usage) + })) + .is_err() + { + log::warn!("usage observer panicked"); + } + } } } @@ -833,6 +862,17 @@ impl OpenAICompatibleLLMProvider { "stream": stream, "messages": messages, }); + if stream + && self.usage_observer.is_some() + && self.config.provider_id == "openai" + && url::Url::parse(&self.config.base_url) + .ok() + .and_then(|url| url.host_str().map(str::to_owned)) + .as_deref() + == Some("api.openai.com") + { + body["stream_options"] = json!({"include_usage": true}); + } if let Some(temperature) = self.config.temperature { // OpenAI 官方 gpt-5 / gpt-6 系列在 Chat Completions 只接受默认 temperature=1， // 传 0.3 会被 400 拒绝（issue #857 / #1101）。官方渠道的这些模型不下发该字段， @@ -944,6 +984,14 @@ impl OpenAICompatibleLLMProvider { }); } + if self.usage_observer.is_some() { + if let Ok(value) = serde_json::from_str::(&body_text) { + if let Some(usage) = crate::usage_ledger::ReportedTokens::from_response(&value) { + self.report_usage(request_id, usage); + } + } + } + let extracted = crate::llm_protocol::extract_text_raw(self.config.protocol.format, &body_text)?; let cleaned = clean_polish_output(&extracted); @@ -1128,6 +1176,7 @@ impl OpenAICompatibleLLMProvider { if !cancelled { events.finish()?; } + self.report_usage(request_id, events.reported_tokens()); let cleaned = clean_polish_output(&full_text); log::info!( "[llm] stream done request_id={} cancelled={} delta_count={} extracted_chars={} extracted_hash={} cleaned_chars={} cleaned_hash={} extracted_near_duplicates={} cleaned_near_duplicates={}", @@ -2505,6 +2554,154 @@ mod tests { assert!(err.to_string().contains("middle of a UTF-8 codepoint")); } + #[tokio::test] + async fn provider_reports_explicit_streaming_usage_without_changing_text() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let address = listener.local_addr().unwrap(); + let server = thread::spawn(move || { + let (mut stream, _) = listener.accept().unwrap(); + let _ = read_http_request(&mut stream); + let events = b"data: {\"choices\":[{\"delta\":{\"content\":\"hello\"}}]}\n\ndata: {\"choices\":[],\"usage\":{\"prompt_tokens\":21,\"completion_tokens\":4}}\n\ndata: [DONE]\n\n"; + write_chunked_sse_response(&mut stream, &[events]); + }); + let observed = std::sync::Arc::new(StdMutex::new(Vec::new())); + let output = observed.clone(); + let provider = OpenAICompatibleLLMProvider::new(OpenAICompatibleConfig::new( + "custom", + "Custom", + format!("http://{address}"), + "", + "fixture-model", + )) + .with_usage_observer(std::sync::Arc::new(move |id, usage| { + output.lock().unwrap().push((id, usage)) + })); + let text = provider + .chat_completion_messages_streaming( + vec![serde_json::json!({"role":"user","content":"private text"})], + StreamingTimeouts { + first_token: Duration::from_secs(2), + idle: Duration::from_secs(2), + }, + |_| {}, + || false, + ) + .await + .unwrap(); + assert_eq!(text, "hello"); + let observations = observed.lock().unwrap(); + assert_eq!(observations.len(), 1); + assert_eq!(observations[0].1.input, Some(21)); + assert_eq!(observations[0].1.output, Some(4)); + server.join().unwrap(); + } + + #[tokio::test] + async fn provider_reports_explicit_non_streaming_usage() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let address = listener.local_addr().unwrap(); + let server = thread::spawn(move || { + let (mut stream, _) = listener.accept().unwrap(); + let _ = read_http_request(&mut stream); + let body = r#"{"choices":[{"message":{"content":"hello"}}],"usage":{"prompt_tokens":33,"completion_tokens":6}}"#; + write!(stream, "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", body.len(), body).unwrap(); + }); + let observed = std::sync::Arc::new(StdMutex::new(Vec::new())); + let output = observed.clone(); + let provider = OpenAICompatibleLLMProvider::new(OpenAICompatibleConfig::new( + "custom", + "Custom", + format!("http://{address}"), + "", + "fixture-model", + )) + .with_usage_observer(std::sync::Arc::new(move |id, usage| { + output.lock().unwrap().push((id, usage)) + })); + assert_eq!( + provider + .chat_completion("system", "private text", Duration::from_secs(2)) + .await + .unwrap(), + "hello" + ); + let observations = observed.lock().unwrap(); + assert_eq!(observations.len(), 1); + assert_eq!(observations[0].1.input, Some(33)); + assert_eq!(observations[0].1.output, Some(6)); + server.join().unwrap(); + } + + #[test] + fn usage_observer_panic_does_not_escape_provider() { + let provider = OpenAICompatibleLLMProvider::new(OpenAICompatibleConfig::new( + "custom", + "Custom", + "http://127.0.0.1:1", + "", + "fixture-model", + )) + .with_usage_observer(std::sync::Arc::new(|_, _| panic!("fixture observer"))); + provider.report_usage( + uuid::Uuid::new_v4(), + crate::usage_ledger::ReportedTokens { + input: Some(1), + ..Default::default() + }, + ); + } + + #[test] + fn missing_provider_usage_does_not_emit_zero_counts() { + let calls = std::sync::Arc::new(std::sync::atomic::AtomicU64::new(0)); + let captured = calls.clone(); + let provider = OpenAICompatibleLLMProvider::new(OpenAICompatibleConfig::new( + "custom", + "Custom", + "http://127.0.0.1:1", + "", + "fixture-model", + )) + .with_usage_observer(std::sync::Arc::new(move |_, _| { + captured.fetch_add(1, Ordering::Relaxed); + })); + provider.report_usage(uuid::Uuid::new_v4(), Default::default()); + assert_eq!(calls.load(Ordering::Relaxed), 0); + } + + #[test] + fn usage_stream_option_is_limited_to_official_openai_chat_requests() { + let observer: LlmUsageObserver = std::sync::Arc::new(|_, _| {}); + let official = OpenAICompatibleLLMProvider::new(OpenAICompatibleConfig::new( + "openai", + "OpenAI", + "https://api.openai.com/v1", + "", + "gpt-4o-mini", + )) + .with_usage_observer(observer.clone()); + assert_eq!( + official.chat_body(true, vec![])["stream_options"]["include_usage"], + true + ); + assert!(official + .chat_body(false, vec![]) + .get("stream_options") + .is_none()); + let proxy = OpenAICompatibleLLMProvider::new(OpenAICompatibleConfig::new( + "openai", + "Proxy", + "https://proxy.example/v1", + "", + "gpt-4o-mini", + )) + .with_usage_observer(observer); + assert!(proxy + .chat_body(true, vec![]) + .get("stream_options") + .is_none()); + } + #[tokio::test] async fn polish_streaming_handles_multibyte_split_in_http_chunk() { let listener = TcpListener::bind("127.0.0.1:0").unwrap(); diff --git a/openless-all/app/crates/openless-core/src/shared_types.rs b/openless-all/app/crates/openless-core/src/shared_types.rs index 6ed413bd8..ca050ffef 100644 --- a/openless-all/app/crates/openless-core/src/shared_types.rs +++ b/openless-all/app/crates/openless-core/src/shared_types.rs @@ -843,6 +843,9 @@ pub struct UserPreferences { /// Observe corrections locally after insertion; independent of LLM context. #[serde(default)] pub vocabulary_learning_enabled: bool, + /// Local consent for observing formatting edits as abstract expression evidence. + #[serde(default)] + pub expression_learning_enabled: bool, #[serde(default)] pub vocabulary_learning_settings: VocabularyLearningSettings, /// Whether the Overview shows the "yearly activity" heatmap card. @@ -1231,6 +1234,8 @@ struct UserPreferencesWire { cursor_context_enabled: bool, #[serde(default)] vocabulary_learning_enabled: bool, + #[serde(default)] + expression_learning_enabled: bool, vocabulary_learning_settings: VocabularyLearningSettings, #[serde(default = "default_true")] show_overview_activity_heatmap: bool, @@ -1437,6 +1442,7 @@ impl Default for UserPreferencesWire { macos_composition_input: prefs.macos_composition_input, cursor_context_enabled: prefs.cursor_context_enabled, vocabulary_learning_enabled: prefs.vocabulary_learning_enabled, + expression_learning_enabled: prefs.expression_learning_enabled, vocabulary_learning_settings: prefs.vocabulary_learning_settings, show_overview_activity_heatmap: prefs.show_overview_activity_heatmap, stacked_row_layout: prefs.stacked_row_layout, @@ -1678,6 +1684,7 @@ impl<'de> Deserialize<'de> for UserPreferences { macos_composition_input: wire.macos_composition_input, cursor_context_enabled: wire.cursor_context_enabled, vocabulary_learning_enabled: wire.vocabulary_learning_enabled, + expression_learning_enabled: wire.expression_learning_enabled, vocabulary_learning_settings: wire.vocabulary_learning_settings.normalized(), show_overview_activity_heatmap: wire.show_overview_activity_heatmap, stacked_row_layout: wire.stacked_row_layout, @@ -2062,6 +2069,7 @@ impl Default for UserPreferences { macos_composition_input: false, cursor_context_enabled: true, vocabulary_learning_enabled: false, + expression_learning_enabled: false, vocabulary_learning_settings: VocabularyLearningSettings::default(), show_overview_activity_heatmap: true, stacked_row_layout: false, diff --git a/openless-all/app/crates/openless-core/src/usage_ledger.rs b/openless-all/app/crates/openless-core/src/usage_ledger.rs new file mode 100644 index 000000000..a819c74f4 --- /dev/null +++ b/openless-all/app/crates/openless-core/src/usage_ledger.rs @@ -0,0 +1,575 @@ +//! Local service-use metadata. Content, credentials and inferred prices are never stored. + +use crate::errors::BackendError; +use crate::persistence::{atomic_write, persistence_error}; +use serde::{Deserialize, Serialize}; +use std::collections::HashSet; +use std::io::Read; +use std::path::PathBuf; +use std::sync::{ + atomic::{AtomicU64, Ordering}, + mpsc, Arc, Mutex, +}; +use uuid::Uuid; + +const MAX_RECORDS: usize = 10_000; +const MAX_BYTES: u64 = 8 * 1024 * 1024; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum ServiceKind { + Asr, + Llm, + Tts, + Multimodal, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum ServicePurpose { + Dictation, + DictationPostprocess, + Translation, + Question, + Selection, + QuickNote, + VoiceEdit, + CloudNote, + Other, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum ServiceOutcome { + Succeeded, + Failed, + Cancelled, + Unknown, +} + +/// Values are retained as the provider reported them. Cache accounting differs +/// between protocols, so these fields must not be summed implicitly. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct ReportedTokens { + pub input: Option, + pub output: Option, + pub cached_input: Option, + pub cache_write_input: Option, + pub reasoning: Option, +} + +impl ReportedTokens { + pub fn from_response(value: &serde_json::Value) -> Option { + let usage = value + .get("usage") + .filter(|usage| usage.is_object()) + .or_else(|| { + value + .pointer("/response/usage") + .filter(|usage| usage.is_object()) + }) + .or_else(|| { + value + .pointer("/message/usage") + .filter(|usage| usage.is_object()) + }) + .or_else(|| value.get("usageMetadata").filter(|usage| usage.is_object()))?; + let tokens = Self { + input: usage + .get("input_tokens") + .or_else(|| usage.get("prompt_tokens")) + .or_else(|| usage.get("promptTokenCount")) + .and_then(|value| value.as_u64()), + output: usage + .get("output_tokens") + .or_else(|| usage.get("completion_tokens")) + .or_else(|| usage.get("candidatesTokenCount")) + .and_then(|value| value.as_u64()), + cached_input: usage + .pointer("/input_tokens_details/cached_tokens") + .or_else(|| usage.pointer("/prompt_tokens_details/cached_tokens")) + .or_else(|| usage.get("prompt_cache_hit_tokens")) + .or_else(|| usage.get("cache_read_input_tokens")) + .or_else(|| usage.get("cachedContentTokenCount")) + .and_then(|value| value.as_u64()), + cache_write_input: usage + .get("cache_creation_input_tokens") + .or_else(|| usage.pointer("/input_tokens_details/cache_write_tokens")) + .and_then(|value| value.as_u64()), + reasoning: usage + .pointer("/output_tokens_details/reasoning_tokens") + .or_else(|| usage.pointer("/completion_tokens_details/reasoning_tokens")) + .and_then(|value| value.as_u64()), + }; + (tokens != Self::default()).then_some(tokens) + } + + pub fn merge_report(&mut self, newer: Self) { + if newer.input.is_some() { + self.input = newer.input; + } + if newer.output.is_some() { + self.output = newer.output; + } + if newer.cached_input.is_some() { + self.cached_input = newer.cached_input; + } + if newer.cache_write_input.is_some() { + self.cache_write_input = newer.cache_write_input; + } + if newer.reasoning.is_some() { + self.reasoning = newer.reasoning; + } + } +} + +/// Quantities are optional because provider responses may not disclose them. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct UsageRecord { + pub id: Uuid, + pub occurred_at_ms: i64, + pub kind: ServiceKind, + pub purpose: ServicePurpose, + pub outcome: ServiceOutcome, + pub provider: Option, + pub channel: Option, + pub model: Option, + pub audio_ms: Option, + pub input_tokens: Option, + pub output_tokens: Option, + pub cached_input_tokens: Option, + pub cache_write_input_tokens: Option, + pub reasoning_tokens: Option, + pub synthesized_characters: Option, +} + +fn identifier_valid(value: &str) -> bool { + let lower = value.to_ascii_lowercase(); + !value.is_empty() + && value.len() <= 160 + && !value.contains("://") + && !lower.starts_with("sk-") + && !lower.starts_with("bearer") + && value + .chars() + .all(|character| character.is_alphanumeric() || "._-/+:".contains(character)) +} + +/// Adapters may omit untrusted or invalid identifiers without dropping usage. +pub fn sanitized_identifier(value: &str) -> Option { + let value = value.trim(); + identifier_valid(value).then(|| value.to_owned()) +} + +impl UsageRecord { + fn valid(&self) -> bool { + !self.id.is_nil() + && self.occurred_at_ms > 0 + && self.provider.as_deref().is_none_or(identifier_valid) + && self.channel.as_deref().is_none_or(identifier_valid) + && self.model.as_deref().is_none_or(identifier_valid) + } +} + +#[derive(Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +struct UsageDocument { + version: u32, + records: Vec, +} + +/// A single shared instance must own a given file within a process. +pub struct UsageLedger { + path: PathBuf, + records: Mutex>>, + generation: AtomicU64, +} + +enum WriterCommand { + Append(u64, Box), + Flush(mpsc::Sender<()>), + Clear(mpsc::Sender>), +} + +/// A bounded worker keeps disk writes off recognition and model response paths. +pub struct UsageRecorder { + ledger: Arc, + sender: mpsc::SyncSender, + dropped_or_failed: Arc, +} + +impl UsageRecorder { + pub fn start(ledger: Arc) -> Result { + let (sender, receiver) = mpsc::sync_channel(256); + let worker_ledger = Arc::clone(&ledger); + let dropped_or_failed = Arc::new(AtomicU64::new(0)); + let failures = Arc::clone(&dropped_or_failed); + std::thread::Builder::new() + .name("openless-usage-ledger".into()) + .spawn(move || { + while let Ok(command) = receiver.recv() { + match command { + WriterCommand::Append(generation, record) => { + if worker_ledger.append(generation, *record).is_err() { + failures.fetch_add(1, Ordering::Relaxed); + } + } + WriterCommand::Flush(reply) => { + let _ = reply.send(()); + } + WriterCommand::Clear(reply) => { + let _ = reply.send(worker_ledger.clear()); + } + } + } + }) + .map_err(|_| persistence_error("start service usage worker"))?; + Ok(Self { + ledger, + sender, + dropped_or_failed, + }) + } + + pub fn generation(&self) -> u64 { + self.ledger.generation() + } + + pub fn submit(&self, generation: u64, record: UsageRecord) -> bool { + if self + .sender + .try_send(WriterCommand::Append(generation, Box::new(record))) + .is_ok() + { + true + } else { + self.dropped_or_failed.fetch_add(1, Ordering::Relaxed); + false + } + } + + /// Call from a blocking worker, never from an audio or UI thread. + pub fn snapshot(&self) -> Result<(Vec, u64), BackendError> { + let (send, receive) = mpsc::channel(); + self.sender + .try_send(WriterCommand::Flush(send)) + .map_err(|_| persistence_error("flush service usage"))?; + receive + .recv_timeout(std::time::Duration::from_secs(5)) + .map_err(|_| persistence_error("flush service usage"))?; + Ok(( + self.ledger.list()?, + self.dropped_or_failed.load(Ordering::Relaxed), + )) + } + + /// Queued records precede the clear; in-flight pre-clear requests are rejected later. + pub fn clear(&self) -> Result<(), BackendError> { + let (send, receive) = mpsc::channel(); + self.sender + .try_send(WriterCommand::Clear(send)) + .map_err(|_| persistence_error("clear service usage"))?; + receive + .recv_timeout(std::time::Duration::from_secs(5)) + .map_err(|_| { + BackendError::new( + crate::errors::BackendErrorCode::OutcomeUnknown, + "service usage clear outcome is unknown", + ) + })? + } +} + +impl UsageLedger { + /// Replacing an unreadable ledger requires an explicit user reset. + pub fn reset(path: PathBuf) -> Result { + atomic_write(&path, br#"{"version":1,"records":[]}"#)?; + Self::open(path) + } + + pub fn open(path: PathBuf) -> Result { + let mut records = Vec::new(); + match std::fs::File::open(&path) { + Ok(file) => { + let mut bytes = Vec::new(); + file.take(MAX_BYTES + 1) + .read_to_end(&mut bytes) + .map_err(|_| persistence_error("read service usage"))?; + if bytes.len() as u64 > MAX_BYTES { + return Err(persistence_error("service usage too large")); + } + let document: UsageDocument = serde_json::from_slice(&bytes) + .map_err(|_| persistence_error("decode service usage"))?; + if document.version != 1 || document.records.len() > MAX_RECORDS { + return Err(persistence_error("invalid service usage")); + } + let mut ids = HashSet::new(); + if document + .records + .iter() + .any(|record| !record.valid() || !ids.insert(record.id)) + { + return Err(persistence_error("invalid service usage record")); + } + records = document.records; + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(_) => return Err(persistence_error("read service usage")), + } + Ok(Self { + path, + records: Mutex::new(Some(records)), + generation: AtomicU64::new(0), + }) + } + + pub fn generation(&self) -> u64 { + self.generation.load(Ordering::Acquire) + } + + pub fn append( + &self, + expected_generation: u64, + record: UsageRecord, + ) -> Result { + if !record.valid() { + return Err(persistence_error("invalid service usage record")); + } + let mut state = self + .records + .lock() + .map_err(|_| persistence_error("lock service usage"))?; + if self.generation.load(Ordering::Acquire) != expected_generation { + return Ok(false); + } + let mut next = state + .as_ref() + .ok_or_else(|| persistence_error("service usage needs reload"))? + .clone(); + if next.iter().any(|existing| existing.id == record.id) { + return Ok(false); + } + next.push(record); + next.sort_unstable_by_key(|record| record.occurred_at_ms); + if next.len() > MAX_RECORDS { + next.drain(..next.len() - MAX_RECORDS); + } + if let Err(error) = self.persist(&next) { + *state = None; + return Err(error); + } + *state = Some(next); + Ok(true) + } + + pub fn list(&self) -> Result, BackendError> { + self.records + .lock() + .map_err(|_| persistence_error("lock service usage"))? + .clone() + .ok_or_else(|| persistence_error("service usage needs reload")) + } + + pub fn clear(&self) -> Result<(), BackendError> { + let mut state = self + .records + .lock() + .map_err(|_| persistence_error("lock service usage"))?; + self.generation.fetch_add(1, Ordering::AcqRel); + if state.is_none() { + return Err(persistence_error("service usage needs reload")); + } + if let Err(error) = self.persist(&[]) { + *state = None; + return Err(error); + } + *state = Some(Vec::new()); + Ok(()) + } + + fn persist(&self, records: &[UsageRecord]) -> Result<(), BackendError> { + let bytes = serde_json::to_vec(&UsageDocument { + version: 1, + records: records.to_vec(), + }) + .map_err(|_| persistence_error("encode service usage"))?; + atomic_write(&self.path, &bytes) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn record() -> UsageRecord { + UsageRecord { + id: Uuid::new_v4(), + occurred_at_ms: 1_800_000_000_000, + kind: ServiceKind::Llm, + purpose: ServicePurpose::DictationPostprocess, + outcome: ServiceOutcome::Succeeded, + provider: Some("provider".into()), + channel: Some("channel-1".into()), + model: Some("model-v1".into()), + audio_ms: None, + input_tokens: Some(80), + output_tokens: Some(20), + cached_input_tokens: None, + cache_write_input_tokens: None, + reasoning_tokens: None, + synthesized_characters: None, + } + } + + #[test] + fn persists_reported_quantities_and_deduplicates_without_text() { + let dir = std::env::temp_dir().join(format!("openless-usage-{}", Uuid::new_v4())); + let path = dir.join("usage.json"); + let ledger = UsageLedger::open(path.clone()).unwrap(); + let event = record(); + assert!(ledger.append(ledger.generation(), event.clone()).unwrap()); + assert!(!ledger.append(ledger.generation(), event.clone()).unwrap()); + let content = std::fs::read_to_string(&path).unwrap(); + assert!(!content.contains("transcript")); + assert!(!content.contains("apiKey")); + drop(ledger); + assert_eq!( + UsageLedger::open(path).unwrap().list().unwrap(), + vec![event] + ); + std::fs::remove_dir_all(dir).unwrap(); + } + + #[test] + fn clear_invalidates_old_writers_and_unknown_files_are_preserved() { + let dir = std::env::temp_dir().join(format!("openless-usage-{}", Uuid::new_v4())); + let path = dir.join("usage.json"); + let ledger = UsageLedger::open(path.clone()).unwrap(); + let generation = ledger.generation(); + assert!(ledger.append(generation, record()).unwrap()); + ledger.clear().unwrap(); + assert!(!ledger.append(generation, record()).unwrap()); + assert!(ledger.list().unwrap().is_empty()); + drop(ledger); + let unknown = br#"{"version":2,"records":[],"newField":true}"#; + std::fs::write(&path, unknown).unwrap(); + assert!(UsageLedger::open(path.clone()).is_err()); + assert_eq!(std::fs::read(&path).unwrap(), unknown); + assert!(UsageLedger::reset(path).unwrap().list().unwrap().is_empty()); + std::fs::remove_dir_all(dir).unwrap(); + } + + #[test] + fn older_records_without_optional_fields_remain_readable() { + let dir = std::env::temp_dir().join(format!("openless-usage-{}", Uuid::new_v4())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("usage.json"); + let mut value = serde_json::to_value(record()).unwrap(); + value.as_object_mut().unwrap().remove("channel"); + value + .as_object_mut() + .unwrap() + .remove("cacheWriteInputTokens"); + value.as_object_mut().unwrap().remove("reasoningTokens"); + std::fs::write( + &path, + serde_json::to_vec(&serde_json::json!({"version":1,"records":[value]})).unwrap(), + ) + .unwrap(); + let saved = UsageLedger::open(path).unwrap().list().unwrap(); + assert_eq!(saved[0].channel, None); + assert_eq!(saved[0].cache_write_input_tokens, None); + assert_eq!(saved[0].reasoning_tokens, None); + std::fs::remove_dir_all(dir).unwrap(); + } + + #[test] + fn recorder_flushes_and_rejects_inflight_records_after_clear() { + let dir = std::env::temp_dir().join(format!("openless-usage-{}", Uuid::new_v4())); + let ledger = Arc::new(UsageLedger::open(dir.join("usage.json")).unwrap()); + let recorder = UsageRecorder::start(ledger).unwrap(); + let generation = recorder.generation(); + assert!(recorder.submit(generation, record())); + assert_eq!(recorder.snapshot().unwrap().0.len(), 1); + recorder.clear().unwrap(); + assert!(recorder.submit(generation, record())); + assert!(recorder.snapshot().unwrap().0.is_empty()); + let mut invalid = record(); + invalid.model = Some("sk-secret".into()); + assert!(recorder.submit(recorder.generation(), invalid)); + assert_eq!(recorder.snapshot().unwrap().1, 1); + drop(recorder); + std::fs::remove_dir_all(dir).unwrap(); + } + + #[test] + fn rejects_invalid_identifiers_without_inventing_token_semantics() { + let dir = std::env::temp_dir().join(format!("openless-usage-{}", Uuid::new_v4())); + let ledger = UsageLedger::open(dir.join("usage.json")).unwrap(); + let mut event = record(); + event.model = Some("https://example.invalid/?key=secret".into()); + assert!(ledger.append(ledger.generation(), event).is_err()); + let mut event = record(); + event.model = Some("sk-example-secret".into()); + assert!(ledger.append(ledger.generation(), event).is_err()); + assert_eq!(sanitized_identifier("sk-example-secret"), None); + assert_eq!(sanitized_identifier(" model-v1 "), Some("model-v1".into())); + let mut event = record(); + event.cached_input_tokens = Some(81); + assert!(ledger.append(ledger.generation(), event).unwrap()); + } + + #[test] + fn reads_only_provider_reported_token_fields() { + use serde_json::json; + let chat = ReportedTokens::from_response(&json!({"usage":{"prompt_tokens":100,"completion_tokens":8,"prompt_tokens_details":{"cached_tokens":30}}})).unwrap(); + assert_eq!( + chat, + ReportedTokens { + input: Some(100), + output: Some(8), + cached_input: Some(30), + cache_write_input: None, + reasoning: None, + } + ); + let response = ReportedTokens::from_response( + &json!({"response":{"usage":{"input_tokens":40,"output_tokens":5}}}), + ) + .unwrap(); + assert_eq!(response.input, Some(40)); + assert_eq!(response.output, Some(5)); + let reasoning = ReportedTokens::from_response( + &json!({"usage":{"output_tokens":10,"output_tokens_details":{"reasoning_tokens":7}}}), + ) + .unwrap(); + assert_eq!(reasoning.reasoning, Some(7)); + let nested = ReportedTokens::from_response( + &json!({"usage":null,"response":{"usage":{"input_tokens":40}}}), + ) + .unwrap(); + assert_eq!(nested.input, Some(40)); + let mut messages = ReportedTokens::from_response( + &json!({"message":{"usage":{"input_tokens":10,"cache_read_input_tokens":90,"cache_creation_input_tokens":12}}}), + ) + .unwrap(); + messages.merge_report( + ReportedTokens::from_response(&json!({"usage":{"output_tokens":7}})).unwrap(), + ); + assert_eq!( + messages, + ReportedTokens { + input: Some(10), + output: Some(7), + cached_input: Some(90), + cache_write_input: Some(12), + reasoning: None, + } + ); + let gemini = ReportedTokens::from_response(&json!({"usageMetadata":{"promptTokenCount":12,"candidatesTokenCount":3,"cachedContentTokenCount":4}})).unwrap(); + assert_eq!(gemini.cached_input, Some(4)); + assert!(ReportedTokens::from_response(&json!({"usage":{"total_tokens":999}})).is_none()); + assert!(ReportedTokens::from_response(&json!({"text":"never stored"})).is_none()); + } +} diff --git a/openless-all/app/e2e/ask-anything.spec.ts b/openless-all/app/e2e/ask-anything.spec.ts index ff59e264b..b8a7666a2 100644 --- a/openless-all/app/e2e/ask-anything.spec.ts +++ b/openless-all/app/e2e/ask-anything.spec.ts @@ -19,6 +19,14 @@ test('translation uses the shared popup and native headroom; save failure preser await picker.click(); const popup = page.locator('.ol-select-popup'); await expect(popup).toBeVisible(); + await expect(popup).toHaveClass(/ol-capsule-language-popup/); + await expect(popup).toHaveCSS('backdrop-filter', 'none'); + await popup.evaluate((node) => node.setAttribute('data-test-retained', 'true')); + for (let i = 0; i < 8; i += 1) { + await page.evaluate((level) => (window as any).fixture.setCapsule({ level }), i / 10); + await expect(popup).toHaveAttribute('data-test-retained', 'true'); + await expect(popup).toHaveCSS('opacity', '1'); + } const clipped = await popup.evaluate((node) => getComputedStyle(node).clipPath.startsWith('shape('), ); diff --git a/openless-all/app/e2e/control-geometry-regressions.spec.ts b/openless-all/app/e2e/control-geometry-regressions.spec.ts index 1ab9af29f..f3ba12a22 100644 --- a/openless-all/app/e2e/control-geometry-regressions.spec.ts +++ b/openless-all/app/e2e/control-geometry-regressions.spec.ts @@ -136,8 +136,23 @@ test('history list keeps its rounded surface and native trackpad scrolling; corr }); expect(wheelPrevented).toBe(false); await page.screenshot({ path: test.info().outputPath('history-list.png') }); - await page.getByRole('button', { name: '工具', exact: true }).click(); - await page.getByRole('button', { name: '纠正规则', exact: true }).click(); + const tools = page.getByRole('button', { name: '工具', exact: true }); + await tools.click(); + await tools.evaluate(async (node) => { + await Promise.all( + node + .parentElement!.getAnimations({ subtree: true }) + .map((animation) => animation.finished.catch(() => {})), + ); + }); + const corrections = page.getByRole('button', { name: '纠正规则', exact: true }); + await corrections.click(); + await expect(corrections).toHaveAttribute('aria-current', 'page'); + await expect(page.locator('[data-ol-page="corrections"]')).toHaveAttribute( + 'data-transition-phase', + 'settled', + { timeout: 15_000 }, + ); await settleNavigation(page); const add = page.getByRole('button', { name: '添加', exact: true }); await expect(add).toBeVisible(); diff --git a/openless-all/app/e2e/live-input-settings.spec.ts b/openless-all/app/e2e/live-input-settings.spec.ts new file mode 100644 index 000000000..96d0c5658 --- /dev/null +++ b/openless-all/app/e2e/live-input-settings.spec.ts @@ -0,0 +1,29 @@ +import { expect, test } from '@playwright/test'; + +for (const [os, ua, platform] of [ + ['macOS', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0)', 'MacIntel'], + ['Windows', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)', 'Win32'], +] as const) { + test(`live transcript input availability on ${os}`, async ({ page }) => { + await page.addInitScript( + ({ ua, platform }) => { + Object.defineProperty(navigator, 'userAgent', { value: ua }); + Object.defineProperty(navigator, 'platform', { value: platform }); + Object.defineProperty(navigator, 'userAgentData', { value: undefined }); + }, + { ua, platform }, + ); + await page.goto('/'); + await page.getByRole('button', { name: '设置', exact: true }).click(); + const dialog = page.getByRole('dialog', { name: '设置', exact: true }); + await dialog.getByRole('button', { name: '高级功能', exact: true }).click(); + if (os === 'macOS') { + await expect(dialog.locator('[data-ol-advanced-entry="liveInput"]')).toHaveCount(0); + return; + } + await dialog.locator('[data-ol-advanced-entry="liveInput"]').click(); + const panel = dialog.locator('[data-ol-advanced-page="liveInput"]'); + await expect(panel).toBeVisible(); + await expect(panel.getByText('实时原文显示在输入框', { exact: true })).toBeVisible(); + }); +} diff --git a/openless-all/app/e2e/motion.spec.ts b/openless-all/app/e2e/motion.spec.ts index e8d81d9c8..6dc232fd8 100644 --- a/openless-all/app/e2e/motion.spec.ts +++ b/openless-all/app/e2e/motion.spec.ts @@ -331,6 +331,39 @@ test('multimodal settings fit one desktop page and keep inactive notices below l }); }); +test('pipeline selection stays evenly inset in both modes', async ({ page }) => { + await page.emulateMedia({ reducedMotion: 'reduce' }); + await page.getByRole('button', { name: '设置', exact: true }).click(); + const dialog = page.getByRole('dialog', { name: '设置', exact: true }); + await dialog.getByRole('button', { name: 'AI 与模型', exact: true }).click(); + const track = dialog.locator('.ol-service-pipeline-track'); + for (const name of ['传统模式', '多模态模式']) { + await dialog.getByRole('button', { name, exact: true }).click(); + await expect + .poll(() => + track.evaluate((element, selectedName) => { + const outer = element.getBoundingClientRect(); + const selected = element.querySelector('button[aria-pressed="true"]')!; + const chip = element.querySelector('.ol-segmented-indicator')!; + const item = selected.getBoundingClientRect(); + const inner = chip.getBoundingClientRect(); + return Math.max( + Math.abs(inner.top - item.top), + Math.abs(inner.left - item.left), + Math.abs(inner.width - item.width), + Math.abs(inner.height - item.height), + Math.abs(inner.top - outer.top - (outer.bottom - inner.bottom)), + Math.abs( + (selectedName === '传统模式' ? inner.left - outer.left : outer.right - inner.right) - + (inner.top - outer.top), + ), + ); + }, name), + ) + .toBeLessThan(0.75); + } +}); + test('selection highlights move and settle correctly after a rapid mode return', async ({ page, }) => { diff --git a/openless-all/app/e2e/refinements.spec.ts b/openless-all/app/e2e/refinements.spec.ts index fd85274c4..000b95e0c 100644 --- a/openless-all/app/e2e/refinements.spec.ts +++ b/openless-all/app/e2e/refinements.spec.ts @@ -2,6 +2,20 @@ import { openMicrophoneSettings, settleNavigation } from './support/navigation'; import { test, expect, type Page } from '@playwright/test'; const fixture = 'http://127.0.0.1:5326/e2e/fixtures/refinements.html'; + +test('style and dictionary action buttons retain compact corners', async ({ page }) => { + await page.goto('/'); + await page.getByRole('button', { name: '风格', exact: true }).click(); + await page.getByRole('button', { name: '润色模式', exact: true }).click(); + await expect(page.getByRole('button', { name: '导入 ZIP' })).toHaveCSS('border-radius', '8px'); + await expect(page.getByRole('button', { name: '激活', exact: true }).first()).toHaveCSS( + 'border-radius', + '8px', + ); + await page.getByRole('button', { name: '词典', exact: true }).click(); + await expect(page.locator('.ol-action-button').first()).toHaveCSS('border-radius', '8px'); +}); + async function openShortcuts(page: Page) { await page.goto('/'); await page.getByRole('button', { name: '设置', exact: true }).click(); @@ -9,6 +23,36 @@ async function openShortcuts(page: Page) { await settleNavigation(page); } +test('shortcut search remains visible without focus', async ({ page }) => { + await openShortcuts(page); + const search = page.getByRole('searchbox', { name: '搜索命令或按键' }); + const colors = await search.evaluate((element) => ({ + search: getComputedStyle(element).backgroundColor, + page: getComputedStyle(element.closest('.ol-settings-content') ?? document.body) + .backgroundColor, + })); + expect(colors.search).not.toBe('rgba(0, 0, 0, 0)'); + expect(colors.search).not.toBe(colors.page); +}); + +test('awareness settings have a dedicated advanced page', async ({ page }) => { + await page.goto('/'); + await page.getByRole('button', { name: '设置', exact: true }).click(); + const dialog = page.getByRole('dialog', { name: '设置', exact: true }); + await dialog.getByRole('button', { name: /^(高级功能|实验与扩展)$/ }).click(); + await settleNavigation(dialog); + await dialog.locator('[data-ol-advanced-entry="cursorContext"]').click(); + await expect(dialog.locator('[data-ol-advanced-page="cursorContext"]')).toBeVisible(); + await expect(dialog.getByText('手改学词', { exact: true })).toBeVisible(); + const expression = dialog.getByRole('switch', { name: '学习表达习惯', exact: true }); + await expect(expression).toBeVisible(); + await expect(expression).toHaveAttribute('aria-checked', 'false'); + await expression.click(); + await expect(expression).toHaveAttribute('aria-checked', 'true'); + await dialog.getByRole('button', { name: '清除表达习惯', exact: true }).click(); + await expect(dialog.getByText('已形成 0 项稳定偏好')).toBeVisible(); +}); + test('shortcut search and categories live in the flat settings pane; keyboard close preserves exit', async ({ page, }) => { diff --git a/openless-all/app/e2e/settings-alignment.spec.ts b/openless-all/app/e2e/settings-alignment.spec.ts new file mode 100644 index 000000000..bfaf68c2f --- /dev/null +++ b/openless-all/app/e2e/settings-alignment.spec.ts @@ -0,0 +1,72 @@ +import { expect, test, type Locator } from '@playwright/test'; +import { openMicrophoneSettings, settleNavigation } from './support/navigation'; + +async function centered(group: Locator) { + await expect(group).toBeVisible(); + for (const zoom of ['1', '1.1']) { + await group.evaluate((node, value) => { + node.style.zoom = value; + }, zoom); + for (const option of await group.getByRole('button').all()) { + await option.click(); + await expect + .poll(() => + group.evaluate((node) => { + const selected = node + .querySelector('button[aria-pressed="true"]')! + .getBoundingClientRect(); + const indicator = node + .querySelector('.ol-segmented-indicator')! + .getBoundingClientRect(); + const track = node.getBoundingClientRect(); + return Math.max( + Math.abs(selected.top - indicator.top), + Math.abs(selected.left - indicator.left), + Math.abs(selected.width - indicator.width), + Math.abs(selected.height - indicator.height), + Math.abs(indicator.top - track.top - (track.bottom - indicator.bottom)), + ); + }), + ) + .toBeLessThan(0.3); + } + } +} + +test('shared selectors remain centered at 100 and 110 percent; awareness cards are separated', async ({ + page, +}) => { + await page.addInitScript(() => { + Object.defineProperty(navigator, 'platform', { value: 'MacIntel' }); + Object.defineProperty(navigator, 'userAgent', { + value: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0)', + }); + Object.defineProperty(navigator, 'userAgentData', { value: undefined }); + }); + await page.emulateMedia({ colorScheme: 'light' }); + await page.goto('/'); + await page.getByRole('button', { name: '设置', exact: true }).click(); + const dialog = page.getByRole('dialog', { name: '设置', exact: true }); + await openMicrophoneSettings(dialog); + await centered(dialog.getByRole('group', { name: '录音方式' })); + await page.screenshot({ path: test.info().outputPath('recording-centered.png') }); + await dialog.getByRole('button', { name: 'AI 与模型', exact: true }).click(); + await settleNavigation(page); + await centered(dialog.getByRole('group', { name: '识别管线' })); + await dialog.getByRole('button', { name: '外观与语言', exact: true }).click(); + await settleNavigation(page); + await centered(dialog.getByRole('group', { name: '主题' })); + await page.screenshot({ path: test.info().outputPath('theme-centered.png') }); + await dialog.getByRole('button', { name: '高级功能', exact: true }).click(); + await settleNavigation(page); + await dialog.locator('[data-ol-advanced-entry="cursorContext"]').click(); + await settleNavigation(page); + await expect(dialog.locator('.ol-awareness-sections')).toBeVisible(); + const gaps = await dialog.locator('.ol-awareness-sections').evaluate((node) => { + const cards = Array.from(node.children).map((child) => child.getBoundingClientRect()); + return cards.slice(1).map((card, i) => card.top - cards[i].bottom); + }); + expect(gaps.length).toBeGreaterThan(0); + for (const gap of gaps) expect(gap).toBeGreaterThanOrEqual(15.5); + await page.screenshot({ path: test.info().outputPath('awareness-separated.png') }); +}); diff --git a/openless-all/app/e2e/usage.spec.ts b/openless-all/app/e2e/usage.spec.ts new file mode 100644 index 000000000..30565f431 --- /dev/null +++ b/openless-all/app/e2e/usage.spec.ts @@ -0,0 +1,100 @@ +import { expect, test } from '@playwright/test'; + +test('usage separates the dashboard, services and history', async ({ page }) => { + await page.goto('/'); + const usage = page.getByRole('button', { name: '用量', exact: true }); + const settings = page.getByRole('button', { name: '设置', exact: true }); + await expect(usage).toHaveCSS( + 'font-size', + await settings.evaluate((node) => getComputedStyle(node).fontSize), + ); + await usage.click(); + await expect(page.getByRole('heading', { name: '用量', exact: true })).toBeVisible(); + await expect(page.getByText('语音识别记录')).toBeVisible(); + await expect(page.getByText('TTS 用量', { exact: true })).toBeVisible(); + await expect(page.getByText('已报告输入 Token', { exact: true })).toBeVisible(); + await expect(page.getByRole('heading', { name: '每日记录' })).toBeVisible(); + await expect(page.getByRole('heading', { name: '服务与用途' })).toHaveCount(0); + await expect(page.locator('[data-ol-page="usage"]')).toHaveAttribute( + 'data-transition-phase', + 'settled', + ); + await page.screenshot({ path: test.info().outputPath('usage-dashboard.png') }); + await page + .getByRole('navigation', { name: '用量页面' }) + .getByRole('button', { name: '服务与用途' }) + .click(); + await expect(page.getByRole('heading', { name: '服务与用途' })).toBeVisible(); + await expect(page.getByRole('heading', { name: '每日记录' })).toHaveCount(0); + await page + .getByRole('navigation', { name: '用量页面' }) + .getByRole('button', { name: '记录' }) + .click(); + const clear = page.getByRole('button', { name: '清除已记录 Token', exact: true }); + await expect(clear).toBeVisible(); + const period = page.getByRole('group', { name: '统计时段' }); + await expect(period).toBeVisible(); + const positions = await clear.evaluate((node) => { + const button = node.getBoundingClientRect(); + const toolbar = node.closest('.ol-usage-toolbar')!.getBoundingClientRect(); + const segment = node + .closest('.ol-usage-toolbar')! + .querySelector('.ol-segmented')! + .getBoundingClientRect(); + return { + right: toolbar.right - button.right, + center: Math.abs(button.top + button.bottom - (segment.top + segment.bottom)) / 2, + }; + }); + expect(positions.right).toBeLessThan(1); + expect(positions.center).toBeLessThan(1); + await page.setViewportSize({ width: 960, height: 600 }); + await page.locator('.ol-usage-page').evaluate((node) => { + node.style.zoom = '1.1'; + }); + const recent = page.getByRole('region', { name: '最近记录列表' }); + await expect(recent).toBeVisible(); + await recent.scrollIntoViewIfNeeded(); + const bounds = await recent.evaluate((node) => ({ + height: node.clientHeight, + content: node.scrollHeight, + overflow: getComputedStyle(node).overflowY, + })); + expect(bounds.overflow).toBe('auto'); + expect(bounds.height).toBeLessThan(400); + expect(bounds.content).toBeGreaterThan(bounds.height); + await recent.hover(); + await page.mouse.wheel(0, 160); + await expect.poll(() => recent.evaluate((node) => node.scrollTop)).toBeGreaterThan(0); + await page.emulateMedia({ colorScheme: 'light' }); + await page.screenshot({ path: test.info().outputPath('usage-history-light-short.png') }); + await page.setViewportSize({ width: 1300, height: 835 }); + await page.locator('.ol-usage-page').evaluate((node) => { + node.style.zoom = '1'; + }); + + const dimensions = await clear.evaluate((node) => ({ + scroll: node.scrollHeight, + client: node.clientHeight, + })); + expect(dimensions.scroll).toBeLessThanOrEqual(dimensions.client + 1); + await page.screenshot({ path: test.info().outputPath('usage-history.png') }); + await clear.click(); + await expect(page.getByText('该时段没有已保存的服务商 token 报告。')).toBeVisible(); + await page.getByRole('button', { name: '近 7 天' }).click(); + await expect(page.getByRole('button', { name: '近 7 天' })).toHaveAttribute( + 'aria-pressed', + 'true', + ); + await page + .getByRole('navigation', { name: '用量页面' }) + .getByRole('button', { name: '概览', exact: true }) + .click(); + await expect(page.getByRole('heading', { name: '每日记录' })).toBeVisible(); + await expect + .poll(() => + page.locator('.ol-usage-chart-dates').evaluate((node) => node.getBoundingClientRect().bottom), + ) + .toBeLessThan(835); + await page.screenshot({ path: test.info().outputPath('usage-dashboard-light.png') }); +}); diff --git a/openless-all/app/harmony/ipc-audit.json b/openless-all/app/harmony/ipc-audit.json index 6d3af1a0a..cf764060e 100644 --- a/openless-all/app/harmony/ipc-audit.json +++ b/openless-all/app/harmony/ipc-audit.json @@ -1,8 +1,8 @@ { "scope": "Static shared IPC command coverage; excludes event subscriptions, dynamic commands and desktop-only calls outside src/lib/ipc.", - "total": 183, + "total": 187, "wired": 103, - "missing": 80, + "missing": 84, "commands": [ { "name": "accept_pending_correction", @@ -109,6 +109,13 @@ ], "implementation": "arkts" }, + { + "name": "clear_expression_profile", + "calledFrom": [ + "src/lib/ipc/expression.ts" + ], + "implementation": "unimplemented" + }, { "name": "clear_history", "calledFrom": [ @@ -116,6 +123,13 @@ ], "implementation": "rust" }, + { + "name": "clear_service_usage", + "calledFrom": [ + "src/lib/ipc/service-usage.ts" + ], + "implementation": "unimplemented" + }, { "name": "cloud_sync_delete", "calledFrom": [ @@ -438,6 +452,13 @@ ], "implementation": "unimplemented" }, + { + "name": "get_expression_profile", + "calledFrom": [ + "src/lib/ipc/expression.ts" + ], + "implementation": "unimplemented" + }, { "name": "get_hotkey_capability", "calledFrom": [ @@ -487,6 +508,13 @@ ], "implementation": "unimplemented" }, + { + "name": "get_service_usage", + "calledFrom": [ + "src/lib/ipc/service-usage.ts" + ], + "implementation": "unimplemented" + }, { "name": "get_settings", "calledFrom": [ diff --git a/openless-all/app/public/licenses/type4me-LICENSE.txt b/openless-all/app/public/licenses/type4me-LICENSE.txt new file mode 100644 index 000000000..dfc042e14 --- /dev/null +++ b/openless-all/app/public/licenses/type4me-LICENSE.txt @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Type4Me Contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/openless-all/app/src-tauri/build.rs b/openless-all/app/src-tauri/build.rs index 5948fe13c..750e8c53a 100644 --- a/openless-all/app/src-tauri/build.rs +++ b/openless-all/app/src-tauri/build.rs @@ -25,6 +25,14 @@ fn main() { .flag("-fobjc-arc") .flag("-mmacosx-version-min=14.0") .compile("openless_playback_reference"); + println!("cargo:rerun-if-changed=src/accessibility_guide_macos.m"); + cc::Build::new() + .file("src/accessibility_guide_macos.m") + .flag("-fobjc-arc") + .flag("-mmacosx-version-min=14.0") + .compile("openless_accessibility_guide"); + println!("cargo:rustc-link-lib=framework=AppKit"); + println!("cargo:rustc-link-lib=framework=ApplicationServices"); println!("cargo:rustc-link-lib=framework=Foundation"); println!("cargo:rustc-link-lib=framework=CoreAudio"); let target_arch = std::env::var("CARGO_CFG_TARGET_ARCH").unwrap_or_default(); diff --git a/openless-all/app/src-tauri/src/accessibility_guide_macos.m b/openless-all/app/src-tauri/src/accessibility_guide_macos.m new file mode 100644 index 000000000..80083fa29 --- /dev/null +++ b/openless-all/app/src-tauri/src/accessibility_guide_macos.m @@ -0,0 +1,96 @@ +#import +#import + +@interface OLAccessibilityDragView : NSView +@property(nonatomic, strong) NSURL *applicationURL; +@end + +@implementation OLAccessibilityDragView +- (NSView *)hitTest:(NSPoint)point { + return [super hitTest:point] ? self : nil; +} +- (NSDragOperation)draggingSession:(NSDraggingSession *)session sourceOperationMaskForDraggingContext:(NSDraggingContext)context { + return NSDragOperationCopy; +} +- (void)mouseDragged:(NSEvent *)event { + if (!self.applicationURL) return; + NSDraggingItem *item = [[NSDraggingItem alloc] initWithPasteboardWriter:self.applicationURL]; + NSImage *icon = [[NSWorkspace sharedWorkspace] iconForFile:self.applicationURL.path]; + [item setDraggingFrame:NSMakeRect(0, 0, 48, 48) contents:icon]; + [self beginDraggingSessionWithItems:@[item] event:event source:self]; +} +@end + +static NSPanel *guidePanel; +static NSTimer *guideTimer; + +static NSString *OLGuideText(NSString *zh, NSString *en) { + return [[[NSLocale preferredLanguages] firstObject] hasPrefix:@"zh"] ? zh : en; +} + +static void OLCloseAccessibilityGuide(void) { + [guideTimer invalidate]; + guideTimer = nil; + [guidePanel close]; + guidePanel = nil; +} + +void openless_show_accessibility_guide(void) { + dispatch_async(dispatch_get_main_queue(), ^{ + if (AXIsProcessTrusted()) return; + NSURL *url = [NSBundle mainBundle].bundleURL; + if (![url.pathExtension.lowercaseString isEqualToString:@"app"]) return; + if (guidePanel) { + [guidePanel orderFrontRegardless]; + return; + } + NSScreen *screen = NSScreen.mainScreen; + if (!screen) return; + NSRect visible = screen.visibleFrame; + NSRect frame = NSMakeRect(NSMidX(visible) - 210, NSMinY(visible) + 20, 420, 138); + guidePanel = [[NSPanel alloc] initWithContentRect:frame + styleMask:NSWindowStyleMaskBorderless | NSWindowStyleMaskNonactivatingPanel + backing:NSBackingStoreBuffered defer:NO]; + guidePanel.level = NSFloatingWindowLevel; + guidePanel.floatingPanel = YES; + guidePanel.hidesOnDeactivate = NO; + guidePanel.releasedWhenClosed = NO; + guidePanel.opaque = NO; + guidePanel.backgroundColor = NSColor.clearColor; + guidePanel.hasShadow = YES; + guidePanel.collectionBehavior = NSWindowCollectionBehaviorCanJoinAllSpaces | NSWindowCollectionBehaviorFullScreenAuxiliary; + + NSView *content = guidePanel.contentView; + content.wantsLayer = YES; + content.layer.backgroundColor = NSColor.windowBackgroundColor.CGColor; + content.layer.cornerRadius = 14; + content.layer.masksToBounds = YES; + NSButton *close = [NSButton buttonWithImage:[NSImage imageWithSystemSymbolName:@"xmark" accessibilityDescription:OLGuideText(@"关闭", @"Close")] + target:guidePanel action:@selector(close)]; + close.frame = NSMakeRect(382, 104, 24, 24); + close.bordered = NO; + [content addSubview:close]; + NSTextField *hint = [NSTextField wrappingLabelWithString:OLGuideText(@"将 OpenLess 拖入辅助功能列表，然后打开开关。", @"Drag OpenLess into the Accessibility list, then turn on its switch.")]; + hint.frame = NSMakeRect(18, 88, 352, 36); + hint.font = [NSFont systemFontOfSize:13 weight:NSFontWeightMedium]; + [content addSubview:hint]; + + OLAccessibilityDragView *drag = [[OLAccessibilityDragView alloc] initWithFrame:NSMakeRect(16, 16, 388, 62)]; + drag.applicationURL = url; + drag.wantsLayer = YES; + drag.layer.backgroundColor = NSColor.controlBackgroundColor.CGColor; + drag.layer.cornerRadius = 10; + NSImageView *icon = [[NSImageView alloc] initWithFrame:NSMakeRect(12, 9, 44, 44)]; + icon.image = [[NSWorkspace sharedWorkspace] iconForFile:url.path]; + [drag addSubview:icon]; + NSTextField *label = [NSTextField labelWithString:@"OpenLess"]; + label.frame = NSMakeRect(68, 21, 280, 20); + label.font = [NSFont systemFontOfSize:14 weight:NSFontWeightSemibold]; + [drag addSubview:label]; + [content addSubview:drag]; + [guidePanel orderFrontRegardless]; + guideTimer = [NSTimer scheduledTimerWithTimeInterval:0.5 repeats:YES block:^(NSTimer *timer) { + if (AXIsProcessTrusted() || !guidePanel.visible) OLCloseAccessibilityGuide(); + }]; + }); +} diff --git a/openless-all/app/src-tauri/src/commands/dictionary.rs b/openless-all/app/src-tauri/src/commands/dictionary.rs index 52bf7ec08..cde0c136f 100644 --- a/openless-all/app/src-tauri/src/commands/dictionary.rs +++ b/openless-all/app/src-tauri/src/commands/dictionary.rs @@ -43,6 +43,44 @@ pub fn list_correction_rules(core: CoreState<'_>) -> Result, core.list_correction_rules().map_err(|e| e.to_string()) } +#[derive(Serialize)] +#[serde(rename_all = "camelCase")] +pub struct ServiceUsageSnapshot { + records: Vec, + dropped_or_failed: u64, +} + +#[tauri::command] +pub async fn get_service_usage(window: Window, core: CoreState<'_>) -> Result { + if window.label() != "main" { return Err("service usage is only available from the main window".into()); } + let recorder = core.services().usage_recorder.clone().ok_or("service usage is unavailable")?; + let (records, dropped_or_failed) = tauri::async_runtime::spawn_blocking(move || recorder.snapshot()) + .await.map_err(|_| "service usage read failed".to_string())? + .map_err(|error| error.to_string())?; + Ok(ServiceUsageSnapshot { records, dropped_or_failed }) +} + +#[tauri::command] +pub async fn clear_service_usage(window: Window, core: CoreState<'_>) -> Result<(), String> { + if window.label() != "main" { return Err("service usage is only available from the main window".into()); } + let recorder = core.services().usage_recorder.clone().ok_or("service usage is unavailable")?; + tauri::async_runtime::spawn_blocking(move || recorder.clear()) + .await.map_err(|_| "service usage clear failed".to_string())? + .map_err(|error| error.to_string()) +} + +#[tauri::command] +pub fn get_expression_profile(window: Window, core: CoreState<'_>) -> Result, String> { + if window.label() != "main" { return Err("expression profile is only available from the main window".into()); } + core.expression_profile_directions().map_err(|error| error.to_string()) +} + +#[tauri::command] +pub fn clear_expression_profile(window: Window, core: CoreState<'_>) -> Result<(), String> { + if window.label() != "main" { return Err("expression profile is only available from the main window".into()); } + core.clear_expression_profile().map_err(|error| error.to_string()) +} + #[tauri::command] pub fn add_correction_rule( core: CoreState<'_>, diff --git a/openless-all/app/src-tauri/src/commands/permissions_cmds.rs b/openless-all/app/src-tauri/src/commands/permissions_cmds.rs index 1b852af5e..b929e7b14 100644 --- a/openless-all/app/src-tauri/src/commands/permissions_cmds.rs +++ b/openless-all/app/src-tauri/src/commands/permissions_cmds.rs @@ -183,8 +183,15 @@ pub fn open_system_settings(pane: String) -> Result<(), String> { std::process::Command::new("open") .arg(url) .spawn() - .map(|_| ()) - .map_err(|e| e.to_string()) + .map_err(|e| e.to_string())?; + if pane == "accessibility" { + extern "C" { + fn openless_show_accessibility_guide(); + } + // The AppKit helper schedules its work on the main thread. + unsafe { openless_show_accessibility_guide() }; + } + Ok(()) } #[cfg(target_os = "windows")] { diff --git a/openless-all/app/src-tauri/src/coordinator.rs b/openless-all/app/src-tauri/src/coordinator.rs index 1aa9689bd..bbbbcc316 100644 --- a/openless-all/app/src-tauri/src/coordinator.rs +++ b/openless-all/app/src-tauri/src/coordinator.rs @@ -443,16 +443,29 @@ fn shared_backend_from_stores( preferences: prefs.core(), history: history.core(), activity: activity.core(), + expression_profile: openless_core::expression_observation::open_shared_expression_profile( + config.data_dir.join("expression-profile.json"), + ), vocabulary: vocab.core(), correction_rules: correction_rules.core(), style_packs: style_packs.core(), }; + #[cfg(not(mobile))] + let usage_recorder = match openless_core::usage_ledger::UsageLedger::open(config.data_dir.join("service-usage.json")) + .and_then(|ledger| openless_core::usage_ledger::UsageRecorder::start(Arc::new(ledger))) + { + Ok(recorder) => Some(Arc::new(recorder)), + Err(error) => { log::warn!("service usage unavailable: {error}"); None } + }; + #[cfg(mobile)] + let usage_recorder = None; let backend_slot = crate::core_adapters::backend_slot(); let mut dependencies = crate::core_adapters::backend_dependencies( app, Arc::clone(&backend_slot), native_asr, Arc::clone(&repositories.preferences), + usage_recorder, hotkey_status, qa_context, ); diff --git a/openless-all/app/src-tauri/src/core_adapters.rs b/openless-all/app/src-tauri/src/core_adapters.rs index ed357d2a2..4bd4d532f 100644 --- a/openless-all/app/src-tauri/src/core_adapters.rs +++ b/openless-all/app/src-tauri/src/core_adapters.rs @@ -92,6 +92,7 @@ pub(crate) fn backend_dependencies( backend: BackendSlot, native_asr_dependencies: TauriNativeAsrDependencies, preferences: Arc, + usage_recorder: Option>, hotkey_status: Arc>, qa_host_context: Arc, ) -> openless_core::BackendDependencies { @@ -251,9 +252,11 @@ pub(crate) fn backend_dependencies( ) .expect("Apple system speech provider ID"); let polisher = Arc::new(openless_core::TextPolisherRouter::default()); - let production_polisher: Arc = Arc::new( - openless_core::SharedCloudTextPolisher::new(Arc::clone(&credential_store)), - ); + let production_polisher = openless_core::SharedCloudTextPolisher::new(Arc::clone(&credential_store)); + let production_polisher: Arc = Arc::new(match &usage_recorder { + Some(recorder) => production_polisher.with_usage_recorder(Arc::clone(recorder)), + None => production_polisher, + }); for provider_type in openless_core::SHARED_CLOUD_LLM_PROVIDER_TYPES { polisher .register(*provider_type, Arc::clone(&production_polisher)) @@ -307,6 +310,7 @@ pub(crate) fn backend_dependencies( .expect("built-in Omni provider ids are non-empty"); } let mut dependencies = openless_core::BackendDependencies::unsupported(); + dependencies.services.usage_recorder = usage_recorder; if let Some(model_store) = model_store.clone() { dependencies.services.configure_model_store(model_store); } @@ -4120,25 +4124,8 @@ impl CoreTextInserter for TauriTextInserter { #[cfg(target_os = "windows")] let ime_target = crate::windows_ime_target::capture_ime_submit_target(); #[cfg(target_os = "macos")] - let live_capture = if self - .backend - .lock() - .as_ref() - .and_then(|b| b.upgrade()) - .is_some_and(|b| b.get_preferences().macos_composition_input) - { - let target = insertion_target.clone(); - Some(Arc::new(Mutex::new(Some( - tauri::async_runtime::spawn_blocking(move || { - if !crate::selection::selection_insertion_target_is_current(&target) { - return Err("live input target changed before capture".into()); - } - crate::host_document::LiveComposition::capture() - }), - )))) - } else { - None - }; + // Legacy macOS composition preferences must not start AX range capture. + let live_capture = None; Some(CapturedTextTarget { inserter: Arc::new(Self { insertion_target: Some(insertion_target.clone()), @@ -4913,7 +4900,8 @@ impl TauriTextInsertionSession { .as_ref() .and_then(|b| b.upgrade()) .is_some_and(|b| { - b.get_preferences().vocabulary_learning_enabled + let prefs = b.get_preferences(); + prefs.vocabulary_learning_enabled || prefs.expression_learning_enabled }) }, || { @@ -5000,17 +4988,6 @@ impl TauriTextInsertionSession { terminal .join_or_spawn( move || async move { - let live = Arc::clone(&self.live_composition); - let live_final = final_text.clone(); - let live_result = tauri::async_runtime::spawn_blocking(move || { - let mut guard = live.lock(); - guard.take().map(|mut live| match live_final { - Some(text) => live.replace(&text), - None => live.cancel(), - }) - }) - .await - .map_err(|e| BackendError::new(BackendErrorCode::Platform, e.to_string()))?; let worker = self.streaming_worker.clone(); let session = &self; finish_mac_insertion_after_barrier( @@ -5024,36 +5001,7 @@ impl TauriTextInsertionSession { if session.cancel_requested.load(Ordering::Acquire) { return Ok(MacInsertionCompletion::Cancelled); } - if let Some(result) = live_result { - return match result { - Ok(()) => Ok(MacInsertionCompletion::Finished( - InsertOutcome::Inserted, - )), - Err(_) => match final_text { - Some(text) => session - .copy_fallback(text) - .await - .map(MacInsertionCompletion::Finished), - None => Ok(MacInsertionCompletion::Cancelled), - }, - }; - } - if session.context.insertion.composition_input { - return match final_text { - Some(text) => session.copy_fallback(text).await.map(MacInsertionCompletion::Finished), - None => Ok(MacInsertionCompletion::Cancelled), - }; - } - match final_text { - Some(text) if !text.is_empty() => session - .insert_final(text) - .await - .map(MacInsertionCompletion::Finished), - Some(_) => { - Ok(MacInsertionCompletion::Finished(InsertOutcome::Inserted)) - } - None => Ok(MacInsertionCompletion::Cancelled), - } + finish_mac_final_text(final_text, |text| session.insert_final(text)).await }, || session.restore_platform_state(), ) @@ -5067,6 +5015,58 @@ impl TauriTextInsertionSession { } } +/// Core passes only the undelivered suffix here; an empty suffix is already delivered. +/// Final delivery must not probe AX composition support or replace earlier streamed text. +#[cfg(target_os = "macos")] +async fn finish_mac_final_text( + text: Option, + insert: impl FnOnce(String) -> F, +) -> Result +where + F: std::future::Future>, +{ + match text { + Some(text) if !text.is_empty() => insert(text).await.map(MacInsertionCompletion::Finished), + Some(_) => Ok(MacInsertionCompletion::Finished(InsertOutcome::Inserted)), + None => Ok(MacInsertionCompletion::Cancelled), + } +} + +#[cfg(target_os = "macos")] +fn replace_live_composition( + slot: &mut Option, + text: &str, +) -> Result<(), String> { + replace_unwritten_target( + slot, + text, + crate::host_document::LiveComposition::capture, + |live, value| live.replace(value), + |live| live.has_written(), + ) +} + +#[cfg(any(test, target_os = "macos"))] +fn replace_unwritten_target( + slot: &mut Option, + text: &str, + capture: impl FnOnce() -> Result, + mut replace: impl FnMut(&mut T, &str) -> Result<(), String>, + written: impl Fn(&T) -> bool, +) -> Result<(), String> { + if let Some(live) = slot.as_mut() { + match replace(live, text) { + Ok(()) => return Ok(()), + Err(error) if written(live) => return Err(error), + Err(_) => *slot = None, + } + } + let mut live = capture()?; + let result = replace(&mut live, text); + *slot = Some(live); + result +} + impl TextInsertionSession for TauriTextInsertionSession { fn supports_streaming(&self) -> bool { #[cfg(target_os = "macos")] @@ -5104,9 +5104,10 @@ impl TextInsertionSession for TauriTextInsertionSession { } tauri::async_runtime::spawn_blocking(move || { let mut live = session.live_composition.lock(); - live.as_mut() - .ok_or_else(|| "live input is unavailable in this editor".to_string())? - .replace(&text) + match replace_live_composition(&mut live, &text) { + Err(_) if live.as_ref().is_none_or(|target| !target.has_written()) => Ok(()), + result => result, + } }) .await .map_err(|e| BackendError::new(BackendErrorCode::Platform, e.to_string()))? @@ -5224,6 +5225,37 @@ mod mac_insertion_lifecycle_tests { tokio::spawn(task); } + #[tokio::test] + async fn final_text_uses_normal_delivery() { + let calls = Mutex::new(Vec::new()); + let result = finish_mac_final_text(Some("final text".to_string()), |text| async { + calls.lock().push(text); + Ok(InsertOutcome::Inserted) + }) + .await + .unwrap(); + assert_eq!(result, MacInsertionCompletion::Finished(InsertOutcome::Inserted)); + assert_eq!(*calls.lock(), ["final text"]); + } + + #[tokio::test] + async fn fully_streamed_text_finishes_without_an_ax_probe_or_second_insertion() { + let result = finish_mac_final_text(Some(String::new()), |_| async { + panic!("already delivered text must not invoke insertion or clipboard fallback") + }) + .await + .unwrap(); + assert_eq!(result, MacInsertionCompletion::Finished(InsertOutcome::Inserted)); + assert_eq!( + finish_mac_final_text(None, |_| async { + panic!("cancelled text must not invoke insertion") + }) + .await + .unwrap(), + MacInsertionCompletion::Cancelled + ); + } + #[tokio::test] async fn worker_start_failure_restores_the_previous_source_once() { let restored = Arc::new(Mutex::new(Vec::new())); @@ -5595,6 +5627,72 @@ fn map_tauri_error(error: impl std::fmt::Display) -> BackendError { #[cfg(test)] mod tests { + #[test] + fn live_input_retargets_only_before_any_write() { + #[derive(Debug)] + struct Target { + written: bool, + current: bool, + } + let mut slot = Some(Target { + written: false, + current: false, + }); + let capture = || Ok(Target { + written: false, + current: true, + }); + let replace = |target: &mut Target, _: &str| { + if !target.current { + return Err("target changed".into()); + } + target.written = true; + Ok(()) + }; + let written = |target: &Target| target.written; + super::replace_unwritten_target(&mut slot, "text", capture, replace, written).unwrap(); + assert!(slot.as_ref().unwrap().written); + slot.as_mut().unwrap().current = false; + let result = super::replace_unwritten_target( + &mut slot, + "more", + || panic!("a written target must never be replaced"), + replace, + written, + ); + assert_eq!(result.unwrap_err(), "target changed"); + } + + #[test] + fn live_input_keeps_target_when_write_outcome_is_uncertain() { + let mut slot = None; + let result = super::replace_unwritten_target( + &mut slot, + "text", + || Ok(false), + |written, _| { + *written = true; + Err("verification failed".into()) + }, + |written| *written, + ); + assert_eq!(result.unwrap_err(), "verification failed"); + assert_eq!(slot, Some(true)); + } + + #[cfg(target_os = "macos")] + #[test] + #[ignore = "requires the foreground disposable native-live-input-fixture app"] + fn live_input_captures_late_focused_native_editor() { + let (_, bundle) = crate::selection::current_front_app_parts(); + assert_eq!(bundle.as_deref(), Some("org.openless.live-input-fixture")); + let mut slot = None; + super::replace_live_composition(&mut slot, "实时原文").unwrap(); + let mut live = slot.expect("captured live editor"); + assert!(live.has_written()); + live.cancel().unwrap(); + } + use super::*; #[cfg(target_os = "windows")] diff --git a/openless-all/app/src-tauri/src/host_document/macos.rs b/openless-all/app/src-tauri/src/host_document/macos.rs index 62fb99fd8..8a2e2ad85 100644 --- a/openless-all/app/src-tauri/src/host_document/macos.rs +++ b/openless-all/app/src-tauri/src/host_document/macos.rs @@ -1669,8 +1669,23 @@ pub(crate) struct LiveComposition { } impl LiveComposition { + pub(crate) fn has_written(&self) -> bool { + self.wrote + } + pub(crate) fn capture() -> Result { + match crate::selection::current_front_app_pid() { + Some(pid) if pid != std::process::id() as i32 => {} + _ => return Err("liveInputTargetUnavailable".into()), + } let (_, _, target) = capture_voice_edit_target()?; + let mut owner_pid = 0; + if unsafe { AXUIElementGetPid(target.element.as_ref(), &mut owner_pid) } != AX_ERROR_SUCCESS + || owner_pid <= 0 + || owner_pid == std::process::id() as i32 + { + return Err("liveInputTargetUnavailable".into()); + } let start = utf16_offset_to_char_offset(&target.text, target.selection_utf16.location as usize); let end = utf16_offset_to_char_offset( @@ -1735,13 +1750,14 @@ impl LiveComposition { K_CF_STRING_ENCODING_UTF8, ); let attr = cfstring_from_static(b"AXSelectedText\0").ok_or("liveInputUnavailable")?; + // A failed AX call can have an unknown outcome; never retarget after submission. + self.wrote = true; let result = AXUIElementSetAttributeValue(self.target.element.as_ref(), attr, value); CFRelease(attr); CFRelease(value); if result != AX_ERROR_SUCCESS { return Err("liveInputUnsupportedEditor".into()); } - self.wrote = true; let expected = format!("{start}{text}{suffix}"); if voice_edit_text(self.target.element.as_ref()).as_deref() != Some(&expected) { return Err("liveInputEditorTransformedText".into()); diff --git a/openless-all/app/src-tauri/src/lib.rs b/openless-all/app/src-tauri/src/lib.rs index fd496f94e..c0c84b527 100644 --- a/openless-all/app/src-tauri/src/lib.rs +++ b/openless-all/app/src-tauri/src/lib.rs @@ -365,6 +365,10 @@ macro_rules! app_invoke_handler_desktop { commands::set_vocab_enabled, commands::update_vocab, commands::list_correction_rules, + commands::get_expression_profile, + commands::clear_expression_profile, + commands::get_service_usage, + commands::clear_service_usage, commands::add_correction_rule, commands::remove_correction_rule, commands::set_correction_rule_enabled, @@ -664,6 +668,10 @@ macro_rules! app_invoke_handler_mobile { $crate::commands::set_vocab_enabled, $crate::commands::update_vocab, $crate::commands::list_correction_rules, + $crate::commands::get_expression_profile, + $crate::commands::clear_expression_profile, + $crate::commands::get_service_usage, + $crate::commands::clear_service_usage, $crate::commands::add_correction_rule, $crate::commands::remove_correction_rule, $crate::commands::set_correction_rule_enabled, diff --git a/openless-all/app/src/components/CapsuleStyles.css b/openless-all/app/src/components/CapsuleStyles.css index 7e1db9b47..e0d7937ce 100644 --- a/openless-all/app/src/components/CapsuleStyles.css +++ b/openless-all/app/src/components/CapsuleStyles.css @@ -105,6 +105,12 @@ pointer-events: auto; max-width: 174px; } +.ol-select-popup.ol-capsule-language-popup { + background: var(--ol-select-popover-bg); + backdrop-filter: none; + -webkit-backdrop-filter: none; + transition: none; +} .ol-translation-language .ol-select-trigger { height: 22px; min-width: 92px; diff --git a/openless-all/app/src/components/CapsuleTranslationBadge.tsx b/openless-all/app/src/components/CapsuleTranslationBadge.tsx index 74d7d13f6..ac8093712 100644 --- a/openless-all/app/src/components/CapsuleTranslationBadge.tsx +++ b/openless-all/app/src/components/CapsuleTranslationBadge.tsx @@ -25,6 +25,7 @@ export function CapsuleTranslationBadge({ visible = true }: { visible?: boolean const [menuOpen, setMenuOpen] = useState(false); const menuEpoch = useRef(0); const menuOwner = useRef(null); + const opening = useRef(false); const closeTimer = useRef>(); const shown = useRef(visible); shown.current = visible; @@ -96,12 +97,19 @@ export function CapsuleTranslationBadge({ visible = true }: { visible?: boolean [i18n.language, target], ); const selected = languages.find((language) => language.nativeName === target); - const options = languages.filter( - (language) => - language.nativeName === target || - !isTranslationTargetRedundant(language.nativeName, snapshot?.workingLanguages ?? []), + const options = useMemo( + () => + languages + .filter( + (language) => + language.nativeName === target || + !isTranslationTargetRedundant(language.nativeName, snapshot?.workingLanguages ?? []), + ) + .map((language) => ({ value: language.nativeName, label: language.displayName })), + [languages, target, snapshot?.workingLanguages], ); const openMenu = async (next: boolean) => { + if (next && opening.current) return; const epoch = ++menuEpoch.current; if (closeTimer.current) clearTimeout(closeTimer.current); if (!next) { @@ -122,6 +130,7 @@ export function CapsuleTranslationBadge({ visible = true }: { visible?: boolean if (!shown.current || !active?.mutable) return; pickerSession.current = active.sessionId; menuOwner.current = active.sessionId; + opening.current = true; try { await setCapsuleLanguageMenu(true, active.sessionId); await new Promise((resolve) => @@ -139,6 +148,8 @@ export function CapsuleTranslationBadge({ visible = true }: { visible?: boolean } } catch { if (epoch === menuEpoch.current) setError(true); + } finally { + opening.current = false; } }; useEffect(() => { @@ -207,10 +218,7 @@ export function CapsuleTranslationBadge({ visible = true }: { visible?: boolean
({ - value: language.nativeName, - label: language.displayName, - }))} + options={options} placeholder={selected?.displayName || target || '…'} ariaLabel={t('translation.target.title')} disabled={!visible || busy || !snapshot?.mutable} @@ -219,6 +227,7 @@ export function CapsuleTranslationBadge({ visible = true }: { visible?: boolean onChange={(value) => void change(value)} side="top" popupStyle={{ width: 240 }} + popupClassName="ol-capsule-language-popup" />
{error && ( diff --git a/openless-all/app/src/components/FloatingShell.tsx b/openless-all/app/src/components/FloatingShell.tsx index 6520b5bf8..af87b16ad 100644 --- a/openless-all/app/src/components/FloatingShell.tsx +++ b/openless-all/app/src/components/FloatingShell.tsx @@ -21,6 +21,7 @@ import type { ServiceViewId } from '../pages/settings/navigation'; import { CloudSyncSetupPrompt } from './CloudSyncSetupPrompt'; import { Overview } from '../pages/Overview'; const History = lazy(() => import('../pages/History').then((m) => ({ default: m.History }))); +const Usage = lazy(() => import('../pages/Usage').then((m) => ({ default: m.Usage }))); const Vocab = lazy(() => import('../pages/Vocab').then((m) => ({ default: m.Vocab }))); const Style = lazy(() => import('../pages/Style').then((m) => ({ default: m.Style }))); const Marketplace = lazy(() => @@ -70,6 +71,7 @@ const SIDEBAR_WIDTH = 226; const PAGE_CMP: Record, ComponentType> = { overview: Overview, history: History, + usage: Usage, vocab: Vocab, style: Style, marketplace: Marketplace, @@ -150,7 +152,6 @@ function FloatingShellBody({ const [providerPromptOpen, setProviderPromptOpen] = useState(false); const [hotkeyModePromptOpen, setHotkeyModePromptOpen] = useState(false); const settingsMount = useExitMount(settingsOpen); - const [settingsPainted, setSettingsPainted] = useState(false); const providerPromptMount = useExitMount(providerPromptOpen); const hotkeyPromptMount = useExitMount(hotkeyModePromptOpen); const [moreOpen, setMoreOpen] = useState(false); @@ -308,7 +309,6 @@ function FloatingShellBody({
{t('shell.betaTag')} @@ -481,18 +481,21 @@ function FloatingShellBody({
- {/* Footer holds only the settings button. */} -
+
+
)} @@ -654,7 +676,6 @@ function FloatingShellBody({ key={`${settingsInitialSection ?? 'default'}:${settingsInitialServiceView ?? 'default'}`} os={os} closing={settingsMount.closing} - onPaintedChange={setSettingsPainted} initialSettingsSection={settingsInitialSection} initialServiceView={settingsInitialServiceView} onClose={() => setSettingsOpen(false)} @@ -789,7 +810,7 @@ function MobileTopBar({ - + {options.map((option) => ( @@ -104,6 +107,7 @@ export function SelectLite({ open: controlledOpen, side, popupStyle, + popupClassName, }} onOpenChange={handleOpenChange} /> @@ -124,6 +128,7 @@ function SearchableSelect({ open, side = 'bottom', popupStyle, + popupClassName, }: SelectLiteProps) { const [query, setQuery] = useState(''); const inputRef = useRef(null); @@ -181,7 +186,11 @@ function SearchableSelect({ sideOffset={8} collisionPadding={8} > - +
{ + return invokeOrMock('get_expression_profile', undefined, () => []); +} + +export function clearExpressionProfile(): Promise { + return invokeOrMock('clear_expression_profile', undefined, () => undefined); +} diff --git a/openless-all/app/src/lib/ipc/mock-data.ts b/openless-all/app/src/lib/ipc/mock-data.ts index 4af0e21c8..d6aa84490 100644 --- a/openless-all/app/src/lib/ipc/mock-data.ts +++ b/openless-all/app/src/lib/ipc/mock-data.ts @@ -139,6 +139,7 @@ export let mockSettings: UserPreferences = { macosCompositionInput: false, cursorContextEnabled: true, vocabularyLearningEnabled: false, + expressionLearningEnabled: false, vocabularyLearningSettings: { observationSeconds: 60, suggestionSeconds: 10, maxPhraseChars: 12 }, showOverviewActivityHeatmap: true, stackedRowLayout: false, diff --git a/openless-all/app/src/lib/ipc/service-usage.ts b/openless-all/app/src/lib/ipc/service-usage.ts new file mode 100644 index 000000000..6d27a244c --- /dev/null +++ b/openless-all/app/src/lib/ipc/service-usage.ts @@ -0,0 +1,38 @@ +import { invokeOrMock } from './shared'; + +export interface ServiceUsageRecord { + id: string; + occurredAtMs: number; + kind: 'asr' | 'llm' | 'tts' | 'multimodal'; + purpose: + | 'dictation' + | 'dictationPostprocess' + | 'translation' + | 'question' + | 'selection' + | 'quickNote' + | 'voiceEdit' + | 'cloudNote' + | 'other'; + outcome: 'succeeded' | 'failed' | 'cancelled' | 'unknown'; + provider: string | null; + channel: string | null; + model: string | null; + audioMs: number | null; + inputTokens: number | null; + outputTokens: number | null; + cachedInputTokens: number | null; + cacheWriteInputTokens: number | null; + reasoningTokens: number | null; + synthesizedCharacters: number | null; +} +export interface ServiceUsageSnapshot { + records: ServiceUsageRecord[]; + droppedOrFailed: number; +} +export function getServiceUsage(): Promise { + return invokeOrMock('get_service_usage', undefined, () => ({ records: [], droppedOrFailed: 0 })); +} +export function clearServiceUsage(): Promise { + return invokeOrMock('clear_service_usage', undefined, () => undefined); +} diff --git a/openless-all/app/src/lib/motion.ts b/openless-all/app/src/lib/motion.ts index 076b9fd91..99f461eef 100644 --- a/openless-all/app/src/lib/motion.ts +++ b/openless-all/app/src/lib/motion.ts @@ -234,11 +234,39 @@ export function useSelectionMotion( indicator.style.visibility = 'hidden'; return; } - // Layout offsets stay correct while the parent dialog is scaled on entry. - const y = underline ? group.clientHeight - 2 : button.offsetTop; - const target = `translate3d(${button.offsetLeft}px, ${y}px, 0)`; - const width = `${button.offsetWidth}px`; - const targetKey = `${target}:${width}`; + // Preserve fractional layout coordinates under CSS zoom and dialog transforms. + // offsetTop/offsetWidth round half-pixel padding and visibly uncenter the chip. + const outer = group.getBoundingClientRect(); + const inner = button.getBoundingClientRect(); + const style = getComputedStyle(group); + const borderX = parseFloat(style.borderLeftWidth) || 0; + const borderY = parseFloat(style.borderTopWidth) || 0; + const boxWidth = + parseFloat(style.width) + + (style.boxSizing === 'border-box' + ? 0 + : (parseFloat(style.paddingLeft) || 0) + + (parseFloat(style.paddingRight) || 0) + + borderX + + (parseFloat(style.borderRightWidth) || 0)); + const boxHeight = + parseFloat(style.height) + + (style.boxSizing === 'border-box' + ? 0 + : (parseFloat(style.paddingTop) || 0) + + (parseFloat(style.paddingBottom) || 0) + + borderY + + (parseFloat(style.borderBottomWidth) || 0)); + const scaleX = outer.width / boxWidth || 1; + const scaleY = outer.height / boxHeight || 1; + const x = (inner.left - outer.left) / scaleX - borderX; + const y = underline + ? boxHeight - borderY - (parseFloat(style.borderBottomWidth) || 0) - 2 + : (inner.top - outer.top) / scaleY - borderY; + const height = underline ? 2 : inner.height / scaleY; + const target = `translate3d(${x}px, ${y}px, 0)`; + const width = `${inner.width / scaleX}px`; + const targetKey = `${target}:${width}:${height}`; if (targetKey === lastTarget) return; const from = interrupted.current ?? { ...currentFrame(indicator), @@ -248,7 +276,7 @@ export function useSelectionMotion( motion?.cancel(); indicator.style.transform = target; indicator.style.width = width; - indicator.style.height = `${underline ? 2 : button.offsetHeight}px`; + indicator.style.height = `${height}px`; indicator.style.visibility = 'visible'; lastTarget = targetKey; if (moving && !reduced && !keyboardInteraction() && typeof indicator.animate === 'function') { diff --git a/openless-all/app/src/lib/serviceUsageMetrics.test.ts b/openless-all/app/src/lib/serviceUsageMetrics.test.ts new file mode 100644 index 000000000..c17b96308 --- /dev/null +++ b/openless-all/app/src/lib/serviceUsageMetrics.test.ts @@ -0,0 +1,98 @@ +import { + summarizeReportedUsage, + formatUsageNumber, + reportedCacheHitRate, +} from './serviceUsageMetrics'; +import type { ServiceUsageRecord } from './ipc/service-usage'; + +function equal(actual: unknown, expected: unknown, label: string) { + if (JSON.stringify(actual) !== JSON.stringify(expected)) throw new Error(label); +} +const make = (overrides: Partial): ServiceUsageRecord => ({ + id: 'id', + occurredAtMs: 100, + kind: 'llm', + purpose: 'dictationPostprocess', + outcome: 'unknown', + provider: 'example', + channel: 'channel', + model: 'model', + audioMs: null, + inputTokens: null, + outputTokens: null, + cachedInputTokens: null, + cacheWriteInputTokens: null, + reasoningTokens: null, + synthesizedCharacters: null, + ...overrides, +}); +const summary = summarizeReportedUsage( + [ + make({ id: 'a', inputTokens: 50, outputTokens: 0, cachedInputTokens: 20 }), + make({ id: 'b', purpose: 'translation', inputTokens: null, outputTokens: 7 }), + make({ id: 'c', occurredAtMs: 99, inputTokens: 999 }), + make({ id: 'd', occurredAtMs: 201, inputTokens: 999 }), + ], + 100, + 200, +); +equal(summary.totals, { input: 50, output: 7 }, 'totals use reported fields in range'); +equal( + summary.byPurpose, + [ + { purpose: 'dictationPostprocess', input: 50, output: 0 }, + { purpose: 'translation', input: null, output: 7 }, + ], + 'usage is grouped without adding cache counts', +); +equal( + summarizeReportedUsage([], 0, 200).totals, + { input: null, output: null }, + 'missing is unknown', +); +equal( + summarizeReportedUsage( + [make({ inputTokens: Number.MAX_SAFE_INTEGER }), make({ inputTokens: 1 })], + 0, + 200, + ).totals.input, + null, + 'unsafe sums are unknown', +); +console.log('reported service usage metrics tests passed'); + +for (const [value, expected] of [ + [null, '—'], + [-1, '—'], + [999, '999'], + [1000, '1K'], + [12500, '12.5K'], + [999999, '1M'], + [1250000, '1.3M'], + [2300000000, '2.3B'], + [Number.MAX_SAFE_INTEGER + 1, '—'], +] as const) { + equal(formatUsageNumber(value), expected, `compact number ${value}`); +} +equal( + reportedCacheHitRate([ + make({ provider: 'deepseek', inputTokens: 100, cachedInputTokens: 80 }), + make({ provider: 'openai', inputTokens: 300, cachedInputTokens: 0 }), + ]), + 0.2, + 'cache rate weights compatible requests by reported input', +); +for (const record of [ + make({ provider: 'anthropic', inputTokens: 100, cachedInputTokens: 80 }), + make({ provider: 'openai', inputTokens: 100, cachedInputTokens: null }), + make({ provider: 'openai', inputTokens: 0, cachedInputTokens: 0 }), + make({ provider: 'openai', inputTokens: 100, cachedInputTokens: 101 }), + make({ provider: 'openai', inputTokens: 100, cachedInputTokens: 80, cacheWriteInputTokens: 0 }), +]) { + equal( + reportedCacheHitRate([record]), + null, + 'unknown, additive or invalid cache reports do not create a rate', + ); +} +console.log('compact usage numbers and reported cache rate tests passed'); diff --git a/openless-all/app/src/lib/serviceUsageMetrics.ts b/openless-all/app/src/lib/serviceUsageMetrics.ts new file mode 100644 index 000000000..409497ac9 --- /dev/null +++ b/openless-all/app/src/lib/serviceUsageMetrics.ts @@ -0,0 +1,78 @@ +import type { ServiceUsageRecord } from './ipc/service-usage'; + +export interface ReportedUsageTotals { + input: number | null; + output: number | null; +} + +function sumKnown(values: Array): number | null { + const known = values.filter( + (value): value is number => value !== null && Number.isSafeInteger(value) && value >= 0, + ); + if (!known.length) return null; + const total = known.reduce((sum, value) => sum + value, 0); + return Number.isSafeInteger(total) ? total : null; +} + +export function summarizeReportedUsage( + records: ServiceUsageRecord[], + fromMs: number, + toMs: number, +) { + const rows = records.filter( + (record) => + Number.isSafeInteger(record.occurredAtMs) && + record.occurredAtMs >= fromMs && + record.occurredAtMs <= toMs, + ); + const summarize = (group: ServiceUsageRecord[]): ReportedUsageTotals => ({ + input: sumKnown(group.map((record) => record.inputTokens)), + output: sumKnown(group.map((record) => record.outputTokens)), + }); + const groups = new Map(); + for (const row of rows) { + const group = groups.get(row.purpose) ?? []; + group.push(row); + groups.set(row.purpose, group); + } + return { + rows, + totals: summarize(rows), + byPurpose: [...groups].map(([purpose, group]) => ({ purpose, ...summarize(group) })), + }; +} + +export function formatUsageNumber(value: number | null | undefined): string { + if (value == null || !Number.isSafeInteger(value) || value < 0) return '—'; + if (value < 1_000) return String(value); + const units = [ + [1_000_000_000, 'B'], + [1_000_000, 'M'], + [1_000, 'K'], + ] as const; + // Choose the unit after rounding so 999,999 becomes 1M, never 1000K. + for (let index = 0; index < units.length; index++) { + const [divisor, suffix] = units[index]; + if (value >= divisor || (index < units.length - 1 && value / divisor >= 0.99995)) { + return `${Number((value / divisor).toFixed(1))}${suffix}`; + } + } + return String(value); +} + +export function reportedCacheHitRate(records: ServiceUsageRecord[]): number | null { + const compatible = records.filter( + (record) => + ['openai', 'deepseek'].includes(record.provider?.toLowerCase() ?? '') && + record.cacheWriteInputTokens == null && + Number.isSafeInteger(record.inputTokens) && + record.inputTokens! > 0 && + Number.isSafeInteger(record.cachedInputTokens) && + record.cachedInputTokens! >= 0 && + record.cachedInputTokens! <= record.inputTokens!, + ); + if (!compatible.length) return null; + const input = sumKnown(compatible.map((record) => record.inputTokens)); + const cached = sumKnown(compatible.map((record) => record.cachedInputTokens)); + return input !== null && input > 0 && cached !== null ? cached / input : null; +} diff --git a/openless-all/app/src/lib/types.ts b/openless-all/app/src/lib/types.ts index 6b5e4ec50..21e2292e6 100644 --- a/openless-all/app/src/lib/types.ts +++ b/openless-all/app/src/lib/types.ts @@ -575,6 +575,7 @@ export interface UserPreferences { * managers / terminals are always hard-blocked. */ cursorContextEnabled: boolean; vocabularyLearningEnabled: boolean; + expressionLearningEnabled: boolean; vocabularyLearningSettings: { observationSeconds: number; suggestionSeconds: number; diff --git a/openless-all/app/src/pages/Overview.tsx b/openless-all/app/src/pages/Overview.tsx index 5dcb0e918..d1d1aae0c 100644 --- a/openless-all/app/src/pages/Overview.tsx +++ b/openless-all/app/src/pages/Overview.tsx @@ -777,6 +777,7 @@ function SegmentedToggle({ }) { return (
({ padding: 2, borderRadius: 'var(--ol-control-radius)', background: 'var(--ol-surface-2)', - border: '0.5px solid var(--ol-line)', + boxShadow: 'inset 0 0 0 0.5px var(--ol-line)', }} > {options.map((option) => { diff --git a/openless-all/app/src/pages/Usage.tsx b/openless-all/app/src/pages/Usage.tsx new file mode 100644 index 000000000..aa2031557 --- /dev/null +++ b/openless-all/app/src/pages/Usage.tsx @@ -0,0 +1,495 @@ +import { useCallback, useEffect, useMemo, useRef, useState } from 'react'; +import { useTranslation } from 'react-i18next'; +import { listHistory } from '../lib/ipc/history'; +import { + clearServiceUsage, + getServiceUsage, + type ServiceUsageSnapshot, +} from '../lib/ipc/service-usage'; +import type { DictationSession } from '../lib/types'; +import { + summarizeReportedUsage, + formatUsageNumber, + reportedCacheHitRate, +} from '../lib/serviceUsageMetrics'; +import { Segmented } from '../components/ui/Segmented'; +import './usage.css'; +import { Btn, Card, PageHeader } from './_atoms'; + +export function Usage() { + const { i18n } = useTranslation(); + const zh = i18n.resolvedLanguage?.startsWith('zh') ?? false; + const text = (cn: string, en: string) => (zh ? cn : en); + const [history, setHistory] = useState(null); + const [error, setError] = useState(false); + const [days, setDays] = useState(30); + const [view, setView] = useState<'overview' | 'services' | 'history'>('overview'); + const [usage, setUsage] = useState(null); + const [usageError, setUsageError] = useState(false); + const [clearError, setClearError] = useState(false); + const [clearing, setClearing] = useState(false); + const mounted = useRef(false); + const clearingRef = useRef(false); + const request = useRef(0); + const refresh = useCallback(async () => { + if (clearingRef.current) return; + const id = ++request.current; + const [historyResult, usageResult] = await Promise.allSettled([ + listHistory(), + getServiceUsage(), + ]); + if (!mounted.current || request.current !== id) return; + if (historyResult.status === 'fulfilled') { + setHistory(historyResult.value); + setError(false); + } else { + setHistory(null); + setError(true); + } + if (usageResult.status === 'fulfilled') { + setUsage(usageResult.value); + setUsageError(false); + setClearError(false); + } else { + setUsage(null); + setUsageError(true); + } + }, []); + useEffect(() => { + mounted.current = true; + void refresh(); + const onFocus = () => void refresh(); + window.addEventListener('focus', onFocus); + return () => { + mounted.current = false; + request.current += 1; + window.removeEventListener('focus', onFocus); + }; + }, [refresh]); + const rows = useMemo(() => { + const cutoff = Date.now() - days * 86_400_000; + return (history ?? []).filter( + (row) => + (!row.source || row.source === 'voice') && new Date(row.createdAt).getTime() >= cutoff, + ); + }, [history, days]); + const asr = rows.filter((row) => row.pipelineMode !== 'multimodal' && row.asrProvider); + const llm = rows.filter((row) => row.pipelineMode !== 'multimodal' && row.llmProvider); + const omni = rows.filter((row) => row.pipelineMode === 'multimodal' && row.llmProvider); + const now = Date.now(); + const { + rows: reported, + totals, + byPurpose, + } = summarizeReportedUsage(usage?.records ?? [], now - days * 86_400_000, now); + const purposeName = (purpose: string) => { + const labels: Record = { + dictationPostprocess: ['听写后处理', 'Dictation processing'], + translation: ['翻译', 'Translation'], + selection: ['选区处理', 'Selection'], + question: ['问答', 'Questions'], + quickNote: ['速记', 'Quick note'], + voiceEdit: ['语音编辑', 'Voice editing'], + cloudNote: ['云笔记', 'Cloud note'], + }; + const label = labels[purpose]; + return label ? text(...label) : text('其他', 'Other'); + }; + const safeTokens = (value: number | null) => ( + = 0 + ? value.toLocaleString() + : undefined + } + > + {formatUsageNumber(value)} + + ); + const cacheRate = reportedCacheHitRate(reported); + const seconds = rows.reduce((total, row) => total + (row.durationMs ?? 0), 0) / 1000; + const stats: Array<[string, string, string?]> = [ + [text('语音识别记录', 'ASR records'), formatUsageNumber(asr.length), String(asr.length)], + [ + text('文字处理记录', 'Text processing records'), + formatUsageNumber(llm.length), + String(llm.length), + ], + [text('多模态记录', 'Multimodal records'), formatUsageNumber(omni.length), String(omni.length)], + [text('TTS 用量', 'TTS usage'), '—'], + [ + text('已报告输入 Token', 'Reported input tokens'), + formatUsageNumber(totals.input), + totals.input?.toLocaleString(), + ], + [ + text('已报告输出 Token', 'Reported output tokens'), + formatUsageNumber(totals.output), + totals.output?.toLocaleString(), + ], + [ + text('缓存命中率', 'Cache hit rate'), + cacheRate === null ? '—' : `${(cacheRate * 100).toFixed(1)}%`, + text('可比较记录的缓存命中率', 'Cache hit rate for comparable reported calls'), + ], + [ + text('录音时长', 'Recording duration'), + rows.some((row) => row.durationMs == null) + ? '—' + : `${Math.round(seconds / 60)} ${text('分钟', 'min')}`, + ], + ]; + const grouped = new Map(); + for (const row of rows) { + const stages = + row.pipelineMode === 'multimodal' + ? [[text('多模态', 'Multimodal'), row.llmProvider, row.llmModel]] + : [ + [text('语音识别', 'ASR'), row.asrProvider, row.asrModel], + [ + row.translationActive + ? text('翻译', 'Translation') + : text('听写后处理', 'Dictation processing'), + row.llmProvider, + row.llmModel, + ], + ]; + for (const [stage, provider, model] of stages) { + if (!provider) continue; + const key = [stage, provider, model].filter(Boolean).join(' · '); + grouped.set(key, (grouped.get(key) ?? 0) + 1); + } + } + const daily = Array.from({ length: days }, (_, index) => { + const date = new Date(); + date.setHours(0, 0, 0, 0); + date.setDate(date.getDate() - (days - index - 1)); + const end = new Date(date); + end.setDate(end.getDate() + 1); + return { + date, + count: rows.filter((row) => { + const time = new Date(row.createdAt).getTime(); + return time >= date.getTime() && time < end.getTime(); + }).length, + }; + }); + const maxDaily = Math.max(1, ...daily.map((day) => day.count)); + const clearControl = view === 'history' && usage && ( + { + clearingRef.current = true; + request.current += 1; + setClearing(true); + setClearError(false); + try { + await clearServiceUsage(); + if (mounted.current) setUsage({ records: [], droppedOrFailed: usage.droppedOrFailed }); + } catch { + if (mounted.current) setClearError(true); + } finally { + clearingRef.current = false; + if (mounted.current) setClearing(false); + } + }} + > + {text('清除已记录 Token', 'Clear recorded tokens')} + + ); + return ( +
+ void refresh()}> + {text('刷新', 'Refresh')} + + } + /> + +
+ setDays(Number(value))} + /> + {clearControl} +
+ {usageError && ( +

+ {text('无法读取本地 Token 账本。', 'Could not read the local token ledger.')} +

+ )} + {clearError && ( +

+ {text( + '清除结果未确认，请刷新后检查。', + 'The clear result is unconfirmed. Refresh to check.', + )} +

+ )} + {usage && usage.droppedOrFailed > 0 && ( +

+ {text( + '本次运行中部分用量未能保存，统计不完整。', + 'Some usage could not be saved during this run.', + )} +

+ )} + {error ? ( +

{text('无法读取历史记录。', 'Could not load history.')}

+ ) : history === null ? ( +

{text('加载中…', 'Loading…')}

+ ) : ( + <> + {view === 'overview' && ( + <> +
+ {stats.map(([label, value, exact]) => ( + +
{label}
+
+ {value} +
+ + ))} +
+ +

+ {text('每日记录', 'Daily records')} +

+
+ {daily.map(({ date, count }) => ( +
+ ))} +
+
+ + {daily[0].date.toLocaleDateString(undefined, { + month: 'short', + day: 'numeric', + })} + + + {daily[daily.length - 1].date.toLocaleDateString(undefined, { + month: 'short', + day: 'numeric', + })} + +
+ + + )} + {view === 'services' && ( + +

+ {text('服务与用途', 'Services and purposes')} +

+ {grouped.size === 0 ? ( +

{text('没有可归属服务的记录。', 'No attributable records.')}

+ ) : ( + [...grouped.entries()] + .sort((a, b) => b[1] - a[1]) + .map(([name, count]) => ( +
+ {name} + {count} +
+ )) + )} + + )} + {view === 'history' && ( +
+ +

+ {text('已报告的模型用量', 'Reported model usage')} +

+

+ {text( + '缓存和推理字段由各服务商报告，可能已包含在输入或输出中，不能直接相加。', + 'Provider-reported cache and reasoning fields may already be included in input or output; do not add them directly.', + )} +

+
+ {byPurpose.map(({ purpose, input, output }) => ( +
+ {purposeName(purpose)} + + {text('已报告输入', 'Reported input')}: {safeTokens(input)} ·{' '} + {text('已报告输出', 'Reported output')}: {safeTokens(output)} + +
+ ))} + {reported.length === 0 ? ( +

+ {text( + '该时段没有已保存的服务商 token 报告。', + 'No provider token reports were saved for this period.', + )} +

+ ) : ( + [...reported] + .sort((a, b) => b.occurredAtMs - a.occurredAtMs) + .slice(0, 30) + .map((record) => ( +
+ + {purposeName(record.purpose)} + + {[ + ...new Set( + [record.provider, record.channel, record.model].filter(Boolean), + ), + ].join(' · ') || '—'} + + + {text('输入', 'Input')}: {safeTokens(record.inputTokens)} ·{' '} + {text('输出', 'Output')}: {safeTokens(record.outputTokens)} +
+ {text('缓存读取', 'Cache read')}: {safeTokens(record.cachedInputTokens)}{' '} + · {text('缓存写入', 'Cache write')}:{' '} + {safeTokens(record.cacheWriteInputTokens)} +
+ {text('推理', 'Reasoning')}: {safeTokens(record.reasoningTokens)} + +
+ )) + )} +
+ + +

+ {text('最近记录', 'Recent records')} +

+
+ {rows.map((row) => ( +
+ + + {[row.asrProvider, row.llmProvider].filter(Boolean).join(' · ') || '—'} + +
+ ))} +
+ +
+ )} + + )} +
+ ); +} diff --git a/openless-all/app/src/pages/Vocab.tsx b/openless-all/app/src/pages/Vocab.tsx index efb29e8f4..bda6c5731 100644 --- a/openless-all/app/src/pages/Vocab.tsx +++ b/openless-all/app/src/pages/Vocab.tsx @@ -610,7 +610,7 @@ export function Vocab() { icon={wordAdded ? 'check' : 'plus'} onClick={() => void addNewWord()} style={{ - borderRadius: 'var(--ol-radius)', + borderRadius: 8, height: 38, minWidth: 92, justifyContent: 'center', diff --git a/openless-all/app/src/pages/_atoms.tsx b/openless-all/app/src/pages/_atoms.tsx index 21634806a..6844767d4 100644 --- a/openless-all/app/src/pages/_atoms.tsx +++ b/openless-all/app/src/pages/_atoms.tsx @@ -242,7 +242,7 @@ export function Btn({ const muted = disabled && (variant === 'primary' || variant === 'blue'); return ( +

+ {t('settings.expressionLearning.clearHint')} +

+ {error &&

{t('settings.expressionLearning.error')}

} + + ); +} diff --git a/openless-all/app/src/pages/settings/RecordingInputSection.tsx b/openless-all/app/src/pages/settings/RecordingInputSection.tsx index f866cc2fc..a4260d22f 100644 --- a/openless-all/app/src/pages/settings/RecordingInputSection.tsx +++ b/openless-all/app/src/pages/settings/RecordingInputSection.tsx @@ -900,28 +900,6 @@ export function RecordingInputSection({ onToggle={(next) => void savePrefs({ ...prefs, streamingInsertSaveClipboard: next })} /> - {capability.adapter === 'macEventTap' && ( - - void savePrefs({ ...prefs, macosCompositionInput: next })} - /> - - )} - {capability.adapter === 'windowsLowLevel' && prefs.windowsInsertionMode === 'tsf' && ( - - void savePrefs({ ...prefs, windowsCompositionInput: next })} - /> - - )} )} {surface === 'text' && showDesktopStartup && ( diff --git a/openless-all/app/src/pages/settings/navigation.test.ts b/openless-all/app/src/pages/settings/navigation.test.ts index 8e9ff19b6..344c56076 100644 --- a/openless-all/app/src/pages/settings/navigation.test.ts +++ b/openless-all/app/src/pages/settings/navigation.test.ts @@ -94,14 +94,7 @@ assert.equal( ); assert.deepEqual( visibleAdvancedPages('android', 'android').map((item) => item.id), - [ - 'voiceEnhancement', - 'cursorContext', - 'startupDiagnostics', - 'vocabularyLearning', - 'customSyncServer', - 'debug', - ], + ['voiceEnhancement', 'cursorContext', 'startupDiagnostics', 'customSyncServer', 'debug'], 'the pipeline switch lives in Services on every platform', ); console.log('settings navigation tests passed'); @@ -127,8 +120,29 @@ assert.equal( const advancedIds: readonly string[] = ADVANCED_PAGES.map((item) => item.id); assert.equal( advancedIds.includes('vocabularyLearning'), + false, + 'correction learning shares the awareness page rather than a duplicate menu entry', +); + +assert.equal( + advancedIds.includes('cursorContext'), + true, + 'awareness remains reachable from advanced features', +); +assert.equal( + visibleAdvancedPages('desktop', 'mac').some((item) => item.id === 'liveInput'), + false, + 'retired macOS in-field streaming is not offered', +); +assert.equal( + visibleAdvancedPages('desktop', 'win').some((item) => item.id === 'liveInput'), true, - 'manual correction learning remains reachable from advanced features', + 'Windows retains its advanced text insertion page', +); +assert.equal( + visibleAdvancedPages('android', 'android').some((item) => item.id === 'liveInput'), + false, + 'Windows text insertion is not offered to Android', ); const sectionIds = SETTINGS_SECTIONS.map((item) => item.id); diff --git a/openless-all/app/src/pages/settings/navigation.ts b/openless-all/app/src/pages/settings/navigation.ts index 525fc4adb..908df3305 100644 --- a/openless-all/app/src/pages/settings/navigation.ts +++ b/openless-all/app/src/pages/settings/navigation.ts @@ -20,6 +20,12 @@ export const ADVANCED_PAGES = [ titleKey: 'settings.advancedPages.voiceEnhancementTitle', beta: true, }, + { + id: 'liveInput', + icon: 'keyboard', + titleKey: 'settings.advancedPages.liveInputTitle', + beta: true, + }, { id: 'cursorContext', icon: 'search', @@ -31,12 +37,6 @@ export const ADVANCED_PAGES = [ icon: 'bolt', titleKey: 'settings.advancedPages.startupDiagnosticsTitle', }, - { - id: 'vocabularyLearning', - icon: 'pencil', - titleKey: 'settings.vocabularyLearning.title', - beta: true, - }, { id: 'customSyncServer', icon: 'cloud', titleKey: 'cloudSyncE2ee.customServerTitle' }, { id: 'debug', icon: 'bolt', titleKey: 'settings.debug.title' }, ] as const; @@ -46,8 +46,8 @@ export type AdvancedPageId = AdvancedPage['id']; export function visibleAdvancedPages(platform: PlatformKind | undefined, os: OS): AdvancedPage[] { return ADVANCED_PAGES.filter((page) => { + if (page.id === 'liveInput') return platform === 'desktop' && os === 'win'; if (page.id === 'localModels') return platform === 'desktop' && os === 'mac'; - if (page.id === 'vocabularyLearning') return platform === 'desktop' || platform === 'android'; if (page.id === 'debug') return platform === 'desktop' || platform === 'android' || platform === 'harmony'; return true; diff --git a/openless-all/app/src/pages/settings/tabs.tsx b/openless-all/app/src/pages/settings/tabs.tsx index 63e33c700..0cf2727a7 100644 --- a/openless-all/app/src/pages/settings/tabs.tsx +++ b/openless-all/app/src/pages/settings/tabs.tsx @@ -1,4 +1,3 @@ -import { VocabularyLearningSection } from './VocabularyLearningSection'; // Content page for each sidebar tab in the settings dialog. Each tab is a vertical stack of section cards; // the real logic lives in the *Section files — this only decides which sections belong to which tab. @@ -27,6 +26,8 @@ import { LocalMigrationSection } from './LocalMigrationSection'; import { LocalModelsSection } from './models/LocalModelsSection'; import { LocalModelsNavContext } from './models/modelsNav'; import { DebugToolsSection } from './DebugToolsSection'; +import { VocabularyLearningSection } from './VocabularyLearningSection'; +import { ExpressionLearningSection } from './ExpressionLearningSection'; import { AudioDspSettings } from './AudioDspSettings'; import { BetaChannelSection } from './BetaChannelSection'; import { AutoUpdateSection } from './AutoUpdateSection'; @@ -617,6 +618,31 @@ function CursorContextSection() { ); } +function LiveInputSection() { + const { t } = useTranslation(); + const { prefs, updatePrefs } = useHotkeySettings(); + if (!prefs || detectOS() !== 'win') return null; + return ( + + + {t('settings.advancedPages.liveInputTitle')} + + + + void updatePrefs((current) => ({ ...current, windowsCompositionInput: next })) + } + /> + + + ); +} + // Privacy: local-first explanation + permission management · data storage. export function PrivacyTab() { const { t } = useTranslation(); @@ -672,6 +698,7 @@ export function AdvancedTab({ onOpenAsr: () => Promise; }) { const { t } = useTranslation(); + const platformCaps = usePlatformCaps(); const entries = pages; const list = (
@@ -705,8 +732,20 @@ export function AdvancedTab({ )} {item.id === 'voiceEnhancement' && } - {item.id === 'cursorContext' && } - {item.id === 'vocabularyLearning' && } + {item.id === 'liveInput' && } + {item.id === 'cursorContext' && ( +
+ + {showsCursorContextControl(platformCaps?.platform, detectOS()) && ( + + )} + {platformCaps?.platform === 'desktop' && + (detectOS() === 'mac' || detectOS() === 'win') && } +
+ )} {item.id === 'startupDiagnostics' && } {item.id === 'customSyncServer' && page === item.id && } {item.id === 'debug' && } diff --git a/openless-all/app/src/pages/usage.css b/openless-all/app/src/pages/usage.css new file mode 100644 index 000000000..68ec3c387 --- /dev/null +++ b/openless-all/app/src/pages/usage.css @@ -0,0 +1,51 @@ +.ol-usage-toolbar { + display: flex; + align-items: center; + justify-content: space-between; + gap: 12px; + margin: 8px 0 16px; +} +.ol-usage-chart { + display: flex; + align-items: flex-end; + gap: 4px; + height: clamp(120px, calc(100dvh - 650px), 240px); +} +.ol-usage-chart-dates { + display: flex; + justify-content: space-between; + margin-top: 10px; + color: var(--ol-ink-3); + font-size: 12px; +} +.ol-usage-history-grid { + display: grid; + grid-template-columns: repeat(2, minmax(0, 1fr)); + gap: 14px; +} +.ol-usage-record-card { + display: flex; + flex-direction: column; + min-width: 0; + height: clamp(260px, calc(100dvh - 320px), 650px); + overflow: hidden; +} +.ol-usage-record-card > h2, +.ol-usage-record-card > p { + flex-shrink: 0; +} +.ol-usage-record-scroll { + min-height: 0; + flex: 1; + overflow: auto; + overscroll-behavior: contain; + overflow-wrap: anywhere; +} +@media (max-width: 1000px) { + .ol-usage-history-grid { + grid-template-columns: minmax(0, 1fr); + } + .ol-usage-record-card { + height: clamp(240px, 55dvh, 480px); + } +} diff --git a/openless-all/app/src/state/useAppState.ts b/openless-all/app/src/state/useAppState.ts index e9e9bcd1b..268d1f5d2 100644 --- a/openless-all/app/src/state/useAppState.ts +++ b/openless-all/app/src/state/useAppState.ts @@ -5,6 +5,7 @@ import { useState } from 'react'; export type AppTab = | 'overview' | 'history' + | 'usage' | 'vocab' | 'style' | 'marketplace' diff --git a/openless-all/app/src/styles/global.css b/openless-all/app/src/styles/global.css index a0d34cd0e..d9993505d 100644 --- a/openless-all/app/src/styles/global.css +++ b/openless-all/app/src/styles/global.css @@ -1083,6 +1083,9 @@ html[data-ol-conservative-layout='true'] .ol-conservative-scope .ol-select-lite- /* Segmented control (components/ui/Segmented), shared by every segmented choice in Settings: a white track with the controls' hairline, a neutral chip that slides under the selected option (useSelectionMotion), labels in the selection language. */ +.ol-overview-segment > button { + --ol-corner-radius: calc(var(--ol-control-radius, 8px) - 2px); +} .ol-segmented { position: relative; isolation: isolate; @@ -1090,12 +1093,24 @@ html[data-ol-conservative-layout='true'] .ol-conservative-scope .ol-select-lite- flex-wrap: wrap; gap: 2px; max-width: 100%; - padding: 2.5px; + padding: 3px; border: 0; box-shadow: inset 0 0 0 0.5px var(--ol-line-strong); border-radius: var(--ol-control-radius); background: var(--ol-surface); } +.ol-service-pipeline-track { + padding: 3px; + border: 0; + box-shadow: inset 0 0 0 0.5px var(--ol-line-strong); +} +.ol-segmented > :is(.ol-segmented-indicator, .ol-segmented-option) { + --ol-corner-radius: calc(var(--ol-control-radius, 8px) - 3px); + border-radius: var(--ol-corner-radius); +} +.ol-action-button { + --ol-corner-radius: 8px; +} .ol-segmented-indicator, .ol-service-view-indicator { position: absolute; diff --git a/openless-all/app/src/styles/refinements.css b/openless-all/app/src/styles/refinements.css index 48959e746..ff0c067ec 100644 --- a/openless-all/app/src/styles/refinements.css +++ b/openless-all/app/src/styles/refinements.css @@ -150,17 +150,21 @@ button:disabled { min-width: 0; height: 36px; padding: 0 11px; - border: 1px solid var(--ol-line-strong); + border: 0; border-radius: var(--ol-control-radius); color: var(--ol-ink); - background: var(--ol-surface); + background: var(--ol-surface-2); + box-shadow: inset 0 0 0 1px var(--ol-line-strong); font: inherit; font-size: 13px; } .ol-settings-search-field:focus-visible { outline: 2px solid var(--ol-blue-ring); outline-offset: 1px; - border-color: var(--ol-blue); + box-shadow: inset 0 0 0 1px var(--ol-blue); +} +.ol-expression-clear { + --ol-corner-radius: 8px; } .ol-shortcuts-toolbar { display: flex;