diff --git a/.github/i18n/README.ar.md b/.github/i18n/README.ar.md index 2457966a..a7d2da06 100644 --- a/.github/i18n/README.ar.md +++ b/.github/i18n/README.ar.md @@ -276,6 +276,38 @@ memory({ mode: "import", inputPath: "./memories.json" }); `chatMessage.maxMemories` على `0` إلى تعطيل حقن ذكريات المشروع، مع استمرار إمكانية حقن ملف المستخدم عند تفعيله. +### Update-on-capture (اختياري) + +مسار اختياري لإبقاء الذكريات حديثة عند تغيّر الحقائق. عند +`updateOnCaptureEnabled: true` يضمّن الالتقاط التلقائي الملخص الجديد، ويبحث عن +ذكريات مشابهة فوق `updateOnCaptureSimilarityThreshold` (الافتراضي `0.75`)، ويترك +لنموذج الالتقاط الاختيار بين `add` أو `update` (إعادة كتابة في المكان) أو +`supersede` (حفظ الجديد وحذف القديم soft-delete إلى سلة المهملات). تُتخطى الذكريات +المثبّتة. الافتراضي معطّل ليبقى السلوك السابق add-only. + +```jsonc +"updateOnCaptureEnabled": true, +"updateOnCaptureSimilarityThreshold": 0.75, +"updateOnCaptureMaxCandidates": 5, +``` + +ملاحظة: مع `Xenova/nomic-embed-text-v1` غالبًا ما يحسّن +`embeddingUseTaskPrefixes: true` (وخفض `similarityThreshold` للاستدعاء عند الحاجة) +جودة المطابقة للذكريات الأطول. تبقى البادئات opt-in افتراضيًا. + +### Dreaming (توحيد ذاكرة اختياري) + +تمريرة خلفية اختيارية تجد الذكريات **ذات الصلة** عبر +`dreaming.relatedThreshold` (الافتراضي `0.75`، مستقل عن +`deduplicationSimilarityThreshold`)، وتطلب من `opencodeProvider` / +`opencodeModel` تنفيذ `merge` / `update` / `supersede` / `keep` / `flag` +(تناقضات)، وتحذف الصفوف المستبدلة soft-delete إلى سلة قابلة للاستعادة. الذكريات +المثبّتة لا تُمس. مع `dreaming.crossProject: true` (أو +`memory.defaultScope: "all-projects"`) تُجمَّع عبر وسوم المشاريع؛ نطاق المستخدم +يبقى منفصلًا. الإعداد تحت `dreaming` في `opencode-mem.jsonc` (`enabled` افتراضي +`false`). واجهة الويب تعرض **Trash** و**Dream**. تُحذف الصفوف soft-deleted نهائيًا +بعد `dreaming.trashRetentionDays` أثناء التنظيف. + ### اختيار / إعداد التضمينات تشغّل التضمينات البحث بالتشابه للذكريات وملف المستخدم. أعدها في نفس الملف (`~/.config/opencode/opencode-mem.jsonc`). **لا توجد خلفية MLX** — التضمينات المحلية تستخدم `@huggingface/transformers` مع ONNX، وليس Apple MLX. diff --git a/.github/i18n/README.de.md b/.github/i18n/README.de.md index 7561ed56..995977db 100644 --- a/.github/i18n/README.de.md +++ b/.github/i18n/README.de.md @@ -279,14 +279,36 @@ Memories verfügbar wären. Mit `chatMessage.maxMemories` auf `0` wird die Injek von Projekt-Memories deaktiviert; das Nutzerprofil kann weiterhin injiziert werden, wenn es aktiviert ist. +### Update-on-capture (optional) + +Opt-in-Pfad, der Memories aktuell hält, wenn sich Fakten ändern. Bei +`updateOnCaptureEnabled: true` bettet Auto-Capture die neue Summary ein, sucht ähnliche +Memories oberhalb von `updateOnCaptureSimilarityThreshold` (Standard `0.75`) und lässt das +Capture-Modell zwischen `add`, `update` (in-place überschreiben) und `supersede` (neu +speichern + alt soft-deleten in den Papierkorb) wählen. Gepinnte Memories werden übersprungen. +Standard ist aus, damit bestehende Installationen add-only bleiben. + +```jsonc +"updateOnCaptureEnabled": true, +"updateOnCaptureSimilarityThreshold": 0.75, +"updateOnCaptureMaxCandidates": 5, +``` + +Tipp: Bei `Xenova/nomic-embed-text-v1` verbessert `embeddingUseTaskPrefixes: true` (und ggf. +ein niedrigeres Recall-`similarityThreshold`) oft die Trefferqualität bei längeren Memories. +Prefixes bleiben standardmäßig opt-in. + ### Dreaming (optionale Memory-Konsolidierung) -Opt-in-Hintergrundlauf, der Near-Duplikate findet (gleicher Similarity-Schwellwert wie -Deduplizierung), sie über `opencodeProvider`/`opencodeModel` zusammenführt und ersetzte -Einträge soft-deleten (wiederherstellbarer Papierkorb). Gepinnte Memories bleiben unberührt. -Konfiguration unter `dreaming` in `opencode-mem.jsonc` (`enabled` Standard `false`). Die -Web-UI bietet **Papierkorb** und **Dreaming**. Soft-gelöschte Zeilen werden nach -`dreaming.trashRetentionDays` beim Cleanup hart bereinigt. +Opt-in-Hintergrundlauf, der **verwandte** Memories findet (über `dreaming.relatedThreshold`, +Standard `0.75`, unabhängig von `deduplicationSimilarityThreshold`), sie über +`opencodeProvider`/`opencodeModel` mit `merge` / `update` / `supersede` / `keep` / `flag` +(Widersprüche) konsolidiert und ersetzte Einträge soft-deleten (wiederherstellbarer +Papierkorb). Gepinnte Memories bleiben unberührt. Mit `dreaming.crossProject: true` (oder +`memory.defaultScope: "all-projects"`) gruppiert Dreaming über Projekt-Tags; User-Scope +bleibt getrennt. Konfiguration unter `dreaming` in `opencode-mem.jsonc` (`enabled` Standard +`false`). Die Web-UI bietet **Papierkorb** und **Dreaming**. Soft-gelöschte Zeilen werden +nach `dreaming.trashRetentionDays` beim Cleanup hart bereinigt. ### Embeddings wählen / konfigurieren diff --git a/.github/i18n/README.nl.md b/.github/i18n/README.nl.md index 3a0a7b82..9f36c50e 100644 --- a/.github/i18n/README.nl.md +++ b/.github/i18n/README.nl.md @@ -277,6 +277,40 @@ in die voor geschikte herinneringen beschikbaar zijn. Met `chatMessage.maxMemori op `0` wordt het injecteren van projectherinneringen uitgeschakeld; het gebruikersprofiel kan nog steeds worden geïnjecteerd als dit is ingeschakeld. +### Update-on-capture (optioneel) + +Optioneel pad om herinneringen actueel te houden wanneer feiten veranderen. Met +`updateOnCaptureEnabled: true` embedt auto-capture de nieuwe samenvatting, zoekt +vergelijkbare herinneringen boven `updateOnCaptureSimilarityThreshold` (standaard +`0.75`) en laat het capture-model kiezen tussen `add`, `update` (in-place +herschrijven) of `supersede` (nieuw opslaan + oud soft-deleten naar de prullenbak). +Vastgezette (pinned) herinneringen worden overgeslagen. Standaard uit, zodat +bestaande installaties add-only blijven. + +```jsonc +"updateOnCaptureEnabled": true, +"updateOnCaptureSimilarityThreshold": 0.75, +"updateOnCaptureMaxCandidates": 5, +``` + +Tip: bij `Xenova/nomic-embed-text-v1` verbetert `embeddingUseTaskPrefixes: true` +(en eventueel een lagere recall-`similarityThreshold`) vaak de matchkwaliteit voor +langere herinneringen. Prefixes blijven standaard opt-in. + +### Dreaming (optionele geheugenconsolidatie) + +Optionele achtergrondpass die **gerelateerde** herinneringen vindt via +`dreaming.relatedThreshold` (standaard `0.75`, onafhankelijk van +`deduplicationSimilarityThreshold`), ze via `opencodeProvider` / `opencodeModel` +met `merge` / `update` / `supersede` / `keep` / `flag` (tegenstrijdigheden) +consolideert, en vervangen rijen soft-deletet naar herstelbare prullenbak. +Vastgezette herinneringen blijven onaangeroerd. Met `dreaming.crossProject: true` +(of `memory.defaultScope: "all-projects"`) groepeert dreaming over projecttags; +user-scope blijft gescheiden. Configureer onder `dreaming` in +`opencode-mem.jsonc` (`enabled` standaard `false`). De web-UI toont **Trash** en +**Dream**. Soft-verwijderde rijen worden na `dreaming.trashRetentionDays` hard +opgeschoond. + ### Embeddings kiezen / configureren Embeddings voeden similarity search voor herinneringen en het gebruikersprofiel. Configureer ze in hetzelfde bestand (`~/.config/opencode/opencode-mem.jsonc`). Er is **geen MLX-backend** — lokale embeddings gebruiken `@huggingface/transformers` met ONNX, niet Apple MLX. diff --git a/.github/i18n/README.tr.md b/.github/i18n/README.tr.md index 24825e4e..6a84eb98 100644 --- a/.github/i18n/README.tr.md +++ b/.github/i18n/README.tr.md @@ -278,6 +278,39 @@ Oturum ve yaşa göre hariç tutma kuralları, enjekte edilen bellek sayısı s doldurmaz. `chatMessage.maxMemories` değerini `0` yapmak proje belleği enjeksiyonunu devre dışı bırakır; kullanıcı profili etkinleştirilmişse yine de enjekte edilebilir. +### Update-on-capture (isteğe bağlı) + +Gerçekler değiştikçe bellekleri güncel tutmak için isteğe bağlı yol. +`updateOnCaptureEnabled: true` iken otomatik yakalama yeni özeti gömer, +`updateOnCaptureSimilarityThreshold` (varsayılan `0.75`) üzerindeki benzer +bellekleri arar ve yakalama modelinin `add`, `update` (yerinde yeniden yazma) +veya `supersede` (yeniyi kaydet + eskisini çöp kutusuna soft-delete) seçmesini +sağlar. Sabitlenmiş (pinned) bellekler atlanır. Varsayılan kapalıdır; mevcut +kurulumlar add-only kalır. + +```jsonc +"updateOnCaptureEnabled": true, +"updateOnCaptureSimilarityThreshold": 0.75, +"updateOnCaptureMaxCandidates": 5, +``` + +İpucu: `Xenova/nomic-embed-text-v1` için `embeddingUseTaskPrefixes: true` +(ve isteğe bağlı olarak daha düşük recall `similarityThreshold`) uzun belleklerde +eşleşme kalitesini çoğu zaman iyileştirir. Önekler varsayılan olarak opt-in kalır. + +### Dreaming (isteğe bağlı bellek birleştirme) + +`dreaming.relatedThreshold` (varsayılan `0.75`, `deduplicationSimilarityThreshold` +ile bağımsız) ile **ilgili** bellekleri bulan isteğe bağlı arka plan geçişi; +yapılandırılmış `opencodeProvider` / `opencodeModel` ile `merge` / `update` / +`supersede` / `keep` / `flag` (çelişkiler) uygular ve geçersiz kılınan satırları +geri yüklenebilir çöp kutusuna soft-delete eder. Sabitlenmiş bellekler dokunulmaz. +`dreaming.crossProject: true` (veya `memory.defaultScope: "all-projects"`) ile +proje etiketleri arasında gruplanır; kullanıcı kapsamı ayrı kalır. +`opencode-mem.jsonc` içinde `dreaming` altında yapılandırın (`enabled` varsayılan +`false`). Web arayüzü **Trash** ve **Dream** sunar. Soft-silinen satırlar +`dreaming.trashRetentionDays` sonra temizlikte kalıcı silinir. + ### Embedding seçimi / yapılandırması Embedding’ler bellekler ve kullanıcı profili için benzerlik aramasını güçlendirir. Bunları aynı dosyada yapılandırın (`~/.config/opencode/opencode-mem.jsonc`). **MLX backend yoktur** — yerel embedding’ler Apple MLX değil, ONNX ile `@huggingface/transformers` kullanır. diff --git a/.github/i18n/README.zh.md b/.github/i18n/README.zh.md index 38e9e505..7d8951f7 100644 --- a/.github/i18n/README.zh.md +++ b/.github/i18n/README.zh.md @@ -275,6 +275,22 @@ memory({ mode: "import", inputPath: "./memories.json" }); 不会占用原本可供符合条件的记忆使用的名额。将 `chatMessage.maxMemories` 设为 `0` 会禁用项目记忆注入，但启用用户画像后仍可注入用户画像。 +### Update-on-capture（可选） + +可选路径，用于在事实变化时保持记忆最新。当 `updateOnCaptureEnabled: true` 时，自动捕获会嵌入新摘要，搜索高于 `updateOnCaptureSimilarityThreshold`（默认 `0.75`）的相似记忆，并由捕获模型选择 `add`、`update`（原地改写）或 `supersede`（保存新条目并将旧条目软删除到回收站）。已固定（pinned）的记忆会被跳过。默认关闭，以便现有安装保持仅新增行为。 + +```jsonc +"updateOnCaptureEnabled": true, +"updateOnCaptureSimilarityThreshold": 0.75, +"updateOnCaptureMaxCandidates": 5, +``` + +提示：对 `Xenova/nomic-embed-text-v1`，启用 `embeddingUseTaskPrefixes: true`（并可视情况降低召回 `similarityThreshold`）通常能改善较长记忆的匹配质量。前缀默认仍为可选。 + +### Dreaming（可选记忆整合） + +可选后台流程，通过 `dreaming.relatedThreshold`（默认 `0.75`，独立于 `deduplicationSimilarityThreshold`）查找**相关**记忆，并由配置的 `opencodeProvider` / `opencodeModel` 执行 `merge` / `update` / `supersede` / `keep` / `flag`（矛盾标记），将过时条目软删除到可恢复回收站。已固定记忆不会被改动。设置 `dreaming.crossProject: true`（或 `memory.defaultScope: "all-projects"`）可跨项目标签分组；用户范围保持分离。在 `opencode-mem.jsonc` 的 `dreaming` 下配置（`enabled` 默认 `false`）。Web UI 提供 **Trash** 与 **Dream**。软删除行在常规清理时于 `dreaming.trashRetentionDays` 后硬清除。 + ### 选择 / 配置嵌入模型 嵌入为记忆与用户画像提供相似性搜索能力。在同一文件中配置（`~/.config/opencode/opencode-mem.jsonc`）。**没有 MLX 后端** — 本地嵌入使用带 ONNX 的 `@huggingface/transformers`，而非 Apple MLX。 diff --git a/README.md b/README.md index e4b92b34..11eb51e1 100644 --- a/README.md +++ b/README.md @@ -300,12 +300,15 @@ memories. Prefixes stay opt-in by default. ### Dreaming (optional memory consolidation) -Opt-in background pass that finds near-duplicate memories (same similarity threshold as -deduplication), asks the configured `opencodeProvider` / `opencodeModel` to merge them, -and soft-deletes superseded rows into recoverable trash. Pinned memories are never -touched. Configure under `dreaming` in `opencode-mem.jsonc` (`enabled` default `false`). -The web UI exposes **Trash** (restore) and **Dream** (manual run). Soft-deleted rows are -hard-purged after `dreaming.trashRetentionDays` during normal cleanup. +Opt-in background pass that finds **related** memories (via `dreaming.relatedThreshold`, +default `0.75`, independent of `deduplicationSimilarityThreshold`), asks the configured +`opencodeProvider` / `opencodeModel` to `merge`, `update`, `supersede`, `keep`, or `flag` +contradictions, and soft-deletes superseded rows into recoverable trash. Pinned memories +are never touched. Set `dreaming.crossProject: true` (or `memory.defaultScope: +"all-projects"`) to group across project tags; user-scope stays separate. Configure under +`dreaming` in `opencode-mem.jsonc` (`enabled` default `false`). The web UI exposes +**Trash** (restore) and **Dream** (manual run). Soft-deleted rows are hard-purged after +`dreaming.trashRetentionDays` during normal cleanup. Deeper related-threshold dreaming (update/supersede/flag, cross-project) lands in the follow-up PR for #464. diff --git a/src/services/api-handlers.ts b/src/services/api-handlers.ts index 0ddc260a..7e462ff0 100644 --- a/src/services/api-handlers.ts +++ b/src/services/api-handlers.ts @@ -506,6 +506,7 @@ export async function handleListTrash(limit = 200): Promise< content: string; deletedAt: number; mergedInto?: string; + supersededBy?: string; projectName?: string; containerTag: string; }> @@ -518,6 +519,7 @@ export async function handleListTrash(limit = 200): Promise< content: string; deletedAt: number; mergedInto?: string; + supersededBy?: string; projectName?: string; containerTag: string; }> = []; @@ -531,6 +533,7 @@ export async function handleListTrash(limit = 200): Promise< content: String(row.content), deletedAt: Number(row.deleted_at), mergedInto: typeof meta.mergedInto === "string" ? meta.mergedInto : undefined, + supersededBy: typeof meta.supersededBy === "string" ? meta.supersededBy : undefined, projectName: row.project_name ? String(row.project_name) : undefined, containerTag: String(row.container_tag), }); @@ -554,10 +557,18 @@ export async function handleRestoreMemory(id: string): Promise = { + add: { + title: "Memory Captured", + message: "Project memory saved from conversation", + }, + update: { + title: "Memory Updated", + message: "Similar existing memory was rewritten with current facts", + }, + supersede: { + title: "Memory Superseded", + message: "New memory saved; outdated entry moved to trash", + }, + }; + const toast = toastByAction[action] ?? toastByAction.add!; await ctx.client?.tui .showToast({ body: { - title: "Memory Captured", - message: "Project memory saved from conversation", + title: toast.title, + message: toast.message, variant: "success", duration: 3000, }, @@ -752,9 +768,14 @@ async function persistAutoCaptureMemory( } const scope = CONFIG.memory.defaultScope === "all-projects" ? "all-projects" : "project"; + // Over-fetch before pin-filtering so pinned hits don't starve the candidate list. + const searchLimit = Math.min( + 20, + Math.max(CONFIG.updateOnCaptureMaxCandidates * 3, CONFIG.updateOnCaptureMaxCandidates + 5) + ); const search = await memoryClient.searchMemories(summaryWithTags, containerTag, scope, { similarityThreshold: CONFIG.updateOnCaptureSimilarityThreshold, - limit: CONFIG.updateOnCaptureMaxCandidates, + limit: searchLimit, }); const rawCandidates: UpdateOnCaptureCandidate[] = (search.success ? search.results : []).map( diff --git a/src/services/dreaming-service.ts b/src/services/dreaming-service.ts index 04b6e4e2..ab36d320 100644 --- a/src/services/dreaming-service.ts +++ b/src/services/dreaming-service.ts @@ -13,6 +13,8 @@ import { parseExtractedVector, formatTagsForEmbedding } from "./turso/vector-uti import { getV2Client } from "./ai/opencode-provider.js"; import { tryAcquireDreamingLock } from "./dreaming-lock.js"; +export const DREAMING_MAX_GROUP_SIZE = 5; + export interface DreamingNearDupGroup { representative: { id: string; @@ -20,19 +22,95 @@ export interface DreamingNearDupGroup { containerTag: string; createdAt: number; isPinned: boolean; + scope: "user" | "project"; }; duplicates: Array<{ id: string; content: string; similarity: number; isPinned: boolean; + containerTag: string; + createdAt: number; }>; } +export type DreamingMemoryEntry = { + id: string; + content: string; + containerTag: string; + createdAt: number; + vector: Float32Array; +}; + +/** + * Pure grouping used by dreaming (and unit tests). Pinned rows must be + * excluded by the caller before invoking this. + */ +export function groupRelatedMemoryEntries( + entries: DreamingMemoryEntry[], + options: { + threshold: number; + allowCrossTag: boolean; + scope: "user" | "project"; + maxGroupSize?: number; + } +): DreamingNearDupGroup[] { + const threshold = options.threshold; + const allowCrossTag = options.allowCrossTag; + const maxGroupSize = options.maxGroupSize ?? DREAMING_MAX_GROUP_SIZE; + const groups: DreamingNearDupGroup[] = []; + const processedIds = new Set(); + + for (let i = 0; i < entries.length; i++) { + const mem1 = entries[i]!; + if (processedIds.has(mem1.id)) continue; + + const similarGroup: DreamingNearDupGroup = { + representative: { + id: mem1.id, + content: mem1.content, + containerTag: mem1.containerTag, + createdAt: mem1.createdAt, + isPinned: false, + scope: options.scope, + }, + duplicates: [], + }; + + for (let j = i + 1; j < entries.length; j++) { + const mem2 = entries[j]!; + if (processedIds.has(mem2.id)) continue; + if (!allowCrossTag && mem1.containerTag !== mem2.containerTag) continue; + + const similarity = cosineSimilarity(mem1.vector, mem2.vector); + if (similarity >= threshold && similarity < 1.0) { + similarGroup.duplicates.push({ + id: mem2.id, + content: mem2.content, + similarity, + isPinned: false, + containerTag: mem2.containerTag, + createdAt: mem2.createdAt, + }); + processedIds.add(mem2.id); + if (similarGroup.duplicates.length >= maxGroupSize - 1) break; + } + } + + if (similarGroup.duplicates.length > 0) { + processedIds.add(mem1.id); + groups.push(similarGroup); + } + } + + return groups; +} + export interface DreamingRunResult { groupsConsidered: number; groupsMerged: number; groupsSkipped: number; + groupsFlagged: number; memoriesSoftDeleted: number; } @@ -40,8 +118,8 @@ interface DreamingState { lastRunAt: number; } -const mergeSchema = z.object({ - action: z.enum(["merge", "skip"]), +const dreamingActionSchema = z.object({ + action: z.enum(["merge", "update", "supersede", "keep", "flag"]), content: z.string().optional(), keepId: z.string().optional(), dropIds: z.array(z.string()).optional(), @@ -69,79 +147,103 @@ function writeState(state: DreamingState): void { writeFileSync(statePath(), JSON.stringify(state), "utf-8"); } +export function isDreamingCrossProjectEnabled(): boolean { + return Boolean(CONFIG.dreaming.crossProject) || CONFIG.memory.defaultScope === "all-projects"; +} + +type MemoryRow = { + id: unknown; + content: unknown; + container_tag: unknown; + created_at: unknown; + is_pinned?: unknown; + vector_json?: unknown; +}; + +/** + * Group related memories for dreaming. Uses dreaming.relatedThreshold + * (not deduplicationSimilarityThreshold). Cross-project grouping is opt-in / + * follows memory.defaultScope; user-scope never mixes with project-scope. + */ export async function findNearDuplicateGroupsForDreaming(): Promise { await ensureTursoReady(); const userShards = await tursoShardManager.getAllShards("user", ""); const projectShards = await tursoShardManager.getAllShards("project", ""); - const allShards = [...userShards, ...projectShards]; + const threshold = CONFIG.dreaming.relatedThreshold; + const crossProject = isDreamingCrossProjectEnabled(); const groups: DreamingNearDupGroup[] = []; - const threshold = CONFIG.deduplicationSimilarityThreshold; - - for (const shard of allShards) { - const db = await tursoConnectionManager.getConnection(shard.dbPath); - const memories = await tursoVectorSearch.getAllMemoriesWithExtractedVectors(db); - const processedIds = new Set(); - - for (let i = 0; i < memories.length; i++) { - const mem1 = memories[i]!; - if (Number(mem1.is_pinned ?? 0) === 1) continue; - const vector1 = parseExtractedVector(mem1.vector_json); - if (!vector1 || processedIds.has(String(mem1.id))) continue; - - const similarGroup: DreamingNearDupGroup = { - representative: { - id: String(mem1.id), - content: String(mem1.content), - containerTag: String(mem1.container_tag), - createdAt: Number(mem1.created_at), - isPinned: false, - }, - duplicates: [], - }; - - for (let j = i + 1; j < memories.length; j++) { - const mem2 = memories[j]!; - if (Number(mem2.is_pinned ?? 0) === 1) continue; - const vector2 = parseExtractedVector(mem2.vector_json); - if (!vector2 || processedIds.has(String(mem2.id))) continue; - if (mem1.container_tag !== mem2.container_tag) continue; - - const similarity = cosineSimilarity(vector1, vector2); - if (similarity >= threshold && similarity < 1.0) { - similarGroup.duplicates.push({ - id: String(mem2.id), - content: String(mem2.content), - similarity, - isPinned: false, - }); - processedIds.add(String(mem2.id)); - } - } - if (similarGroup.duplicates.length > 0) { - processedIds.add(String(mem1.id)); - groups.push(similarGroup); + const collectFromShardPool = async ( + shards: Awaited>, + scope: "user" | "project", + allowCrossTag: boolean + ) => { + const entries: DreamingMemoryEntry[] = []; + + for (const shard of shards) { + const db = await tursoConnectionManager.getConnection(shard.dbPath); + const memories = await tursoVectorSearch.getAllMemoriesWithExtractedVectors(db); + for (const mem of memories as MemoryRow[]) { + if (Number(mem.is_pinned ?? 0) === 1) continue; + const vector = parseExtractedVector(mem.vector_json); + if (!vector) continue; + entries.push({ + id: String(mem.id), + content: String(mem.content), + containerTag: String(mem.container_tag), + createdAt: Number(mem.created_at), + vector, + }); } } - } + + groups.push( + ...groupRelatedMemoryEntries(entries, { + threshold, + allowCrossTag, + scope, + maxGroupSize: DREAMING_MAX_GROUP_SIZE, + }) + ); + }; + + // User-scope: always same container_tag only (never mix with projects). + await collectFromShardPool(userShards, "user", false); + // Project-scope: cross-tag when crossProject / defaultScope all-projects. + await collectFromShardPool(projectShards, "project", crossProject); return groups; } -function buildMergePrompt(group: DreamingNearDupGroup): string { +export function buildMergePrompt(group: DreamingNearDupGroup): string { const items = [ - { id: group.representative.id, content: group.representative.content }, - ...group.duplicates.map((d) => ({ id: d.id, content: d.content })), + { + id: group.representative.id, + content: group.representative.content, + createdAt: group.representative.createdAt, + containerTag: group.representative.containerTag, + }, + ...group.duplicates.map((d) => ({ + id: d.id, + content: d.content, + createdAt: d.createdAt, + containerTag: d.containerTag, + similarity: Number(d.similarity.toFixed(3)), + })), ]; - return `You consolidate near-duplicate project memories into one clearer entry. + return `You consolidate related project/user memories. They may be near-duplicates OR newer facts that replace older ones. Rules: -- Prefer action "merge" only when the memories clearly describe the same fact/fix/gotcha. -- The merged content MUST retain important details, including why something failed when present. +- "merge": same fact worded differently → one clearer entry (keepId + dropIds + content). +- "update": rewrite keepId with the current correct content; soft-delete dropIds. +- "supersede": newer fact replaces older → keep newest correct content on keepId (or prefer newest id), soft-delete outdated dropIds. +- "keep": related but all still valid / unsafe to change — no mutation. +- "flag": clear contradiction you cannot resolve — no mutation; we mark metadata. +- Prefer the newest correct fact when one clearly supersedes another. +- Merged/updated content MUST retain important details, including why something failed when present. - Do NOT invent facts. Do NOT drop unique constraints, commands, paths, or error messages. -- keepId must be one of the input ids (prefer the most complete/recent). -- dropIds must list every other id when merging. -- Use action "skip" when memories are related but not safe to merge. +- keepId must be one of the input ids. dropIds lists every other id when merging/updating/superseding. +- Use "keep" when memories are only loosely related. Return JSON matching the schema. @@ -159,6 +261,9 @@ export class DreamingService { intervalHours: CONFIG.dreaming.intervalHours, maxGroupsPerRun: CONFIG.dreaming.maxGroupsPerRun, trashRetentionDays: CONFIG.dreaming.trashRetentionDays, + relatedThreshold: CONFIG.dreaming.relatedThreshold, + crossProject: CONFIG.dreaming.crossProject, + crossProjectEffective: isDreamingCrossProjectEnabled(), isRunning: this.isRunning, lastRunAt: state.lastRunAt || null, }; @@ -201,6 +306,7 @@ export class DreamingService { groupsConsidered: 0, groupsMerged: 0, groupsSkipped: 0, + groupsFlagged: 0, memoriesSoftDeleted: 0, }; @@ -233,19 +339,37 @@ export class DreamingService { systemPrompt: "You are a careful memory consolidation assistant. Return only structured JSON.", userPrompt: buildMergePrompt(group), - schema: mergeSchema, + schema: dreamingActionSchema, directory, }); - if (decision.action !== "merge" || !decision.content?.trim()) { - result.groupsSkipped++; - continue; - } - const memberIds = new Set([ group.representative.id, ...group.duplicates.map((d) => d.id), ]); + + if (decision.action === "keep") { + result.groupsSkipped++; + continue; + } + + if (decision.action === "flag") { + const flagged = await this.applyFlag(group, decision.reason); + if (flagged) result.groupsFlagged++; + else result.groupsSkipped++; + continue; + } + + if ( + (decision.action !== "merge" && + decision.action !== "update" && + decision.action !== "supersede") || + !decision.content?.trim() + ) { + result.groupsSkipped++; + continue; + } + const keepId = decision.keepId && memberIds.has(decision.keepId) ? decision.keepId @@ -261,15 +385,19 @@ export class DreamingService { continue; } - const applied = await this.applyMerge( + const metaKey = decision.action === "supersede" ? "supersededBy" : "mergedInto"; + // Always preserve the keep row's own container_tag (critical for + // cross-project groups where keepId may not be the representative). + const outcome = await this.applyDreamingAction( keepId, decision.content.trim(), finalDrop, - group.representative.containerTag + metaKey, + decision.action ); - if (applied) { + if (outcome.applied) { result.groupsMerged++; - result.memoriesSoftDeleted += finalDrop.length; + result.memoriesSoftDeleted += outcome.softDeleted; } else { result.groupsSkipped++; } @@ -288,12 +416,37 @@ export class DreamingService { } } - private async applyMerge( + private async applyFlag(group: DreamingNearDupGroup, reason?: string): Promise { + const ids = [group.representative.id, ...group.duplicates.map((d) => d.id)]; + const shards = [ + ...(await tursoShardManager.getAllShards("user", "")), + ...(await tursoShardManager.getAllShards("project", "")), + ]; + let any = false; + for (const id of ids) { + for (const shard of shards) { + const db = await tursoConnectionManager.getConnection(shard.dbPath); + const row = await tursoVectorSearch.getMemoryById(db, id); + if (!row) continue; + const ok = await tursoVectorSearch.patchMemoryMetadata(db, id, { + contradictionWith: ids.filter((other) => other !== id), + contradictionReason: reason || "Unresolved contradiction during dreaming", + contradictionFlaggedAt: Date.now(), + }); + if (ok) any = true; + break; + } + } + return any; + } + + private async applyDreamingAction( keepId: string, content: string, dropIds: string[], - containerTag: string - ): Promise { + dropMetaKey: "mergedInto" | "supersededBy", + action: "merge" | "update" | "supersede" + ): Promise<{ applied: boolean; softDeleted: number }> { const shards = [ ...(await tursoShardManager.getAllShards("user", "")), ...(await tursoShardManager.getAllShards("project", "")), @@ -310,8 +463,32 @@ export class DreamingService { break; } } - if (!keepShard || !keepRow) return false; - if (Number(keepRow.is_pinned ?? 0) === 1) return false; + if (!keepShard || !keepRow) return { applied: false, softDeleted: 0 }; + if (Number(keepRow.is_pinned ?? 0) === 1) return { applied: false, softDeleted: 0 }; + + // Soft-delete drops first so we never rewrite keep if nothing can be retired. + let softDeleted = 0; + for (const dropId of dropIds) { + for (const shard of shards) { + const dropDb = await tursoConnectionManager.getConnection(shard.dbPath); + const dropRow = await tursoVectorSearch.getMemoryById(dropDb, dropId); + if (!dropRow) continue; + if (Number(dropRow.is_pinned ?? 0) === 1) continue; + const ok = await tursoVectorSearch.softDeleteVector(dropDb, dropId, { + [dropMetaKey]: keepId, + }); + if (ok) softDeleted++; + break; + } + } + if (softDeleted === 0) { + log("dreaming: skipped keep rewrite (no drops soft-deleted)", { + keepId, + dropIds, + action, + }); + return { applied: false, softDeleted: 0 }; + } await embeddingService.warmup(); const vector = await embeddingService.embedWithTimeout(content, { task: "document" }); @@ -339,10 +516,12 @@ export class DreamingService { } meta.mergedFrom = mergedFrom; meta.source = typeof meta.source === "string" ? meta.source : "dreaming"; + meta.dreamingAction = action; const db = await tursoConnectionManager.getConnection(keepShard.dbPath); const updatedAt = Date.now(); const isPinned = Number(keepRow.is_pinned ?? 0) === 1 ? 1 : 0; + const keepContainerTag = String(keepRow.container_tag); await db.transaction("write", async (tx) => { await tx.execute({ sql: `DELETE FROM memories WHERE id = ?`, args: [keepId] }); @@ -359,7 +538,7 @@ export class DreamingService { content, JSON.stringify(Array.from(vector)), ...(tagsVector ? [JSON.stringify(Array.from(tagsVector))] : []), - containerTag || String(keepRow.container_tag), + keepContainerTag, tags.length > 0 ? tags.join(",") : null, keepRow.type ? String(keepRow.type) : null, Number(keepRow.created_at), @@ -377,18 +556,7 @@ export class DreamingService { }); }); - for (const dropId of dropIds) { - for (const shard of shards) { - const dropDb = await tursoConnectionManager.getConnection(shard.dbPath); - const dropRow = await tursoVectorSearch.getMemoryById(dropDb, dropId); - if (!dropRow) continue; - if (Number(dropRow.is_pinned ?? 0) === 1) continue; - await tursoVectorSearch.softDeleteVector(dropDb, dropId, { mergedInto: keepId }); - break; - } - } - - return true; + return { applied: true, softDeleted }; } } diff --git a/src/services/turso/vector-search.ts b/src/services/turso/vector-search.ts index f6249d2e..76fdf033 100644 --- a/src/services/turso/vector-search.ts +++ b/src/services/turso/vector-search.ts @@ -361,6 +361,23 @@ export class TursoVectorSearch { return changes > 0; } + async patchMemoryMetadata( + db: TursoDb, + memoryId: string, + metaPatch: Record + ): Promise { + await this.prepareShardDb(db); + const existing = await this.getMemoryById(db, memoryId, { includeDeleted: false }); + if (!existing) return false; + const meta = parseMetadata(existing.metadata) || {}; + Object.assign(meta, metaPatch); + const changes = await db.run( + `UPDATE memories SET metadata = ?, updated_at = ? WHERE id = ? AND ${ACTIVE_MEMORY_SQL}`, + [JSON.stringify(meta), Date.now(), memoryId] + ); + return changes > 0; + } + async restoreVector(db: TursoDb, memoryId: string): Promise { await this.prepareShardDb(db); const changes = await db.run( diff --git a/src/services/update-on-capture.ts b/src/services/update-on-capture.ts index ee8d0489..613d1915 100644 --- a/src/services/update-on-capture.ts +++ b/src/services/update-on-capture.ts @@ -71,7 +71,7 @@ export function filterUpdateOnCaptureCandidates( maxCandidates = CONFIG.updateOnCaptureMaxCandidates ): UpdateOnCaptureCandidate[] { return candidates - .filter((c) => !c.isPinned) + .filter((c) => Number(c.isPinned) !== 1 && c.isPinned !== true) .sort((a, b) => b.similarity - a.similarity) .slice(0, Math.max(1, maxCandidates)); } @@ -171,16 +171,23 @@ export async function applyUpdateOnCaptureDecision( // supersede const added = await client.addMemory(input.summary, input.containerTag, input.metadata as any); - if (!added.success) { - return { success: false, action: "supersede", error: added.error }; + if (!added.success || !added.id) { + return { success: false, action: "supersede", error: added.error || "add failed" }; } const deleted = await client.deleteMemory(targetId, { supersededBy: added.id }); if (!deleted.success) { - log("update-on-capture: supersede soft-delete failed", { + log("update-on-capture: supersede soft-delete failed; rolling back new memory", { targetId, newId: added.id, error: deleted.error, }); + // Keep store consistent: don't leave both old (active) and new (active). + await client.deleteMemory(added.id, { supersededRollback: true }); + return { + success: false, + action: "supersede", + error: deleted.error || "failed to soft-delete superseded memory", + }; } return { success: true, id: added.id, action: "supersede" }; } diff --git a/tests/dreaming-grouping.test.ts b/tests/dreaming-grouping.test.ts new file mode 100644 index 00000000..617b0174 --- /dev/null +++ b/tests/dreaming-grouping.test.ts @@ -0,0 +1,108 @@ +import { describe, expect, it } from "bun:test"; +import { + groupRelatedMemoryEntries, + isDreamingCrossProjectEnabled, + type DreamingMemoryEntry, +} from "../src/services/dreaming-service.js"; +import { CONFIG } from "../src/config.js"; + +function vec(...values: number[]): Float32Array { + return new Float32Array(values); +} + +function entry( + id: string, + containerTag: string, + vector: Float32Array, + createdAt = 1 +): DreamingMemoryEntry { + return { id, content: id, containerTag, createdAt, vector }; +} + +describe("groupRelatedMemoryEntries", () => { + it("groups related pairs at relatedThreshold but not only near-dup 0.9", () => { + // Identical direction → similarity 1.0 is excluded (< 1.0 required). + // Nearly parallel vectors → high similarity (~0.99). + // Distinct-ish vectors → ~0.8 range. + const a = entry("a", "p1", vec(1, 0, 0)); + const b = entry("b", "p1", vec(0.8, 0.6, 0)); // cos ≈ 0.8 + const c = entry("c", "p1", vec(0, 1, 0)); // cos with a = 0 + + const at075 = groupRelatedMemoryEntries([a, b, c], { + threshold: 0.75, + allowCrossTag: false, + scope: "project", + }); + expect(at075).toHaveLength(1); + expect(at075[0]!.representative.id).toBe("a"); + expect(at075[0]!.duplicates.map((d) => d.id)).toEqual(["b"]); + + const at09 = groupRelatedMemoryEntries([a, b, c], { + threshold: 0.9, + allowCrossTag: false, + scope: "project", + }); + expect(at09).toHaveLength(0); + }); + + it("keeps same-tag only when cross-tag is disabled", () => { + const a = entry("a", "p1", vec(1, 0)); + const b = entry("b", "p2", vec(0.99, 0.01)); + const groups = groupRelatedMemoryEntries([a, b], { + threshold: 0.75, + allowCrossTag: false, + scope: "project", + }); + expect(groups).toHaveLength(0); + }); + + it("allows cross-tag grouping when enabled", () => { + const a = entry("a", "p1", vec(1, 0)); + const b = entry("b", "p2", vec(0.99, 0.01)); + const groups = groupRelatedMemoryEntries([a, b], { + threshold: 0.75, + allowCrossTag: true, + scope: "project", + }); + expect(groups).toHaveLength(1); + expect(groups[0]!.duplicates[0]!.containerTag).toBe("p2"); + }); + + it("caps group size", () => { + const base = entry("r", "p1", vec(1, 0, 0)); + const others = [0.1, 0.15, 0.2, 0.25, 0.3].map((y, i) => + entry(`d${i}`, "p1", vec(1, y, 0), i + 2) + ); + const groups = groupRelatedMemoryEntries([base, ...others], { + threshold: 0.7, + allowCrossTag: false, + scope: "project", + maxGroupSize: 3, + }); + expect(groups.length).toBeGreaterThanOrEqual(1); + expect(groups.every((g) => g.duplicates.length <= 2)).toBe(true); + expect(groups[0]!.duplicates.length).toBe(2); + }); +}); + +describe("isDreamingCrossProjectEnabled", () => { + it("follows dreaming.crossProject or memory.defaultScope", () => { + const prevCross = CONFIG.dreaming.crossProject; + const prevScope = CONFIG.memory.defaultScope; + try { + CONFIG.dreaming.crossProject = false; + CONFIG.memory.defaultScope = "project"; + expect(isDreamingCrossProjectEnabled()).toBe(false); + + CONFIG.memory.defaultScope = "all-projects"; + expect(isDreamingCrossProjectEnabled()).toBe(true); + + CONFIG.memory.defaultScope = "project"; + CONFIG.dreaming.crossProject = true; + expect(isDreamingCrossProjectEnabled()).toBe(true); + } finally { + CONFIG.dreaming.crossProject = prevCross; + CONFIG.memory.defaultScope = prevScope; + } + }); +}); diff --git a/tests/dreaming-prompt.test.ts b/tests/dreaming-prompt.test.ts index c06226b6..246222d0 100644 --- a/tests/dreaming-prompt.test.ts +++ b/tests/dreaming-prompt.test.ts @@ -2,9 +2,11 @@ import { describe, expect, it } from "bun:test"; import { normalizeDreamingIntervalHours, normalizeDreamingMaxGroupsPerRun, + normalizeDreamingRelatedThreshold, normalizeDreamingTrashRetentionDays, } from "../src/config.js"; import { mergeBoth, applyProjectBoost } from "../src/services/memory-recall.js"; +import { buildMergePrompt, type DreamingNearDupGroup } from "../src/services/dreaming-service.js"; describe("dreaming config helpers", () => { it("normalizes dreaming numeric knobs", () => { @@ -13,6 +15,39 @@ describe("dreaming config helpers", () => { expect(normalizeDreamingMaxGroupsPerRun(2.9)).toBe(2); expect(normalizeDreamingTrashRetentionDays(7)).toBe(7); expect(normalizeDreamingTrashRetentionDays(-1)).toBe(30); + expect(normalizeDreamingRelatedThreshold(0.8)).toBe(0.8); + expect(normalizeDreamingRelatedThreshold(2)).toBe(1); + expect(normalizeDreamingRelatedThreshold(undefined)).toBe(0.75); + }); +}); + +describe("dreaming prompt actions", () => { + it("mentions supersede/update/flag actions", () => { + const group: DreamingNearDupGroup = { + representative: { + id: "a", + content: "old partition nvme0n1p3", + containerTag: "p1", + createdAt: 1, + isPinned: false, + scope: "project", + }, + duplicates: [ + { + id: "b", + content: "moved to sda5", + similarity: 0.78, + isPinned: false, + containerTag: "p1", + createdAt: 2, + }, + ], + }; + const prompt = buildMergePrompt(group); + expect(prompt).toContain("supersede"); + expect(prompt).toContain("update"); + expect(prompt).toContain("flag"); + expect(prompt).toContain("keep"); }); }); diff --git a/tests/update-on-capture.test.ts b/tests/update-on-capture.test.ts index 8f5f9c3f..23b8c875 100644 --- a/tests/update-on-capture.test.ts +++ b/tests/update-on-capture.test.ts @@ -119,7 +119,37 @@ describe("applyUpdateOnCaptureDecision", () => { ); expect(result).toEqual({ success: true, id: "mem_new", action: "supersede" }); expect(client.calls).toContain("add"); - expect(client.calls.some((c) => c.startsWith("delete:mem_old:"))).toBe(true); + expect(client.calls.some((c) => c.includes("supersededBy"))).toBe(true); + }); + + it("rolls back new memory when supersede soft-delete fails", async () => { + const calls: string[] = []; + const client: UpdateOnCaptureMemoryClient & { calls: string[] } = { + calls, + addMemory: async () => { + calls.push("add"); + return { success: true, id: "mem_new" }; + }, + updateMemory: async () => ({ success: true }), + getMemoryById: async (id) => ({ + success: true, + memory: { id, content: "old", isPinned: false }, + }), + deleteMemory: async (id, patch) => { + calls.push(`delete:${id}:${JSON.stringify(patch)}`); + if (id === "mem_old") return { success: false, error: "busy" }; + return { success: true }; + }, + }; + const result = await applyUpdateOnCaptureDecision( + client, + { action: "supersede", targetId: "mem_old" }, + baseInput, + new Set(["mem_old"]) + ); + expect(result.success).toBe(false); + expect(result.action).toBe("supersede"); + expect(calls.some((c) => c.includes("supersededRollback"))).toBe(true); }); it("falls back to add for invalid target", async () => { diff --git a/web/src/lib/components/explorer/TrashDialog.svelte b/web/src/lib/components/explorer/TrashDialog.svelte index 992ce507..256d5f04 100644 --- a/web/src/lib/components/explorer/TrashDialog.svelte +++ b/web/src/lib/components/explorer/TrashDialog.svelte @@ -9,6 +9,7 @@ content: string; deletedAt: number; mergedInto?: string; + supersededBy?: string; projectName?: string; containerTag: string; }; @@ -72,6 +73,9 @@ {#if item.mergedInto} {i18n.t("trash-merged-into", { id: item.mergedInto })} {/if} + {#if item.supersededBy} + {i18n.t("trash-superseded-by", { id: item.supersededBy })} + {/if}