From f6463390b596bc97e6df56cfcd86f17792658ed6 Mon Sep 17 00:00:00 2001 From: leemour Date: Tue, 20 Oct 2026 23:30:48 +0200 Subject: [PATCH] test(projects): record second installed gathering observation --- CHANGELOG.md | 2 + performance/projects/PROCESS.md | 2 +- performance/projects/README.md | 2 +- .../projects/installed-observation/README.md | 51 ++++ .../measure-local084.mjs | 225 ++++++++++++++++++ .../second-project084-base-totals.json | 78 ++++++ .../second-project084-expanded-totals.json | 78 ++++++ 7 files changed, 436 insertions(+), 2 deletions(-) create mode 100644 performance/projects/installed-observation/measure-local084.mjs create mode 100644 performance/projects/installed-observation/second-project084-base-totals.json create mode 100644 performance/projects/installed-observation/second-project084-expanded-totals.json diff --git a/CHANGELOG.md b/CHANGELOG.md index 221ebbe..0ca8524 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ Every entry says what changed as a caller sees it, why, and what to watch for ## Unreleased +- Add an opt-in installed Memo 0.84.0 observation runner and aggregate second-project name/contact results. + ### Added - **`performance/projects/restore/check-memo.mjs`** exercises published Memo0.83 through real CLI diff --git a/performance/projects/PROCESS.md b/performance/projects/PROCESS.md index 30f302d..fbce650 100644 --- a/performance/projects/PROCESS.md +++ b/performance/projects/PROCESS.md @@ -21,7 +21,7 @@ data. It is functional release acceptance, separate from retrieval and extractio not establish a complete project backup or a safe reset/restore path. The [installed post-fix observation](installed-observation/README.md) is a separate mutable-archive -measurement. It publishes totals through an allowlisted projection; cases, source references and +measurement, including a second project on Memo 0.84.0 with explicit name/contact hints. It publishes totals through an allowlisted projection; cases, source references and per-item rows remain private. Do not combine it with the frozen synthetic precision/recall scores. | Study | What is measured | Runtime provenance | What it does not establish | diff --git a/performance/projects/README.md b/performance/projects/README.md index dbf88b2..c617042 100644 --- a/performance/projects/README.md +++ b/performance/projects/README.md @@ -6,7 +6,7 @@ - [Published snapshots](snapshot/README.md): invented-data checks of the released read-only annotation SDK; selected scope, source-body omission and proposed-state preservation. - [Installed observation](installed-observation/README.md): allowlisted totals from a private fixed-window - gathering rerun after the document pagination fix; finished pagination remains separate from archive + gathering rerun after the document pagination fix and a second-project name/contact trial; finished pagination remains separate from archive completeness, relevance and interpretation. - [Extraction contract](extraction-contract/README.md): fake-completion checks of released SDK schema, diff --git a/performance/projects/installed-observation/README.md b/performance/projects/installed-observation/README.md index 6686a11..fd1a7a1 100644 --- a/performance/projects/installed-observation/README.md +++ b/performance/projects/installed-observation/README.md @@ -104,3 +104,54 @@ checksum, three phase totals and qualifications. Neither includes the project's source references, source text, per-question answers or labels. Original totals remain unchanged. Finished pagination still leaves remote completeness and semantic interpretation unknown; source review and proposed-finding decisions are separate from this measurement. + +## Second project on Memo 0.84.0 + +The owner asked whether gathering works beyond the first project. A second existing project has a +confirmed key document and person relationship. Its key document supplies alternative names and a +warning that some matching documents were copied into another project. The test asks what ordinary +name/key gathering finds, and how explicit name/contact hints change its candidate set. + +Linux x64, Node 24.19.0, installed registry Memo 0.84.0/Core 0.377.0/Base 0.47.4; +[release run](https://github.com/WireCatLabs/cli-memo/actions/runs/39485604512) succeeded. +A fixed exclusive cutoff and 60-day window are reused throughout. Each invocation allows 100 items +and 96 reader pages; each phase permits at most four invocations. The expanded request adds three +alternative names and one existing person reference without saving aliases or relationships. + +| Request | Direct refs | Messages / documents | Calls per phase | Phase wall times, ms | Pagination | +| --- | ---: | ---: | ---: | --- | --- | +| Existing name/key | 80 | 10 / 70 | 2 | 2,220.24 / 2,638.03 / 1,791.54 | Finished | +| Added names/contact | 115 | 15 / 100 | 4 | 3,623.69 / 3,539.47 / 3,635.13 | Finished | + +All 80 original references remain; 35 are added. The expanded set labels 114 as project candidates +and one as person-only. These labels explain matching lanes, not verified membership. Within each +request, all three phases return identical direct reference/excerpt-hash sets and zero duplicates. +The bounded project-review view is unchanged. Expanded-run before/after runtime fingerprints match; +the baseline was not bracketed by a software fingerprint. No claim of whole-store immutability follows. + +Finished pagination is not finished reading: 76 baseline and 107 expanded excerpts are truncated. +Eleven baseline and three expanded hits report surrounding-context issues. Internal notes are +reported unread. No meetings matched; this does not establish complete meeting coverage. +A bounded qualitative read found a document explicitly warning that its files describe another +project: name matching usefully retrieves a warning but cannot decide the document's project membership. +No semantic accuracy, blind recall, token savings, current business state or controlled speedup is measured. + +Re-run the exact retained private cases only against the installed released binary, using a new +private result path. This intentionally reads the selected real archive; synthetic checks keep temporary +HOME/store/state instead. The script requires explicit opt-in, pins package versions, refuses an existing +output before any archive read, retains references/hashes/coverage, and never writes source bodies: + +```sh +T=$(mktemp -d -p /var/tmp) && HOME=/path/to/selected/home TMPDIR=$T node performance/projects/installed-observation/measure-local084.mjs --live-local /PRIVATE/case.json /PRIVATE/new-result.json /ABSOLUTE/installed/memo +``` + +Project public totals with the unchanged projector above. Cases, runtime fingerprints, source references, +manual reading notes and results are private in `2026-10-20-second-project-gathering.md`. +[Baseline totals](second-project084-base-totals.json) and +[expanded totals](second-project084-expanded-totals.json) contain aggregate counts and input checksums only. +The earlier measurements, cases and frozen synthetic benchmarks remain unchanged. + +The exact executed source is retained privately as `measure-pmed084.mjs`, matching each recorded +runner checksum. The public runner differs only in sorted import names for repository lint. +An isolated occupied-output check refused with EEXIST before any Memo invocation; the existing +aggregate projection sentinel checks pass unchanged. diff --git a/performance/projects/installed-observation/measure-local084.mjs b/performance/projects/installed-observation/measure-local084.mjs new file mode 100644 index 0000000..009c90a --- /dev/null +++ b/performance/projects/installed-observation/measure-local084.mjs @@ -0,0 +1,225 @@ +import assert from "node:assert/strict" +import { execFileSync } from "node:child_process" +import { createHash } from "node:crypto" +import { closeSync, openSync, readFileSync, realpathSync, writeFileSync } from "node:fs" +import { dirname, resolve } from "node:path" +import { performance } from "node:perf_hooks" +import { fileURLToPath } from "node:url" + +const [consent, casePath, output, binaryPath] = process.argv.slice(2) +assert.equal( + consent, + "--live-local", + "explicit --live-local is required; uses the owner's installed CLI and local archive", +) +assert.ok(output, "provide a new aggregate-result JSON path") +assert.ok(casePath && binaryPath, "provide private case, new output and installed binary paths") +const scenario = JSON.parse(readFileSync(resolve(casePath), "utf8")) +const binary = realpathSync(binaryPath) +const packageRoot = resolve(dirname(realpathSync(binary)), "../..") +const manifest = JSON.parse(readFileSync(resolve(packageRoot, "package.json"), "utf8")) +assert.equal(manifest.version, "0.84.0", "this measured protocol pins published Memo 0.84.0") +assert.equal(manifest.dependencies["@wirecat/cli-core"], "0.377.0") +const sha = (value) => createHash("sha256").update(value).digest("hex") +const textMeta = (item) => ({ + ref: item.ref, + fingerprint: item.fingerprint ?? null, + kind: item.kind ?? null, + time_kind: item.time_kind ?? null, + context: typeof item.context === "string" ? item.context : null, + text_truncated: item.text_truncated ?? null, + around_issue: item.around_issue ?? null, + lanes: item.lanes ?? [], + text_sha256: sha(item.text ?? ""), + text_utf8_bytes: Buffer.byteLength(item.text ?? ""), + around: (item.around ?? []).map((one) => ({ + ref: one.ref, + found: one.found ?? null, + text_sha256: sha(one.text ?? ""), + text_utf8_bytes: Buffer.byteLength(one.text ?? ""), + })), +}) +const invoke = (args) => { + const start = performance.now() + let raw + try { + raw = execFileSync(binary, args, { + env: process.env, + encoding: "utf8", + timeout: 60000, + maxBuffer: 16 * 1024 * 1024, + stdio: ["ignore", "pipe", "pipe"], + }) + } catch (error) { + const safe = new Error( + `installed Memo failed (exit ${error.status ?? "none"}, signal ${error.signal ?? "none"}); captured output is withheld`, + ) + throw safe + } + const envelope = JSON.parse(raw) + assert.ok(envelope.item && envelope.meta, "expected a complete item envelope") + return { + envelope, + wall_ms: +(performance.now() - start).toFixed(2), + stdout_utf8_bytes: Buffer.byteLength(raw), + } +} +const contextSnapshot = () => { + const { envelope } = invoke(["projects", "context", scenario.project, "--json"]) + const value = envelope.item + return { + project: value.project.ref, + key_documents: value.key_documents.map((item) => ({ + ref: item.ref, + confirmed: item.confirmed, + })), + relationships: value.relationships.map((item) => ({ + id: item.id, + confirmed: item.confirmed, + })), + facts: value.review.facts.items.map((item) => ({ + ref: item.ref, + status: item.status, + })), + findings_batches: value.review.meeting_actions.items.map((item) => item.id), + open_tasks: value.open_tasks.items.map((item) => item.ref), + partial_view: true, + } +} +const report = { + case_id: scenario.case_id, + started_at: new Date().toISOString(), + node: process.version, + memo: manifest.version, + dependencies: { + core: manifest.dependencies["@wirecat/cli-core"], + base: manifest.dependencies["@wirecat/cli-base"], + }, + case_sha256: sha(readFileSync(resolve(casePath))), + runner_sha256: sha(readFileSync(fileURLToPath(import.meta.url))), + request: scenario, + runs: [], + llm_tokens: null, + remote_fetches_requested: 0, + qualification: + "Live local observation; no raw source text saved, no project reset, no knowledge proposals. Default catch-up may update local indexes. Time is CLI wall time, not agent analysis time. Owner archive is mutable; references and hashes detect drift but are not an offline corpus snapshot.", +} +const outputFd = openSync(resolve(output), "wx", 0o600) +try { + const before = contextSnapshot() + for (const [name, catchUp] of [ + ["index_as_is", false], + ["default_first", true], + ["default_repeat", true], + ]) { + const args = [ + "projects", + "evidence", + scenario.project, + "--source", + scenario.source, + "--since-time", + scenario.since_time, + "--before-time", + scenario.before_time, + "--limit", + String(scenario.limit), + "--max-pages", + String(scenario.max_pages), + ...(!catchUp ? ["--no-catch-up"] : []), + ...(scenario.aliases ?? []).flatMap((value) => ["--alias", value]), + ...(scenario.contacts ?? []).flatMap((value) => ["--contact", value]), + "--json", + ] + const run = { + name, + args, + pages: [], + items: [], + stagnation_detected: false, + } + let cursor + let previousProgress + for (let page = 0; page < scenario.max_invocations; page++) { + const { envelope, wall_ms, stdout_utf8_bytes } = invoke([...args, ...(cursor ? ["--cursor", cursor] : [])]) + const value = envelope.item + assert.equal(value.window.before, Date.parse(scenario.before_time), "all phases must use the fixed exclusive end") + run.pages.push({ + wall_ms, + stdout_utf8_bytes, + meta: { duration_ms: envelope.meta.duration_ms }, + window: value.window, + scope: value.scope, + lanes: value.lanes, + usage: value.usage, + totals: value.totals, + index: value.index, + stopped: value.stopped ?? null, + complete: value.complete, + resumed: value.resumed, + completeness: value.completeness, + units: value.units, + unread_sources: value.unread_sources, + oversized: (value.oversized ?? []).map(({ ref, bytes, lanes, source }) => ({ ref, bytes, lanes, source })), + matched_again: (value.matchedAgain ?? value.matched_again ?? []).map(({ ref, lanes }) => ({ ref, lanes })), + checkpoint_present: Boolean(value.checkpoint), + }) + run.items.push(...value.items.map(textMeta)) + cursor = value.checkpoint + const progress = JSON.stringify( + value.units.map(({ lane, source, state, pages, issue }) => ({ + lane, + source, + state, + pages, + issue, + })), + ) + if (cursor && value.items.length === 0 && progress === previousProgress) { + run.stagnation_detected = true + break + } + previousProgress = progress + if (!cursor) break + } + run.finished_pagination = !cursor + run.wall_ms = +run.pages.reduce((sum, page) => sum + page.wall_ms, 0).toFixed(2) + run.unique_refs = new Set(run.items.map((item) => item.ref)).size + run.duplicate_refs = run.items.length - run.unique_refs + run.by_kind = Object.fromEntries( + [...new Set(run.items.map((item) => item.kind))].map((kind) => [ + kind, + run.items.filter((item) => item.kind === kind).length, + ]), + ) + const direct = new Set(run.items.map((item) => item.ref)) + const context = new Set(run.items.flatMap((item) => item.around.map((one) => one.ref))) + run.anchor_checks = (scenario.questions ?? []).map((question) => ({ + id: question.id, + anchors: question.anchors.map((ref) => ({ + ref, + direct: direct.has(ref), + around: context.has(ref), + })), + })) + run.ref_set_sha256 = sha([...direct].sort().join("\n")) + run.content_set_sha256 = sha( + run.items + .map((item) => `${item.ref}:${item.text_sha256}`) + .sort() + .join("\n"), + ) + report.runs.push(run) + process.stdout.write( + `${name}: ${run.unique_refs} unique references, ${run.wall_ms} ms, pagination ${run.finished_pagination ? "finished" : "capped"}\n`, + ) + } + const after = contextSnapshot() + report.project_review_view_unchanged = JSON.stringify(before) === JSON.stringify(after) + report.project_review_snapshot = before + report.finished_at = new Date().toISOString() + writeFileSync(outputFd, `${JSON.stringify(report, null, 2)}\n`) + process.stdout.write(`Saved aggregate metadata only: ${resolve(output)}\n`) +} finally { + closeSync(outputFd) +} diff --git a/performance/projects/installed-observation/second-project084-base-totals.json b/performance/projects/installed-observation/second-project084-base-totals.json new file mode 100644 index 0000000..6d63d03 --- /dev/null +++ b/performance/projects/installed-observation/second-project084-base-totals.json @@ -0,0 +1,78 @@ +{ + "memo": "0.84.0", + "core": "0.377.0", + "base": "0.47.4", + "node": "v24.19.0", + "private_measurement_sha256": "371455fc81a2e78cfa8d51d5d8a0e448283b585f0c902a6c882e8b11ab1fdd12", + "phases": [ + { + "phase": "index_as_stored", + "invocations": 2, + "wall_ms": 2220.24, + "direct_references": 80, + "surrounding_references": 64, + "combined_references": 80, + "direct_by_kind": { + "message": 10, + "document": 70, + "meeting": 0 + }, + "direct_excerpt_bytes": 36976, + "distinct_surrounding_excerpt_bytes": 53669, + "distinct_surrounding_excerpts": 256, + "charged_read_budget_bytes": 2944014, + "charged_surrounding_budget_bytes": 53672, + "pagination_finished": true, + "no_progress_stop": false, + "oversized_references": 4, + "resolved_sources": 14 + }, + { + "phase": "default_catch_up", + "invocations": 2, + "wall_ms": 2638.03, + "direct_references": 80, + "surrounding_references": 64, + "combined_references": 80, + "direct_by_kind": { + "message": 10, + "document": 70, + "meeting": 0 + }, + "direct_excerpt_bytes": 36976, + "distinct_surrounding_excerpt_bytes": 53669, + "distinct_surrounding_excerpts": 256, + "charged_read_budget_bytes": 2944014, + "charged_surrounding_budget_bytes": 53672, + "pagination_finished": true, + "no_progress_stop": false, + "oversized_references": 4, + "resolved_sources": 14 + }, + { + "phase": "default_repeat", + "invocations": 2, + "wall_ms": 1791.54, + "direct_references": 80, + "surrounding_references": 64, + "combined_references": 80, + "direct_by_kind": { + "message": 10, + "document": 70, + "meeting": 0 + }, + "direct_excerpt_bytes": 36976, + "distinct_surrounding_excerpt_bytes": 53669, + "distinct_surrounding_excerpts": 256, + "charged_read_budget_bytes": 2944014, + "charged_surrounding_budget_bytes": 53672, + "pagination_finished": true, + "no_progress_stop": false, + "oversized_references": 4, + "resolved_sources": 14 + } + ], + "app_model_calls": 0, + "model_tokens": null, + "qualified_as": "Mutable local archive observation; not complete remote coverage, semantic recall, model quality, agent time, actual I/O or cold-cache speedup." +} diff --git a/performance/projects/installed-observation/second-project084-expanded-totals.json b/performance/projects/installed-observation/second-project084-expanded-totals.json new file mode 100644 index 0000000..b600ad5 --- /dev/null +++ b/performance/projects/installed-observation/second-project084-expanded-totals.json @@ -0,0 +1,78 @@ +{ + "memo": "0.84.0", + "core": "0.377.0", + "base": "0.47.4", + "node": "v24.19.0", + "private_measurement_sha256": "d379ab3145b686ad999cfd8e8fdf4edcbcb9b81ce1a316061bbe7978bea44722", + "phases": [ + { + "phase": "index_as_stored", + "invocations": 4, + "wall_ms": 3623.69, + "direct_references": 115, + "surrounding_references": 144, + "combined_references": 167, + "direct_by_kind": { + "message": 15, + "document": 100, + "meeting": 0 + }, + "direct_excerpt_bytes": 51328, + "distinct_surrounding_excerpt_bytes": 65261, + "distinct_surrounding_excerpts": 360, + "charged_read_budget_bytes": 5547144, + "charged_surrounding_budget_bytes": 65763, + "pagination_finished": true, + "no_progress_stop": false, + "oversized_references": 6, + "resolved_sources": 14 + }, + { + "phase": "default_catch_up", + "invocations": 4, + "wall_ms": 3539.47, + "direct_references": 115, + "surrounding_references": 144, + "combined_references": 167, + "direct_by_kind": { + "message": 15, + "document": 100, + "meeting": 0 + }, + "direct_excerpt_bytes": 51328, + "distinct_surrounding_excerpt_bytes": 65261, + "distinct_surrounding_excerpts": 360, + "charged_read_budget_bytes": 5547144, + "charged_surrounding_budget_bytes": 65763, + "pagination_finished": true, + "no_progress_stop": false, + "oversized_references": 6, + "resolved_sources": 14 + }, + { + "phase": "default_repeat", + "invocations": 4, + "wall_ms": 3635.13, + "direct_references": 115, + "surrounding_references": 144, + "combined_references": 167, + "direct_by_kind": { + "message": 15, + "document": 100, + "meeting": 0 + }, + "direct_excerpt_bytes": 51328, + "distinct_surrounding_excerpt_bytes": 65261, + "distinct_surrounding_excerpts": 360, + "charged_read_budget_bytes": 5547144, + "charged_surrounding_budget_bytes": 65763, + "pagination_finished": true, + "no_progress_stop": false, + "oversized_references": 6, + "resolved_sources": 14 + } + ], + "app_model_calls": 0, + "model_tokens": null, + "qualified_as": "Mutable local archive observation; not complete remote coverage, semantic recall, model quality, agent time, actual I/O or cold-cache speedup." +}