Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ Every entry says what changed as a caller sees it, why, and what to watch for

## Unreleased

- Add an opt-in installed Memo 0.84.0 observation runner and aggregate second-project name/contact results.

### Added

- **`performance/projects/restore/check-memo.mjs`** exercises published Memo0.83 through real CLI
Expand Down
2 changes: 1 addition & 1 deletion performance/projects/PROCESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ data. It is functional release acceptance, separate from retrieval and extractio
not establish a complete project backup or a safe reset/restore path.

The [installed post-fix observation](installed-observation/README.md) is a separate mutable-archive
measurement. It publishes totals through an allowlisted projection; cases, source references and
measurement, including a second project on Memo 0.84.0 with explicit name/contact hints. It publishes totals through an allowlisted projection; cases, source references and
per-item rows remain private. Do not combine it with the frozen synthetic precision/recall scores.

| Study | What is measured | Runtime provenance | What it does not establish |
Expand Down
2 changes: 1 addition & 1 deletion performance/projects/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
- [Published snapshots](snapshot/README.md): invented-data checks of the released read-only annotation SDK; selected scope, source-body omission and proposed-state preservation.

- [Installed observation](installed-observation/README.md): allowlisted totals from a private fixed-window
gathering rerun after the document pagination fix; finished pagination remains separate from archive
gathering rerun after the document pagination fix and a second-project name/contact trial; finished pagination remains separate from archive
completeness, relevance and interpretation.

- [Extraction contract](extraction-contract/README.md): fake-completion checks of released SDK schema,
Expand Down
51 changes: 51 additions & 0 deletions performance/projects/installed-observation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,3 +104,54 @@ checksum, three phase totals and qualifications. Neither includes the project's
source references, source text, per-question answers or labels. Original totals remain unchanged.
Finished pagination still leaves remote completeness and semantic interpretation unknown; source
review and proposed-finding decisions are separate from this measurement.

## Second project on Memo 0.84.0

The owner asked whether gathering works beyond the first project. A second existing project has a
confirmed key document and person relationship. Its key document supplies alternative names and a
warning that some matching documents were copied into another project. The test asks what ordinary
name/key gathering finds, and how explicit name/contact hints change its candidate set.

Linux x64, Node 24.19.0, installed registry Memo 0.84.0/Core 0.377.0/Base 0.47.4;
[release run](https://github.com/WireCatLabs/cli-memo/actions/runs/39485604512) succeeded.
A fixed exclusive cutoff and 60-day window are reused throughout. Each invocation allows 100 items
and 96 reader pages; each phase permits at most four invocations. The expanded request adds three
alternative names and one existing person reference without saving aliases or relationships.

| Request | Direct refs | Messages / documents | Calls per phase | Phase wall times, ms | Pagination |
| --- | ---: | ---: | ---: | --- | --- |
| Existing name/key | 80 | 10 / 70 | 2 | 2,220.24 / 2,638.03 / 1,791.54 | Finished |
| Added names/contact | 115 | 15 / 100 | 4 | 3,623.69 / 3,539.47 / 3,635.13 | Finished |

All 80 original references remain; 35 are added. The expanded set labels 114 as project candidates
and one as person-only. These labels explain matching lanes, not verified membership. Within each
request, all three phases return identical direct reference/excerpt-hash sets and zero duplicates.
The bounded project-review view is unchanged. Expanded-run before/after runtime fingerprints match;
the baseline was not bracketed by a software fingerprint. No claim of whole-store immutability follows.

Finished pagination is not finished reading: 76 baseline and 107 expanded excerpts are truncated.
Eleven baseline and three expanded hits report surrounding-context issues. Internal notes are
reported unread. No meetings matched; this does not establish complete meeting coverage.
A bounded qualitative read found a document explicitly warning that its files describe another
project: name matching usefully retrieves a warning but cannot decide the document's project membership.
No semantic accuracy, blind recall, token savings, current business state or controlled speedup is measured.

Re-run the exact retained private cases only against the installed released binary, using a new
private result path. This intentionally reads the selected real archive; synthetic checks keep temporary
HOME/store/state instead. The script requires explicit opt-in, pins package versions, refuses an existing
output before any archive read, retains references/hashes/coverage, and never writes source bodies:

```sh
T=$(mktemp -d -p /var/tmp) && HOME=/path/to/selected/home TMPDIR=$T node performance/projects/installed-observation/measure-local084.mjs --live-local /PRIVATE/case.json /PRIVATE/new-result.json /ABSOLUTE/installed/memo
```

Project public totals with the unchanged projector above. Cases, runtime fingerprints, source references,
manual reading notes and results are private in `2026-10-20-second-project-gathering.md`.
[Baseline totals](second-project084-base-totals.json) and
[expanded totals](second-project084-expanded-totals.json) contain aggregate counts and input checksums only.
The earlier measurements, cases and frozen synthetic benchmarks remain unchanged.

The exact executed source is retained privately as `measure-pmed084.mjs`, matching each recorded
runner checksum. The public runner differs only in sorted import names for repository lint.
An isolated occupied-output check refused with EEXIST before any Memo invocation; the existing
aggregate projection sentinel checks pass unchanged.
225 changes: 225 additions & 0 deletions performance/projects/installed-observation/measure-local084.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,225 @@
import assert from "node:assert/strict"
import { execFileSync } from "node:child_process"
import { createHash } from "node:crypto"
import { closeSync, openSync, readFileSync, realpathSync, writeFileSync } from "node:fs"
import { dirname, resolve } from "node:path"
import { performance } from "node:perf_hooks"
import { fileURLToPath } from "node:url"

const [consent, casePath, output, binaryPath] = process.argv.slice(2)
assert.equal(
consent,
"--live-local",
"explicit --live-local is required; uses the owner's installed CLI and local archive",
)
assert.ok(output, "provide a new aggregate-result JSON path")
assert.ok(casePath && binaryPath, "provide private case, new output and installed binary paths")
const scenario = JSON.parse(readFileSync(resolve(casePath), "utf8"))
const binary = realpathSync(binaryPath)
const packageRoot = resolve(dirname(realpathSync(binary)), "../..")
const manifest = JSON.parse(readFileSync(resolve(packageRoot, "package.json"), "utf8"))
assert.equal(manifest.version, "0.84.0", "this measured protocol pins published Memo 0.84.0")
assert.equal(manifest.dependencies["@wirecat/cli-core"], "0.377.0")
const sha = (value) => createHash("sha256").update(value).digest("hex")
const textMeta = (item) => ({
ref: item.ref,
fingerprint: item.fingerprint ?? null,
kind: item.kind ?? null,
time_kind: item.time_kind ?? null,
context: typeof item.context === "string" ? item.context : null,
text_truncated: item.text_truncated ?? null,
around_issue: item.around_issue ?? null,
lanes: item.lanes ?? [],
text_sha256: sha(item.text ?? ""),
text_utf8_bytes: Buffer.byteLength(item.text ?? ""),
around: (item.around ?? []).map((one) => ({
ref: one.ref,
found: one.found ?? null,
text_sha256: sha(one.text ?? ""),
text_utf8_bytes: Buffer.byteLength(one.text ?? ""),
})),
})
const invoke = (args) => {
const start = performance.now()
let raw
try {
raw = execFileSync(binary, args, {
env: process.env,
encoding: "utf8",
timeout: 60000,
maxBuffer: 16 * 1024 * 1024,
stdio: ["ignore", "pipe", "pipe"],
})
} catch (error) {
const safe = new Error(
`installed Memo failed (exit ${error.status ?? "none"}, signal ${error.signal ?? "none"}); captured output is withheld`,
)
throw safe
}
const envelope = JSON.parse(raw)
assert.ok(envelope.item && envelope.meta, "expected a complete item envelope")
return {
envelope,
wall_ms: +(performance.now() - start).toFixed(2),
stdout_utf8_bytes: Buffer.byteLength(raw),
}
}
const contextSnapshot = () => {
const { envelope } = invoke(["projects", "context", scenario.project, "--json"])
const value = envelope.item
return {
project: value.project.ref,
key_documents: value.key_documents.map((item) => ({
ref: item.ref,
confirmed: item.confirmed,
})),
relationships: value.relationships.map((item) => ({
id: item.id,
confirmed: item.confirmed,
})),
facts: value.review.facts.items.map((item) => ({
ref: item.ref,
status: item.status,
})),
findings_batches: value.review.meeting_actions.items.map((item) => item.id),
open_tasks: value.open_tasks.items.map((item) => item.ref),
partial_view: true,
}
}
const report = {
case_id: scenario.case_id,
started_at: new Date().toISOString(),
node: process.version,
memo: manifest.version,
dependencies: {
core: manifest.dependencies["@wirecat/cli-core"],
base: manifest.dependencies["@wirecat/cli-base"],
},
case_sha256: sha(readFileSync(resolve(casePath))),
runner_sha256: sha(readFileSync(fileURLToPath(import.meta.url))),
request: scenario,
runs: [],
llm_tokens: null,
remote_fetches_requested: 0,
qualification:
"Live local observation; no raw source text saved, no project reset, no knowledge proposals. Default catch-up may update local indexes. Time is CLI wall time, not agent analysis time. Owner archive is mutable; references and hashes detect drift but are not an offline corpus snapshot.",
}
const outputFd = openSync(resolve(output), "wx", 0o600)
try {
const before = contextSnapshot()
for (const [name, catchUp] of [
["index_as_is", false],
["default_first", true],
["default_repeat", true],
]) {
const args = [
"projects",
"evidence",
scenario.project,
"--source",
scenario.source,
"--since-time",
scenario.since_time,
"--before-time",
scenario.before_time,
"--limit",
String(scenario.limit),
"--max-pages",
String(scenario.max_pages),
...(!catchUp ? ["--no-catch-up"] : []),
...(scenario.aliases ?? []).flatMap((value) => ["--alias", value]),
...(scenario.contacts ?? []).flatMap((value) => ["--contact", value]),
"--json",
]
const run = {
name,
args,
pages: [],
items: [],
stagnation_detected: false,
}
let cursor
let previousProgress
for (let page = 0; page < scenario.max_invocations; page++) {
const { envelope, wall_ms, stdout_utf8_bytes } = invoke([...args, ...(cursor ? ["--cursor", cursor] : [])])
const value = envelope.item
assert.equal(value.window.before, Date.parse(scenario.before_time), "all phases must use the fixed exclusive end")
run.pages.push({
wall_ms,
stdout_utf8_bytes,
meta: { duration_ms: envelope.meta.duration_ms },
window: value.window,
scope: value.scope,
lanes: value.lanes,
usage: value.usage,
totals: value.totals,
index: value.index,
stopped: value.stopped ?? null,
complete: value.complete,
resumed: value.resumed,
completeness: value.completeness,
units: value.units,
unread_sources: value.unread_sources,
oversized: (value.oversized ?? []).map(({ ref, bytes, lanes, source }) => ({ ref, bytes, lanes, source })),
matched_again: (value.matchedAgain ?? value.matched_again ?? []).map(({ ref, lanes }) => ({ ref, lanes })),
checkpoint_present: Boolean(value.checkpoint),
})
run.items.push(...value.items.map(textMeta))
cursor = value.checkpoint
const progress = JSON.stringify(
value.units.map(({ lane, source, state, pages, issue }) => ({
lane,
source,
state,
pages,
issue,
})),
)
if (cursor && value.items.length === 0 && progress === previousProgress) {
run.stagnation_detected = true
break
}
previousProgress = progress
if (!cursor) break
}
run.finished_pagination = !cursor
run.wall_ms = +run.pages.reduce((sum, page) => sum + page.wall_ms, 0).toFixed(2)
run.unique_refs = new Set(run.items.map((item) => item.ref)).size
run.duplicate_refs = run.items.length - run.unique_refs
run.by_kind = Object.fromEntries(
[...new Set(run.items.map((item) => item.kind))].map((kind) => [
kind,
run.items.filter((item) => item.kind === kind).length,
]),
)
const direct = new Set(run.items.map((item) => item.ref))
const context = new Set(run.items.flatMap((item) => item.around.map((one) => one.ref)))
run.anchor_checks = (scenario.questions ?? []).map((question) => ({
id: question.id,
anchors: question.anchors.map((ref) => ({
ref,
direct: direct.has(ref),
around: context.has(ref),
})),
}))
run.ref_set_sha256 = sha([...direct].sort().join("\n"))
run.content_set_sha256 = sha(
run.items
.map((item) => `${item.ref}:${item.text_sha256}`)
.sort()
.join("\n"),
)
report.runs.push(run)
process.stdout.write(
`${name}: ${run.unique_refs} unique references, ${run.wall_ms} ms, pagination ${run.finished_pagination ? "finished" : "capped"}\n`,
)
}
const after = contextSnapshot()
report.project_review_view_unchanged = JSON.stringify(before) === JSON.stringify(after)
report.project_review_snapshot = before
report.finished_at = new Date().toISOString()
writeFileSync(outputFd, `${JSON.stringify(report, null, 2)}\n`)
process.stdout.write(`Saved aggregate metadata only: ${resolve(output)}\n`)
} finally {
closeSync(outputFd)
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
{
"memo": "0.84.0",
"core": "0.377.0",
"base": "0.47.4",
"node": "v24.19.0",
"private_measurement_sha256": "371455fc81a2e78cfa8d51d5d8a0e448283b585f0c902a6c882e8b11ab1fdd12",
"phases": [
{
"phase": "index_as_stored",
"invocations": 2,
"wall_ms": 2220.24,
"direct_references": 80,
"surrounding_references": 64,
"combined_references": 80,
"direct_by_kind": {
"message": 10,
"document": 70,
"meeting": 0
},
"direct_excerpt_bytes": 36976,
"distinct_surrounding_excerpt_bytes": 53669,
"distinct_surrounding_excerpts": 256,
"charged_read_budget_bytes": 2944014,
"charged_surrounding_budget_bytes": 53672,
"pagination_finished": true,
"no_progress_stop": false,
"oversized_references": 4,
"resolved_sources": 14
},
{
"phase": "default_catch_up",
"invocations": 2,
"wall_ms": 2638.03,
"direct_references": 80,
"surrounding_references": 64,
"combined_references": 80,
"direct_by_kind": {
"message": 10,
"document": 70,
"meeting": 0
},
"direct_excerpt_bytes": 36976,
"distinct_surrounding_excerpt_bytes": 53669,
"distinct_surrounding_excerpts": 256,
"charged_read_budget_bytes": 2944014,
"charged_surrounding_budget_bytes": 53672,
"pagination_finished": true,
"no_progress_stop": false,
"oversized_references": 4,
"resolved_sources": 14
},
{
"phase": "default_repeat",
"invocations": 2,
"wall_ms": 1791.54,
"direct_references": 80,
"surrounding_references": 64,
"combined_references": 80,
"direct_by_kind": {
"message": 10,
"document": 70,
"meeting": 0
},
"direct_excerpt_bytes": 36976,
"distinct_surrounding_excerpt_bytes": 53669,
"distinct_surrounding_excerpts": 256,
"charged_read_budget_bytes": 2944014,
"charged_surrounding_budget_bytes": 53672,
"pagination_finished": true,
"no_progress_stop": false,
"oversized_references": 4,
"resolved_sources": 14
}
],
"app_model_calls": 0,
"model_tokens": null,
"qualified_as": "Mutable local archive observation; not complete remote coverage, semantic recall, model quality, agent time, actual I/O or cold-cache speedup."
}
Loading
Loading