Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ Every entry says what changed as a caller sees it, why, and what to watch for

### Added

- **`performance/projects/restore/check-memo.mjs`** exercises published Memo0.83 through real CLI
processes and stdio MCP, with an exact retained dependency lock. It checks private export, explicit
target preview, owner/agent restore policy and historical proposed state using invented stores.

- **`performance/projects/restore/check.mjs`** verifies selected annotation reconstruction against
published Core0.374 with independent IDs, stale-preview refusal and proposed state preserved.
It records release provenance and does not reset a live project or claim full backup coverage.
Expand Down
3 changes: 2 additions & 1 deletion performance/projects/PROCESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ retained bytes; it does not run the applications or establish that a historical
## Existing evidence and its limits

The [published selected restore check](restore/README.md) uses independent synthetic stores and explicit
bindings. It verifies a stale preview and repeat refusal, not a complete project backup or live reset.
bindings. Its Memo CLI/stdio check retains the installed dependency lock and exercises caller policy.
These verify selected restoration, not a complete project backup or live reset.

The [published snapshot check](snapshot/README.md) exercises a selected annotation read with invented
data. It is functional release acceptance, separate from retrieval and extraction scores. It does
Expand Down
2 changes: 1 addition & 1 deletion performance/projects/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Projects

- [Published selected restore](restore/README.md): independent destination bindings, stale preview refusal,
proposed-state preservation and repeat refusal against the released SDK.
proposed-state preservation and repeat refusal against the released SDK and Memo CLI/stdio MCP.

- [Published snapshots](snapshot/README.md): invented-data checks of the released read-only annotation SDK; selected scope, source-body omission and proposed-state preservation.

Expand Down
29 changes: 29 additions & 0 deletions performance/projects/restore/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,32 @@ links. Notes, specialized meeting memberships, tasks, memories, labels and findi
It does not measure retrieval quality, model extraction, performance, connector authorization,
full project backup, injected mid-transaction rollback or concurrent processes. Those require
separate fixtures; no live source was reset. Memo CLI/MCP acceptance is a separate host check.

## Check the Memo command boundary

A working SDK does not establish that the CLI publishes a private file, enforces its caller policy
or transports validation errors through MCP. The [Memo0.83 result](result-memo083.json) records a fresh
npm installation with Core0.374, Base0.47.4 and MCP client2.2.0. Its registry tarball matches release
validation, SHA-256 `8df6a8f5c2d290c32e88a21b65f92f30f4bfa29446f3f033164078912d728852`.

Use the retained [manifest](package-memo083.json) and [lock](lock-memo083.json) to reproduce the same
dependency graph rather than resolving newer transitive packages:

```sh
T=$(mktemp -d -p /var/tmp) && mkdir "$T/package" && cp performance/projects/restore/package-memo083.json "$T/package/package.json" && cp performance/projects/restore/lock-memo083.json "$T/package/package-lock.json" && HOME=$T XDG_CONFIG_HOME=$T/c XDG_DATA_HOME=$T/d XDG_CACHE_HOME=$T/k MESSAGING_STORE=$T/store WIRECAT_STATE_DIR=$T/s TMPDIR=$T NPM_CONFIG_USERCONFIG=/dev/null npm ci --prefix "$T/package" --ignore-scripts --no-audit --no-fund && HOME=$T XDG_CONFIG_HOME=$T/c XDG_DATA_HOME=$T/d XDG_CACHE_HOME=$T/k MESSAGING_STORE=$T/store WIRECAT_STATE_DIR=$T/s TMPDIR=$T node performance/projects/restore/check-memo.mjs "$T/package/node_modules/@wirecat/cli-core/dist/store/index.js" "$T/package/node_modules/@wirecat/cli-memo/dist/bin/memo.js" "$T/package/node_modules/@modelcontextprotocol/client" 0.83.0
```

The driver creates both stores through the shared invented fixture; it accepts no existing store.
Export/check/restore run as real CLI processes. Snapshot/preview run through an actual stdio MCP
client, including read-only mode, unknown-field errors and semantic snapshot refusal. It verifies
private export permissions on Unix, preview target bytes, agent restore refusal, repeated restore
refusal, preserved proposed facts and unrelated records. CLI error JSON is on stderr with empty
stdout; successful CLI and MCP answers keep their standard envelopes.

Owner-labelled and agent-labelled processes are exercised only on newly created synthetic stores.
Caller environment labels are a CLI policy, not operating-system isolation. MCP calls retain their
normal audit writes, so only the export's source bytes and the preview's target bytes are compared.
The driver never claims the entire source remains byte-identical after MCP auditing.

The shared fixture was extracted without changing the earlier SDK assertions or frozen results.
This host check adds no retrieval/extraction quality, OAuth, live-source or full-project backup claim.
137 changes: 137 additions & 0 deletions performance/projects/restore/check-memo.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
import assert from "node:assert/strict"
import { execFileSync, spawnSync } from "node:child_process"
import { existsSync, readFileSync, statSync, writeFileSync } from "node:fs"
import { dirname, join, resolve } from "node:path"
import { pathToFileURL } from "node:url"
import { annotationFixture } from "./fixture.mjs"

const [sdkEntry, memoEntry, clientPackage, expectedVersion] = process.argv.slice(2)
assert(
sdkEntry && memoEntry && clientPackage && expectedVersion,
"pass the installed Core store entry, Memo entry, MCP client package directory and expected Memo version",
)
assert(process.env.HOME && process.env.MESSAGING_STORE && process.env.WIRECAT_STATE_DIR)
assert.equal(existsSync(process.env.MESSAGING_STORE), false)
const sdk = await import(pathToFileURL(resolve(sdkEntry)).href)
const { Client } = await import(pathToFileURL(resolve(clientPackage, "dist/index.mjs")).href)
const { StdioClientTransport } = await import(pathToFileURL(resolve(clientPackage, "dist/stdio.mjs")).href)
const f = await annotationFixture(sdk)
const env = { ...process.env, MESSAGING_STORE: f.source, MEMO_NO_UPDATE_CHECK: "1", AI_AGENT: "", CLAUDECODE: "" }
const invoke = (...args) =>
execFileSync(process.execPath, [resolve(memoEntry), ...args], { env, encoding: "utf8", timeout: 15000 })
assert.equal(invoke("--version").trim(), expectedVersion)
const run = (...args) => {
const answer = JSON.parse(invoke(...args, "--json"))
assert(answer.meta)
assert(answer.item)
return answer.item
}
const directory = dirname(f.source),
snapshotFile = join(directory, "snapshot.json"),
bindingsFile = join(directory, "bindings.json")
const bindings = {
[f.project.ref]: f.destination.ref,
[f.sourceDocument.ref]: f.targetDocument.ref,
[`bot:${f.sourceFact.author.id}`]: `bot:${f.targetFact.author.id}`,
}
writeFileSync(bindingsFile, JSON.stringify(bindings))
const sourceBefore = readFileSync(f.source),
targetBefore = readFileSync(f.target)
const exported = run("projects", "annotations", "export", "REX", "--output", snapshotFile)
assert.deepEqual(exported.counts, { aliases: 1, facts: 1, links: 1 })
assert(!readFileSync(snapshotFile, "utf8").includes(f.body))
assert.deepEqual(readFileSync(f.source), sourceBefore)
if (process.platform !== "win32") assert.equal(statSync(snapshotFile).mode & 0o777, 0o600)
const options = ["--target-store", f.target, "--project", f.destination.ref, "--bindings", bindingsFile]
const preview = run("projects", "annotations", "check", snapshotFile, ...options)
assert.equal(preview.ready, true)
assert.deepEqual(readFileSync(f.target), targetBefore)
const transport = new StdioClientTransport({
command: process.execPath,
args: [resolve(memoEntry), "mcp"],
env: Object.fromEntries(
Object.entries({ ...env, AI_AGENT: "synthetic-host-check", MEMO_READ_ONLY: "1" }).filter(
([, value]) => value !== undefined,
),
),
stderr: "pipe",
})
const client = new Client({ name: "synthetic-annotation-check", version: "1" })
try {
await client.connect(transport)
const tools = (await client.listTools()).tools.map((tool) => tool.name)
assert(tools.includes("project_annotations_snapshot"))
assert(tools.includes("project_annotations_preview"))
assert(!tools.includes("project_annotations_restore"))
const snapshot = await client.callTool({ name: "project_annotations_snapshot", arguments: { project: "REX" } })
assert(!snapshot.isError)
assert(snapshot.structuredContent.meta)
const args = {
snapshot: snapshot.structuredContent.item.snapshot,
target_store: f.target,
project: f.destination.ref,
bindings,
}
const checked = await client.callTool({ name: "project_annotations_preview", arguments: args })
assert(!checked.isError)
assert(checked.structuredContent.meta)
assert.equal(checked.structuredContent.item.ready, true)
const refused = await client.callTool({
name: "project_annotations_snapshot",
arguments: { project: "REX", typo: true },
})
assert.equal(refused.isError, true)
assert.equal(refused.structuredContent.error.code, "validation_error")
assert.match(refused.structuredContent.error.message, /typo/)
const invalid = structuredClone(args)
invalid.snapshot.project.description = " "
const semantic = await client.callTool({ name: "project_annotations_preview", arguments: invalid })
assert.equal(semantic.isError, true)
assert.match(semantic.structuredContent.error.message, /project.description/)
} finally {
await client.close()
}
assert.deepEqual(readFileSync(f.target), targetBefore)
const restore = [
"projects",
"annotations",
"restore",
snapshotFile,
...options,
"--fingerprint",
preview.fingerprint,
"--json",
]
const agent = spawnSync(process.execPath, [resolve(memoEntry), ...restore], {
env: { ...env, AI_AGENT: "synthetic-host-check" },
encoding: "utf8",
timeout: 15000,
})
assert.equal(agent.status, 5)
assert.equal(agent.stdout, "")
assert.equal(JSON.parse(agent.stderr).error.code, "permission_error")
assert.deepEqual(readFileSync(f.target), targetBefore)
const receipt = run(...restore.slice(0, -1))
assert.deepEqual(receipt.counts, { aliases: 1, facts: 1, links: 1 })
assert(Array.isArray(receipt.mappings))
const repeated = spawnSync(process.execPath, [resolve(memoEntry), ...restore], {
env,
encoding: "utf8",
timeout: 15000,
})
assert.equal(repeated.status, 2)
assert.equal(repeated.stdout, "")
assert.equal(JSON.parse(repeated.stderr).error.code, "validation_error")
const target = await sdk.openStore({ path: f.target, readOnly: true })
try {
const facts = await target.facts.list({ project: f.destination.ref })
assert.equal(facts.length, 1)
assert.equal(facts[0].status, "proposed")
assert.equal((await target.facts.list({ project: f.unrelated.ref })).length, 2)
} finally {
await target.close()
}
assert.equal(existsSync(process.env.MESSAGING_STORE), false)
process.stdout.write(
`${JSON.stringify({ memo_version: expectedVersion, cli_export_check_restore: true, stdio_snapshot_preview: true, standard_envelopes: true, unknown_field_named: true, semantic_validation_preserved: true, no_restore_tool: true, agent_restore_refused: true, repeat_refused: true, proposed_state_preserved: true, unrelated_facts_preserved: true, implicit_store_created: false, owner_data_used: false })}\n`,
)
75 changes: 16 additions & 59 deletions performance/projects/restore/check.mjs
Original file line number Diff line number Diff line change
@@ -1,70 +1,27 @@
import assert from "node:assert/strict"
import { existsSync, mkdtempSync, readFileSync } from "node:fs"
import { tmpdir } from "node:os"
import { join, resolve } from "node:path"
import { existsSync, readFileSync } from "node:fs"
import { resolve } from "node:path"
import { pathToFileURL } from "node:url"

import { annotationFixture } from "./fixture.mjs"

const entry = process.argv[2]
assert(entry, "pass the independently installed published package store entry")
assert(process.env.HOME && process.env.MESSAGING_STORE && process.env.WIRECAT_STATE_DIR)
assert.equal(existsSync(process.env.MESSAGING_STORE), false)
const sdk = await import(pathToFileURL(resolve(entry)).href)
const directory = mkdtempSync(join(tmpdir(), "published-project-restore-"))
const source = join(directory, "source.db"),
target = join(directory, "target.db")
const body = "INVENTED_EXTERNAL_BODY_SENTINEL"
let project, destination, unrelated, sourceDocument, targetDocument, sourceFact, targetFact
const seed = async (path, targetSide) => {
const store = await sdk.openStore({ path, now: () => 1700000000000 })
try {
if (targetSide) {
unrelated = await store.knowledge.addProject({ name: "Unrelated Example", key: "PAD" })
await store.facts.add({ type: "statement", statement: "Padding", project: unrelated.ref, by: { bot: "padding" } })
targetFact = await store.facts.add({
type: "statement",
statement: "Keep this unrelated fact",
project: unrelated.ref,
by: { bot: "target-reviewer" },
})
}
const selected = await store.knowledge.addProject({
name: "Research Example",
key: "REX",
...(targetSide ? {} : { description: "Invented portable project" }),
})
const folder = await store.notes.addFolder({ name: "Invented sources" })
if (targetSide)
await store.notes.saveFileNote({ folderId: folder.id, path: "padding.md", title: "Padding", text: "Other text" })
const { note } = await store.notes.saveFileNote({
folderId: folder.id,
path: "evidence.md",
title: "Evidence",
text: body,
})
if (targetSide) {
destination = selected
targetDocument = note
} else {
project = selected
sourceDocument = note
await store.knowledge.addVocabularyAlias({ kind: "project", id: selected.id }, "Example study")
sourceFact = await store.facts.add({
type: "statement",
statement: "A report awaits review",
project: selected.ref,
by: { bot: "source-reviewer" },
evidence: [note.ref],
})
}
} finally {
await store.close()
}
}
await seed(source, false)
await seed(target, true)
assert.notEqual(project.ref, destination.ref)
assert.notEqual(sourceDocument.ref, targetDocument.ref)
assert.notEqual(sourceFact.author.id, targetFact.author.id)
const {
source,
target,
body,
project,
destination,
unrelated,
sourceDocument,
targetDocument,
sourceFact,
targetFact,
} = await annotationFixture(sdk)
const sourceBefore = readFileSync(source),
targetBefore = readFileSync(target)
const snapshot = await sdk.snapshotProjectAnnotations({ source, project: project.ref })
Expand Down
86 changes: 86 additions & 0 deletions performance/projects/restore/fixture.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
import assert from "node:assert/strict"
import { mkdtempSync } from "node:fs"
import { tmpdir } from "node:os"
import { join } from "node:path"

export const annotationFixture = async (sdk) => {
const directory = mkdtempSync(join(tmpdir(), "published-project-restore-"))
const source = join(directory, "source.db"),
target = join(directory, "target.db")
const body = "INVENTED_EXTERNAL_BODY_SENTINEL"
let project, destination, unrelated, sourceDocument, targetDocument, sourceFact, targetFact
const seed = async (path, targetSide) => {
const store = await sdk.openStore({ path, now: () => 1700000000000 })
try {
if (targetSide) {
unrelated = await store.knowledge.addProject({ name: "Unrelated Example", key: "PAD" })
await store.facts.add({
type: "statement",
statement: "Padding",
project: unrelated.ref,
by: { bot: "padding" },
})
targetFact = await store.facts.add({
type: "statement",
statement: "Keep this unrelated fact",
project: unrelated.ref,
by: { bot: "target-reviewer" },
})
}
const selected = await store.knowledge.addProject({
name: "Research Example",
key: "REX",
...(targetSide ? {} : { description: "Invented portable project" }),
})
const folder = await store.notes.addFolder({ name: "Invented sources" })
if (targetSide)
await store.notes.saveFileNote({
folderId: folder.id,
path: "padding.md",
title: "Padding",
text: "Other text",
})
const { note } = await store.notes.saveFileNote({
folderId: folder.id,
path: "evidence.md",
title: "Evidence",
text: body,
})
if (targetSide) {
destination = selected
targetDocument = note
} else {
project = selected
sourceDocument = note
await store.knowledge.addVocabularyAlias({ kind: "project", id: selected.id }, "Example study")
sourceFact = await store.facts.add({
type: "statement",
statement: "A report awaits review",
project: selected.ref,
by: { bot: "source-reviewer" },
evidence: [note.ref],
})
}
} finally {
await store.close()
}
}
await seed(source, false)
await seed(target, true)
assert.notEqual(project.ref, destination.ref)
assert.notEqual(sourceDocument.ref, targetDocument.ref)
assert.notEqual(sourceFact.author.id, targetFact.author.id)

return {
source,
target,
body,
project,
destination,
unrelated,
sourceDocument,
targetDocument,
sourceFact,
targetFact,
}
}
Loading
Loading