diff --git a/performance/projects/PROCESS.md b/performance/projects/PROCESS.md index 36666ee..eeae699 100644 --- a/performance/projects/PROCESS.md +++ b/performance/projects/PROCESS.md @@ -123,6 +123,10 @@ The existing study READMEs own their exact installation commands and bounds: prefix. The public SDK seeds invented documents; actual CLI subprocesses run under the existing external-I/O guard. Keep the bound of one initial call and two resumes plus the separate tiny control. Record the result as a known-limitation reproduction, not a passing retrieval-quality measurement. +5. [Evidence on large documents](evidence-large-document/README.md#setup): a separate test beside the frozen + historical evidence, which stays unchanged. The runner takes a built Memo checkout like the historical one, + but runs its CLI under the external-I/O guard and refuses unless every state folder is inside one fresh + `/var/tmp` directory. Labels are in its own `baseline.json`. For example, after creating the reviewed isolated package prefix as the Drive guide describes: diff --git a/performance/projects/README.md b/performance/projects/README.md index 6ae3d29..e900622 100644 --- a/performance/projects/README.md +++ b/performance/projects/README.md @@ -25,5 +25,8 @@ - [evidence](evidence/README.md): what `memo projects evidence` finds and misses on a synthetic store against a frozen baseline — the project's names, extra words and people, the 60-day window, resume, unread sources. +- [evidence-large-document](evidence-large-document/README.md): the same command on large imported documents — + paragraphs found, paragraphs shown around a hit, pages, charged bytes, `oversized` and resume. A separate test: the + frozen one above is unchanged. - [review](review/README.md): what reviewing an agent's proposals costs with and without context, and how a changed cited text and agent decisions are handled. diff --git a/performance/projects/evidence-large-document/README.md b/performance/projects/evidence-large-document/README.md new file mode 100644 index 0000000..1837bef --- /dev/null +++ b/performance/projects/evidence-large-document/README.md @@ -0,0 +1,171 @@ +# Project evidence on large documents + +## Why we ran it + +The [frozen project evidence test](../evidence/README.md) holds 13 short texts, so it cannot show how +`memo projects evidence` behaves on a large imported document. Two things were never checked: what the command +returns when one document is far larger than a page and its matching paragraphs lie far apart, and which +paragraphs the `around` context shows for a document hit. The risk was recorded as RISK-1268. The owner decided +(NEED-1572, option A) to measure it in a separate test and leave the frozen fixture, its `baseline.json` and its +results unchanged. The decision this feeds: whether a reader who gets a document hit can trust the excerpt and the +`around` paragraphs to point at what matched, and whether a long search finishes when it is cut into steps. + +## Setup + +- AMD Ryzen AI 9 HX 470, Linux 7.0, Node v24.19.0. +- Memo 0.82.0, built from `main` at `579e0af` (`v0.82.0-3-g579e0af`), with `@wirecat/cli-core` 0.371.0 and + `@wirecat/cli-base` 0.47.1. Measured 2026-10-20. +- Limits from Core's [project gathering guide](https://github.com/WireCatLabs/cli-core/blob/main/docs/dev/PROJECT-GATHERING.md): + a page is 64 KiB (65,536 bytes); a record above that is read alone, up to 256 KiB; above 256 KiB it is listed + under `oversized` and not read; the read budget is 2 MiB per invocation. +- Synthetic store of 11 documents in one imported notes folder, project "Lantern Works" with key `LNW` and alias + "Beacon Desk" (three searches, shown as `l1` name, `l2` key, `l3` alias). Paragraphs are split at blank lines and + start with a marker (`p004`), so the paragraph a result shows can be read off its text. + + | Document | Bytes | Paragraphs that name the project | + | --- | --- | --- | + | `large` | 144,829 | p4 (name), p60 (key), p150 (alias), p215 (name) | + | `huge` | 284,459 | p10 (name) | + | `m1` to `m6` | 42,829 each | p30 (name) | + | `small` | 76 | p2 (alias) | + | `decoy-lantern` | 29,865 | none: "lantern" and "works" in the same paragraph | + | `decoy-garden` | 29,841 | none | + +- The right answers are in [`baseline.json`](baseline.json), written before the first run: which documents must be + found, which paragraphs `around` must show, that `huge` is `oversized`, and what the resume must do. The runner + [`evidence-large-document.ts`](evidence-large-document.ts) builds the store through the build's store module, runs + the built `memo projects evidence LNW --json` three ways (default; `--before-n 5 --after-n 5`; `--max-pages 2` and + `--cursor` until it stops) and prints the scores. It runs under the existing + [external-I/O guard](../../documents/drive-shared-store/guard.mjs). It does not touch the owner's store; it + refuses to start unless HOME, the XDG folders, `MESSAGING_STORE`, `WIRECAT_STATE_DIR` and `TMPDIR` are inside + one fresh `/var/tmp` folder. +- The runner reuses no helper from [`../fixture.ts`](../fixture.ts): that file seeds the Harbour Bureau store and + cannot take other documents. The same sandbox idea is repeated, as in the frozen test. + +Command, from the repository root, with a built Memo checkout: + +```sh +S=$(mktemp -d -p /var/tmp ld-XXX) && HOME=$S XDG_CONFIG_HOME=$S/config XDG_DATA_HOME=$S/data XDG_CACHE_HOME=$S/cache MESSAGING_STORE=$S/store.db WIRECAT_STATE_DIR=$S/state TMPDIR=$S node performance/projects/evidence-large-document/evidence-large-document.ts /path/to/cli-memo > performance/projects/evidence-large-document/result.json +``` + +The full output of this run is [`result.json`](result.json). + +## Results + +Default run, one invocation: 13 reads, 12 pages, `complete: true`, no checkpoint, 104 ms. + +| Check | Label (before the run) | Observed | +| --- | --- | --- | +| Documents found | `large`, `m1` to `m6`, `small` | all eight found | +| Decoys not found | `decoy-lantern`, `decoy-garden` | `decoy-garden` not found; **`decoy-lantern` found** by the name search | +| `huge` (284,459 bytes) | listed under `oversized`, charged 0 | `oversized` lists it (284,515 bytes, name search); no item | +| `large` found by | three searches | all three (`l1`, `l2`, `l3`) | +| Charged bytes | at most 1.2 times the size of the documents read | 432,246 for 431,744 bytes of found documents: 1.00 | +| Pages | name search at least 4 | name search 9 (one record per page), key search 1, alias search 2 | +| Mediums and `small`: `around` | p28 to p32 and p1 to p3 | exactly as labelled, the hit paragraph marked `found` | +| `large`: `around` | p2 to p6, around the first paragraph that names the project (p4) | **p58 to p62, around p60** (the key paragraph) | +| `large` with `--before-n 5 --after-n 5` | p1 to p9 | **p55 to p65**, around p60 | +| Paragraphs that name the project, shown in `around` | 8 of 12 | 8 of 12, but one of them is the wrong one: `large` p60 is shown instead of p4; p150, p215 and the paragraph in `huge` (not read) are not | +| `around` bytes (`usage.around_bytes`) | not labelled | 8,171 default; 17,887 with 5 and 5 | + +The excerpt of a hit is the first 300 characters of the document, and `around` shows 200-character excerpts +(Core's guide). So of the four paragraphs in `large` that name the project, a reader sees one, and which one depends +on the search that found the document first, not on the position in the document. + +Resume with `--max-pages 2` and `--cursor`: + +| Invocation | New references | Pages so far | Charged bytes | `complete` | +| --- | --- | --- | --- | --- | +| 1 | 1 | 1 | 42,881 | false | +| 2 | 1 | 2 | 144,887 | false | +| 3 | 2 | 4 | 43,015 | false | +| 4 | 0 | 4 | 0 | false | + +It never completes: 4 of the 9 references the one-shot run returns, no duplicates, and invocation 4 and every later +one read nothing and return the same checkpoint (the runner stops there; the first run went on to 41 invocations +with the same result). `stopped` is `null`, so the output does not say why it stopped. + +## What it showed + +1. **Large-document paging works in one shot.** A 145 KB document and six 43 KB ones are found in a single + invocation, each read once (1.00 of the bytes), and the 284 KB document is reported as `oversized` instead of + stalling the run. +2. **BUG-1: a resumed run stalls** (fixed, see [After the fixes](#after-the-fixes-memo-0840-core-03770)). With `--max-pages 2` the units end `partial` and the checkpoint is returned, + but a resume makes no read and returns the same checkpoint, forever. Tried with `--max-pages 3` and `4`: the + same, at other points. A caller that follows `checkpoint` as the guide says loops without progress. A + likely cause, not confirmed, is the `huge` record or the `read-limited` state of `large` that the first + invocation reports (its units show `read-limited` with `validation_error`); the run was not repeated without + `huge`. +3. **FIND-1: `around` for a document follows the search that found it first, not the document.** Core's guide says + the context is around "the first one that names a lane word, in document order". In `large` that would be p4; + the output is around p60, the paragraph of the key search. The three other matching paragraphs are not shown + and nothing in the item says there are more. Fixed, see [After the fixes](#after-the-fixes-memo-0840-core-03770). +4. **FIND-2: the project name is searched as separate words.** `decoy-lantern` has no "Lantern Works" in it, only + "lantern" and "works" in one paragraph, and the name search returns it. The `--help` text calls each name "its + own literal search". For a project with a common-word name this gives false hits. + +## Not measured + +Timing beyond one wall-clock figure, memory, a store with many thousand documents, messages or mail next to the +documents, a document between 256 KiB and a few MiB through other paths, other budgets (the CLI sets only +`--max-pages`), whether `--max-pages` without `huge` completes, and any real data. All data here is invented. + +## After the fixes (Memo 0.84.0, Core 0.377.0) + +Run 2026-10-20 with the same runner and the same synthetic store. The two findings above were fixed in Core +0.377.0 ([cli-core#1690](https://github.com/WireCatLabs/cli-core/pull/1690), resume; +[cli-core#1694](https://github.com/WireCatLabs/cli-core/pull/1694), document `around` and the new `moreMatches` +field), released in Memo 0.84.0 (`@wirecat/cli-memo` on npm). Setup: Memo built from the tag `v0.84.0` +(`ca74d71`) with `@wirecat/cli-core` 0.377.0, same machine and Node. The runner gained one output field, +`more_matches`, and nothing else; the full output is +[`result-2026-10-20-memo-0.84.0.json`](result-2026-10-20-memo-0.84.0.json). `result.json` and `baseline.json` are +unchanged. + +Default run, one invocation: 13 reads, 12 pages, `complete: true`, no checkpoint, 99 ms. + +| Check | Label (before the run) | Observed | +| --- | --- | --- | +| Documents found | `large`, `m1` to `m6`, `small` | all eight found | +| Decoys not found | `decoy-lantern`, `decoy-garden` | `decoy-garden` not found; `decoy-lantern` found by the name search (expected, see below) | +| `huge` (284,459 bytes) | listed under `oversized`, charged 0 | `oversized` lists it (284,515 bytes, name search); no item | +| `large` found by | three searches | all three (`l1`, `l2`, `l3`) | +| Charged bytes | at most 1.2 times the size of the documents read | 432,246 for 431,744 bytes of found documents: 1.00 | +| Pages | name search at least 4 | name search 9, key search 1, alias search 2 | +| Mediums and `small`: `around` | p28 to p32 and p1 to p3 | exactly as labelled, the hit paragraph marked `found` | +| `large`: `around` | p2 to p6, around the first paragraph that names the project (p4) | **p2 to p6, found p4** (was p58 to p62) | +| `large` with `--before-n 5 --after-n 5` | p1 to p9 | **p1 to p9, found p4** (was p55 to p65) | +| Paragraphs that name the project, shown in `around` | 8 of 12 | 8 of 12, and now the right ones: `large` p4 is shown; p60, p150, p215 and the paragraph in `huge` (not read) are not | +| `around` bytes (`usage.around_bytes`) | not labelled | 8,171 default; 17,483 with 5 and 5 | + +Checks that failed before and pass now: the `large` `around` position (both runs), and the resume. Everything else +is unchanged. + +`moreMatches` in the default run: `large` 3 (p60, p150 and p215 are the other matching paragraphs); every other +document 0. So the item now says that more paragraphs match than the `around` shows, though it still does not say +which. + +Resume with `--max-pages 2` and `--cursor`: + +| Invocation | New references | Pages so far | Charged bytes | `complete` | +| --- | --- | --- | --- | --- | +| 1 | 1 | 1 | 42,881 | false | +| 2 | 1 | 2 | 144,887 | false | +| 3 | 2 | 4 | 43,015 | false | +| 4 | 1 | 6 | 42,881 | false | +| 5 | 2 | 8 | 85,762 | false | +| 6 | 2 | 10 | 72,820 | false | +| 7 | 0 | 12 | 0 | true | + +The resume completes in 7 invocations with the same 9 references as the one-shot run, no duplicates, 12 pages and +432,246 charged bytes in total (the same as the one-shot run), and no invocation reads without progress. + +What is still open: + +- **`decoy-lantern` is found. This is expected behaviour, not a bug.** The owner decided (NEED-1579, option B) to + keep matching the words of a project name anywhere in one text, so "lantern" and "works" in one paragraph match + "Lantern Works". FIND-2 stays as a description of that behaviour, and a project with a common-word name will get + such hits. The baseline still labels the decoy as "not found"; that label is kept as written and is not + re-scored. +- Only the first matching paragraph of a document is shown in `around`; `moreMatches` gives the count of the + others, not their position. +- Everything under "Not measured" above is still not measured. diff --git a/performance/projects/evidence-large-document/baseline.json b/performance/projects/evidence-large-document/baseline.json new file mode 100644 index 0000000..bba0904 --- /dev/null +++ b/performance/projects/evidence-large-document/baseline.json @@ -0,0 +1,105 @@ +{ + "frozen": "2026-10-20", + "written_before_first_run": true, + "request": { + "project": "LNW", + "source": "notes:", + "since_time": "60d", + "around": "default 2 before, 2 after" + }, + "thresholds": { "page_bytes": 65536, "single_record_read_bytes": 262144, "default_read_budget_bytes": 2097152 }, + "documents": { + "large": { + "approx_bytes": 150000, + "paragraphs": 220, + "label": "relevant", + "why": "above one page (64 KiB), below the 256 KiB single-record limit; four paragraphs name the project, far apart", + "hit_paragraphs": { "4": "name", "60": "key", "150": "alias", "215": "name" }, + "around_default": [2, 3, 4, 5, 6], + "around_default_found": [4], + "around_five": [1, 2, 3, 4, 5, 6, 7, 8, 9] + }, + "huge": { + "approx_bytes": 300000, + "paragraphs": 430, + "label": "relevant", + "why": "above 256 KiB: expected to be listed under oversized, not read", + "hit_paragraphs": { "10": "name" } + }, + "m1": { + "approx_bytes": 45000, + "paragraphs": 65, + "label": "relevant", + "why": "medium document, one hit in the middle", + "hit_paragraphs": { "30": "name" }, + "around_default": [28, 29, 30, 31, 32] + }, + "m2": { + "approx_bytes": 45000, + "paragraphs": 65, + "label": "relevant", + "why": "medium document, one hit in the middle", + "hit_paragraphs": { "30": "name" }, + "around_default": [28, 29, 30, 31, 32] + }, + "m3": { + "approx_bytes": 45000, + "paragraphs": 65, + "label": "relevant", + "why": "medium document, one hit in the middle", + "hit_paragraphs": { "30": "name" }, + "around_default": [28, 29, 30, 31, 32] + }, + "m4": { + "approx_bytes": 45000, + "paragraphs": 65, + "label": "relevant", + "why": "medium document, one hit in the middle", + "hit_paragraphs": { "30": "name" }, + "around_default": [28, 29, 30, 31, 32] + }, + "m5": { + "approx_bytes": 45000, + "paragraphs": 65, + "label": "relevant", + "why": "medium document, one hit in the middle", + "hit_paragraphs": { "30": "name" }, + "around_default": [28, 29, 30, 31, 32] + }, + "m6": { + "approx_bytes": 45000, + "paragraphs": 65, + "label": "relevant", + "why": "medium document, one hit in the middle", + "hit_paragraphs": { "30": "name" }, + "around_default": [28, 29, 30, 31, 32] + }, + "small": { + "approx_bytes": 300, + "paragraphs": 3, + "label": "relevant", + "why": "tiny document that names the project only by its alias", + "hit_paragraphs": { "2": "alias" }, + "around_default": [1, 2, 3] + }, + "decoy-lantern": { + "approx_bytes": 30000, + "label": "irrelevant", + "why": "says lantern and works separately, never the project's name" + }, + "decoy-garden": { "approx_bytes": 30000, "label": "irrelevant", "why": "another project's plan" } + }, + "expect": { + "found": ["large", "m1", "m2", "m3", "m4", "m5", "m6", "small"], + "oversized": ["huge"], + "not_found": ["decoy-lantern", "decoy-garden"], + "paragraphs_relevant_total": 12, + "paragraphs_surfaced_by_around": "only those within 2 of the first hit paragraph of each document: large 1 of 4, each medium 1 of 1, small 1 of 1", + "large_lanes": ["name", "key", "alias"], + "pages_name_lane_at_least": 4, + "charged_bytes_per_document_size_max": 1.2, + "charged_bytes_for_oversized_document": 0, + "complete": true, + "resume_two_pages": { "completes": true, "duplicates": 0, "same_references_as_one_run": true } + } +} diff --git a/performance/projects/evidence-large-document/evidence-large-document.ts b/performance/projects/evidence-large-document/evidence-large-document.ts new file mode 100644 index 0000000..fb36125 --- /dev/null +++ b/performance/projects/evidence-large-document/evidence-large-document.ts @@ -0,0 +1,239 @@ +// Project evidence on large imported documents, scored against baseline.json (written before the first run): +// one document above the 64 KiB page, one above the 256 KiB single-record limit, six medium ones that need several +// pages, and decoys. Measures which paragraphs are found, which show in `around`, pages, charged bytes, `oversized` +// and `--max-pages` + `--cursor`. The frozen performance/projects/evidence test cannot reach any of this. +// +// node evidence-large-document.ts +// +// Run it with HOME, XDG_*, MESSAGING_STORE, WIRECAT_STATE_DIR and TMPDIR inside one fresh /var/tmp directory +// (see README.md). Results: README.md. Nothing here opens the owner's store. +import { execFileSync } from "node:child_process" +import { existsSync, readFileSync, realpathSync } from "node:fs" +import { join } from "node:path" +import { pathToFileURL } from "node:url" + +const [memo] = process.argv.slice(2) +if (!memo) throw new Error("usage: node evidence-large-document.ts ") +const baseline = JSON.parse(readFileSync(new URL("baseline.json", import.meta.url), "utf8")) + +const home = realpathSync(process.env.HOME ?? "") +if (!home.startsWith("/var/tmp/")) throw new Error("run with HOME inside a fresh /var/tmp directory") +for (const key of [ + "XDG_CONFIG_HOME", + "XDG_DATA_HOME", + "XDG_CACHE_HOME", + "MESSAGING_STORE", + "WIRECAT_STATE_DIR", + "TMPDIR", +]) + if (!process.env[key]?.startsWith(home)) throw new Error(`${key} must stay inside HOME`) +if (existsSync(process.env.MESSAGING_STORE as string)) throw new Error("refuse an existing store") +const env = { ...process.env, PATH: process.env.PATH ?? "" } + +const filler = [ + "The harbour crane was inspected on schedule and the report was filed without remarks.", + "Quarterly figures were compared with the plan and the difference was explained in the appendix.", + "A short note on vendor invoices: three were paid, two wait for a signature, one was disputed.", + "The meeting room was booked for Thursday and the agenda was sent to all attendees in advance.", + "Weather delays moved the delivery by two days, which the schedule absorbed without further changes.", + "Training material was revised after feedback and the new version replaces the earlier handout.", + "Open questions were collected in a list; each has an owner and a date for the next update.", +] +const paragraph = (index: number, hit?: string) => { + const id = `p${String(index).padStart(3, "0")}` + const body = Array.from({ length: 7 }, (_, i) => filler[(index + i) % filler.length]).join(" ") + return hit ? `${id} ${hit} ${body.slice(0, 420)}` : `${id} ${body}` +} +const document = (paragraphs: number, hits: Record = {}) => + Array.from({ length: paragraphs }, (_, i) => paragraph(i + 1, hits[i + 1])).join("\n\n") + +const NAME = "Lantern Works" +const KEY = "LNW" +const ALIAS = "Beacon Desk" +const said = { + name: `${NAME} confirmed the review.`, + key: `${KEY} review is confirmed.`, + alias: `${ALIAS} confirmed the review.`, +} +const texts: Record = { + large: document(220, { 4: said.name, 60: said.key, 150: said.alias, 215: said.name }), + huge: document(430, { 10: said.name }), + small: "p001 Notes.\n\np002 The Beacon Desk call is on Friday.\n\np003 Bring the folder.", + "decoy-lantern": document(45).replace( + "p020", + "p020 A lantern festival is held in the square and the town works late that night.", + ), + "decoy-garden": document(45).replace("p020", "p020 Garden plan: beds, water and compost for the spring."), +} +for (const key of ["m1", "m2", "m3", "m4", "m5", "m6"]) texts[key] = document(65, { 30: said.name }) + +const { openStore } = await import(pathToFileURL(join(memo, "node_modules/@wirecat/cli-core/dist/store/index.js")).href) +const store = await openStore({ env }) +const project = await store.knowledge.addProject({ name: NAME, key: KEY }) +await store.knowledge.addVocabularyAlias({ kind: "project", id: project.id }, ALIAS) +const folder = await store.notes.addFolder({ name: "Large documents" }) +const keyOf: Record = {} +for (const [key, text] of Object.entries(texts)) { + const saved = await store.notes.saveFileNote({ folderId: folder.id, path: `${key}.md`, title: key, text }) + keyOf[saved.note.ref] = key +} +await store.fillSearchIndex?.() +await store.close() +const size = (key: string) => Buffer.byteLength(texts[key] as string) + +const memoBin = join(memo, "dist/bin/memo.js") +const guard = new URL("../../documents/drive-shared-store/guard.mjs", import.meta.url).pathname +const evidence = (...args: string[]) => + JSON.parse( + execFileSync( + process.execPath, + ["--import", guard, memoBin, "projects", "evidence", KEY, "--source", `notes:${folder.id}`, ...args, "--json"], + { env, encoding: "utf8", timeout: 120_000, maxBuffer: 16 * 1024 * 1024 }, + ), + ).item + +type Item = { + ref: string + lanes: string[] + text: string + around?: { ref: string; text: string; found?: boolean }[] | null + moreMatches?: number + more_matches?: number +} +const marker = (text: string) => Number(/^p(\d{3})/.exec(text)?.[1] ?? Number.NaN) +const summarise = (run: { + items: Item[] + oversized: { ref: string; bytes: number; lanes: string[] }[] + units: { state: string; pages: number; issue: string | null; [k: string]: unknown }[] + usage: Record + totals: Record + complete: boolean + checkpoint: string | null + stopped: string | null + matched_again: unknown[] +}) => { + const found = run.items.map((item) => keyOf[item.ref] ?? `unknown:${item.ref}`) + const around = Object.fromEntries( + run.items.map((item) => [ + keyOf[item.ref], + { + paragraphs: (item.around ?? []).map((one) => marker(one.text)), + found_marked: (item.around ?? []).filter((one) => one.found).map((one) => marker(one.text)), + in_other_document: (item.around ?? []).filter((one) => one.ref !== item.ref).length, + }, + ]), + ) + return { + found, + more_matches: Object.fromEntries( + run.items.map((item) => [keyOf[item.ref], item.moreMatches ?? item.more_matches ?? null]), + ), + lanes_per_document: Object.fromEntries(run.items.map((item) => [keyOf[item.ref], item.lanes])), + around, + oversized: run.oversized.map((one) => ({ + document: keyOf[one.ref] ?? one.ref, + bytes: one.bytes, + lanes: one.lanes, + })), + units: run.units.map(({ state, pages, issue, ...rest }) => ({ ...rest, state, pages, issue })), + pages_total: run.units.reduce((sum, unit) => sum + unit.pages, 0), + usage: run.usage, + totals: run.totals, + complete: run.complete, + has_checkpoint: run.checkpoint !== null, + stopped: run.stopped, + matched_again: run.matched_again.length, + } +} + +const expect = baseline.expect +const docs = baseline.documents as Record< + string, + { hit_paragraphs?: Record; around_default?: number[] } +> +const sameSet = (a: string[], b: string[]) => [...a].sort().join() === [...b].sort().join() + +const one = evidence() +const base = summarise(one) +const wide = summarise(evidence("--before-n", "5", "--after-n", "5")) +const readable = expect.found as string[] +const bytesRead = base.found.reduce((sum, key) => sum + size(key), 0) + +const aroundOk = Object.fromEntries( + readable.map((key) => [ + key, + sameSet((base.around[key]?.paragraphs ?? []).map(String), (docs[key]?.around_default ?? []).map(String)), + ]), +) +const relevantParagraphs = Object.entries(docs).flatMap(([key, doc]) => + Object.keys(doc.hit_paragraphs ?? {}).map((index) => ({ key, index: Number(index) })), +) +const surfaced = relevantParagraphs.filter(({ key, index }) => base.around[key]?.paragraphs.includes(index)) + +const pages: { new_refs: number; pages: number; charged_bytes: number; complete: boolean }[] = [] +const refs: string[] = [] +let stalled = false +let step = evidence("--max-pages", "2") +for (let resumes = 0; ; resumes++) { + refs.push(...step.items.map((item: Item) => item.ref)) + pages.push({ + new_refs: step.items.length, + pages: step.units.reduce((sum: number, unit: { pages: number }) => sum + unit.pages, 0), + charged_bytes: step.usage.read_bytes, + complete: step.complete, + }) + if (!step.checkpoint || resumes >= 40) break + if (resumes > 0 && step.usage.reads === 0 && step.items.length === 0) { + stalled = true + break + } + step = evidence("--max-pages", "2", "--cursor", step.checkpoint) +} + +process.stdout.write( + `${JSON.stringify( + { + memo: execFileSync(process.execPath, [memoBin, "--version"], { env, encoding: "utf8" }).trim(), + node: process.version, + document_bytes: Object.fromEntries(Object.keys(texts).map((key) => [key, size(key)])), + default_run: { + ...base, + found_all_readable: sameSet(base.found, readable), + unexpected_found: base.found.filter((key) => !readable.includes(key)), + charged_bytes: base.usage.read_bytes, + found_document_bytes: bytesRead, + charged_per_document_byte: +(base.usage.read_bytes / bytesRead).toFixed(2), + oversized_ok: sameSet( + base.oversized.map((one) => one.document), + expect.oversized, + ), + around_matches_label: aroundOk, + relevant_paragraphs: relevantParagraphs.length, + relevant_paragraphs_in_around: surfaced.length, + relevant_paragraphs_not_surfaced: relevantParagraphs + .filter((one) => !surfaced.includes(one) && !base.oversized.some((over) => over.document === one.key)) + .map(({ key, index }) => `${key}:p${index}`), + }, + around_five_each_side: { + large: wide.around.large, + expected_large: baseline.documents.large.around_five, + around_bytes: wide.usage.around_bytes, + }, + resume_two_pages: { + invocations: pages.length, + per_invocation: pages, + references: refs.length, + duplicates: refs.length - new Set(refs).size, + same_references_as_one_run: sameSet( + [...new Set(refs)].map((ref) => keyOf[ref] ?? ref), + base.found, + ), + cumulative_charged_bytes: pages.reduce((sum, one) => sum + one.charged_bytes, 0), + completed: pages.at(-1)?.complete ?? false, + stalled_without_a_read: stalled, + }, + }, + null, + 2, + )}\n`, +) diff --git a/performance/projects/evidence-large-document/result-2026-10-20-memo-0.84.0.json b/performance/projects/evidence-large-document/result-2026-10-20-memo-0.84.0.json new file mode 100644 index 0000000..7630c92 --- /dev/null +++ b/performance/projects/evidence-large-document/result-2026-10-20-memo-0.84.0.json @@ -0,0 +1,224 @@ +{ + "memo": "0.84.0", + "node": "v24.19.0", + "document_bytes": { + "large": 144829, + "huge": 284459, + "small": 76, + "decoy-lantern": 29865, + "decoy-garden": 29841, + "m1": 42829, + "m2": 42829, + "m3": 42829, + "m4": 42829, + "m5": 42829, + "m6": 42829 + }, + "default_run": { + "found": ["m6", "large", "small", "m5", "m4", "m3", "m2", "m1", "decoy-lantern"], + "more_matches": { + "m6": 0, + "large": 3, + "small": 0, + "m5": 0, + "m4": 0, + "m3": 0, + "m2": 0, + "m1": 0, + "decoy-lantern": 0 + }, + "lanes_per_document": { + "m6": ["l1"], + "large": ["l2", "l3", "l1"], + "small": ["l3"], + "m5": ["l1"], + "m4": ["l1"], + "m3": ["l1"], + "m2": ["l1"], + "m1": ["l1"], + "decoy-lantern": ["l1"] + }, + "around": { + "m6": { + "paragraphs": [28, 29, 30, 31, 32], + "found_marked": [30], + "in_other_document": 0 + }, + "large": { + "paragraphs": [2, 3, 4, 5, 6], + "found_marked": [4], + "in_other_document": 0 + }, + "small": { + "paragraphs": [1, 2, 3], + "found_marked": [2], + "in_other_document": 0 + }, + "m5": { + "paragraphs": [28, 29, 30, 31, 32], + "found_marked": [30], + "in_other_document": 0 + }, + "m4": { + "paragraphs": [28, 29, 30, 31, 32], + "found_marked": [30], + "in_other_document": 0 + }, + "m3": { + "paragraphs": [28, 29, 30, 31, 32], + "found_marked": [30], + "in_other_document": 0 + }, + "m2": { + "paragraphs": [28, 29, 30, 31, 32], + "found_marked": [30], + "in_other_document": 0 + }, + "m1": { + "paragraphs": [28, 29, 30, 31, 32], + "found_marked": [30], + "in_other_document": 0 + }, + "decoy-lantern": { + "paragraphs": [18, 19, 20, 21, 22], + "found_marked": [20], + "in_other_document": 0 + } + }, + "oversized": [ + { + "document": "huge", + "bytes": 284515, + "lanes": ["l1"] + } + ], + "units": [ + { + "lane": "l1", + "source": 0, + "after": 1787347295456, + "before": 1792531295456, + "state": "done", + "pages": 9, + "issue": null + }, + { + "lane": "l2", + "source": 0, + "after": 1787347295456, + "before": 1792531295456, + "state": "done", + "pages": 1, + "issue": null + }, + { + "lane": "l3", + "source": 0, + "after": 1787347295456, + "before": 1792531295456, + "state": "done", + "pages": 2, + "issue": null + } + ], + "pages_total": 12, + "usage": { + "reads": 13, + "read_bytes": 432246, + "around_bytes": 8171, + "items": 9, + "elapsed_ms": 102 + }, + "totals": { + "reads": 13, + "read_bytes": 432246, + "items": 9, + "invocations": 1 + }, + "complete": true, + "has_checkpoint": false, + "stopped": null, + "matched_again": 0, + "found_all_readable": false, + "unexpected_found": ["decoy-lantern"], + "charged_bytes": 432246, + "found_document_bytes": 431744, + "charged_per_document_byte": 1, + "oversized_ok": true, + "around_matches_label": { + "large": true, + "m1": true, + "m2": true, + "m3": true, + "m4": true, + "m5": true, + "m6": true, + "small": true + }, + "relevant_paragraphs": 12, + "relevant_paragraphs_in_around": 8, + "relevant_paragraphs_not_surfaced": ["large:p60", "large:p150", "large:p215"] + }, + "around_five_each_side": { + "large": { + "paragraphs": [1, 2, 3, 4, 5, 6, 7, 8, 9], + "found_marked": [4], + "in_other_document": 0 + }, + "expected_large": [1, 2, 3, 4, 5, 6, 7, 8, 9], + "around_bytes": 17483 + }, + "resume_two_pages": { + "invocations": 7, + "per_invocation": [ + { + "new_refs": 1, + "pages": 1, + "charged_bytes": 42881, + "complete": false + }, + { + "new_refs": 1, + "pages": 2, + "charged_bytes": 144887, + "complete": false + }, + { + "new_refs": 2, + "pages": 4, + "charged_bytes": 43015, + "complete": false + }, + { + "new_refs": 1, + "pages": 6, + "charged_bytes": 42881, + "complete": false + }, + { + "new_refs": 2, + "pages": 8, + "charged_bytes": 85762, + "complete": false + }, + { + "new_refs": 2, + "pages": 10, + "charged_bytes": 72820, + "complete": false + }, + { + "new_refs": 0, + "pages": 12, + "charged_bytes": 0, + "complete": true + } + ], + "references": 9, + "duplicates": 0, + "same_references_as_one_run": true, + "cumulative_charged_bytes": 432246, + "completed": true, + "stalled_without_a_read": false + } +} diff --git a/performance/projects/evidence-large-document/result.json b/performance/projects/evidence-large-document/result.json new file mode 100644 index 0000000..c1eac95 --- /dev/null +++ b/performance/projects/evidence-large-document/result.json @@ -0,0 +1,195 @@ +{ + "memo": "0.82.0", + "node": "v24.19.0", + "document_bytes": { + "large": 144829, + "huge": 284459, + "small": 76, + "decoy-lantern": 29865, + "decoy-garden": 29841, + "m1": 42829, + "m2": 42829, + "m3": 42829, + "m4": 42829, + "m5": 42829, + "m6": 42829 + }, + "default_run": { + "found": ["m6", "large", "small", "m5", "m4", "m3", "m2", "m1", "decoy-lantern"], + "lanes_per_document": { + "m6": ["l1"], + "large": ["l2", "l3", "l1"], + "small": ["l3"], + "m5": ["l1"], + "m4": ["l1"], + "m3": ["l1"], + "m2": ["l1"], + "m1": ["l1"], + "decoy-lantern": ["l1"] + }, + "around": { + "m6": { + "paragraphs": [28, 29, 30, 31, 32], + "found_marked": [30], + "in_other_document": 0 + }, + "large": { + "paragraphs": [58, 59, 60, 61, 62], + "found_marked": [60], + "in_other_document": 0 + }, + "small": { + "paragraphs": [1, 2, 3], + "found_marked": [2], + "in_other_document": 0 + }, + "m5": { + "paragraphs": [28, 29, 30, 31, 32], + "found_marked": [30], + "in_other_document": 0 + }, + "m4": { + "paragraphs": [28, 29, 30, 31, 32], + "found_marked": [30], + "in_other_document": 0 + }, + "m3": { + "paragraphs": [28, 29, 30, 31, 32], + "found_marked": [30], + "in_other_document": 0 + }, + "m2": { + "paragraphs": [28, 29, 30, 31, 32], + "found_marked": [30], + "in_other_document": 0 + }, + "m1": { + "paragraphs": [28, 29, 30, 31, 32], + "found_marked": [30], + "in_other_document": 0 + }, + "decoy-lantern": { + "paragraphs": [18, 19, 20, 21, 22], + "found_marked": [20], + "in_other_document": 0 + } + }, + "oversized": [ + { + "document": "huge", + "bytes": 284515, + "lanes": ["l1"] + } + ], + "units": [ + { + "lane": "l1", + "source": 0, + "after": 1787326090153, + "before": 1792510090153, + "state": "done", + "pages": 9, + "issue": null + }, + { + "lane": "l2", + "source": 0, + "after": 1787326090153, + "before": 1792510090153, + "state": "done", + "pages": 1, + "issue": null + }, + { + "lane": "l3", + "source": 0, + "after": 1787326090153, + "before": 1792510090153, + "state": "done", + "pages": 2, + "issue": null + } + ], + "pages_total": 12, + "usage": { + "reads": 13, + "read_bytes": 432246, + "around_bytes": 8171, + "items": 9, + "elapsed_ms": 104 + }, + "totals": { + "reads": 13, + "read_bytes": 432246, + "items": 9, + "invocations": 1 + }, + "complete": true, + "has_checkpoint": false, + "stopped": null, + "matched_again": 0, + "found_all_readable": false, + "unexpected_found": ["decoy-lantern"], + "charged_bytes": 432246, + "found_document_bytes": 431744, + "charged_per_document_byte": 1, + "oversized_ok": true, + "around_matches_label": { + "large": false, + "m1": true, + "m2": true, + "m3": true, + "m4": true, + "m5": true, + "m6": true, + "small": true + }, + "relevant_paragraphs": 12, + "relevant_paragraphs_in_around": 8, + "relevant_paragraphs_not_surfaced": ["large:p4", "large:p150", "large:p215"] + }, + "around_five_each_side": { + "large": { + "paragraphs": [55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65], + "found_marked": [60], + "in_other_document": 0 + }, + "expected_large": [1, 2, 3, 4, 5, 6, 7, 8, 9], + "around_bytes": 17887 + }, + "resume_two_pages": { + "invocations": 4, + "per_invocation": [ + { + "new_refs": 1, + "pages": 1, + "charged_bytes": 42881, + "complete": false + }, + { + "new_refs": 1, + "pages": 2, + "charged_bytes": 144887, + "complete": false + }, + { + "new_refs": 2, + "pages": 4, + "charged_bytes": 43015, + "complete": false + }, + { + "new_refs": 0, + "pages": 4, + "charged_bytes": 0, + "complete": false + } + ], + "references": 4, + "duplicates": 0, + "same_references_as_one_run": false, + "cumulative_charged_bytes": 230783, + "completed": false, + "stalled_without_a_read": true + } +}