Build tooling work, commits C25a,e,f, C26-C30.

WIP.Harness.md

@@ -429,11 +429,21 @@ Four smaller things it knows, each of which cost a run:
   was measured: a `[RunAfterBuild]` Sub that shifts a `Single` (BUGS-TO-REPORT.md) builds with
   `[LINKER] SUCCESS`, the console adds `[BUILD] Executing 'DocSamples.Probe.Run'...` and the
   codegen line, and nothing in the Sub runs, so `tbrun` had returned that log with exit 0.
-- **A callee's code-generation failure is invisible.** When the failing shift is in a
-  procedure the probe calls, the codegen line naming that procedure comes straight after the
-  `[BUILD] Executing` line, before the probe's first statement runs. The probe's `Debug.Cls`
-  erases it, the probe prints what comes before the call and stops there, and `tbrun` exits 0
-  with that partial output. Measured with and without `Debug.Cls` on BETA 983; not fixed.
+- **A callee's code-generation failure is erased by the probe's own `Debug.Cls`.** When the
+  failing shift is in a procedure the probe calls, the codegen line naming that procedure
+  comes straight after the `[BUILD] Executing` line, before the probe's first statement runs.
+  The probe's `Debug.Cls` erases it, the probe prints what comes before the call and stops
+  there, and `tbrun` used to exit 0 with that partial output (measured with and without
+  `Debug.Cls` on BETA 983). Since the tooling review's C25a, `tbrun` wraps the page's global
+  `clearDebugConsole()` before it presses Build, and keeps what each clear erases
+  (`keepClears` in `tb-ide.mjs`). A `BUILD_FAILED` line in that record after the last
+  `[BUILD] Executing` line exits 2, naming the line and printing the partial output. `main.js`
+  calls that function by name from the compiler's `event_clearDebugConsole`, which
+  `Debug.Cls` raises, from the pane's Clear command, and on closing the project; no other
+  path that empties the console was found. The IDE does not clear the console when a build
+  starts, so the probe's first `Debug.Cls` erases the build log from `[BUILD] Starting...` on.
+  `event_clearDebugConsole` writes an empty line after its clear, so the record of a second
+  `Debug.Cls` begins with one.
 
 A reader of the console that is not `tbrun` should **compare the whole console before and
 after, not read on from an index**: new text can be appended to an entry that is still open.

docs/Documentation/Tools.md

@@ -694,9 +694,9 @@ probe's output, with exit 0. Run it again: both failures seen so far passed on a
 A `[RunAfterBuild]` Sub that fails code generation exits 2 the same way: the build succeeds,
 the console adds `[LINKER] compilation (codegen) error detected in '<module>.<procedure>'`,
 and nothing in the Sub runs, `Debug.Cls` included. A procedure the probe *calls* that fails
-code generation is not caught. Its error line is written before the probe's first statement,
-which erases it, and the probe stops at the call, so `tbrun` exits 0 with the output printed
-up to that point.
+code generation exits 2 as well. Its error line is written before the probe's first
+statement, so the probe's `Debug.Cls` erases it, and the probe stops at the call. `tbrun`
+keeps what each clear erases, so it names that line and prints the output up to the call.
 
 **The capture is complete however much a probe prints**, so there is no reason to keep one
 short. `tbrun` reads the console's backing array rather than the pane, which is a virtualised
@@ -731,8 +731,9 @@ comes back as `A&`.
 | `--show` / `--hide` | As for [`tbbuild.mjs`](#tbbuild): your own desktop or a private one, with `TBBUILD_SHOW` setting the default. |
 
 Exit codes: **0** captured output, **1** the project has compile errors (the diagnostics are
-printed), **2** the harness failed or the build did after a clean compile, **3** nothing reached
-the console before the timeout.
+printed), **2** the harness failed or the build did after a clean compile, **3** no output:
+nothing reached the console before the timeout, or the probe ran and printed nothing after its
+last `Debug.Cls`.
 
 **A probe that activates a COM server can leak one per run.** `CreateObject("Excel.Application")`
 is activated by DCOM, so the `EXCEL.EXE` that appears is a child of `svchost.exe` rather than
@@ -853,7 +854,8 @@ registry. It deletes the scratch key when it ends.
 
 It is not a gate and is not in `test.bat`, because it needs Windows and a real registry and
 the CI runners have neither. Run it by hand after changing `tb-registry.mjs`. Exit code
-**0** when every check holds, **1** when one does not.
+**0** when every check holds, **1** when one does not, **2** when something else stops it,
+such as PowerShell failing.
 
 ### check_examples.mjs
 {: #check-examples }

docs/Documentation/Wisdom.md

@@ -75,11 +75,13 @@ node wisdom/wisdom.mjs extract --since 2025-06-01
 
 The `--since` mode writes to a sideband file (`staging-since-<date>.md`) and does not touch the canonical `staging.md` or the watermark.
 
+For `export`, `--since` limits only the channels and threads not exported yet, which are fetched from the date on, and a later export without `--force` does not go back for their older messages. One already exported is brought up to date as in a run without `--since`, so no stored history is lost. A channel or thread with no message since the date gets no file, so a later export without `--since` fetches its whole history.
+
 The extract step automatically partitions large thread sets into batches of 200, so filtering is optional --- but `--since` and `--channel` reduce the number of threads analysed (and therefore agent invocations and API costs).
 
 ## Reviewing staging.md
 
-`staging.md` is the long-lived review file. Each `## ` section is one proposed documentation addition, with structured metadata at the bottom (source thread IDs, confidence level, date range, optional reviewer note). Sections are grouped by target page and delimited by `---` lines.
+`staging.md` is the long-lived review file. Each `## ` section is one proposed documentation addition, with structured metadata at the bottom (source thread IDs, confidence level, date range, optional reviewer note). Sections are grouped by target page and delimited by `---` lines. The first line after a `---`, blank lines aside, must be a section's `## ` heading: the next merge stops at anything else, naming its line, rather than drop it.
 
 **Removing sections.** Delete any section that is not useful --- the removal is stable. If the source thread is unchanged on the next run, the watermark filter skips it entirely and the section stays gone. If the thread later receives new messages, the thread re-enters the pipeline and the agent may produce a fresh finding that accounts for the new context; it reappears with a `[REFINED?]` marker so the reviewer knows it is a revision of something already triaged.
 
@@ -103,11 +105,11 @@ Fetches messages from Discord channels and forum threads.
 node wisdom/wisdom.mjs export
 ```
 
-Outputs raw JSON under `wisdom/data/raw/`. Supports incremental runs --- a manifest tracks the highest message ID per channel, so re-running fetches only new messages. Use `--force` to re-fetch everything.
+Outputs raw JSON under `wisdom/data/raw/`. Supports incremental runs --- a manifest tracks the highest message ID per channel and thread, so re-running fetches only the new messages, and only from the channels and threads that have some. Use `--force` to re-fetch everything.
 
 | Flag | Effect |
 |------|--------|
-| `--since <date>` | Only fetch messages after this ISO 8601 date |
+| `--since <date>` | Fetch a channel or thread not exported yet only from this ISO 8601 date on; one already exported is brought up to date as without it |
 | `--channel <id>` | Restrict to one channel (repeatable) |
 | `--dry-run` | Discover channels/threads; do not fetch messages |
 | `--force` | Ignore manifest; re-fetch all history |
@@ -177,6 +179,7 @@ wisdom/
   wisdom.mjs              Entry point --- CLI parser, runExport(), dispatch
   config.mjs              Load config.jsonc, apply CLI overrides
   config.jsonc             Server/channel/rate-limit configuration
+  files.mjs               Atomic writes (temp file + rename); JSON reads that report a bad file by path
 
   discord/                 Phase 1 --- Discord API layer
     api.mjs                HTTP client, auth, rate-limiter, snowflake utilities
@@ -217,19 +220,22 @@ Also defines `runConcurrent(items, concurrency, fn)` --- a simple worker-pool: s
 ### Phase 1 control flow
 
 1. **Discover** (`discord/discover.mjs`): fetch the full channel list from `/guilds/{id}/channels`, filter by type (text/forum) and exclude patterns. For forums, paginate `/channels/{id}/threads/archived/public` and (bot-only) `/guilds/{id}/threads/active`, deduplicate, and filter by `min_message_count`.
-2. **Fetch members** (`discord/discover.mjs`): paginate `/guilds/{id}/members` (bot-only; user tokens get an empty map). Write `guild.json` and `members.json`.
+2. **Fetch members** (`discord/discover.mjs`): paginate `/guilds/{id}/members`. A user token, or a bot the endpoint refuses, gets an empty map, and messages then show authors by their global name rather than their server nickname. Write `guild.json` and `members.json`.
 3. **Build target list**: merge text channels and forum threads into a single list. Targets that previously returned 403 (tracked in `denied.json`) sort to the end.
 4. **Fetch messages** (`discord/messages.mjs`): run targets through `runConcurrent`. For each target:
-   - Check manifest: if the target's highest-seen snowflake is recorded and the output file exists, skip (up-to-date).
+   - Check manifest: if the target's output file exists, its snowflake is recorded, and the `last_message_id` discovery reported for it is no newer, skip it (up-to-date) without a request.
    - Call `fetchMessages(client, channelId, afterSnowflake)`:
-     - **Incremental** (afterSnowflake set): page forward with `?after=`, collecting new messages.
-     - **Full** (afterSnowflake null): page backward with `?before=`, collecting all history.
+     - **Incremental** (the output file exists and its snowflake is recorded, with or without `--since`): page forward with `?after=` from that snowflake, collecting new messages.
+     - **Since** (otherwise, under `--since`): page forward with `?after=` from the date's snowflake.
+     - **Full** (otherwise, which includes every target under `--force` without `--since`): page backward with `?before=`, collecting all history.
      - Sort chronologically (ascending snowflake).
-   - Write `{ channel | thread, messages }` to `raw/channels/{id}.json` or `raw/threads/{id}.json`.
-   - Update manifest with `highestSnowflake(messages)` and flush to disk after each target.
+   - Write `{ channel | thread, messages }` to `raw/channels/{id}.json` or `raw/threads/{id}.json`. An incremental fetch appends its messages to those already in the file and stores the channel or thread object discovery returned; a since or full fetch writes the file whole.
+   - Update the manifest to the newest snowflake now on disk for the target, and flush it after each target. That is the newest message, or discovery's `last_message_id` when that is newer because its message was deleted, so the next run does not fetch the target again for nothing. A target with no file gets no entry.
 
 The export manifest (`raw/manifest.json`) is a flat `{ channelOrThreadId: highestSnowflake }` object. It governs incremental fetches --- on the next run, only messages newer than the stored snowflake are requested.
 
+Every file the export writes goes to a `.tmp` file first, which is then renamed over the old one, so a run that stops during a write leaves the previous file whole. A manifest or `denied.json` that does not parse stops the next export with an error that names it, unless `--force` is given, which ignores both files. So does a stored channel or thread file that has new messages to append.
+
 ### Phase 2 control flow
 
 1. Load `guild.json` to build `channelMap` (id to channel object) and `tagMap` (forum tag id to tag name).
@@ -288,7 +294,7 @@ The pipeline runs both stages without a barrier --- Stage 2 for group A starts a
 1. **Collect results**: read all `extract-results-*.json` files and concatenate their additions arrays.
 
 2. **Graft into staging.md** (`extract/merger.mjs` --- `graftAdditions`):
-   - Parse existing `staging.md` into `{ preamble, sections[] }`. The parser splits on `---` delimiter lines, then parses each chunk into heading (target_page + section + optional marker), body lines, and trailing meta lines (source threads, confidence, date range, reviewer note).
+   - Parse existing `staging.md` into `{ preamble, sections[] }`. The parser splits on `---` delimiter lines, then parses each chunk into heading (target_page + section + optional marker), body lines, and trailing meta lines (source threads, confidence, date range, reviewer note). A chunk that does not start with a `## ` heading stops the merge, naming its first line, before anything is written.
    - For each addition, compute a match key: `(target_page, section, sorted finding_ids)`.
      - **Key exists in staging.md** (and section is not `[LOCKED]`): replace the section body and meta in place.
      - **Key not in staging, but in the emission log** (from `extract-state.json`): this was previously emitted, reviewed, and removed. Insert with a `[REFINED?]` marker.
@@ -337,6 +343,7 @@ data/
     guild.json
     members.json
     manifest.json
+    denied.json
     channels/*.json
     threads/*.json
   threads/                          Phase 2 output

scripts/build_dot_metrics.mjs

@@ -36,6 +36,12 @@ import path from "node:path";
 import { fileURLToPath, pathToFileURL } from "node:url";
 import puppeteer from "puppeteer";
 
+// A crash is the harness failing, not a finding: exit 2, as Extending.md's gate
+// conventions require, where 1 is --check finding the table stale. This file
+// runs at top level, so there is no main().catch to do it; the handler also
+// catches a rejected top-level await.
+process.on("uncaughtException", (err) => { console.error(err); process.exit(2); });
+
 const REPO = path.dirname(path.dirname(fileURLToPath(import.meta.url)));
 const OUT = path.join(REPO, "builder", "inter-metrics.json");
 

scripts/check_links_diff.mjs

@@ -403,6 +403,12 @@ const SIDES = {
   },
 };
 
+// A build this harness needed and could not get: main() reports it as an
+// error: line, exit 2, having already printed the build's own output.
+class HarnessError extends Error {}
+
+const exitOf = (r) => `exit ${r.status}${r.error ? `, ${r.error.message}` : ""}`;
+
 // Build once per distinct --baseurl / --dest and cache the findings the
 // build wrote. The trees the script side reads are the ones this build
 // produced, so both sides are looking at the same bytes.
@@ -429,7 +435,7 @@ function fusedBuild({ baseurl = "", dest = null, src = "docs", offline = false }
   if (!fs.existsSync(out)) {
     console.error(r.stdout ?? "");
     console.error(r.stderr ?? "");
-    throw new Error(`the build wrote no findings file (exit ${r.status})`);
+    throw new HarnessError(`the build wrote no findings file (${exitOf(r)})`);
   }
   const findings = JSON.parse(fs.readFileSync(out, "utf8"));
   fs.rmSync(out, { force: true });
@@ -522,7 +528,7 @@ function ensureBasePathTree(dir, allowBuild) {
   if (r.status !== 0) {
     console.error(r.stdout ?? "");
     console.error(r.stderr ?? "");
-    throw new Error(`base-path build failed (exit ${r.status})`);
+    throw new HarnessError(`base-path build failed (${exitOf(r)})`);
   }
   return true;
 }
@@ -643,12 +649,24 @@ function main() {
     return 2;
   }
 
+  // A build that failed, or a throw, means nothing was compared: exit 2, not
+  // the 1 that says the sides differ. The fixture folder goes either way.
+  try {
+    return compare(opts);
+  } catch (e) {
+    console.error(e instanceof HarnessError ? `error: ${e.message}` : e);
+    return 2;
+  } finally {
+    if (opts.fixtureDir) fs.rmSync(opts.fixtureDir, { recursive: true, force: true });
+  }
+}
+
+function compare(opts) {
   console.log(`check_links_diff: ${opts.a} vs ${opts.b}`);
 
   // findings[side][case]
   const findings = { [opts.a]: {}, [opts.b]: {} };
   let differences = 0;
-  let fixtureDir = null;
   const skipped = [];
 
   const usesFused = opts.a === "fused" || opts.b === "fused";
@@ -680,7 +698,6 @@ function main() {
     }
     if (c.needsFixture && !opts.fixtureDir) {
       opts.fixtureDir = fs.mkdtempSync(path.join(os.tmpdir(), "link-check-fixture-"));
-      fixtureDir = opts.fixtureDir;
       writeFixture(opts.fixtureDir);
     }
     const root = c.root(opts);
@@ -747,8 +764,6 @@ function main() {
     }
   }
 
-  if (fixtureDir) fs.rmSync(fixtureDir, { recursive: true, force: true });
-
   if (skipped.length) {
     console.log(`\nskipped: ${skipped.join(", ")}`);
   }

scripts/check_publish_policy.mjs

@@ -150,43 +150,45 @@ if (!failures) console.log(`  ok    build-only types (${[...BUILD_EXTENSIONS].jo
 // If either stops holding, the message is wrong and this says so.
 {
   const tmp = await fs.mkdtemp(path.join(os.tmpdir(), "publish-policy-"));
-  const probe = async (name, body) => {
-    const dir = path.join(tmp, name);
-    await fs.mkdir(dir, { recursive: true });
-    await fs.writeFile(path.join(dir, "probe.md"), body);
-    try {
-      const { pages } = await discover(dir, []);
-      return pages.length ? "page" : "static";
-    } catch (err) {
-      return "throws: " + err.message.split("\n")[0];
+  try {
+    const probe = async (name, body) => {
+      const dir = path.join(tmp, name);
+      await fs.mkdir(dir, { recursive: true });
+      await fs.writeFile(path.join(dir, "probe.md"), body);
+      try {
+        const { pages } = await discover(dir, []);
+        return pages.length ? "page" : "static";
+      } catch (err) {
+        return "throws: " + err.message.split("\n")[0];
+      }
+    };
+    const FM = "---\ntitle: X\npermalink: /x\n---\nbody\n";
+
+    // Written as an escape, not as the character: a literal BOM inside a
+    // string literal is invisible, and an editor stripping it would turn
+    // this assertion into a no-op that still passes.
+    const bom = await probe("bom", "\u{FEFF}" + FM);
+    if (bom !== "page") {
+      fail(`a UTF-8 BOM now yields "${bom}", not a page -- discover.mjs's ` +
+           `stripBom() is gone, and publish-policy.mjs's .md message should ` +
+           `name the BOM again`);
     }
-  };
-  const FM = "---\ntitle: X\npermalink: /x\n---\nbody\n";
-
-  // Written as an escape, not as the character: a literal BOM inside a
-  // string literal is invisible, and an editor stripping it would turn
-  // this assertion into a no-op that still passes.
-  const bom = await probe("bom", "\u{FEFF}" + FM);
-  if (bom !== "page") {
-    fail(`a UTF-8 BOM now yields "${bom}", not a page -- discover.mjs's ` +
-         `stripBom() is gone, and publish-policy.mjs's .md message should ` +
-         `name the BOM again`);
-  }
 
-  const bad = await probe("badyaml", "---\ntitle: [unclosed\n---\nbody\n");
-  if (!bad.startsWith("throws:")) {
-    fail(`malformed frontmatter YAML now yields "${bad}" instead of its own ` +
-         `error -- it would reach the publish policy, whose .md message does ` +
-         `not mention it`);
-  }
+    const bad = await probe("badyaml", "---\ntitle: [unclosed\n---\nbody\n");
+    if (!bad.startsWith("throws:")) {
+      fail(`malformed frontmatter YAML now yields "${bad}" instead of its own ` +
+           `error -- it would reach the publish policy, whose .md message does ` +
+           `not mention it`);
+    }
 
-  const lead = await probe("leadingblank", "\n" + FM);
-  if (lead !== "static") {
-    fail(`a blank line before the opening delimiter now yields "${lead}" -- ` +
-         `the .md message's advice no longer describes a real fault`);
+    const lead = await probe("leadingblank", "\n" + FM);
+    if (lead !== "static") {
+      fail(`a blank line before the opening delimiter now yields "${lead}" -- ` +
+           `the .md message's advice no longer describes a real fault`);
+    }
+  } finally {
+    await fs.rm(tmp, { recursive: true, force: true });
   }
-
-  await fs.rm(tmp, { recursive: true, force: true });
   if (!failures) console.log("  ok    the .md refusal names a fault that can actually occur");
 }