diff --git a/WIP.Harness.md b/WIP.Harness.md index 4ee68712..e85efeb9 100644 --- a/WIP.Harness.md +++ b/WIP.Harness.md @@ -429,11 +429,21 @@ Four smaller things it knows, each of which cost a run: was measured: a `[RunAfterBuild]` Sub that shifts a `Single` (BUGS-TO-REPORT.md) builds with `[LINKER] SUCCESS`, the console adds `[BUILD] Executing 'DocSamples.Probe.Run'...` and the codegen line, and nothing in the Sub runs, so `tbrun` had returned that log with exit 0. -- **A callee's code-generation failure is invisible.** When the failing shift is in a - procedure the probe calls, the codegen line naming that procedure comes straight after the - `[BUILD] Executing` line, before the probe's first statement runs. The probe's `Debug.Cls` - erases it, the probe prints what comes before the call and stops there, and `tbrun` exits 0 - with that partial output. Measured with and without `Debug.Cls` on BETA 983; not fixed. +- **A callee's code-generation failure is erased by the probe's own `Debug.Cls`.** When the + failing shift is in a procedure the probe calls, the codegen line naming that procedure + comes straight after the `[BUILD] Executing` line, before the probe's first statement runs. + The probe's `Debug.Cls` erases it, the probe prints what comes before the call and stops + there, and `tbrun` used to exit 0 with that partial output (measured with and without + `Debug.Cls` on BETA 983). Since the tooling review's C25a, `tbrun` wraps the page's global + `clearDebugConsole()` before it presses Build, and keeps what each clear erases + (`keepClears` in `tb-ide.mjs`). A `BUILD_FAILED` line in that record after the last + `[BUILD] Executing` line exits 2, naming the line and printing the partial output. `main.js` + calls that function by name from the compiler's `event_clearDebugConsole`, which + `Debug.Cls` raises, from the pane's Clear command, and on closing the project; no other + path that empties the console was found. The IDE does not clear the console when a build + starts, so the probe's first `Debug.Cls` erases the build log from `[BUILD] Starting...` on. + `event_clearDebugConsole` writes an empty line after its clear, so the record of a second + `Debug.Cls` begins with one. A reader of the console that is not `tbrun` should **compare the whole console before and after, not read on from an index**: new text can be appended to an entry that is still open. diff --git a/builder/PLAN-TOOLING-REVIEW.md b/builder/PLAN-TOOLING-REVIEW.md index c18512c5..c146e359 100644 --- a/builder/PLAN-TOOLING-REVIEW.md +++ b/builder/PLAN-TOOLING-REVIEW.md @@ -165,7 +165,10 @@ on, a commit's **Landed** note is written in full in the commit that lands it, w history keeps it; at the end of each phase, that phase's landed entries are cut to what later work still needs. The landed entries of C01–C18 and C13a were cut this way on 2026-09-26; their full text is in this file as it stood before the commit `builder: cut the tooling -plan's landed entries to what later work needs`. Line numbers are the +plan's landed entries to what later work needs`. Those of the rest of Phase 1 (C19–C30, with +C22a–C22j, C25a–C25f, C27a and C27b) were cut the same day, and their full text is in this +file as it stood before `builder: cut Phase 1's landed entries in the tooling plan`. A +pointer below to a cut entry's Landed note means that text. Line numbers are the review's, at `fe9ce12b`, and move as the commits land. ### The organising idea @@ -379,721 +382,135 @@ value exits 2." ### C19 — `scripts: close the browser on every exit path, through lib/browser.mjs` -**L3-1 (R1).** `check_a11y.mjs` launches Chromium (`:167`) and closes it only on success -(`:218`), and `main().catch` exits 2 without closing it (`:244-247`). The comment at `:229` -names the path that throws: a `PAGE_STATES` applier's failed assertion, which throws by -design. On Linux, where CI runs this gate, Chromium stays up for the rest of the job. The -three sibling tools guard their launch. - -**Change.** `scripts/lib/browser.mjs` takes `axe-scan.mjs`'s `launchBrowser` and -`LAUNCH_ARGS` (`:258-264`), with their reasons, and adds `withBrowser(fn, options)`, which -closes the browser in a `finally`. `axe-scan.mjs` re-exports `launchBrowser`, so its -importers need not change. `check_a11y.mjs`, `check_a11y_fingerprint.mjs`, `sweep_a11y.mjs` -and `check_axe_patch_equiv.mjs` use `withBrowser`. C44 moves the two diagram tools onto the -same module. - -**Verify.** With an applier made to fail, a Chromium started by the run is left over before -the change and none after. Count by the Puppeteer cache path in each process's command -line, never by image name, since the owner's own Chrome has the same one. `check_a11y.mjs`'s -findings unchanged; `check_axe_patch_equiv.mjs` passes. - -**Landed.** The premise was wrong; the change stands, restated (see [Where the plan was -wrong](#where-the-plan-was-wrong)). At HEAD, with `"details.section-links"` changed to -`"details.section-links-x"` in the first `PAGE_STATES` applier, `check_a11y.mjs` exits 2 after -8 s with the applier's error and leaves no Chromium running. `@puppeteer/browsers` subscribes -to Node's `exit` event for every launch (`lib/launch.js:178`) and kills the browser there -synchronously (`:232`): `taskkill /pid /T /F` on Windows (`:268`), a `SIGKILL` of the -browser's detached process group elsewhere (`:151`, `:283`). Linux was read, not measured. -What the failed run left was the browser's temporary profile, one -`%TEMP%\puppeteer_dev_chrome_profile-*` folder of 4.3 MB. Puppeteer deletes it only when the -browser process's own `exit` event arrives (`puppeteer-core`'s `BrowserLauncher.js:64`, `:82`), -and `process.exit` ends Node before that. A clean run left none. On 2026-09-26 the owner kept -C19 as planned and had the 211 such folders then in `%TEMP%` (389 MB, dated February to -September 2026) deleted. - -`scripts/lib/browser.mjs` holds `launchBrowser`, `LAUNCH_ARGS` and `withBrowser(fn, -options)`. `LAUNCH_ARGS` is no longer exported, since nothing imported it. `axe-scan.mjs` -re-exports `launchBrowser` for the four `perf/` rigs that import it, and drops its `puppeteer` -import; `builder/link-check.mjs`'s comment, which said `axe-scan.mjs` owns puppeteer, now says -it loads puppeteer through `browser.mjs`. In `check_a11y.mjs`, `buildMatrix` now runs before -the launch instead of after it. -`sweep_a11y.mjs`'s header said `--recycle-every` restarts the browser; the code opens a fresh -tab, as its own comment says, and the header now says so too. - -After the change the same failing run exits 2 with the same error and leaves no folder, and -no Chromium. `check.bat`'s a11y line is unchanged: `13 pages x 2 theme(s) x 2 viewport(s) + 8 -state audit(s) checked: 0 violation(s), 42 incomplete check(s)`. `check_a11y_fingerprint.mjs`'s -self-test reports `60/60 audits identical -- gate PASSES` before and after. `sweep_a11y.mjs ---limit 4 --recycle-every 2 --theme light --viewport desktop` writes four records that match -HEAD's apart from `runMs`. `check_axe_patch_equiv.mjs` reports `20/20 colour values -identical`. +**Carried forward.** `scripts/lib/browser.mjs` holds `launchBrowser` and `withBrowser(fn, +options)`, which closes the browser in a `finally`; `LAUNCH_ARGS` is no longer exported. +`axe-scan.mjs` re-exports `launchBrowser` for the `perf/` rigs that import it. C44 moves +`check_dot_fit.mjs` and `build_dot_metrics.mjs` onto the same module, adding a file-access +launch option to it. ### C20 — `a11y: validate --theme and --viewport wherever a matrix is built` -**L1-1 (R1).** `check_a11y.mjs:82-95`'s `pick()` was written after `--theme drak` labelled a -light run "drak"; `sweep_a11y.mjs:120-121` and `check_a11y_fingerprint.mjs:134-135`, the gate -for an axe upgrade, never received it. `buildMatrix` labels the report from the unvalidated -string and `gotoPage` applies it; the dark CSS matches only `[data-theme=dark]`, so an -unknown value renders light under a report that says otherwise. - -**Change.** `pick()` moves into `axe-scan.mjs`, beside `THEMES` and `VIEWPORTS`, and all three -tools use it. `buildMatrix` also refuses an unknown value, as the backstop for a future -caller. - -**Verify.** `--theme drak` and `--viewport tiny` fail with a usage error in all three tools; -`check_a11y.mjs`'s findings unchanged. - -**Landed.** Reproduced first, after C19. `sweep_a11y.mjs --theme drak --viewport desktop ---limit 1` exited 0 and recorded `/404.html [drak, desktop]`. `--theme light --viewport tiny` -also exited 0, recording `[light, tiny]`: `setViewport(undefined)` does not throw, so the audit -ran at a size nobody chose. `check_a11y_fingerprint.mjs --pages /404.html` reported `1/1 -audits identical -- gate PASSES` with either value. - -`pick()` moved from `check_a11y.mjs` into `axe-scan.mjs`, exported, beside `THEMES`, and its -comment gained the viewport case. All three tools call it at module level, so the usage error -comes before any browser starts. `buildMatrix` throws for a theme not in `THEMES` or a -viewport that is not an own key of `VIEWPORTS` (`Object.hasOwn`, so `toString` is refused -too). `sweep_a11y.mjs` builds its own matrix, so the backstop covers the other two. - -The six cases now exit 2 with one line each, `unknown --theme "drak"; expected one of light, -dark or both` or `unknown --viewport "tiny"; expected one of desktop, mobile or both`. A -scratch probe of `buildMatrix` got 60 entries by default and 30 for one theme or one -viewport, and a throw for `drak`, `tiny` and `toString`. Valid single values still run: -`sweep_a11y.mjs --theme dark --viewport mobile --limit 1` recorded `/404.html [dark, mobile]`, -and the fingerprint self-test with the same values passed. `check.bat`'s a11y line is -unchanged. +Landed. ### C21 — `builder: the Gantt chart draws Check, vendorAssets and Other` -**A1-1 (R1).** `gantt.mjs:39-49` draws only `Seeds`, `Spine` (which also takes `Render`) -and `Write`, so -`checkBook` and `checkReport` (section `Check`) and `vendorAssets` (no `GANTT_SECTION` entry; -`tbdocs.mjs:539-562`) are drawn on no build, while their durations still stretch the time -axis. `COLORS.Other` (`:12`) is never used, and `Builder.md:410` says a task with no section -falls into an "Other" bucket. - -**Change.** `mainSections` gains `Check` and `Other`, and `vendorAssets` gets a -`GANTT_SECTION` entry. - -**Verify.** The built `gantt.svg` names `checkBook`, `checkReport` and `vendorAssets` after -the change and not before. The tree comparison identical apart from its normalised Gantt -regions. `Builder.md:410` re-read against the result. - -**Landed.** Before, both built `gantt.svg` files (at 0ba42460) named none of the three tasks -and had no Check band. After, each names all three once and has a Check band, and so do both -copies inlined into `BuildInfo.html`. `vendorAssets` charts in Spine: it runs on the main -thread after `discover`, and `markdownInit` waits for it. Check's colour is a pink (`#e59ac6` -light, `#b35c8c` dark); its label contrast is in the range of the other bars', and both themes -were looked at through puppeteer. - -**At the owner's request, a task the chart cannot draw fails the build**, so `Other` was not -added: nothing can reach it, and `COLORS.Other` and its `.gb-other` rules are gone. -`groupGanttTimings` throws for a task with no section, and `renderGantt` for a main-thread -task whose section has no band or a worker task whose section has no colour. With each defect put back -by a scratch edit, the check fixture's build exited 1 with `gantt: task vendorAssets has no -section; add it to GANTT_SECTION`, and with `gantt: the chart has no place for task checkBook -in section "Check"`. Without `--check` the Check tasks are not charted, so the second fails -only a checked build, which `build.bat` and both CI workflows are. WIP.md's gate table gains -the check beside nav integrity; Tools.md lists no build-internal checks, so nothing is -registered there. - -Docs: Builder.md's `Other` sentence rewritten and `vendorAssets` moved from Seeds to Spine in -its section list; Extending.md's row on Gantt sections rewritten, its counts corrected (32 -static tasks, 30 in the map, none setting `ganttSection` on its definition, where it said 31, -28 and `dispatch`); Pipeline-Stages.md's `ganttSection` values gain `Check` and `renderGantt`'s -row says it throws; BuildInfo.md's alt text says five bands. `Builder.md:410`, re-read (now -`:409-413`): the `Other` sentence was the one the entry named; the paragraph's claim that -boot timings form a row group of their own, and more in the section lists above it, is wrong -and is recorded under Found while implementing. - -`compare_trees` replaces the chart whole, so it cannot see this change; the built chart is the -oracle. It exits 1 on the four edited pages, online and offline, the search data and -`book.html`, and BuildInfo.html's difference is its alt text. `check.bat`'s a11y line is -unchanged (0 violations, 42 incomplete), though BuildInfo.html is in the sample and the chart -gained four labels. `test.bat` and lint pass. +Landed. ### C22 — `scripts: crawl_check follows every link attribute the build checks` -**A4-3 (R1).** `crawl_check.mjs:91-108`, the only checker that runs against the deployed -site, handles five tag and attribute pairs, where `link-check.mjs:38-60`'s `LINK_ATTR_TABLE` -has 21 tags and 26 pairs. It never follows `srcset`, `poster`, `cite`, `formaction`, -`action`, `data` or `longdesc`. - -**Change.** Export `LINK_ATTR_TABLE` and `splitSrcset` from `link-check.mjs`, and let the -table decide `crawl_check.mjs`'s tag handling; its HTTP concerns (concurrency, redirects, -HEAD then GET) stay its own. This is separate from decision 5, which covers the two -filesystem checkers only. - -**Verify.** Run it before and after against `serve.bat`'s local server if it takes a base -URL, and otherwise ask before crawling the live site. The after run requests the `srcset` and -`poster` targets, and reports nothing the before run did not, apart from any link in those -attributes that is actually broken. - -**Landed**, with one change of shape. Exporting the table and `splitSrcset` would have left a -second copy of the loop over them in `crawl_check.mjs`, including which attribute is a -srcset. So `link-check.mjs` exports one function instead, `forEachLink(name, attribs, fn)`, -which walks the table and splits a srcset; `extractFromHtml` and `crawl_check`'s tag handler -both call it, and the table and `splitSrcset` stay private. `crawl_check`'s id capture is -unchanged. Tools.md's paragraph says it follows every link the build's check follows. - -`extractFromHtml` gives the same results: a scratch script ran HEAD's copy and the edited one -over every `.html` file in the three built trees, the check fixture's trees and the fixture -below (2,414 files, 1,777,900 links), with every option on and with every option off, and no -field of any result differed. `check_links_diff.mjs --a script --b fused` found no -differences across 6 cases. - -The tool takes a start URL, so both runs used a scratch static server on `127.0.0.1` that -resolves a path as `serve.mjs` does and, as GitHub Pages does, redirects a folder URL without -its trailing slash to the slash form. Without the redirect, 626 links came back broken, all -from folder pages fetched without the slash; an agent confirmed the redirect on both live -sites (a 301 with an absolute `Location`), and found no GitHub documentation of it. Node's -server also closes an idle keep-alive socket after 5 s, which failed 30 fetches until the -scratch server kept its sockets longer. - -Against the built site, with `--skip-external`, before and after: 1,247 pages crawled, 3,228 -unique links, 0 broken, 0 missing anchors, the two reports identical apart from the elapsed -time. The site uses none of the newly followed attributes: nothing in `_site` has a `srcset`, -`poster`, `cite`, `action`, `data` or `longdesc`. So the fixture carried the test: one page -with a missing target for each of the table's 26 pairs, 28 URLs in all, since both srcsets -list two. Before, 5 broken (`a`, `link`, `img src`, `script`, `iframe`); after, all 28, the -srcset and poster targets included. `compare_trees`: Tools.html online and offline, the search -data and `book.html`, nothing else. Two defects found on the way are recorded under Found -while implementing. +Landed. ### C22a — `scripts: crawl_check sets its exit code instead of calling process.exit` -**Found while verifying C22; the owner asked on 2026-09-26 for it to be fixed before C23** -(see Found while implementing). `crawl_check.mjs` ended `main()` with `process.exit()` straight -after printing its report, and its crash handler called `process.exit(2)`. On Windows (Node -24.13.0) that can abort on a libuv assertion, `!(handle->flags & UV_HANDLE_CLOSING)` in -`src\win\async.c:76`: the report is complete, and the exit code is 0xC0000409 (Git Bash shows -127) instead of 1. - -**Change.** `main()` sets `process.exitCode` and returns, and the crash handler sets it to 2. -The two usage exits stay, since they run before any fetch. The header and Tools.md's -paragraph give all three exit codes: a missing anchor exits 1 as well. - -**Landed**, with one addition. With `process.exit` gone, a crawl of the site printed its -report at 66 s and never exited: idle, its CPU time flat, 90 connections to the server still -established, until it was stopped 269 s later. `crawlOne` GETs every same-site URL but read -the body only of an HTML page answered 2xx, and an unread body keeps its connection busy; -`process.exit` had been cutting that short. `discardBody` now cancels every body that is not -read, in `crawlOne` and after `checkUrl`'s HEAD and GET. - -The assertion does not need unread bodies. On scratch probes against the fixture server, 28 -concurrent fetches followed by `process.exit(1)` aborted 10 times of 10 with or without -cancelling the bodies, and one fetch, read or unread, exited 1 all 10 times. What else it -needs is not established: HEAD's crawl of the site reaches `process.exit` with those 90 -connections open, and did not abort in C22's four runs. - -Against the C22 fixture, through the kit's static server with `--skip-external`: before, 5 -runs of 5 aborted after reporting 28 broken; after, 10 of 10 exit 1 with 28 broken and no -assertion, the process ending 25–30 ms after its report. Against the site, three runs after: -exit 0, 1,247 pages crawled, 3,228 unique links, 1,247 status checks, 0 broken, 0 missing -anchors, as in C22's runs, in 66 to 79 s, ending 26–43 ms after the report. A crash -mid-crawl, injected by a preload that makes `Response#url` throw from its 200th read so that -`crawlOne` throws outside its `try` blocks with other fetches in flight: HEAD's copy aborted -5 of 5 with 0xC0000409, and after, 5 of 5 exit 2, about 40 ms after the error. An unknown flag -and a missing URL exit 2, as before. `compare_trees`: Tools.html online and offline, the search -data and `book.html`, nothing else. +Landed. ### C22b — `builder: serve.bat redirects a folder URL to its trailing slash` -**Found while verifying C22; the owner asked on 2026-09-26 for it to be fixed before C23** -(see Found while implementing). `serve.mjs`'s static handler answered a folder URL without its -trailing slash with the folder's `index.html`, where GitHub Pages answers 301 to the slash -form. A browser then resolves the page's relative links against the parent folder, one level -too high. The built site links 74 folder pages without the slash, e.g. -`../../tB/Modules/Collection` from Permanent-Links, so a `serve.bat` preview reached through -one of those links shows a page whose links are broken. - -**Change.** When the only file that matches is the folder's `index.html` and the URL path -lacks its slash, answer 301 to the path plus `/`, keeping any query string. - -**Landed.** `resolveFile` returns `{ file }` or `{ redirect }`. The `Location` is built from -the folder under the destination rather than from the request, percent-encoded segment by -segment, so a request for `//tB/Packages` redirects to `/tB/Packages/` and not to a host named -`tB`. The redirect carries the page's `no-store` cache header, so a browser does not keep it -after a folder page becomes a single-file one. A URL that names a page is served as before: -`/tB/Core/Dim` from `Dim.html`. No page under `docs/Documentation/` describes how the serve -resolves a URL. - -Verified on a test serve (port 4393, `--dest docs/_serve-c22b`, through a temporary -`.claude/launch.json` entry; both removed afterwards). Before, `/tB/Packages`, -`/tB/Modules/Interaction` and `/tB/Packages/CEF` answered 200. After, each answers 301 to its -slash form and the slash forms 200; `/tB/Packages?x=1&y=2` redirects to -`/tB/Packages/?x=1&y=2`, and a temporary folder named `Ä b` to `/%C3%84%20b/`. -`crawl_check --skip-external` against HEAD's serve: 1,850 pages crawled, 3,831 unique links, -623 broken, of which 603 were HTTP errors, such as a 404 for `/Core/Attributes`, and 20 were -`fetch failed`, and 2 missing anchors. After: 1,247 pages crawled, 3,228 unique links and 0 -missing anchors, as in C22's crawl of `_site`, and 20 broken, every one a `fetch failed` -caused by `read ECONNRESET`, and none an HTTP error. The resets are a separate defect, -recorded under Found while implementing: with the serve's keep-alive timeout raised as a -scratch experiment, two crawls of two had none. `compare_trees`: identical. +Landed. ### C22c — `docs: Builder.md's task sections match the chart and the task graph` -**Found while re-reading Builder.md's Gantt paragraph for C21; the owner asked on 2026-09-26 -for it to be fixed before C23** (see Found while implementing). "Task DAG by section" says the -chart's five sections organise its discussion, and disagreed with the chart and with `TASKS`: -the lists put `discover` in Seeds and `warmInit` and `renderEnvInit` in Seeds and Render; the -Seeds discussion said a seed has no predecessors and listed `scss`, `prepDest` and -`prepPageDirs`, which have; the Spine sketch drew `loadData → highlighterInit` off `discover`; -and the Gantt paragraph put the start-up bars in a row group of their own, and a lane's bars -in completion order. - -**Change.** The five lists follow `GANTT_SECTION`, and a sentence says `warmInit` and -`renderEnvInit` are in none of them. Each task's bullet and its row in What runs where move -under its chart section; the two per-lane start-up tasks are described under Render. The -Seeds introduction says which seeds wait for another task, the Spine sketch is redrawn from -`expected`, and the Gantt paragraph says where the bands and the start-up bars are drawn. - -**Landed**, with three additions. The Render sketch drew the `flush:i` column feeding -`renderJoin`; it is redrawn with each `render:i` feeding `renderJoin` and each `flush:i` -feeding `flushJoin`, as `dispatch` wires them. `vendorAssets`, in the Spine list, had neither -a bullet nor a row in What runs where, and gains both, from Pipeline-Stages.md's section. And -the table, which says it lists every task, had no row for the three Check tasks. Every edge -in the two sketches was checked against `expected` and `dispatch`'s dynamic edges (the Spine -sketch leaves out `deriveRedirects → dispatch`, which `markdownInit` implies), every section -against `GANTT_SECTION`, and the Gantt paragraph against `gantt.mjs` (the four bands, a lane's -bars sorted by `workerStart`, the `cold` / `warm` / `env` labels) and `tbdocs.mjs` (no -cold-start bars on a rebuild; the `Join` tasks skipped). A scratch script confirmed the -sketches' vertical connectors line up. `scheduler-dag.dot` already draws `config → -highlighterInit → loadData`. Extending.md's account of the four surfaces still holds. -Pipeline-Stages.md files the same tasks by stage rather than by chart section, which is -recorded under Found while implementing. `compare_trees`: Builder.html online and offline, the -search data and `book.html`, nothing else. +Landed. ### C22d — `scripts: crawl_check retries a request that fails before any response` -**Found while verifying C22b; the owner asked on 2026-09-26 for it to be fixed before C23, -with two retries** (see Found while implementing). A crawl of `serve.bat` reported about 20 -links broken with `fetch failed`, each caused by `read ECONNRESET`: the serve closes an idle -keep-alive connection after Node's default 5 s, `fetch` reuses one just as it closes, and -`crawl_check` reported the first failure as the link's. - -**Change.** `fetchWithTimeout`, which every request goes through (`crawlOne`'s GET, -`checkUrl`'s HEAD and its GET after a 405 or 501), becomes `fetchWithRetry`: when `fetch` -rejects, whether reset or timed out, it tries twice more, each attempt with the full -`--timeout`, and throws the last error. An HTTP error status is a response and is not -retried, and neither is a failure while reading a body. The header and Tools.md's paragraph -say so. - -**Landed.** A test serve (port 4393, `--dest docs/_serve-c22d`, through a temporary -`.claude/launch.json` entry; both removed afterwards), crawled with `--skip-external` through -the kit's `crawl-tally.mjs`: before, exit 1 with 10 broken, every one a `fetch failed` from -`read ECONNRESET` (the twelfth session's crawls had 20, 21 and 20); after, two crawls, each -exit 0, 1,247 pages crawled, 3,228 unique links, 0 broken and 0 missing anchors, while the -preload logged 20 failed attempts in each, all `read ECONNRESET`. A scratch server, the kit's -`c22d-retry.mjs`, resets the first two requests to `/r2` and `/h2` and the first three to -`/r3` and `/h3`, and never answers `/slow`; the crawl runs with `--timeout 1000`, and the `r` -paths are same-origin GETs, the others cross-origin HEADs. HEAD's copy requested each path -once and reported all five. After, each path was requested three times: `/r2` and `/h2` -succeeded on the third, and `/r3`, `/h3` and `/slow` were reported, the last as `timeout`. The -C22 fixture through the kit's static server: three runs of three exit 1 with 28 broken, as -before. A host that never answers now costs three timeouts, 45 s at the default, before its -link is reported. `compare_trees`: Tools.html online and offline, the search data and -`book.html`, nothing else. +Landed. ### C22e — `docs: Pipeline-Stages.md's task sections match the chart and the task graph` -**Found while fixing Builder.md's copy of the same lists in C22c; the owner asked on -2026-09-26 for it to be fixed before C23** (see Found while implementing). Extending.md says -each task's `###` heading on the page sits under the numbered section matching its Gantt -section, and eight did not: Section 1 (Seed tasks) held `scss`, `dot`, `prepDest` and -`prepPageDirs`, Section 2 (Spine) `loadData` and `dispatch`, and Section 3 (Render fan-out) -`flush:i` and `flushJoin`. Section 1 also held `warmInit`, which the chart draws as a start-up -bar in each worker's row, and its introduction said its tasks have no predecessors. - -**Change.** Each `###` section moves under its chart section, in Builder.md's order, and -`warmInit` joins `renderEnvInit` under Render, as in Builder.md; the four introductions say -what each section now holds. No heading's text changes, so every anchor keeps its id. -Extending.md's rule gains the case the chart gives no section: a per-lane start-up task goes -under Render. - -**Landed**, with three additions. `markdownInit`'s `expected` line lacked `deriveRedirects`, -which it waits for only to count the redirect stubs for the `{{tbdocs:...}}` counts -(`tbdocs.mjs:598-599`), and its prose did not mention the counts; both are corrected, and so -is `deriveRedirects`'s list of consumers. `renderJoin` unblocks `symbolIndex` as well as -`searchData` and `writePdf`, and `flushJoin` unblocks `linkJoin` as well as `writeAux` and -`writePdf`. And `highlighterInit` gains the `expected` block that every other task with a -predecessor has. A scratch script moved the sections and would not write unless the region's -non-blank lines came out the same multiset; `git diff --color-moved` shows no other line -changed. The kit's `c22e-verify.mjs` checks the page against `tbdocs.mjs`: each block's -section against `GANTT_SECTION`, with `render:i` in Render and `flush:i` in Write as -`dispatch.submit` gives them and the two start-up tasks in Render; each `expected` line -against `TASKS`; and a block for every static task. On HEAD's page it reports 10 problems, -nine misplaced blocks and `markdownInit`'s line; after, none. `build.bat`'s link check passes, -so no link into the page lost its anchor. `compare_trees`: Extending.html and -Pipeline-Stages.html online and offline, the search data and `book.html`, nothing else. Two -defects found on the way are recorded under Found while implementing. +Landed. ### C22f — `docs: export tables for counts.mjs and page-baseline.mjs` -**Found while fixing Pipeline-Stages.md's task sections in C22e; the owner asked on -2026-09-26 for it to be fixed before C23** (see Found while implementing). The page says its -second half covers every module, with the full export table for each, and had none for -`builder/counts.mjs` or `builder/page-baseline.mjs`. - -**Change.** A `counts.mjs` section after `render.mjs`'s, since `countPlugin` is the last -plugin `createMarkdownIt` applies, and a `page-baseline.mjs` section after -`symbol-baseline.mjs`'s, the other drift guard: every export, in the order the module declares -them, in the neighbouring tables' form. - -**Landed.** A Sonnet agent drafted both tables from the modules, and every row was checked -against the source; five were corrected. `validateCountNames` returns a message per unknown -reference, not per name, and offers the nearest known name only within an edit distance of -three (`counts.mjs:251-258`). `countPlugin` runs after `replacements` because its core rule is -pushed last, not because it is the last plugin. `checkPageBaseline`'s row is rewritten so that -each case reads on its own (`page-baseline.mjs:120-175`). `GUARDED_SRC` is passed in the two -scripts' probes rather than building fixtures. And `deriveCounts`'s folder-style indexes are -reference pages, not only classes. Nothing imports `COUNT_NAMES`, which is recorded under -Found while implementing. `compare_trees`: Pipeline-Stages.html online and offline, the search -data and `book.html`, nothing else. +Landed. ### C22g — `builder: tbdocs.mjs's task-graph comment points to TASKS and the docs` -**Found while checking C22e's edges; the owner asked on 2026-09-26 for it to be fixed before -C23, with a pointer rather than a correction** (see Found while implementing). The comment -above `TASKS` described the graph in prose, and contradicted `TASKS` in three places. - -**Change.** The prose goes. The comment says that `TASKS`, with the `render:i` and `flush:i` -tasks `dispatch.submit` adds, is the graph, and names Pipeline-Stages.md and -`scheduler-dag.dot` as its descriptions; the sentence on `runBuild()` stays. No page cites the -comment. - -**Landed.** A comment-only change: `compare_trees` identical, lint clean. +Landed. ### C22h — `builder: delete counts.mjs's unused COUNT_NAMES` -**Found while checking C22f's table; the owner asked on 2026-09-26 for it to be deleted -before C23** (see Found while implementing). `COUNT_NAMES` called `deriveCounts` with an empty -state at module load, and nothing read it: `validateCountNames` checks a page against the -keys of the counts it is given, and `countPlugin` substitutes from the same object. - -**Change.** The export goes, with its row in Pipeline-Stages.md's table and the clause of the -`deriveCounts` row that named it. It was the only call that omitted `extra`, so `extra`'s -default and the `?? 0` behind `redirectStubs` go too, and the JSDoc and the table's signature -make `extra` required; `tbdocs.mjs:606`, the one caller left, always passes it. Extending.md -says instead that the returned object's keys are the names a page may use. - -**Landed.** `compare_trees`: Extending.html and Pipeline-Stages.html online and offline, the -search data and `book.html`, nothing else. Lint clean. +Landed. ### C22i — `scripts: crawl_check reports a page whose body cannot be read` -**Found while implementing C22d; the owner asked on 2026-09-26 for it to be fixed before C23, -with no retry** (see Found while implementing). `crawlOne` recorded a same-site page as -reachable before reading its body, and returned in silence when the read failed: the page's -links were never extracted, its ids never indexed, and the report said nothing. - -**Change.** When the read fails, `crawlOne` records the page broken, with its status and the -error prefixed `body:`, and returns. It does not retry: the server has answered, and C22d's -retries already cover the common reset, which comes before any response. The part of the body -that arrived is not parsed. The header and Tools.md's paragraph say so. - -**Landed.** The kit's `c22i-body.mjs` serves `/`, linking `/cut` and `/ok`; `/cut` answers -200 `text/html` with a `Content-Length` of 5000, sends a link to `/missing` and closes the -socket 50 ms later. HEAD's copy: exit 0, 3 pages crawled, 2 unique links, 3 status checks, 0 -broken and 0 missing anchors. After: exit 1, the same counts with 1 broken, `[ERR body: -terminated] http://localhost:4395/cut`, where `terminated` is undici's message. Both requested -each of `/`, `/cut` and `/ok` once and `/missing` never. The kit's `c22d-retry.mjs` gives -C22d's result unchanged: each failing path requested three times, `/r2` and `/h2` recovering, -and `/r3`, `/h3` and `/slow` reported. The C22 fixture through the kit's static server: three -runs of three exit 1 with 28 broken. `compare_trees`: Tools.html online and offline, the -search data and `book.html`, nothing else. Lint clean. +Landed. ### C22j — `scripts: crawl_check's --timeout covers a page's body` -**Found while implementing C22i; the owner asked on 2026-09-26 for it to be fixed before -C23** (see Found while implementing). `fetchWithRetry` cleared its timer once `fetch` -resolved, which is when the headers arrive, so `--timeout` did not bound the read of a page's -body, and a body that stalled after its headers held the crawl until undici gave up. - -**Change.** Each attempt passes `AbortSignal.timeout(timeoutMs)` as its signal, so the timeout -runs on through the body. Such a signal fails with a `TimeoutError`, not an `AbortError`, so -the three places that record an error take its text from one function, `errorText`: -`timeout` for a timeout, the error's message otherwise. A body still arriving when the time -runs out is reported `body: timeout`, without a retry, as C22i decided for a body that breaks -off. The header, the retry comment and Tools.md's paragraph say so. - -**Landed.** The kit's `c22i-stall.mjs` serves `/`, linking `/stall` and `/ok`; `/stall` -answers 200 `text/html` with a `Content-Length` of 5000, sends a link to `/missing`, then -sends nothing and keeps the socket open. With `--timeout 1000`, C22i's commit exited 1 after -305.6 s, reporting `[ERR body: terminated]` for `/stall`; after, it exits 1 after 1.2 s, -reporting `[ERR body: timeout]`. Both requested each of `/`, `/stall` and `/ok` once and -`/missing` never. `c22i-body.mjs` still reports `body: terminated` for `/cut`; -`c22d-retry.mjs` gives C22d's result unchanged, `/slow` reported as `timeout`; and the C22 -fixture gives three runs of three exit 1 with 28 broken. A crawl of the built site through -the kit's static server: exit 0, 1,247 pages crawled, 3,228 unique links, 0 broken and 0 -missing anchors in 88.8 s, so the default 15 s, which now covers each body, stopped no page. -`compare_trees`: Tools.html online and offline, the search data and `book.html`, nothing -else. Lint clean. +Landed. ### C23 — `scripts: check_examples restores the registry after a spawn failure` -**L3-2 (R2)**, with V4's note that `check_examples.mjs` has no process-level handler at all. -`buildStaged`'s spawn (`:601-612`) has no `'error'` listener and waits for `'exit'`. A spawn -failure is then an uncaught exception at the emitter, outside both `main()`'s catch and -`main().catch`, so the step that restores the tbIDE registry never runs. - -**Change.** Listen for `'error'` and wait for `'close'`, as `addin_test.mjs:180,185` and -`check_regex_safety.mjs:347-348` do. An `uncaughtException` and `unhandledRejection` handler -restores the registry and exits 2. - -**Verify.** With the spawn pointed at a missing executable (a scratch edit), the run exits 2 -and the registry is as it was found. The `examples.bat` summary unchanged (a harness run). - -**Landed.** Waiting for `'close'` also means `out` holds all of tbbuild's output when its JSON -is parsed, which `'exit'` did not promise. The handler is `die`, installed at the bottom -beside `main().catch`, whose body it takes over; the probes' `fakeLane` already has a -parameter named `crash`. A scratch edit pointed the spawn at `C:\no-such-folder\node.exe`, -run with `--jobs 1 --only "^Reference/Core/"` (186 samples from 87 pages in 12 projects). -HEAD: exit 1 after 3.6 s, on Node's own report of the uncaught `spawn -C:\no-such-folder\node.exe ENOENT`, which `examples.bat` reads as a sample that does not -compile, and nothing tidied. After: exit 2 after 3.4 s, `check_examples: spawn -C:\no-such-folder\node.exe ENOENT`, from `main()`'s catch around the lanes, which finishes -the tidy. A scratch throw from `process.nextTick` and a scratch rejection that nothing -awaits, each in `buildStaged` before the spawn, exit 2 through `die` with the error printed. -A read-only snapshot of the keys the tidy covers (`reg export` of the IDE's settings key and -the two association keys) came out identical around every run; in the failing runs no IDE -starts, so HEAD leaves the registry as found as well, and the full run is what shows the tidy -still puts it back. `examples.bat`: exit 0 after 122.2 s, `1129 sample(s), 1129 compile, 0 -finding(s), 120.2s -- clean`, from 597 pages in 43 projects on 4 lanes, the snapshot -identical before and after. A lane that fails while another builds was checked as well, -because the catch then tidies while the other lane's IDE still runs, which `finishTidy`'s -comment forbids: with lane 1 failing 6 s in on two lanes, the run exited 2 after 9.8 s, no -tbbuild or IDE process was left, and the snapshot was identical, because Node ends the -children it spawned when it exits, and they end theirs (`tb-ide.mjs:145-150`). -`compare_trees` identical. Lint clean. +Landed. ### C24 — `scripts: tb-operate stops on an afterReveal timeout` -**A7-2 (R2).** `afterReveal` (`tb-operate.mjs:421-429`) returns `false` on a timeout, and -`openFile` (`:453`), `setCursor` (`:461`) and `select` (`:475`) discard the result. That -reopens the cursor-reset race the function exists to prevent: `:401-409` records the measured -`"xyz"` to `"zy"` corruption. - -**Change.** Each call site checks the result, retries once, then throws naming the file and -position. - -**Verify.** `addin-test.bat` green, all ten lanes (a harness run). - -**Landed**, without the retry: waiting again only makes the wait longer, which is what -`afterReveal`'s `timeout` is for, and opening the file again would start a new reveal (see -Where the plan was wrong). The three call sites share one unexported helper, `settledAt`, -which throws naming the file and the place; `setCursor` and `select` are not given the file -and read it from `editorState`. `afterReveal` still returns `false` on a timeout, for a -caller that waits after an add-in's `Editors.Open`; nothing in the tree calls it but the -three. The kit's `c24-reveal.mjs` drives the three against a fake connection whose reveal -window never closes. HEAD's copy returned from each after 10.1 s, and `setCursor` and -`select` then placed the cursor anyway; after, each throws after 10.1 s, as in `the IDE was -still revealing lines 10 s later, so the cursor could still move: -/AddinHost/Sources/Haystack.twin at 4:9`, and places nothing. With the window closed, each -returns at once and places the cursor as before. WIP.Harness.md's paragraph on the 700 ms -says so. `addin-test.bat`: exit 0 after 131.4 s, `10 of 10 lane(s) ran: 10 passed`, with -`registry: put back (20 project-state, 21 recent-list and 3 association writes)`. -`compare_trees` identical. Lint clean. +Landed. ### C25 — `scripts: tbbuild always tidies; correct tb-registry's -Command note` -**A7-9, A7-6 (R3).** `tbbuild`'s shutdown skips its tidy step when the IDE handle was never -set, which is safe only by an invariant inside `tb-launch.ps1`; `tbrun` always tidies. And -`tb-registry.mjs:49-50` says `tbrun`'s snapshot uses `-EncodedCommand`, where `tbrun.mjs:376` -uses `-Command` with a fixed literal. - -**Change.** `tbbuild` tidies on every shutdown path, as `tbrun` does. The comment is -corrected; the code is safe as it stands. - -**Verify.** `tbbuild` on a probe, and `tbbuild` given a missing `--ide`, both leave the -registry as found (harness runs). - -**Landed.** `shutdown` runs without an IDE only after a `launchIde` that throws, as it does -when the DevTools port stays taken or a hidden launch prints no pid. It now returns early only -under `--keep`: `shutdownIde` does nothing without an IDE, as `tbrun` already relies on, and -`finishTidy` writes only where a value differs. The kit's `c25-reglog.mjs`, preloaded, logs -each registry request a run makes and what each write request changed, and `c25-run.mjs` -brackets a run with `reg-snap.mjs`'s read-only snapshots. With `--ide C:\nope\twinBASIC.exe`, -HEAD exited 2 after 1.4 s having made `startTidy`'s three reads and no other request; after, -it exited 2 after 2.3 s with `finishTidy` run as well: `restoreProjects` wrote 0 values, -`restoreKeys` changed 0, and the build targets were read and left alone. No IDE started, so -the snapshot was identical around both. `tbbuild` on a probe (the `console` template, -packed): exit 0 after 10.7 s, `--- 0 error(s), 0 warning(s), 0 hint(s), 0 info`, with 1 -project-state and 1 recent-list value put back, the snapshot identical before and after, and -no IDE or compiler process left. `tb-registry.mjs`'s comment names only `tb-launch.ps1` as -passed the same way. `compare_trees` identical. Lint clean. +Landed. ### C25a — `scripts: tbrun reports a codegen failure that Debug.Cls erased` -**Found while verifying C16; the owner chose this fix on 2026-09-25** (see Found while -implementing). When a procedure the probe calls fails code generation, the compiler logs -`[LINKER] compilation (codegen) error` straight after `[BUILD] Executing -'..'...`. The probe's first statement, `Debug.Cls`, erases that line, -and the probe prints up to the call and stops. `tbrun` exits 0 with the partial output. - -**Change.** Before it presses Build, `tbrun` wraps the IDE page's global -`clearDebugConsole()`, so each call first saves the lines it is about to erase, read the -way `readConsole` reads them. In BETA 983's `ide/main.js` the compiler's clear event -(`event_clearDebugConsole`) and the Clear command both call that function by name, so the -wrapper sees every clear. After the run, a `BUILD_FAILED` line in a saved segment after the -last `[BUILD] Executing` line makes `tbrun` exit 2, naming that line and printing the -partial output. An IDE page without `clearDebugConsole` is refused, as one without -`dataNodes` is today. Probes keep `Debug.Cls`: nothing asked of a probe changes. `tbrun`'s -header, Tools.md's paragraph and WIP.Harness.md's bullet on failed builds name the case. - -**Verify.** Harness runs, one at a time: probe C (the shift in a procedure the probe calls), -before `exit 0` with `before` as its output, after `exit 2` naming the codegen line; probe A -(the shift in the `[RunAfterBuild]` Sub) still `exit 2`; a clean probe still `exit 0` with -its output; and a clean probe that calls `Debug.Cls` twice `exit 0`, since a saved segment -with no failure line in it is not a failure. +Landed. ### C25b — `scripts: tbbuild refuses a named IDE that is not there` -**Found while implementing C25; the owner asked on 2026-09-26 for it to be fixed before -C25a**, as were C25c and C25d (see Found while implementing). Of the six tools that call -`findIde`, `tbbuild` alone launched a named IDE without checking that it exists. `tbrun`, -`addin_test` and `build_package_api` refuse one with exit 2, `census_attributes` looks for -its `packages` folder, and `check_examples` for the compiler beside it. A wrong `--ide` or -`TB_IDE` was found out only by the launch, which reported it in CLIXML (C25c) or, under -`--show`, crashed (C25d). - -**Change.** The check that refuses a missing IDE also refuses a named one that is not there, -before `startTidy`, naming the path: `no twinBASIC IDE at : pass --ide ...`, exit 2. - -**Landed.** With `--ide C:\nope\twinBASIC.exe`, and with `TB_IDE` naming the same path and no -`--ide`, `tbbuild` exits 2 after 0.1 s with that line and makes no registry request (the kit's -`c25-run.mjs`); at C25's commit it exited 2 after 2.3 s, with the launch's CLIXML, after a -full tidy. `tbbuild` on the probe, with the IDE found on the Desktop: exit 0 after 13.8 s, -`--- 0 error(s), 0 warning(s), 0 hint(s), 0 info`, the snapshot identical before and after. -`compare_trees` identical. Lint clean. +Landed. ### C25c — `scripts: tb-launch.ps1 reports why a launch failed, in plain text` -**Found while implementing C25** (see Found while implementing). When a hidden launch failed, -`launchIde` relayed `tb-launch.ps1`'s stderr, which PowerShell writes in CLIXML when its -streams are redirected: the message read `#< CLIXML` and a line of XML, with the cause inside -an `` record. The cause was wrong as well. `Fail` read the Win32 error after -PowerShell had made calls of its own, which replace it: for `C:\nope\twinBASIC.exe` it -reported 203, "The system could not find the environment option that was entered", where a -C# read straight after the same `CreateProcess` gave 3. `tbbuild`, `tbrun` and the add-in -test lanes all launch through it. - -**Change.** Every Win32 call moves into a C# helper that throws, with the error read straight -after the call: `Desktop`, `KillOnCloseJob`, `Start`, `Assign`, which still ends the -suspended process when it cannot go into the job, and `Resume`. Progress is silenced, and a -`trap` writes a failure's innermost message as one line of UTF-8 on stderr and exits 1. -Setting the job's limit in C# drops the PowerShell workaround of copying the nested struct -out and back. WIP.Harness.md's paragraph on the launcher says why. - -**Landed.** The kit's `c25c-launch.mjs` runs a copy of the script as `tb-ide.mjs` does, -without an IDE. HEAD's wrote CLIXML to stderr in every case, a successful launch included, -for its progress record. After: `C:\nope\twinBASIC.exe` gives `CreateProcess failed: The -directory name is invalid`, because the working folder, the missing `C:\nope`, is checked -first; the install folder gives `Access is denied`, and `C:\Windows\twinBASIC.exe` `The system -cannot find the file specified`. Each is the only line on stderr, with exit 1. A short-lived -`node` child, with the job and without, prints its pid and nothing on stderr. A scratch -variant that passes `Assign` a null job gives `AssignProcessToJobObject failed: The handle is -invalid` and leaves no suspended child; another shows the UTF-8 line is needed, since without -it `éü` arrives as `��`. `tbbuild --ide` naming the install folder: exit 2 after 2.4 s with -that line, the registry as found. `tbbuild` on the probe: exit 0 after 10.7 s, clean. The -encoded script is 24,296 characters, against 20,872 before; a command line stops at 32,767. -`examples.bat`: exit 0, `1129 sample(s), 1129 compile, 0 finding(s), 124.1s -- clean`. -`addin-test.bat`: exit 0 after 130 s, `10 of 10 lane(s) ran: 10 passed`, `registry: put back -(20 project-state, 21 recent-list and 3 association writes)`. `compare_trees` identical. Lint -clean. +Landed. ### C25d — `scripts: launchIde under --show reports a spawn that fails` -**Found while implementing C25** (see Found while implementing). `launchIde`'s `--show` branch -spawned the IDE with no `'error'` listener and returned at once. A spawn that fails is -reported by an `'error'` event, not a throw, so `launchIde` returned a handle with no pid, and -the event then ended the process on Node's report of an unhandled `'error'`, with exit 1, -which `tbbuild` and `tbrun` define as compile errors. Nothing after it ran, the tidy -included. - -**Change.** The branch waits for the child's `'spawn'` or `'error'` event before it returns, -and a failed spawn throws `could not start the IDE: `, which the callers' -catch reports with exit 2. The doc comment says a launch that fails throws, hidden or not. - -**Landed.** The kit's `c25d-show.mjs` calls `launchIde` with `show: true` and no IDE. HEAD's -copy returned `pid undefined` for `C:\nope\twinBASIC.exe` and for the install folder, and the -process then died on the unhandled `'error'` with exit 1; for a one-second `node` child it -returned a live pid. After: the first two throw `could not start the IDE: spawn -ENOENT`, since libuv reports a folder as not found too, and the child's live pid is returned -as before. `tbbuild --show` with the install folder as `--ide`: exit 2 after 2.3 s with that -line, after a tidy that wrote nothing, the registry as found; at C25's commit it exited 1 on -Node's report. No IDE was started on the desktop: the success path is the `node` child's. -`compare_trees` identical. Lint clean. +Landed. -### C26 — `wisdom: parseStaging refuses a chunk it cannot place` +### C25e — `scripts: tbrun's --raw changes only what it prints` -**L3-3 (R1)**, the half that needs no shared module. `parseStaging` (`merger.mjs:114-129`) -splits on any line equal to `---`, and `parseSection` returns `null` for a chunk that does -not start with `## ` (`:162-168`), which the caller drops (`:147-148,152-153`). A bare `---` -inside a fenced sample therefore drops the rest of its section and the section's -`finding_ids` line without a word, against the header's promise never to drop reviewer -content (`:111-112`). Today's file produces no such chunk. +Landed. -**Change.** A chunk that does not start with `## ` is an error naming its line. C36 then -stops a fenced `---` from making such a chunk; this check stays as the guard for any other -malformed one. +### C25f — `scripts: tbrun says when a probe ran and printed nothing` -**Verify.** A synthetic file with a `---` inside a fence: before, the tail disappears; after, -the run fails naming the line. The real `staging.md` parses to the same 1,160 sections and -serialises to the same bytes as before. +Landed. -### C27 — `wisdom: write manifest.json and denied.json atomically` +### C26 — `wisdom: parseStaging refuses a chunk it cannot place` -**A10-5 (R2).** `saveManifest` (`wisdom/discord/messages.mjs:10-12`) and `wisdom.mjs:146` -write with a plain `writeFileSync`, and `loadManifest` parses without a guard, beside -`wisdom/extract/state.mjs:62-75`, which writes to a temp file and renames it. +**Carried forward.** `parseStaging` throws, naming the line, when the chunk after a `---` line +does not start with `## `, so a `---` inside a fenced sample stops the run. C26's oracle, a +scratch script not in the repository, ran it on the real `staging.md` and on synthetic files, +one of them a section whose fenced sample holds a `---`. C36's Verify expects that file to +keep its section's tail and its `_Source threads:_` line. -**Change.** Both writes use the temp-and-rename write `state.mjs` already has, shared within -`wisdom/`; a file that does not parse is reported by name. +### C27 — `wisdom: write manifest.json and denied.json atomically` -**Verify.** With the rename made to throw (a scratch edit), the previous file survives -intact; a truncated manifest gives an error that names it. +Landed. -### C28 — `scripts: exit 2 on a crash in three tools that exit 1` +### C27a — `wisdom: an incremental export fetches new messages in stored targets` -**A5-2 (R2), and the `check_tb_registry.mjs` half of L1-11.** The convention -(`Extending.md:640-648`) keeps 1 for a finding and 2 for a crash. `pick_a11y_sample.mjs`'s -`discover()` (`:159-169`) throws uncaught on a missing tree and exits 1, the same as a -coverage gap (`:310`), and it runs in `check.bat`. `build_dot_metrics.mjs` has no catch -around its browser work, so a crash exits 1, which is also its STALE result. -`check_tb_registry.mjs` exits 1 for a crash and for a failure alike. +Landed. -**Change.** The convention's crash handler, as `check_dot_fit.mjs:31-34` has it, in all -three, exiting 2. C43 folds the handlers into one helper. +### C27b — `wisdom: export --since keeps the history already stored` -**Verify.** In each, a forced crash exits 2 and a real finding still exits 1. -`check_tb_registry.mjs`'s fixtures pass (a harness run). +Landed. -### C29 — `test.bat: cite check_gate_lists for the gate's history` +### C28 — `scripts: exit 2 on a crash in three tools that exit 1` -**A6-2 (R1).** `test.bat:34-43` tells the gate's history in a way that neither -`check_gate_lists.mjs`'s header (about `:30-44`) nor `Tools.md:435-437` supports, and those -two agree with each other. +**Carried forward.** `pick_a11y_sample.mjs`, `build_dot_metrics.mjs` and +`check_tb_registry.mjs` each gained `check_dot_fit.mjs`'s crash handler, installed after +their imports, exiting 2 on a crash; `check_tb_registry.mjs`'s catch passes on everything but +an `AssertionError`. C43 folds these three handlers, and C07's, into `lib/gate-probes.mjs`'s +shared handler. -**Change.** Trim the comment to a citation of the header, matching the file's other seven -comments. +### C29 — `test.bat: cite check_gate_lists for the gate's history` -**Verify.** Re-read against both accounts; `check_gate_lists.mjs` and -`check_ci_workflows.mjs` pass. +Landed. ### C30 — `scripts: tidy check_links_diff's and check_publish_policy's failures` -**L2-6, L3-6 (R3).** `check_publish_policy.mjs:152-189` and `check_links_diff.mjs:651-750` -remove their scratch folders only on success, where four other tools do it in a `finally`. -And `check_links_diff.mjs`'s `fusedBuild` (`:426,432`) and `ensureBasePathTree` -(`:518,525`) call `spawnSync` unguarded, so a failure reaches the terminal as a stack trace -at exit 1 rather than as the tool's `error:` line at 2. - -**Change.** A `finally` for both scratch folders; both spawns report through the tool's error -path. - -**Verify.** A forced failure in each leaves no scratch folder and prints the tool's `error:` -line with exit 2. `check_links_diff.mjs --self-test` and CI's fixture cases unchanged. +Landed. ## Phase 2: shared code, in place, with no change in behaviour @@ -2180,7 +1597,8 @@ Defects the review did not have, found by building something this plan asks for. table and plugin chain` (`57cdaa1d`). - **`tbrun` exits 0 with partial output when a procedure the probe calls fails code - generation**, found while verifying C16. Scheduled as C25a; see its entry for the fix. + generation**, found while verifying C16. Scheduled as C25a, and fixed in `scripts: tbrun + reports a codegen failure that Debug.Cls erased`. - **Builder.md's "Task DAG by section" disagrees with the chart and with the task graph**, found while re-reading its Gantt paragraph for C21. The section lists put `discover` in @@ -2291,6 +1709,38 @@ Defects the review did not have, found by building something this plan asks for. `tbbuild` and `tbrun` give compile errors. Measured with a missing executable and with the install folder. Fixed in `scripts: launchIde under --show reports a spawn that fails`. +- **`tbrun --raw` never sees a failed build**, found while verifying C25a. `--raw` keeps each + console line's timestamp column, and `tbrun` read the console once, with it, for its checks + as well as its output. `BUILD_FAILED` is anchored at a line's start, so under `--raw` no + failure matched: probe A exited 0 after 19.5 s and printed the timestamped build log as the + probe's output. A line holding only a timestamp is not blank either, so a probe that printed + nothing exited 0 after 18.9 s with one such line, where without `--raw` it exits 3. Fixed in + `scripts: tbrun's --raw changes only what it prints`. + +- **`tbrun` says a probe that ran and printed nothing "may not have run"**, found while + verifying C25e. The build log is written and erased within the first 400 ms poll, so such a + probe waits the whole 120 s timeout and exits 3 with the hint to look for a modal, though + the record of clears C25a keeps shows that the Sub started. Fixed in `scripts: tbrun says + when a probe ran and printed nothing`. + +- **An incremental `wisdom.mjs export` never fetched a new message in a target it had + stored**, found while verifying C27. `runExport` skipped every channel and thread with a + manifest entry and a file, and the `?after=` branch ran only when the file was missing, + writing the new messages as the whole file. Wisdom.md said a re-run fetches the new + messages. C27's oracle measured it: a message added between two exports never reached the + file. The stored export was four months old, and the first online pass after the fix + appended 4,105 messages to 28 files and added 96. Fixed in `wisdom: an incremental export + fetches new messages in stored targets`. + +- **`wisdom.mjs export --since` replaced the history already stored**, found while verifying + C27a. Under `--since` the export loaded an empty manifest, so it fetched every target from + the date, wrote each file whole with only the messages after it, and saved a manifest + holding only the targets that had some. A later plain export took each shortened file as up + to date and never fetched the lost messages again: those before the date, and any between + the file's end and the date. C27a's oracle measured it: a stored channel holding d1 and d2 + held d10 alone after an export with `--since` d5. Fixed in `wisdom: export --since keeps the + history already stored`. + ## Open questions Each is settled in the commit named, on the recommendation given there, unless the owner diff --git a/docs/Documentation/Tools.md b/docs/Documentation/Tools.md index 304a684f..062c44a7 100644 --- a/docs/Documentation/Tools.md +++ b/docs/Documentation/Tools.md @@ -694,9 +694,9 @@ probe's output, with exit 0. Run it again: both failures seen so far passed on a A `[RunAfterBuild]` Sub that fails code generation exits 2 the same way: the build succeeds, the console adds `[LINKER] compilation (codegen) error detected in '.'`, and nothing in the Sub runs, `Debug.Cls` included. A procedure the probe *calls* that fails -code generation is not caught. Its error line is written before the probe's first statement, -which erases it, and the probe stops at the call, so `tbrun` exits 0 with the output printed -up to that point. +code generation exits 2 as well. Its error line is written before the probe's first +statement, so the probe's `Debug.Cls` erases it, and the probe stops at the call. `tbrun` +keeps what each clear erases, so it names that line and prints the output up to the call. **The capture is complete however much a probe prints**, so there is no reason to keep one short. `tbrun` reads the console's backing array rather than the pane, which is a virtualised @@ -731,8 +731,9 @@ comes back as `A&`. | `--show` / `--hide` | As for [`tbbuild.mjs`](#tbbuild): your own desktop or a private one, with `TBBUILD_SHOW` setting the default. | Exit codes: **0** captured output, **1** the project has compile errors (the diagnostics are -printed), **2** the harness failed or the build did after a clean compile, **3** nothing reached -the console before the timeout. +printed), **2** the harness failed or the build did after a clean compile, **3** no output: +nothing reached the console before the timeout, or the probe ran and printed nothing after its +last `Debug.Cls`. **A probe that activates a COM server can leak one per run.** `CreateObject("Excel.Application")` is activated by DCOM, so the `EXCEL.EXE` that appears is a child of `svchost.exe` rather than @@ -853,7 +854,8 @@ registry. It deletes the scratch key when it ends. It is not a gate and is not in `test.bat`, because it needs Windows and a real registry and the CI runners have neither. Run it by hand after changing `tb-registry.mjs`. Exit code -**0** when every check holds, **1** when one does not. +**0** when every check holds, **1** when one does not, **2** when something else stops it, +such as PowerShell failing. ### check_examples.mjs {: #check-examples } diff --git a/docs/Documentation/Wisdom.md b/docs/Documentation/Wisdom.md index f98142c2..a3570503 100644 --- a/docs/Documentation/Wisdom.md +++ b/docs/Documentation/Wisdom.md @@ -75,11 +75,13 @@ node wisdom/wisdom.mjs extract --since 2025-06-01 The `--since` mode writes to a sideband file (`staging-since-.md`) and does not touch the canonical `staging.md` or the watermark. +For `export`, `--since` limits only the channels and threads not exported yet, which are fetched from the date on, and a later export without `--force` does not go back for their older messages. One already exported is brought up to date as in a run without `--since`, so no stored history is lost. A channel or thread with no message since the date gets no file, so a later export without `--since` fetches its whole history. + The extract step automatically partitions large thread sets into batches of 200, so filtering is optional --- but `--since` and `--channel` reduce the number of threads analysed (and therefore agent invocations and API costs). ## Reviewing staging.md -`staging.md` is the long-lived review file. Each `## ` section is one proposed documentation addition, with structured metadata at the bottom (source thread IDs, confidence level, date range, optional reviewer note). Sections are grouped by target page and delimited by `---` lines. +`staging.md` is the long-lived review file. Each `## ` section is one proposed documentation addition, with structured metadata at the bottom (source thread IDs, confidence level, date range, optional reviewer note). Sections are grouped by target page and delimited by `---` lines. The first line after a `---`, blank lines aside, must be a section's `## ` heading: the next merge stops at anything else, naming its line, rather than drop it. **Removing sections.** Delete any section that is not useful --- the removal is stable. If the source thread is unchanged on the next run, the watermark filter skips it entirely and the section stays gone. If the thread later receives new messages, the thread re-enters the pipeline and the agent may produce a fresh finding that accounts for the new context; it reappears with a `[REFINED?]` marker so the reviewer knows it is a revision of something already triaged. @@ -103,11 +105,11 @@ Fetches messages from Discord channels and forum threads. node wisdom/wisdom.mjs export ``` -Outputs raw JSON under `wisdom/data/raw/`. Supports incremental runs --- a manifest tracks the highest message ID per channel, so re-running fetches only new messages. Use `--force` to re-fetch everything. +Outputs raw JSON under `wisdom/data/raw/`. Supports incremental runs --- a manifest tracks the highest message ID per channel and thread, so re-running fetches only the new messages, and only from the channels and threads that have some. Use `--force` to re-fetch everything. | Flag | Effect | |------|--------| -| `--since ` | Only fetch messages after this ISO 8601 date | +| `--since ` | Fetch a channel or thread not exported yet only from this ISO 8601 date on; one already exported is brought up to date as without it | | `--channel ` | Restrict to one channel (repeatable) | | `--dry-run` | Discover channels/threads; do not fetch messages | | `--force` | Ignore manifest; re-fetch all history | @@ -177,6 +179,7 @@ wisdom/ wisdom.mjs Entry point --- CLI parser, runExport(), dispatch config.mjs Load config.jsonc, apply CLI overrides config.jsonc Server/channel/rate-limit configuration + files.mjs Atomic writes (temp file + rename); JSON reads that report a bad file by path discord/ Phase 1 --- Discord API layer api.mjs HTTP client, auth, rate-limiter, snowflake utilities @@ -217,19 +220,22 @@ Also defines `runConcurrent(items, concurrency, fn)` --- a simple worker-pool: s ### Phase 1 control flow 1. **Discover** (`discord/discover.mjs`): fetch the full channel list from `/guilds/{id}/channels`, filter by type (text/forum) and exclude patterns. For forums, paginate `/channels/{id}/threads/archived/public` and (bot-only) `/guilds/{id}/threads/active`, deduplicate, and filter by `min_message_count`. -2. **Fetch members** (`discord/discover.mjs`): paginate `/guilds/{id}/members` (bot-only; user tokens get an empty map). Write `guild.json` and `members.json`. +2. **Fetch members** (`discord/discover.mjs`): paginate `/guilds/{id}/members`. A user token, or a bot the endpoint refuses, gets an empty map, and messages then show authors by their global name rather than their server nickname. Write `guild.json` and `members.json`. 3. **Build target list**: merge text channels and forum threads into a single list. Targets that previously returned 403 (tracked in `denied.json`) sort to the end. 4. **Fetch messages** (`discord/messages.mjs`): run targets through `runConcurrent`. For each target: - - Check manifest: if the target's highest-seen snowflake is recorded and the output file exists, skip (up-to-date). + - Check manifest: if the target's output file exists, its snowflake is recorded, and the `last_message_id` discovery reported for it is no newer, skip it (up-to-date) without a request. - Call `fetchMessages(client, channelId, afterSnowflake)`: - - **Incremental** (afterSnowflake set): page forward with `?after=`, collecting new messages. - - **Full** (afterSnowflake null): page backward with `?before=`, collecting all history. + - **Incremental** (the output file exists and its snowflake is recorded, with or without `--since`): page forward with `?after=` from that snowflake, collecting new messages. + - **Since** (otherwise, under `--since`): page forward with `?after=` from the date's snowflake. + - **Full** (otherwise, which includes every target under `--force` without `--since`): page backward with `?before=`, collecting all history. - Sort chronologically (ascending snowflake). - - Write `{ channel | thread, messages }` to `raw/channels/{id}.json` or `raw/threads/{id}.json`. - - Update manifest with `highestSnowflake(messages)` and flush to disk after each target. + - Write `{ channel | thread, messages }` to `raw/channels/{id}.json` or `raw/threads/{id}.json`. An incremental fetch appends its messages to those already in the file and stores the channel or thread object discovery returned; a since or full fetch writes the file whole. + - Update the manifest to the newest snowflake now on disk for the target, and flush it after each target. That is the newest message, or discovery's `last_message_id` when that is newer because its message was deleted, so the next run does not fetch the target again for nothing. A target with no file gets no entry. The export manifest (`raw/manifest.json`) is a flat `{ channelOrThreadId: highestSnowflake }` object. It governs incremental fetches --- on the next run, only messages newer than the stored snowflake are requested. +Every file the export writes goes to a `.tmp` file first, which is then renamed over the old one, so a run that stops during a write leaves the previous file whole. A manifest or `denied.json` that does not parse stops the next export with an error that names it, unless `--force` is given, which ignores both files. So does a stored channel or thread file that has new messages to append. + ### Phase 2 control flow 1. Load `guild.json` to build `channelMap` (id to channel object) and `tagMap` (forum tag id to tag name). @@ -288,7 +294,7 @@ The pipeline runs both stages without a barrier --- Stage 2 for group A starts a 1. **Collect results**: read all `extract-results-*.json` files and concatenate their additions arrays. 2. **Graft into staging.md** (`extract/merger.mjs` --- `graftAdditions`): - - Parse existing `staging.md` into `{ preamble, sections[] }`. The parser splits on `---` delimiter lines, then parses each chunk into heading (target_page + section + optional marker), body lines, and trailing meta lines (source threads, confidence, date range, reviewer note). + - Parse existing `staging.md` into `{ preamble, sections[] }`. The parser splits on `---` delimiter lines, then parses each chunk into heading (target_page + section + optional marker), body lines, and trailing meta lines (source threads, confidence, date range, reviewer note). A chunk that does not start with a `## ` heading stops the merge, naming its first line, before anything is written. - For each addition, compute a match key: `(target_page, section, sorted finding_ids)`. - **Key exists in staging.md** (and section is not `[LOCKED]`): replace the section body and meta in place. - **Key not in staging, but in the emission log** (from `extract-state.json`): this was previously emitted, reviewed, and removed. Insert with a `[REFINED?]` marker. @@ -337,6 +343,7 @@ data/ guild.json members.json manifest.json + denied.json channels/*.json threads/*.json threads/ Phase 2 output diff --git a/scripts/build_dot_metrics.mjs b/scripts/build_dot_metrics.mjs index c5cf462c..da341e09 100644 --- a/scripts/build_dot_metrics.mjs +++ b/scripts/build_dot_metrics.mjs @@ -36,6 +36,12 @@ import path from "node:path"; import { fileURLToPath, pathToFileURL } from "node:url"; import puppeteer from "puppeteer"; +// A crash is the harness failing, not a finding: exit 2, as Extending.md's gate +// conventions require, where 1 is --check finding the table stale. This file +// runs at top level, so there is no main().catch to do it; the handler also +// catches a rejected top-level await. +process.on("uncaughtException", (err) => { console.error(err); process.exit(2); }); + const REPO = path.dirname(path.dirname(fileURLToPath(import.meta.url))); const OUT = path.join(REPO, "builder", "inter-metrics.json"); diff --git a/scripts/check_links_diff.mjs b/scripts/check_links_diff.mjs index 917d4cf8..739a1f16 100644 --- a/scripts/check_links_diff.mjs +++ b/scripts/check_links_diff.mjs @@ -403,6 +403,12 @@ const SIDES = { }, }; +// A build this harness needed and could not get: main() reports it as an +// error: line, exit 2, having already printed the build's own output. +class HarnessError extends Error {} + +const exitOf = (r) => `exit ${r.status}${r.error ? `, ${r.error.message}` : ""}`; + // Build once per distinct --baseurl / --dest and cache the findings the // build wrote. The trees the script side reads are the ones this build // produced, so both sides are looking at the same bytes. @@ -429,7 +435,7 @@ function fusedBuild({ baseurl = "", dest = null, src = "docs", offline = false } if (!fs.existsSync(out)) { console.error(r.stdout ?? ""); console.error(r.stderr ?? ""); - throw new Error(`the build wrote no findings file (exit ${r.status})`); + throw new HarnessError(`the build wrote no findings file (${exitOf(r)})`); } const findings = JSON.parse(fs.readFileSync(out, "utf8")); fs.rmSync(out, { force: true }); @@ -522,7 +528,7 @@ function ensureBasePathTree(dir, allowBuild) { if (r.status !== 0) { console.error(r.stdout ?? ""); console.error(r.stderr ?? ""); - throw new Error(`base-path build failed (exit ${r.status})`); + throw new HarnessError(`base-path build failed (${exitOf(r)})`); } return true; } @@ -643,12 +649,24 @@ function main() { return 2; } + // A build that failed, or a throw, means nothing was compared: exit 2, not + // the 1 that says the sides differ. The fixture folder goes either way. + try { + return compare(opts); + } catch (e) { + console.error(e instanceof HarnessError ? `error: ${e.message}` : e); + return 2; + } finally { + if (opts.fixtureDir) fs.rmSync(opts.fixtureDir, { recursive: true, force: true }); + } +} + +function compare(opts) { console.log(`check_links_diff: ${opts.a} vs ${opts.b}`); // findings[side][case] const findings = { [opts.a]: {}, [opts.b]: {} }; let differences = 0; - let fixtureDir = null; const skipped = []; const usesFused = opts.a === "fused" || opts.b === "fused"; @@ -680,7 +698,6 @@ function main() { } if (c.needsFixture && !opts.fixtureDir) { opts.fixtureDir = fs.mkdtempSync(path.join(os.tmpdir(), "link-check-fixture-")); - fixtureDir = opts.fixtureDir; writeFixture(opts.fixtureDir); } const root = c.root(opts); @@ -747,8 +764,6 @@ function main() { } } - if (fixtureDir) fs.rmSync(fixtureDir, { recursive: true, force: true }); - if (skipped.length) { console.log(`\nskipped: ${skipped.join(", ")}`); } diff --git a/scripts/check_publish_policy.mjs b/scripts/check_publish_policy.mjs index 9efac8db..b328ebb7 100644 --- a/scripts/check_publish_policy.mjs +++ b/scripts/check_publish_policy.mjs @@ -150,43 +150,45 @@ if (!failures) console.log(` ok build-only types (${[...BUILD_EXTENSIONS].jo // If either stops holding, the message is wrong and this says so. { const tmp = await fs.mkdtemp(path.join(os.tmpdir(), "publish-policy-")); - const probe = async (name, body) => { - const dir = path.join(tmp, name); - await fs.mkdir(dir, { recursive: true }); - await fs.writeFile(path.join(dir, "probe.md"), body); - try { - const { pages } = await discover(dir, []); - return pages.length ? "page" : "static"; - } catch (err) { - return "throws: " + err.message.split("\n")[0]; + try { + const probe = async (name, body) => { + const dir = path.join(tmp, name); + await fs.mkdir(dir, { recursive: true }); + await fs.writeFile(path.join(dir, "probe.md"), body); + try { + const { pages } = await discover(dir, []); + return pages.length ? "page" : "static"; + } catch (err) { + return "throws: " + err.message.split("\n")[0]; + } + }; + const FM = "---\ntitle: X\npermalink: /x\n---\nbody\n"; + + // Written as an escape, not as the character: a literal BOM inside a + // string literal is invisible, and an editor stripping it would turn + // this assertion into a no-op that still passes. + const bom = await probe("bom", "\u{FEFF}" + FM); + if (bom !== "page") { + fail(`a UTF-8 BOM now yields "${bom}", not a page -- discover.mjs's ` + + `stripBom() is gone, and publish-policy.mjs's .md message should ` + + `name the BOM again`); } - }; - const FM = "---\ntitle: X\npermalink: /x\n---\nbody\n"; - - // Written as an escape, not as the character: a literal BOM inside a - // string literal is invisible, and an editor stripping it would turn - // this assertion into a no-op that still passes. - const bom = await probe("bom", "\u{FEFF}" + FM); - if (bom !== "page") { - fail(`a UTF-8 BOM now yields "${bom}", not a page -- discover.mjs's ` + - `stripBom() is gone, and publish-policy.mjs's .md message should ` + - `name the BOM again`); - } - const bad = await probe("badyaml", "---\ntitle: [unclosed\n---\nbody\n"); - if (!bad.startsWith("throws:")) { - fail(`malformed frontmatter YAML now yields "${bad}" instead of its own ` + - `error -- it would reach the publish policy, whose .md message does ` + - `not mention it`); - } + const bad = await probe("badyaml", "---\ntitle: [unclosed\n---\nbody\n"); + if (!bad.startsWith("throws:")) { + fail(`malformed frontmatter YAML now yields "${bad}" instead of its own ` + + `error -- it would reach the publish policy, whose .md message does ` + + `not mention it`); + } - const lead = await probe("leadingblank", "\n" + FM); - if (lead !== "static") { - fail(`a blank line before the opening delimiter now yields "${lead}" -- ` + - `the .md message's advice no longer describes a real fault`); + const lead = await probe("leadingblank", "\n" + FM); + if (lead !== "static") { + fail(`a blank line before the opening delimiter now yields "${lead}" -- ` + + `the .md message's advice no longer describes a real fault`); + } + } finally { + await fs.rm(tmp, { recursive: true, force: true }); } - - await fs.rm(tmp, { recursive: true, force: true }); if (!failures) console.log(" ok the .md refusal names a fault that can actually occur"); } diff --git a/scripts/check_tb_registry.mjs b/scripts/check_tb_registry.mjs index 8d05f20a..143bb5db 100644 --- a/scripts/check_tb_registry.mjs +++ b/scripts/check_tb_registry.mjs @@ -2,7 +2,8 @@ // // node scripts/check_tb_registry.mjs // -// Exit: 0 every assertion held, 1 one did not. +// Exit: 0 every assertion held, 1 one did not, 2 something else threw, such as +// PowerShell failing, so the test could not run to its end. // // NOT A GATE, and it must not join test.bat: it needs Windows and a real // registry, and the CI runners are Ubuntu while the rule for test.bat is that @@ -50,6 +51,11 @@ import path from "node:path"; import { fileURLToPath, pathToFileURL } from "node:url"; import * as R from "./lib/tb-registry.mjs"; +// A crash is the harness failing, not a finding: exit 2, as Extending.md's gate +// conventions require. This file runs at top level, so there is no main().catch +// to do it; the catch below passes it everything but a failed assertion. +process.on("uncaughtException", (err) => { console.error(err); process.exit(2); }); + const BASE = "Software\\tbharness-selftest"; const ROOT = BASE + "\\twinBASIC_IDE"; const ASSOC = BASE + "\\Classes\\twinBASIC.ProjectFile"; @@ -264,6 +270,8 @@ try { console.log("check_tb_registry: every assertion holds"); } catch (e) { + // A failed assertion is the finding; anything else is a crash. + if (!(e instanceof assert.AssertionError)) throw e; console.error(`check_tb_registry: ${e.message}`); process.exitCode = 1; } finally { diff --git a/scripts/lib/tb-ide.mjs b/scripts/lib/tb-ide.mjs index eff52ff0..af4208f3 100644 --- a/scripts/lib/tb-ide.mjs +++ b/scripts/lib/tb-ide.mjs @@ -731,6 +731,40 @@ const CONSOLE_MARK_JS = `(() => { */ export const consoleMark = (c) => c.evaluate(CONSOLE_MARK_JS); +// What each clear erased. The page's global clearDebugConsole() empties the +// console, and BETA 983's main.js calls it by name from the compiler's +// event_clearDebugConsole, which a program's Debug.Cls raises, from the pane's +// Clear command, and on closing the project. So a wrapper assigned to that +// global sees each of those clears, and reads the console as readConsole does, +// without timestamps, before letting it go. A page is wrapped once: a second +// call finds the record already there and leaves it alone. +const KEEP_CLEARS_JS = `(() => { + if (typeof clearDebugConsole !== "function") return false; + if (!Array.isArray(window.__tbKeptClears)) { + const clear = clearDebugConsole; + window.__tbKeptClears = []; + clearDebugConsole = function () { + window.__tbKeptClears.push(${consoleJs(false, 0, null)} ?? ""); + return clear.apply(this, arguments); + }; + } + return true; +})()`; + +/** + * Keep what each later clear of the DEBUG CONSOLE erases, for keptClears. + * False when this IDE has no global `clearDebugConsole()` to wrap. + */ +export const keepClears = (c) => c.evaluate(KEEP_CLEARS_JS); + +/** + * What each clear since keepClears erased, oldest first: one string per clear, + * one line per entry, as readConsole returns them. Null when this page keeps + * no such record. + */ +export const keptClears = (c) => + c.evaluate(`Array.isArray(window.__tbKeptClears) ? window.__tbKeptClears : null`); + // The build log, in the compiler's own words (its strings, BETA 983). A build // writes "[BUILD] Starting..." to the DEBUG CONSOLE, and a binary ends with // "[LINKER] SUCCESS created output file ''" or with one of some twenty diff --git a/scripts/pick_a11y_sample.mjs b/scripts/pick_a11y_sample.mjs index f0e6a5ce..fb5e83bc 100644 --- a/scripts/pick_a11y_sample.mjs +++ b/scripts/pick_a11y_sample.mjs @@ -46,6 +46,11 @@ import { readdirSync, readFileSync, existsSync } from "node:fs"; import { resolve, join, relative, sep } from "node:path"; import { DEFAULT_ROOT_DIR, REPO_ROOT, SAMPLE_PAGES } from "./lib/axe-scan.mjs"; +// A crash is the harness failing, not a finding: exit 2, as Extending.md's gate +// conventions require, where 1 is a coverage gap. This file runs at top level, +// so there is no main().catch to do it. +process.on("uncaughtException", (err) => { console.error(err); process.exit(2); }); + // --------------------------------------------------------------------------- // Construct families // --------------------------------------------------------------------------- diff --git a/scripts/tbrun.mjs b/scripts/tbrun.mjs index ede07150..75f2722a 100644 --- a/scripts/tbrun.mjs +++ b/scripts/tbrun.mjs @@ -19,8 +19,10 @@ // // Exit: 0 captured output, 1 the project has compile errors, 2 the harness // failed -- a build that fails after a clean compile included, and a -// [RunAfterBuild] Sub that fails code generation, since the probe never runs -// -- 3 the build produced no console output before the timeout. +// [RunAfterBuild] Sub that fails code generation, since the probe never runs, +// and a procedure the probe calls that fails it, since the probe stops at the +// call -- 3 no output: the build produced none in the console before the +// timeout, or the probe ran and printed none after its last Debug.Cls. // // ---------------------------------------------------------------- why // @@ -71,7 +73,11 @@ // 4. START THE PROBE WITH Debug.Cls. The DEBUG CONSOLE is also where the IDE // writes its own build log, and the linker writes there after the build -- // so without a clear, a probe's output comes back interleaved with -// [LINKER] lines. The script warns when a probe omits it. +// [LINKER] lines. The script warns when a probe omits it. The clear can +// erase a failure as well: a procedure the probe calls that fails code +// generation is reported before the probe's first statement runs. So the +// script wraps the page's clearDebugConsole() before the build, keeps +// what each clear erases, and looks there too (tb-ide's keepClears). // 5. QUIET-PERIOD, NOT A MARKER. Waiting for a sentinel string means every // probe has to print one and the script has to know it. Waiting for the // console to stop changing works for any probe. @@ -96,9 +102,9 @@ import { existsSync, readFileSync, mkdirSync, statSync, readdirSync, rmSync } fr import { tmpdir } from "node:os"; import path from "node:path"; import { compilerExe, findIde } from "./lib/tb-install.mjs"; -import { BUILD_FAILED, TARGETS, attachIde, clickCenter, compileOutcome, killTree, launchIde, - readConsole, setBuildTarget, shutdownIde, summaryLine, waitForCompile, - wantShow } from "./lib/tb-ide.mjs"; +import { BUILD_FAILED, TARGETS, attachIde, clickCenter, compileOutcome, keepClears, keptClears, + killTree, launchIde, readConsole, setBuildTarget, shutdownIde, summaryLine, + waitForCompile, wantShow } from "./lib/tb-ide.mjs"; import { laneProjectId, stageProject } from "./lib/tb-project.mjs"; import { finishTidy, startTidy } from "./lib/tb-registry.mjs"; @@ -291,8 +297,13 @@ if (outcome.counts[0] > 0) { // --------------------------------------------- build the exe, read the console -let captured = null, failure = null; +let captured = null, shown = null, erased = null, failure = null; try { + // (4) Keep what each clear erases, for the check after the run. + if (!await keepClears(cdp)) { + throw new Error("no clearDebugConsole() in this IDE -- a probe's Debug.Cls could erase a " + + "failure unseen. Refusing rather than returning what it left as complete."); + } // (2) a real press/release pair; element.click() is ignored. if (!await clickCenter(cdp, "buildIcon")) { throw new Error("no #buildIcon in the IDE page -- did the project load?"); @@ -303,7 +314,7 @@ try { let last = "", lastChange = Date.now(), seen = false; while (Date.now() - started < timeoutMs) { await new Promise((r) => setTimeout(r, 400)); - const now = await readConsole(cdp, { timestamps: flag("raw") }); + const now = await readConsole(cdp); if (now === null) { throw new Error("no debugConsoleContent.dataNodes in this IDE -- the DEBUG CONSOLE " + "was never created, or this build moved it. Refusing rather than " + @@ -313,6 +324,17 @@ try { else if (seen && Date.now() - lastChange > quietMs) break; } captured = strip(last); + // --raw changes what is printed, never what is checked. BUILD_FAILED needs a + // line that starts where the console's text does, and a line holding only a + // timestamp is never blank, so every check reads without the column; under + // --raw the lines printed are the same entries, read again with it. + shown = flag("raw") ? strip(last, await readConsole(cdp, { timestamps: true })) : captured; + const kept = await keptClears(cdp); + if (!kept) { + throw new Error("the IDE page no longer holds what the DEBUG CONSOLE's clears erased, so " + + "a failure they erased cannot be ruled out"); + } + erased = kept.flatMap((text) => text.split("\n")); cdp.close(); } catch (e) { failure = e.message; @@ -332,7 +354,29 @@ if (failure) die(2, `tbrun: ${failure}`); if (captured.some((l) => BUILD_FAILED.test(l))) { die(2, "tbrun: the build or the probe's code generation failed, so the probe never ran. " + "The console holds the IDE's build log, not the probe's output:\n" + - captured.map((l) => ` ${l}`).join("\n")); + shown.map((l) => ` ${l}`).join("\n")); +} +// A procedure the probe calls that fails code generation is reported straight +// after the "[BUILD] Executing '..'..." line, before the +// probe's first statement runs. So Debug.Cls erases the report, the probe stops +// where it calls that procedure, and the console holds only what it printed +// before then -- which tbrun returned as the whole output, exit 0 (measured, +// BETA 983). A failure line among what the clears erased counts only after the +// last Executing line: before it is the build's own log, which ended in success +// or the probe would not have run. +const started = erased.findLastIndex((l) => /^\[BUILD\] Executing '/.test(l)); +const lost = started < 0 ? undefined : erased.slice(started + 1).find((l) => BUILD_FAILED.test(l)); +if (lost) { + die(2, "tbrun: the probe's Debug.Cls erased a failure the IDE reported as the probe started:\n" + + ` ${lost}\n` + + "What the probe printed, which stops where it called the procedure that failed:\n" + + (shown.length ? shown.map((l) => ` ${l}`).join("\n") : " (nothing)")); +} +// A probe that ran leaves its Executing line among what its Debug.Cls erased, +// so an empty console then means it printed nothing after its last clear, not +// that it never ran. +if (!captured.length && started >= 0) { + die(3, `tbrun: the probe ran (${erased[started]}) but printed nothing after its last Debug.Cls.`); } if (!captured.length) { die(3, "tbrun: the build produced no console output before the timeout.\n" + @@ -342,10 +386,10 @@ if (!captured.length) { } if (flag("json")) { - console.log(JSON.stringify({ exe: builtFile(), arch, lines: captured, idePid: ideRun?.pid ?? null, + console.log(JSON.stringify({ exe: builtFile(), arch, lines: shown, idePid: ideRun?.pid ?? null, reaped }, null, 2)); } else { - for (const l of captured) console.log(l); + for (const l of shown) console.log(l); } // ------------------------------------------------------------------ helpers @@ -353,13 +397,17 @@ if (flag("json")) { // Trim blank lines off both ends. That is all this has to do now: reading // dataNodes rather than the pane means the header, the ">" input prompt and // the timestamp column never arrive in the first place, so the three filters -// that used to live here are gone along with the guesswork in them. -function strip(text) { +// that used to live here are gone along with the guesswork in them. Given `raw`, +// the same console read with its timestamps, it returns the same entries from +// that instead, since a line holding a timestamp is never blank. +function strip(text, raw = null) { if (!text) return []; - const out = text.split("\n").map((l) => l.replace(/\r$/, "")); - while (out.length && !out[0].trim()) out.shift(); - while (out.length && !out[out.length - 1].trim()) out.pop(); - return out; + const lines = (s) => s.split("\n").map((l) => l.replace(/\r$/, "")); + const out = lines(text); + let from = 0, to = out.length; + while (from < to && !out[from].trim()) from++; + while (to > from && !out[to - 1].trim()) to--; + return (raw === null ? out : lines(raw)).slice(from, to); } // (6) End OUR IDE by pid, never by image name. The tree kill takes the probe exe diff --git a/test.bat b/test.bat index 824cd186..fdbfa517 100644 --- a/test.bat +++ b/test.bat @@ -34,14 +34,9 @@ node scripts/check_publish_policy.mjs @if errorlevel 1 goto :fail @rem Tools.md's two numbered gate lists against the two wrappers that @rem actually run them, and then every gate count stated in prose on any -@rem developer page. This rotted three times: round 2 found test.bat -@rem documented as three gates when it had four and fixed it in Tools.md, -@rem Building.md's parallel copy went untouched, a fifth gate landed, and -@rem round 3 found Building.md naming three of five and Extending.md -@rem claiming check.bat runs six. Round 4 found the fix for THAT had put -@rem three more wrong numbers into Building.md and one into README.md, -@rem under a gate that read neither file. None of it broke a link or -@rem failed a gate. Pure text, no tree, no browser, ~50 ms. +@rem developer page. The counts rotted three times without breaking a +@rem link or failing a gate; check_gate_lists.mjs's header has the +@rem history. Pure text, no tree, no browser, ~50 ms. node scripts/check_gate_lists.mjs @if errorlevel 1 goto :fail @rem The same question asked of the two CI workflows, which nothing else diff --git a/wisdom/discord/messages.mjs b/wisdom/discord/messages.mjs index 5b4ac781..4d1e68bc 100644 --- a/wisdom/discord/messages.mjs +++ b/wisdom/discord/messages.mjs @@ -1,14 +1,13 @@ -import { readFileSync, writeFileSync, existsSync } from 'node:fs' import { join } from 'node:path' +import { readJsonFile, writeFileAtomic } from '../files.mjs' export function loadManifest(dir) { - const p = join(dir, 'manifest.json') - if (!existsSync(p)) return {} - return JSON.parse(readFileSync(p, 'utf-8')) + return readJsonFile(join(dir, 'manifest.json'), {}, + 'Delete it, and the next export fetches the whole history of every channel and thread again.') } export function saveManifest(dir, manifest) { - writeFileSync(join(dir, 'manifest.json'), JSON.stringify(manifest, null, 2)) + writeFileAtomic(join(dir, 'manifest.json'), JSON.stringify(manifest, null, 2)) } export async function fetchMessages(client, channelId, afterSnowflake) { @@ -46,15 +45,25 @@ export async function fetchMessages(client, channelId, afterSnowflake) { } } - // Sort chronologically (ascending snowflake) - messages.sort((a, b) => { - const d = BigInt(a.id) - BigInt(b.id) - return d < 0n ? -1 : d > 0n ? 1 : 0 - }) - + messages.sort(bySnowflake) return messages } +// Chronological order (ascending snowflake), the order a target's file keeps. +function bySnowflake(a, b) { + const d = BigInt(a.id) - BigInt(b.id) + return d < 0n ? -1 : d > 0n ? 1 : 0 +} + +/** + * The messages a target's file already holds, followed by those fetched since, + * in chronological order. A fetched message whose id is already held is dropped. + */ +export function appendMessages(stored, fetched) { + const ids = new Set(stored.map(m => m.id)) + return [...stored, ...fetched.filter(m => !ids.has(m.id))].sort(bySnowflake) +} + export function highestSnowflake(messages) { if (!messages.length) return null return messages.reduce( diff --git a/wisdom/extract/merger.mjs b/wisdom/extract/merger.mjs index 520c720c..b47ff340 100644 --- a/wisdom/extract/merger.mjs +++ b/wisdom/extract/merger.mjs @@ -14,8 +14,9 @@ // Atomic writes: temp file + rename, previous staging.md retained as // staging.md.bak for one generation. -import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync, copyFileSync } from 'node:fs' +import { existsSync, mkdirSync, readFileSync, copyFileSync } from 'node:fs' import { join } from 'node:path' +import { writeFileAtomic } from '../files.mjs' import { buildEmissionKeySet, emissionKey } from './state.mjs' const STAGING_FILE = 'staging.md' @@ -80,9 +81,7 @@ export function graftAdditions(outDir, additions, state) { if (existsSync(stagingPath)) { copyFileSync(stagingPath, backupPath) } - const tmpPath = stagingPath + '.tmp' - writeFileSync(tmpPath, serialized) - renameSync(tmpPath, stagingPath) + writeFileAtomic(stagingPath, serialized) return stats } @@ -109,24 +108,34 @@ export function renderSideband(additions) { * sections: Section[] — each section parsed from a chunk delimited by `---` * * The parser is forgiving: anything it can't categorise gets preserved as - * part of a section body so we never silently drop reviewer content. + * part of a section body so we never silently drop reviewer content. A chunk + * after a `---` that does not start with `## ` has no section to go in, so it + * throws, naming the chunk's first line, rather than being dropped. */ export function parseStaging(content) { const text = content.replace(/\r\n/g, '\n') const lines = text.split('\n') - // Split on lines that are exactly "---". + // Split on lines that are exactly "---", keeping each chunk's first line + // number for the error below. const chunks = [] + const starts = [] let current = [] - for (const line of lines) { - if (line === '---') { + let start = 1 + for (let i = 0; i < lines.length; i++) { + if (lines[i] === '---') { chunks.push(current) + starts.push(start) current = [] + start = i + 2 } else { - current.push(line) + current.push(lines[i]) } } - if (current.length || chunks.length === 0) chunks.push(current) + if (current.length || chunks.length === 0) { + chunks.push(current) + starts.push(start) + } // First chunk = preamble + first section. Subsequent chunks = sections. // Trailing chunk after the last `---` is usually blank lines; preserve as @@ -150,7 +159,18 @@ export function parseStaging(content) { for (let i = 1; i < chunks.length; i++) { if (!chunks[i].length) continue const s = parseSection(chunks[i]) - if (s) sections.push(s) + if (s) { + sections.push(s) + continue + } + // parseSection returns null for a chunk of blank lines, which is dropped, + // and for one that does not start with "## ", which is refused. + const first = chunks[i].findIndex((l) => l !== '') + if (first >= 0) { + throw new Error(`staging.md line ${starts[i] + first} follows a "---" line but is not a ` + + `"## " heading, so it belongs to no section (a "---" inside a code sample ` + + `splits the file too): ${chunks[i][first]}`) + } } // Strip leading/trailing blank lines from preamble. diff --git a/wisdom/extract/state.mjs b/wisdom/extract/state.mjs index 53fbc590..32cb0561 100644 --- a/wisdom/extract/state.mjs +++ b/wisdom/extract/state.mjs @@ -26,8 +26,9 @@ // State is updated only on successful merge. A workflow failure mid-pipeline // leaves state untouched, so the next run retries the same threads. -import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from 'node:fs' +import { existsSync, mkdirSync, readFileSync } from 'node:fs' import { join } from 'node:path' +import { writeFileAtomic } from '../files.mjs' const STATE_FILE = 'extract-state.json' const STATE_VERSION = 1 @@ -60,19 +61,16 @@ export function loadState(outDir) { /** * Write state atomically (temp file + rename). Sets lastRun to current ISO - * timestamp. Caller must ensure outDir exists. + * timestamp. */ export function saveState(outDir, state) { mkdirSync(outDir, { recursive: true }) - const path = join(outDir, STATE_FILE) - const tmpPath = path + '.tmp' const serialized = { version: STATE_VERSION, lastRun: new Date().toISOString().replace(/\.\d{3}Z$/, 'Z'), processedThreads: state.processedThreads || {}, } - writeFileSync(tmpPath, JSON.stringify(serialized, null, 2)) - renameSync(tmpPath, path) + writeFileAtomic(join(outDir, STATE_FILE), JSON.stringify(serialized, null, 2)) } /** diff --git a/wisdom/files.mjs b/wisdom/files.mjs new file mode 100644 index 00000000..c12e646f --- /dev/null +++ b/wisdom/files.mjs @@ -0,0 +1,32 @@ +// files.mjs — how wisdom writes and reads the state files it keeps between runs. +// +// A state file is written to `.tmp` and renamed over the old one, so a +// write cut off part way leaves the previous file whole. A JSON state file +// that does not parse is reported by its path, not as a bare SyntaxError. + +import { existsSync, readFileSync, renameSync, writeFileSync } from 'node:fs' + +/** + * Write `text` to `path` through a temp file and a rename. The caller makes + * sure the folder exists. + */ +export function writeFileAtomic(path, text) { + const tmpPath = path + '.tmp' + writeFileSync(tmpPath, text) + renameSync(tmpPath, path) +} + +/** + * Parse the JSON file at `path`, or return `fallback` when there is none. A + * file that does not parse throws an error that names it, followed by + * `remedy`, which says what deleting the file costs. + */ +export function readJsonFile(path, fallback, remedy) { + if (!existsSync(path)) return fallback + const text = readFileSync(path, 'utf-8') + try { + return JSON.parse(text) + } catch (err) { + throw new Error(`${path} is not valid JSON (${err.message}). ${remedy}`) + } +} diff --git a/wisdom/wisdom.mjs b/wisdom/wisdom.mjs index 76bedf19..d36e8c86 100644 --- a/wisdom/wisdom.mjs +++ b/wisdom/wisdom.mjs @@ -1,12 +1,13 @@ #!/usr/bin/env node -import { mkdirSync, writeFileSync, readFileSync, existsSync } from 'node:fs' +import { mkdirSync, existsSync } from 'node:fs' import { join, dirname } from 'node:path' import { fileURLToPath } from 'node:url' import { loadConfig } from './config.mjs' +import { readJsonFile, writeFileAtomic } from './files.mjs' import { createClient, CapReachedError, timestampToSnowflake, EXIT_CAP_REACHED } from './discord/api.mjs' import { discoverChannels, fetchMembers } from './discord/discover.mjs' -import { fetchMessages, loadManifest, saveManifest, highestSnowflake } from './discord/messages.mjs' +import { fetchMessages, appendMessages, loadManifest, saveManifest, highestSnowflake } from './discord/messages.mjs' import { runProcess } from './process/thread.mjs' import { runExtract, runMerge } from './extract/prep.mjs' @@ -43,7 +44,7 @@ function parseArgs(argv) { function writeJson(path, data) { mkdirSync(dirname(path), { recursive: true }) - writeFileSync(path, JSON.stringify(data, null, 2)) + writeFileAtomic(path, JSON.stringify(data, null, 2)) } async function runConcurrent(items, concurrency, fn) { @@ -104,7 +105,7 @@ async function runExport(flags) { writeJson(join(outDir, 'members.json'), members) // Manifest governs incremental fetches - const manifest = (flags.force || flags.since) ? {} : loadManifest(outDir) + const manifest = flags.force ? {} : loadManifest(outDir) const sinceSnowflake = flags.since ? timestampToSnowflake(Date.parse(flags.since)) : null @@ -113,7 +114,8 @@ async function runExport(flags) { const deniedPath = join(outDir, 'denied.json') const denied = flags.force ? {} - : existsSync(deniedPath) ? JSON.parse(readFileSync(deniedPath, 'utf-8')) : {} + : readJsonFile(deniedPath, {}, + 'Delete it: it only lists the targets that refused access, so that they are tried last.') // Fetch targets: text channels + forum threads // Previously denied targets sort to the end @@ -131,19 +133,25 @@ async function runExport(flags) { const subdir = target.kind === 'thread' ? 'threads' : 'channels' const filePath = join(outDir, subdir, `${target.id}.json`) - if (!flags.force && !sinceSnowflake && manifest[target.id] && existsSync(filePath)) { + // A target already on disk is fetched again only when discovery reports a + // message newer than its watermark, and then only for what came after it, + // --since or not. --since limits how far back any other target goes. + const watermark = manifest[target.id] + const stored = Boolean(watermark) && existsSync(filePath) + const lastId = target.obj.last_message_id + if (stored && lastId && BigInt(lastId) <= BigInt(watermark)) { upToDate++ return } - const after = sinceSnowflake || manifest[target.id] || null + const after = stored ? watermark : sinceSnowflake let messages try { messages = await fetchMessages(client, target.id, after) } catch (err) { if (/403/.test(err.message)) { denied[target.id] = new Date().toISOString() - writeFileSync(deniedPath, JSON.stringify(denied, null, 2)) + writeFileAtomic(deniedPath, JSON.stringify(denied, null, 2)) completed++ process.stderr.write(`[wisdom] [${completed}/${targets.length}] ${target.name}: no access; skipping\n`) return @@ -152,13 +160,26 @@ async function runExport(flags) { } if (messages.length) { + const held = stored + ? readJsonFile(filePath, null, + 'Delete it, and the next export fetches the whole history of this target again.').messages + : [] writeJson(filePath, { [target.kind]: target.obj, - messages, + messages: appendMessages(held, messages), }) - const highest = highestSnowflake(messages) - if (highest) manifest[target.id] = highest - saveManifest(outDir, manifest) + } + // Every message from where the target's file starts (its first message, or + // the --since date) up to the newest one discovery reported is now on disk. + // The watermark moves past that one even when it has since been deleted, + // so that the next run does not fetch the target again for nothing. A + // target with no file gets no watermark. + if (messages.length || stored) { + const newest = highestSnowflake([...messages, { id: lastId }, { id: watermark }].filter(m => m.id)) + if (newest !== watermark) { + manifest[target.id] = newest + saveManifest(outDir, manifest) + } } totalMessages += messages.length @@ -192,7 +213,7 @@ Commands: Export options: --guild Guild (server) ID --channel Restrict to this channel (repeatable) - --since Only content after this ISO 8601 date + --since Fetch targets not exported yet only from this ISO 8601 date --force Ignore manifest; re-fetch all history --out Output directory [default: wisdom/data/raw] --concurrency Parallel fetches [default: 3]