diff --git a/BUGS-TO-REPORT.md b/BUGS-TO-REPORT.md index 2690397c..2c1cf74b 100644 --- a/BUGS-TO-REPORT.md +++ b/BUGS-TO-REPORT.md @@ -52,13 +52,15 @@ fixture project on a private desktop and the IDE was killed: once for each varia 1, and in two successive sessions on one project for the third row and the growth from 18 copies to 20. The first row was seen when the key had been deleted and the next harness run recreated it, the second at the end of a `check_examples` run. The -harness records it because its registry tidy (`scripts/lib/tb-registry.mjs`) leaves a list -shorter than 21 entries whenever it removes harness projects, and the next project the user +harness records it because its own runs trip it: every IDE a run starts opens a project, and +a run that began on a list holding one entry ended with seventeen copies of it. The registry +tidy (`scripts/lib/tb-registry.mjs`) now puts the list back as it found it, without the +copies; a list that was short to begin with is left short, and the next project the user opens then trips this. --- -## Compiler crashes on an `Interface` whose name and base are both angle-bracket placeholders +## Compiler crashes on an `Interface` named by an angle-bracket placeholder that has an `Extends` clause **Build:** BETA 983 (`twinBASIC_win32.dll+00141F7A`) **Severity:** crash --- takes the compiler down, three restarts, then the IDE gives up. @@ -74,23 +76,59 @@ The IDE's DEBUG CONSOLE reports `NATIVE EXCEPTION: ACCESS_VIOLATION {no-basic-co `>>> thread 0004: ParsingFileStart, `, then `restarting from MEMORY`, three times over. -**Neither half reproduces it on its own**, which is what makes it worth reporting rather -than shrugging at: +**It takes a placeholder name and an `Extends` clause**, and what the clause names does not +matter: | source | result | |---|---| | `Interface ` + `End Interface` | TB5182 Syntax error, no crash | | `Interface IFoo Extends ` + `End Interface` | TB5182 + TB5079 + TB5127, no crash | | `Interface Extends ` + `End Interface` | **crash** | +| `Interface Extends IBase` + `End Interface`, no `IBase` anywhere | **crash** | +| the same, with `Interface IBase` or `Class IBase` declared in another file | **crash** | -So it takes a placeholder in *both* positions. The input is not real code --- it is a -syntax skeleton, the shape `docs/Reference/Attributes.md` uses to show where an attribute -goes --- but a parser meeting nonsense should diagnose it, and this one dereferences -something instead. +This entry used to say that it takes a placeholder in *both* positions; the last two rows, +measured on 2026-09-24 with a project of its own each, say otherwise. The input is not real +code --- it is a syntax skeleton, the shape `docs/Reference/Attributes.md` uses to show +where an attribute goes --- but a parser meeting nonsense should diagnose it, and this one +dereferences something instead. **Found by** pointing `scripts/check_examples.mjs` at the documentation's own code samples; the skeleton is one of the 1,124 `tb` fences under `docs/`. A crash in a batch of samples -costs the whole batch its result, which is why that tool bisects on exit code 4. +costs the whole batch its result, which is why that tool isolates the sample on exit code 4. + +--- + +## An `Interface` that extends itself compiles without a diagnostic + +**Build:** BETA 983 +**Severity:** invalid code accepted --- the same cycle through a class is refused. + +This two-line file compiles with no error, warning, hint or info: + +``` +Interface IA Extends IA +End Interface +``` + +A cycle through two interfaces is accepted the same way, in one file or split across two: +`Interface IA Extends IB` and `Interface IB Extends IA`. The other kinds of cycle are +diagnosed: + +| source | result | +|---|---| +| `Class CA` + `Inherits CA` + `End Class` | TB5127 circular reference | +| `Class CA` inheriting `CB` and `Class CB` inheriting `CA`, two files | TB5127 circular reference, TB5022 failed to import inherited members | +| `Type TA` holding a `TB` and `Type TB` holding a `TA`, two modules | TB5101 unable to finalize User Defined Type, possible circular reference | + +So a cycle is checked for classes and UDTs, and not for interfaces. What happens when such a +project is built --- its type library has to describe the cycle --- was not tried. + +**Observed** on 2026-09-24 with `tbbuild`, a project of its own for each source: exit 0 and +`0 error(s), 0 warning(s), 0 hint(s), 0 info` for the three interface cases, and the +diagnostics above for the rest. **Found by** looking for a compiler crash that needs two +files, to test `check_examples`' handling of one: a cycle between two files was the likeliest +candidate, and the interface cycle compiled instead of crashing. --- @@ -977,3 +1015,41 @@ with `(Of ...)`, which compile and return the larger value. **Found by** probing round 9's UC-65 answer, whose `Max` uses `>` on a type parameter with nothing to say which types it accepts. + +--- + +## Text that continues a `Debug.Print` line is escaped twice in the DEBUG CONSOLE + +**Build:** BETA 983 +**Severity:** cosmetic, but it changes what a program appears to print: `&`, `<` and `>` in +the continued part of a line show as `&`, `<` and `>`. + +Two statements in a `[RunAfterBuild]` Sub are the whole reproduction: + +``` +Debug.Print "A"; +Debug.Print "&" +``` + +The DEBUG CONSOLE shows `A&`. The text that opens the line comes out right --- +`Debug.Print "a < b";` shows `a < b` --- and everything printed after it until the line +ends is escaped twice: after `Debug.Print "C";`, `Debug.Print "D";` and +`Debug.Print "<&>"`, the line reads `CD<&>`. + +**What does not reproduce it:** a whole line (`Debug.Print "a < b & c"` shows exactly that), +and the same text in one statement (`Debug.Print "B"; "&"` shows `B&`). + +`debugOutputPartial` in `ide/main.js`, which takes all of a program's output, and an +add-in's `PrintText` too, and adds to a line that is still open, passes the new text through +`TEXTtoHTML` twice: once as it builds the text and again as it stores it. When the new +text's colour differs from the line's, the `` it puts in to change +colour goes through the second pass too, so the tags themselves show as text. The colour +comes from the output: a program's plain output is `debugConsoleOutputText`, and a +`PrintText` is `debugConsoleOutputTextYELLOW`. With a line left open in the first, made by +calling `debugOutputPartial` from the page, a `PrintText` from the IDE's own Sample 10 add-in +showed as `Hello there from +WaynesWorldAddIn!`. A program's own open line followed by a `PrintText` was not tried. + +**Observed** on 2026-09-24 with `scripts/tbrun.mjs`, which decodes the console's stored +entries once, as the pane renders them. Found while making the add-in harness read text +that the IDE appends to an open console line. diff --git a/README.md b/README.md index 181a4a67..2db71111 100644 --- a/README.md +++ b/README.md @@ -23,16 +23,17 @@ All help is *very much* appreciated :) The site is rendered by `tbdocs`, a Node.js static site generator kept in [`builder/`](builder/). You need **Node.js 22+**; a PDF or accessibility run additionally needs Chromium, installed once with `npx puppeteer browsers install chrome` (add `--install-deps` on Linux only). ``` -npm ci # once, from the repository root -build.bat # renders _site/, _site-offline/ and _site-pdf/, and link-checks them -serve.bat # localhost:4000 with watch + live reload -check.bat # the gates that read the built site, ending in the accessibility scan -test.bat # the gates that test the toolchain itself -book.bat # renders the PDF book; run build.bat first -examples.bat # compiles the twinBASIC code samples in the pages (Windows + a twinBASIC install) +npm ci # once, from the repository root +build.bat # renders _site/, _site-offline/ and _site-pdf/, and link-checks them +serve.bat # localhost:4000 with watch + live reload +check.bat # the gates that read the built site, ending in the accessibility scan +test.bat # the gates that test the toolchain itself +book.bat # renders the PDF book; run build.bat first +examples.bat # compiles the twinBASIC code samples in the pages (Windows + a twinBASIC install) +addin-test.bat # tests IDE add-ins by operating an IDE (Windows + a twinBASIC install) ``` -A clean `build.bat && check.bat` is the bar for "ready to commit"; add `test.bat` when the change touched anything outside `docs/`. Each wrapper names the gates it runs, in order, on [Tools and Scripts](https://docs.twinbasic.com/Documentation/Development/Tools). On Linux or macOS, run the `node` command inside each batch file directly --- they are thin wrappers. `examples.bat` is the exception to both: it drives the twinBASIC IDE, so it is Windows-only and is deliberately outside every gate and outside CI. +A clean `build.bat && check.bat` is the bar for "ready to commit"; add `test.bat` when the change touched anything outside `docs/`. Each wrapper names the gates it runs, in order, on [Tools and Scripts](https://docs.twinbasic.com/Documentation/Development/Tools). On Linux or macOS, run the `node` command inside each batch file directly --- they are thin wrappers. `examples.bat` and `addin-test.bat` are the exceptions to both: they drive the twinBASIC IDE, so they are Windows-only and deliberately outside every gate and outside CI. Where to read more: diff --git a/WIP.ExamplesBuild.md b/WIP.ExamplesBuild.md index f42fac6b..56a0d932 100644 --- a/WIP.ExamplesBuild.md +++ b/WIP.ExamplesBuild.md @@ -397,11 +397,32 @@ Two things a batch runner must do that a single-fence runner need not: the page, the fence and the line within the page. Emitted while generating, not reconstructed afterwards. The offset differs per slot and the arithmetic must come out the same for all three, which is a probe. -- **Bisect on a compiler crash.** twinBASIC runs the compiler in-process with user code, so - a bad sample can take it down --- and in a batch that loses all hundred with it. On crash, - split and recurse: O(log n) extra builds, paid only on failure. Verified against the real - case, with the crashing sample isolated out of a batch and the rest of the batch still - reporting. +- **Isolate a compiler crash, starting from the sample it names.** twinBASIC runs the + compiler in-process with user code, so a bad sample can take it down --- and in a batch + that loses all hundred with it. `tbbuild`'s crash report names the file the compiler died + parsing, which for a sample is its generated module, so the unit holding it is built alone + and the rest of the batch without it: two builds, paid only on failure. With no sample + named, split and recurse: O(log n) extra builds. On a page of nine samples with the crash + fixture fifth, halving took 7 builds and 48 s and the named start 3 builds and 22 s, with + the same finding and the other eight still reporting. Until then the name went unread --- + `buildStaged` kept `tbbuild`'s report and nothing looked at it --- so every crash paid for + the whole bisect. +- **A crash that needs two samples used to vanish.** Halving separates any pair by the time + it reaches single samples; both halves then build clean, and every sample in the batch + counted as compiling --- the false clean that the crash check exists to prevent, one level + down. Now, when neither part of a crashing batch crashes on its own (the named sample and + the rest, or two halves), `together` searches for the samples the crash needs: holding one + part fixed, whichever half of the other still crashes with it holds them, and when neither + does, each half is searched with the other held. The members are blamed --- each has its + own result, and none is a pass --- and one finding names them all. **No real crash of that + shape is known**, so its tests are probes against a fake lane whose builds crash on the + sample sets a probe chooses; before the fix, the three probe shapes that need a set came + back with nothing crashed, nothing blamed and no finding. Four two-file candidates were + tried for a real one: inheritance cycles through interfaces, classes and UDTs, and the + fixture's placeholder name with a base declared in the other file. None crashes only as a + pair: the class and UDT cycles are diagnosed, the interface cycle is accepted without a + word, which is queued in [BUGS-TO-REPORT.md](BUGS-TO-REPORT.md), and the placeholder + crashes from its own file. ### A diagnostic that lands in a package's own source @@ -463,7 +484,8 @@ implementation. changing" as suspect on this compiler.** - **A two-line syntax skeleton crashes the compiler**, and it is in the corpus: `Interface Extends ` / `End Interface`, in - `Reference/Attributes.md`. Neither half crashes alone. Recorded in + `Reference/Attributes.md`. The placeholder name is what does it, given any `Extends` + clause: `Extends IBase` crashes it too. Recorded in [BUGS-TO-REPORT.md](BUGS-TO-REPORT.md); the classifier now refuses a `` as a declaration name, so it takes an explicit `slot=` to reach the compiler with one. - **`project.buildPath` must be an explicit file.** The default `${SourcePath}\Build\...` diff --git a/WIP.Harness.md b/WIP.Harness.md index 32e16adc..f09c8e77 100644 --- a/WIP.Harness.md +++ b/WIP.Harness.md @@ -1,17 +1,19 @@ # twinBASIC Documentation --- The Compiler Harness How the twinBASIC compiler is reached without a person driving the IDE: getting -at the package sources, censusing what they contain, compiling a probe, and -capturing what one prints. Split out of [WIP.md](WIP.md), which keeps the -invocations and the operational rules under [Driving the twinBASIC -compiler](WIP.md#driving-the-twinbasic-compiler) --- this file is why they are -what they are. +at the package sources, censusing what they contain, compiling a probe, +capturing what one prints, and testing an IDE add-in. Split out of +[WIP.md](WIP.md), which keeps the invocations and the operational rules under +[Driving the twinBASIC compiler](WIP.md#driving-the-twinbasic-compiler) --- this +file is why they are what they are. Read it before changing `scripts/tbbuild.mjs`, `scripts/tbrun.mjs`, -`scripts/lib/tb-ide.mjs`, `scripts/lib/tb-cdp.mjs`, `scripts/lib/tb-launch.ps1`, -`scripts/lib/tb-registry.mjs`, `scripts/lib/tb-ide-copy.mjs` or -`builder/census_attributes.mjs`, and before concluding anything about twinBASIC -syntax from a sweep of exported sources. +`scripts/addin_test.mjs`, `scripts/lib/tb-ide.mjs`, `scripts/lib/tb-cdp.mjs`, +`scripts/lib/tb-launch.ps1`, `scripts/lib/tb-registry.mjs`, +`scripts/lib/tb-ide-copy.mjs`, `scripts/lib/tb-project.mjs`, +`scripts/lib/tb-addin.mjs`, `scripts/lib/tb-operate.mjs`, `scripts/lib/tb-lane.mjs`, +anything under `test/addin/`, or `builder/census_attributes.mjs`, and before +concluding anything about twinBASIC syntax from a sweep of exported sources. ## Getting at the `.twin` sources @@ -184,7 +186,7 @@ handshake talks to the renderer, so it hangs on precisely the state you need to **The mechanics are one library, [scripts/lib/tb-ide.mjs](scripts/lib/tb-ide.mjs)**: starting the IDE, attaching, waiting for the compile, reading the diagnostics and the DEBUG -CONSOLE, clicking, and ending the process tree. `tbbuild` and `tbrun` are command lines +CONSOLE, clicking, building, and ending the process tree. `tbbuild` and `tbrun` are command lines around it, and the add-in harness planned in [WIP.HelpAddin.md](WIP.HelpAddin.md) is built on it. Moving the code there was checked against 14 fixture cases run before and after --- every exit code and every line of output the same, apart from the two fixes below --- and @@ -227,7 +229,7 @@ machine, and which is the same policy [BOOKPLAN.md](BOOKPLAN.md) records blockin --- never comes into it, and no `-ExecutionPolicy Bypass` has to be recommended to anyone. Its inputs arrive as environment variables, so there is no argument quoting to get wrong. -Six things about the harness were learned by getting them wrong, and each is a comment in +Seven things about the harness were learned by getting them wrong, and each is a comment in the file now: - **Pass the project on the IDE's command line, and spawn with an argv array.** @@ -263,9 +265,26 @@ the file now: OPERATIONAL with the counters at zero --- byte-identical to a clean build. A 275-file project reported `0 errors, 0 warnings` and exit 0, twice, reproducibly, while its quarters reported 129, 0, 294 and 448 errors. The console is the record that sampling cannot miss, - because nothing removes an entry from it: `NATIVE EXCEPTION`, then `restarting from - MEMORY`, then a thread dump naming the file being parsed, which is what the exit-4 message - reports. + because nothing removes an entry from it: `NATIVE EXCEPTION` and `restarting from + MEMORY`, three times over, with a thread dump naming the file being parsed from the second + crash on, and that file is what the exit-4 message reports. +- **Poll for the crashed file's name; the first crash never carries it.** Only a compiler in + TRACE-MODE writes the thread dump that names the file, and the IDE switches that on in + answer to the first `NATIVE EXCEPTION` and passes it to a compiler as it starts, so the + restarted compiler's crash is the first to name one. `waitForCompile` re-read the console + once, 2 s after it saw the crash, and on 2026-09-24 one run in six against the crash + fixture printed no file. Over 41 runs of that fixture on BETA 983, the second crash came + 1.6 to 1.9 s after the first on an idle machine and 1.8 to 3.0 s with four IDEs compiling + at once, as `check_examples` runs them. Replayed over the four-lane runs at every phase of + the 1 Hz sample, the single re-read missed the name 31% of the time; the poll that replaced + it, every 250 ms for up to 5 s, missed none and needed at most 3.25 s. That poll is + `awaitCrashName`, and the add-in lanes' `closeProject` had the same gap: closed right after + a first crash, three times, it now waited 1.9 s and named the file where its single read + had `crashed 1x` and no name. The failing run said + `crashed 2x`, which that race does not explain --- a re-read that loses it has seen one + crash --- and no run here had a second crash without a name. If one does, the third crash + names the file as well, but under four lanes it came as late as 6.4 s, after the poll has + given up. - **Kill the process tree, forcibly.** An IDE showing a modal ignores a normal close, and the launcher is not the process holding the compiler, so `taskkill /T /F`. A tree kill still misses a process started while it runs, which a compiler restart can be; the job the IDE @@ -337,6 +356,14 @@ distinct DevTools ports, WebView2 user-data folders and private desktops, so ins not collide. Three projects: **26 s sequentially, 10 s in parallel**, with each run reporting its own diagnostics and no bleed between them. +**A port another IDE holds is refused.** The harness attaches to whatever page answers on +its port, so an IDE already there --- another lane's, or another session's, since several +sessions run this harness on one machine with ports of their own choosing --- would be the +one read and operated. `launchIde` binds the port for a moment first, and gives up after +ten seconds with a message saying which port and why. The wait is for the lane's own +previous IDE: after `shutdownIde` a port came free in 13 and 16 ms, and once in two +seconds. + ### Capturing what a probe prints, not just whether it compiles `tbbuild` answers *does this compile*. [scripts/tbrun.mjs](scripts/tbrun.mjs) answers *what @@ -349,8 +376,10 @@ measure once there was a way to run code. It takes an **exported tree** rather than a `.twinproj`, stages a copy, pins the build path in the copy, packs it, compiles it with the same library calls `tbbuild` makes, clicks -Build, then reads the DEBUG CONSOLE back over CDP. The probe is a module with a `[RunAfterBuild]` Sub, which the IDE runs once -the exe is linked. Reader-facing documentation is the [`tbrun.mjs` entry in +Build, then reads the DEBUG CONSOLE back over CDP. The staging is +[scripts/lib/tb-project.mjs](scripts/lib/tb-project.mjs), which the add-in harness shares. +The probe is a module with a `[RunAfterBuild]` Sub, which the IDE runs once the exe is +linked. Reader-facing documentation is the [`tbrun.mjs` entry in Tools.md](docs/Documentation/Tools.md). **The trap that cost two silent runs, and the reason the script owns the tree.** A project @@ -395,6 +424,22 @@ after, not read on from an index**: new text can be appended to an entry that is Round 8's export probe missed the first `[EXPORT] exporting...` line of every session that way. `tbrun` re-reads the whole backing array on every poll, which is why it never did. +`readConsole`'s `since` does compare, and it is what `buildProject` and `openedUrls` read +with: the mark `consoleMark` takes holds the last entry as well as the count, and the text +appended to that entry comes back as the first line, before the entries after it. The +mechanism is in `ide/main.js`. Everything the compiler's process writes, a program's +`Debug.Print` and an add-in's `PrintText` alike, arrives as an output event and goes through +`debugOutputPartial`, which adds to the last entry in place (`updateItem`) while its line is +open. Output that ends in a line break closes the line, which is why each `PrintText` makes +an entry of its own, and so does `debugOutputLine`, the IDE's own messages, which starts a +new entry. Measured by calling both from the page: from a mark taken on an open line, the +count-only read missed the text appended to it, and the new read returned it first. And +measured for `PrintText`: with a line left open, Sample 10's printed line was appended to +that entry, and the new read returned it. **The +IDE escapes that continued text twice** ([BUGS-TO-REPORT.md](BUGS-TO-REPORT.md)), so a probe +printing `&`, `<` or `>` after a `Debug.Print ...;` reads them back as `&`, `<` and +`>`, which is also what the console shows. + It settles on a quiet period rather than a sentinel, so no probe has to print a marker the script knows about. Distinct `--port` values let probes run concurrently, exactly as `tbbuild`'s do. @@ -414,6 +459,73 @@ on an image allowlist, *and* windowless --- a new one that has a window is repor alone, since that cannot be told from a copy the user opened. `--no-reap` turns it off, and concurrent runs driving the same server should use it and sweep once at the end. +### Building for win64 + +**`tbrun` and `tbbuild` take `--arch win32|win64`, and set it on every run, win32 +included.** Before the option they built whatever target the IDE had for the project, which +for a fresh probe is win32, so no probe could measure 64-bit behaviour --- and a target the +IDE remembered for a reused path decided the build without a word. `tbbuild` needs it as much +as `tbrun`: `#If Win64` and `LongPtr`'s size change what compiles. Measured with a module +declaring a variable of an undeclared type under both branches of `#If Win64`: win32 +reported line 6's `OnlyUnder32Bit`, win64 line 4's `OnlyUnder64Bit`. + +**The target is the toolbar's build configuration box**, the page global +`buildConfigSelector`, whose options are `win32`, `win64` and `nocompile` --- safe mode, +which the IDE sets itself after four compiler crashes in a minute, with a message box. The +IDE's own `tbBuild_SwitchToWin64` and `tbBuild_SwitchToWin32` (Ctrl+F1, Ctrl+F2) set the box +and call its `onchange`, and `setBuildTarget` in `tb-ide.mjs` does the same. The handler, +`changedActiveBuildConfig` in `ide/main2.js`, saves the target for the project's path and +calls `restartCompilerSafely`, which kills the compiler: **every switch restarts it**, as the +target's own compiler, `twinBASIC_win64_noDEP.exe` for win64, and the project compiles +again. A project opening with a remembered target goes through the same switch before it +loads. + +**Wait for the restart by the compiler's pid, not by the clock.** Measured on BETA 983, at +50 ms: the status bar stayed OPERATIONAL for about 250 ms after the switch, read UNAVAILABLE +until the new compiler's pid appeared in `g_CurrentCompilerProcessId` at about 510 ms, and +LIMITED until OPERATIONAL at 1.4 s. `waitForCompile` started straight after the switch can +sample that second of downtime twice after having seen OPERATIONAL, and it counts that as +the compiler going down twice: exit 4, a crash that did not happen. A three-second pause +before it worked when this was first measured, and is a guess. `setBuildTarget` waits for +the pid to change, then for the compile. + +**A win64 probe runs as a 64-bit process.** `[RunAfterBuild]` code runs in the compiler that +built it, not in the binary: + +| | win32 | win64 | +|---|---|---| +| `LenB` of a `LongPtr` | 4 | 8 | +| `#If Win64` | False | True | +| `ProcessorArchitecture()` | 0, `vbArchWin32` | 1, `vbArchWin64` | +| `Environ$("PROCESSOR_ARCHITECTURE")` | x86 | AMD64 | +| `IsWow64Process` | 1 | 0 | +| module path of the process | `bin\twinBASIC_win32_noDEP.exe` | `bin\twinBASIC_win64_noDEP.exe` | +| PE machine of the built file | 0x14c | 0x8664 | + +The first three are decided when compiling; the last three can only come from the running +process, and they agree. + +**The build path's folder has to be explicit; its file name need not be.** `tbrun` used to +pin the output to `tbrun-probe.exe` whatever the build type or target. It now builds into +its own `out` folder under the IDE's own name, `${ProjectName}_${Architecture}.${FileExtension}`, +which gave `ArchProbe_win32.exe` and `ArchProbe_win64.exe` with no Save dialog --- so the +trap above comes from the default `${SourcePath}\Build\...`, not from the variables. The +name is looked for after the build rather than assumed, because `${FileExtension}` follows +the build type. + +**A switch writes the IDE's remembered target for the project's path**, so the registry tidy +has to put it back. Under a harness folder the entry is swept, as every entry there is. A +named project --- `tbbuild` on one of the user's own --- has its entry snapshotted and +restored, as its saved state is ([What a run leaves in the +registry](#what-a-run-leaves-in-the-registry-and-putting-it-back)). Measured end to end with +an entry of `win64` seeded for a named project: the default run said `target: win32 (the IDE +remembered win64 for this project)`, compiled for win32, and left the entry at `win64`; with +the seed removed, the whole value was byte-identical to before. Two limits. Under `--keep` +nothing is tidied, so a kept IDE's switch stays remembered. And the IDE's own save is an +unguarded read-modify-write of one JSON value (`setProjectLastUsedTargetArchitecture`), so two +IDEs switching at the same moment can lose one another's entry --- harmless for a harness +path, which the next sweep deletes anyway. + ## What a run leaves in the registry, and putting it back **Every IDE the harness starts writes to the user's own settings.** They live under @@ -426,7 +538,7 @@ were: the user's own recent projects had gone from the IDE entirely. One `exampl adds 41 values and fills the list. [scripts/lib/tb-registry.mjs](scripts/lib/tb-registry.mjs) puts it back. **The rule is to -leave everything as it was found**, and it takes three forms: +leave everything as it was found**, and it takes four forms: - **A folder only the harness writes to is swept by prefix** --- `check_examples`' and `tbrun`'s work folders. The sweep also runs at the start of a run, which catches what an @@ -440,10 +552,24 @@ leave everything as it was found**, and it takes three forms: - **The `.twinproj` association is restored value by value**, writing only what differs, so an untouched key is never written. The IDE does not rewrite it on every launch: key timestamps show `DefaultIcon` and `shell\open\command` last written when BETA 983 was - installed, through a day of launches of that build. That it rewrites them when the path - differs is inferred from that timing, not yet measured; a private copy of the IDE per - test lane ([WIP.HelpAddin.md](WIP.HelpAddin.md), Stage 1 item 2) is the first thing that - will test it. + installed, through a day of launches of that build. It rewrites them when its own path + differs, which was measured once there was a private copy of the IDE to start ([A private + IDE for every lane](#a-private-ide-for-every-lane)). +- **The build target the IDE remembers for each project is deleted under the same + folders**, before the run and after it. The IDE keeps the target it last built a project + for as one JSON object in `IDESettings\targetArchitectureMemory`, keyed by the project's + path, and a project it opens again starts in that target. A harness path is used run + after run, so one run's entry decides every later run's target without a word: on + 2026-09-24 the object held `win64` for `tbrun`'s work folders on ports 9372 and 9373, so + `tbrun` on either port built 64-bit. `sweepArchitectureMemory` deletes the entries under + the run's folders and leaves every other entry alone, the user's among them. Opening a + project only reads its entry; one is written when the target of an open project changes, + which `--arch` does ([Building for win64](#building-for-win64)). So a **named** project's + entry is snapshotted and put back as its saved state is, by + `restoreArchitectureMemory`: its old value in its old place, and any other spelling of its + path the IDE saved deleted. The object is edited in JavaScript and written back with + `JSON.stringify`, which is how the IDE writes it, so the other entries keep their exact + text and order, and the write is refused if the value changed after it was read. **One process owns the registry per run.** `check_examples` runs four lanes of `tbbuild` children at once; each restoring its own snapshot would put back whatever the registry held @@ -491,14 +617,59 @@ list a new installation has, and it cannot be fixed from outside the IDE. and the end-to-end check below ran its fixture cases one at a time for this reason. - An IDE the user has open reads the recent list when it starts and writes its whole copy back when it opens a project, so it can bring back harness entries that were tidied after - it started. + it started. So can an IDE of a run from another session, which is why a check of the + registry waits until no other session's run is going. + +**A third was closed: an association that named the temp folder is never put back.** A run +that starts while another run's IDE copy is open finds the association pointing at that +copy. Put back at the end, it would point `.twinproj` files at a folder that has been +deleted, and other sessions run `examples.bat` while add-in tests run copies, so the +overlap is ordinary. `startTidy` now notes whether the association it recorded names the +temp folder, and if it did, `finishTidy` leaves the association as the IDEs set it and says +so; the next IDE started from a real install points it at that install. A run still on +older code puts back what it found, so the guarantee holds once every checkout has it. + +**Observed on 2026-09-24: older code in another checkout did exactly that.** An add-in run +put the association back, and a comparison straight afterwards found it as it had been. A +minute later it named that run's IDE copy, deleted by then: another session's harness run, +on the code from before this rule, had recorded the association while the copy held it, and +put that back when it ended. The next add-in run found it naming the temp folder and, by the +rule, left it as its own IDEs set it, which was at its own copy, deleted in turn. It was put +back by hand. So until every checkout has the rule, an association can be left naming a +deleted copy, and `.twinproj` files then open nothing until an IDE is started from a real +install. + +**The recent list is put back as it was found, not only swept.** The sweep alone was exact +on an empty list, and the list was empty when the tidy was first verified. On a real one the +IDE changes the list by itself while a run's projects are on it, in two ways, both in +[BUGS-TO-REPORT.md](BUGS-TO-REPORT.md): it fills a short list's empty slots with copies of +its last entry, and a full list loses its oldest entry for every project the run opens. +Measured on 2026-09-24: an add-in run that began with one entry in the list ended with +seventeen copies of it. Its four IDEs had filled all 21 slots, and the sweep removed only +their four projects. So `snapshotProjects` records the whole list, and +`restoreProjects`, after deleting the run's entries and putting a named project back in its +place as before, keeps no more copies of any path than the list had, and puts the entries +that fell off the end back there. A change made for any other reason is kept: a project the +user opened meanwhile stays on top. **An entry in the temp folder is not brought back**, +because it belongs to some run, whose own tidy may have removed it meanwhile; bringing it +back would leave that run's entry in the list for good. That is also why a run that starts +with another run's leftovers in the list can end without them. **Verified** by [scripts/check_tb_registry.mjs](scripts/check_tb_registry.mjs), which plays out a run on a scratch key and checks every rule above, the guards and the ownership rule -included. It is not a gate: it needs Windows and a real registry, and the CI runners are -Ubuntu. Run it after changing `tb-registry.mjs`. End to end, the 14 fixture cases, run one -at a time, and a full `examples.bat` run leave `ProjectState`, the recent list and the -association keys exactly as they were, value for value. +included. For the recent list it has five cases: the copies of a short list's last entry, a +full list's lost entries, a project opened meanwhile, copies that were there before the run, +and another run's entry that its tidy removed meanwhile. It is not a gate: it needs Windows +and a real registry, and the CI runners are Ubuntu. Run it after changing `tb-registry.mjs`. + +End to end, the 14 fixture cases, run one at a time, and a full `examples.bat` run leave +`ProjectState`, the recent list and the association keys exactly as they were, value for +value, though the recent list was empty when that was first shown. With the user's own +projects planted in it, two of them and then 21, an add-in run left it identical both times. +The add-in checks in [Building an add-in and loading it](#building-an-add-in-and-loading-it) +left `IDESettings` unchanged too, compared value by value through hashes, so that the +comparison copied no value out of the registry; the build targets they planted under their +own folders were gone afterwards. ## A private IDE for every lane @@ -541,6 +712,16 @@ file, so a wrong path cannot empty a folder anybody cares about. Deleting one th still holds fails with `EPERM`, and that is the right report: it is how the leak in the next section was found. +**`removeTree` retries the delete itself, because `rmSync` does not.** An IDE ended a +moment ago still holds some of its files for a while, and `removeIdeCopy` always passed +`rmSync` a `maxRetries` of 10 for that. On Node 24.13 that option does nothing here: +`rmSync` failed with `EPERM` within a millisecond on a folder holding a file another +process had open, with `maxRetries` and `retryDelay` exactly as without them (measured). +It never showed while every caller ran a registry tidy, a second or more of PowerShell, +between ending the IDE and deleting the copy. The lane code deletes the copy the moment the +IDE has gone, and the first delete failed. `removeTree` in `tb-ide-copy.mjs` retries for up +to five seconds, and `removeIdeCopy` and the add-in runner both use it. + **`loadedAddins(c)`** in `tb-ide.mjs` is the check that the copy is what it claims to be. It asks the page's `root.getAddinsList`, which asks the compiler over its root socket (`RequestAddinsStateList`), so the answer is the compiler's own, not an inference from files @@ -594,3 +775,288 @@ cases gave output identical to before the job, and a full `examples.bat` passed 1,116 with no process left afterwards. `--show` still starts the IDE directly, without a launcher or a job: it is for a person watching, and it has not been moved onto the launcher because that would mean putting an untested window on somebody's screen. + +## Building an add-in and loading it + +**An add-in is tested with two IDEs of the lane's copy, one after the other.** The compiler +loads add-ins only as it starts, and an IDE holds one project, so: + +1. `buildAddin` in [scripts/lib/tb-addin.mjs](scripts/lib/tb-addin.mjs) stages the add-in's + exported tree through [scripts/lib/tb-project.mjs](scripts/lib/tb-project.mjs) --- the + staging `tbrun` does, moved there to be shared --- with the build path pinned to + `\out\.dll`. It starts the copy on it, refuses a project with compile + errors (exit code 1, every diagnostic listed), builds, and ends the IDE. +2. `addAddin` in `tb-ide-copy.mjs` puts the DLL in the copy's `addins\win32`. It refuses any + folder that is not a marked copy, so a test add-in cannot reach the real install. +3. The copy starts again, on the project the test opens, and `loadedAddins` asks its + compiler what it loaded. + +Both shipped add-in samples went through it. Sample 10 and Sample 15 each built in about +nine seconds, IDE start included, and the next IDE reported `WaynesWorld AddIn` and +`GlobalSearchAddIn AddIn`, with Sample 10's five `OnProjectLoaded` lines in its DEBUG +CONSOLE and its two toolbar buttons on the page. + +**The DLL is built into the work folder, not into `addins` as the samples' own build path +has it**, because the IDE that builds is the lane's copy too: on a rebuild its compiler +would hold the previous build, loaded from that folder, while the linker tried to replace +it. **A compiler holds every add-in it loaded** (P8 in WIP.HelpAddin.md): while the IDE +runs, overwriting the file fails with `EBUSY` and deleting it with `EPERM`, though renaming +it works. The hold also outlasts the process: with every process of the IDE gone, the first +overwrite still failed and one 25 ms later worked, four runs out of four. So `addAddin` +retries for two seconds. Its copy fails with `EIO` rather than `EBUSY`, and a retry that did +not listen for `EIO` failed three runs out of three. + +**Whether the build worked is read from the build log.** `buildProject` in `tb-ide.mjs` +clicks Build, as `tbrun` does, and waits for the DEBUG CONSOLE. The wording is in the +compiler's strings: `[BUILD] Starting...`, then for a binary either `[LINKER] SUCCESS created +output file ''` or one of about twenty failure lines --- `[LINKER] FAILED ...`, +`[BUILD] FAILED ...`, `[BUILD] ERROR ...`, `[BUILD] failed`, `[LINKER] compilation (codegen) +error ...`. An output file another process held open gave `[LINKER] FAILED to create output +file '...' (error code 32)`, then `LOCKED BY:` and a line naming the process, then `[BUILD] +failed`. Two details: + +- **Only lines added after the click count.** Nothing removes a console entry but a clear. + So the entries from the pre-click count on are new, unless the first entry changed or the + count fell: that means a clear, and then everything is new. A previous build's SUCCESS + line can never be taken for this build's. Text the IDE appends to the entry that was last + at the click counts as new too, since the IDE adds to a line that is still open in place + (`readConsole`'s `since`, under `tbrun` above). +- **A failure line waits two seconds for a success line after it.** The strings include + `[BUILD] failed to use project.iconForm setting`, and whether a build goes on after that + one has not been seen. + +What a package build writes has not been looked at; `buildProject` knows binaries only. + +**win32 only, and the target is checked rather than assumed.** A project path the IDE has +no memory of opens in the first target on its list, win32, and gets +`twinBASIC_win32_noDEP.exe`. `buildAddin` reads the target the IDE chose and refuses any +other, and the [registry tidy](#what-a-run-leaves-in-the-registry-and-putting-it-back) +deletes any target remembered for the lane's paths. **The target decides which folder is +read** (part of P7): a copy holding Sample 10 as `InFolder_win32.dll` in `addins\win32` and +`InFolder_win64.dll` in `addins\win64` loaded the first alone when it opened a project with +no memory. When it opened a project remembered as win64, it started +`twinBASIC_win64_noDEP.exe` with `twinBASIC_nativedbg_win64.exe`, which tried the second +alone and failed, since the add-in is 32-bit. + +**A DLL that fails to load still appears in `loadedAddins`, as `Unknown Addin`.** The DEBUG +CONSOLE says why, on a line that starts with the file name: `[InFolder_win64.dll] Failed to +load addin. LoadLibrary() failed.` So a test looks for the name it expects rather than +counting, and reads the console for the reason when that name is missing. + +**Verified** by a scratch driver, beyond the two samples. An add-in with an undeclared name +failed with exit code 1 and both of its diagnostics. An output file held open by another +process failed with exit code 2 and the linker's line. A lane whose add-in path was planted +in the registry as win64 still built win32, because the tidy deleted the entry first. The 14 +fixture cases gave output identical to before, `tbrun`'s three included. A full +`examples.bat` passed 1,117 of 1,117, and the registry was identical afterwards, +`IDESettings` included. + +## Operating the IDE and reading it + +**A scenario is written with [scripts/lib/tb-operate.mjs](scripts/lib/tb-operate.mjs)**: +click, press keys, type, read the add-ins' tool windows, message boxes, notifications and +list views, open a file, and move or read the code editor's cursor. `readCrash` in +`tb-ide.mjs` says whether the compiler crashed, from the same console record `tbbuild` +reads, and `awaitCrashName` waits for that record to name the file being parsed, which no +first crash does. Every call takes a connection from `attachIde`. Both of Stage 1's acceptance +scenarios were carried out with these calls alone, on a lab IDE with Samples 10 and 15 +built in; [WIP.HelpAddin.md](WIP.HelpAddin.md), Stage 1 item 5, has what they did. + +**Input is real input; reading is from the page's data.** A click is the pointer moving to +the element's centre, pressing and releasing, and a key press is the key-down and key-up a +keyboard sends, because the IDE's own controls ignore `element.click()` and an add-in's +shortcut is matched on the real pair of key events. Reading is the other way round: a list +view draws only the rows that fit, and a tool window is a shadow root that +`document.querySelector` cannot see into, so the calls read `toolWindowsById`, a list +view's `dataNodes` and `window.editor` rather than what is drawn. + +Six things about it were learned on the samples: + +- **A click scrolls its target into view, and checks what is at the point before it + clicks.** Sample 10's tool window is taller than it is shown. Its eleventh button had a + size and a place, but the place was under the window's bottom edge, and the first click + went to the window's resize handle and did nothing. `click` now calls `scrollIntoView`, + finds the element at the centre point through every shadow root, and throws, naming both, + when something else is there. It also throws when there is no such element, or the + element has no size, which is what a hidden tool window's elements have. +- **A click waits for its target, up to five seconds.** What an add-in adds is in the page's + data a moment before it is drawn. The first run of the Sample 15 scenario waited until + the results list held both files' results, read from the list view's data, and clicked a + match under a millisecond later: the list had not drawn the row yet, and the click failed + with "there is no such element". Done by hand, a pause had always come between the two. So + `click` now tries again every 100 ms until the target is there, has a size and is not + covered, and only then throws, with the last reason. +- **Of the elements a selector finds, the target is the first one on screen.** Sample 15's + results list held one file's entry twice in the page while its rendered text had it once: + a list view keeps rows it has drawn before, and a kept row is not on screen. A target can + also be narrowed by its exact text, and can take the last match rather than the first, + for a dialog stacked on another. +- **Which element carries the handler decides what a click does.** Sample 15 puts each + match's `[line,col]` label beside the clickable line, not inside it, so a click on the + label runs the handler of the file's whole entry and opens the file's first match. The + harness was right and the target was wrong; clicking the line opened `Haystack.twin` at + line 4, column 13, as the result said. +- **Typing is one key press per character.** Sample 15 searches on key-up, once typing + pauses for a second, so text put into its box any other way would never be searched. + `pressKey` sends a US keyboard's `key`, `code` and virtual key code, since the IDE names + an add-in shortcut's letters from `code` and every other key from `key`. Modifiers go down + before the key and come up after it, in reverse. Measured in the code editor: End moved + to the end of the line, Shift+End selected to it, Ctrl+A selected all 152 characters, and + a typed `x` followed by Backspace left the text as it was. +- **The editor is `window.editor`**, one Monaco editor given the model of the selected + tab, and `openEditors.selectedEditorNode.name` is that tab's file, + `//Sources/`. Opening a file the way the IDE's Find in Files does --- + `fs.tree.resolvePath("twinbasic:" + path)`, then `openEditors.openFile(node, false, + false, false, line, column)` --- put the cursor at the line and column given, counted + from 1. + +**The connection itself changed in three ways.** They were the gaps item 1 found in +`tbbuild`, and they matter more once a harness clicks into dialogs on purpose: + +- **Every CDP call has a time limit**, thirty seconds unless a call passes its own, and a + connection that closes fails its waiting calls at once. Before, a page blocked by a dialog + or a synchronous host call hung the caller for good. Measured: on a page an `alert()` was + blocking, a call with a three-second limit failed after 3.0 s, with a message naming the + likely causes. +- **`attachIde` records and dismisses every javascript dialog**, in `c.dialogs`, after + sending `Page.enable`. `tbbuild` has always listened for dialogs, but nothing sent + `Page.enable`, and without it CDP reports none, so its list could never fill. Measured: + an `alert()` opened from the page was recorded with its type and text, and dismissed, and + the page answered again. The IDE only ever calls `alert()`, from 37 places, so every + dialog is accepted; a `confirm()` or `prompt()`, which only an add-in could open, would be + cancelled. +- **An alert that opened before the connection existed cannot be answered.** Measured: + `Page.enable` got no answer while it was open, the connection was told of no dialog, and + `Page.handleJavaScriptDialog` replied "No dialog is showing" while the page stayed + blocked. `attachIde` marks such a page (`c.pageBlocked`), and a compile that then never + reports its project open says why, instead of looking like a slow compile. The IDE's + candidates are its "IDE startup failure" alert and "Bad command line syntax.", which + `launchIde`'s single argument never provokes; `--show` puts either where it can be read. + +**Verified:** after the connection changes, the port check and the association rule, the +14 fixture cases gave output identical to before, and a full `examples.bat` passed 1,119 of +1,119 and left nothing in the registry under its folder, with another session's harness +runs going at the same time. + +## An add-in under test opens nothing + +**Every IDE the harness starts has `TB_ADDIN_TEST=1` in its environment**, set by +`launchIde` (`ADDIN_TEST_ENV` in [tb-ide.mjs](scripts/lib/tb-ide.mjs)). An add-in that sees +it does nothing outside the IDE and prints each such action to the DEBUG CONSOLE instead: +`open ` for a URL it would have opened in a browser. A browser started from an IDE on +the private desktop would open where nobody can see it, and outlive the run. `openedUrls` in +[tb-operate.mjs](scripts/lib/tb-operate.mjs) reads those lines back, and with a mark from +`consoleMark` only the ones printed after it; `readConsole` takes the same mark as `since`, +and `buildProject` now reads its build log that way. + +**Every IDE, not only the add-in runner's.** `tbbuild`, `tbrun` and `examples.bat` start the +real install, whose compiler loads whatever add-ins the user has put in its `addins` +folders, and none of those IDEs is on a desktop anybody watches. A caller can still set the +variable otherwise, or leave it out by passing `undefined` as its value: Node leaves such a +variable out of a child's environment even when its own environment has it (measured). + +**Measured on BETA 983 (P10 in WIP.HelpAddin.md):** a probe add-in printed the variable +from `Host_OnProjectLoaded`. Through `launchIde` it read `1`, from `Environ$` and from +`GetEnvironmentVariableW` alike, and with the variable left out both said it was unset. The +path it travels: `tb-launch.ps1` calls `CreateProcess` with no environment block of its own, +so the IDE inherits the launcher's; the IDE starts the compiler, `twinBASIC_win32_noDEP.exe`, +as a direct child; and the add-in runs inside the compiler's process --- the process id it +read was the compiler's. The toolbar's restart button ends the compiler and starts a new +process, and the add-in that process loaded read `1` too. The control, +`WEBVIEW2_USER_DATA_FOLDER`, arrived with the lane's port in it. + +**The console gives back exactly what was printed, a whole line at a time.** `PrintText` +stores an add-in's text escaped, `` as `<b>` and `&` as `&`, and `readConsole` +decodes it, so a URL with `&` in its query string comes back unchanged. Text that continues +a line left open is the exception: the IDE escapes it twice (under `tbrun` above). Each +`PrintText` ends its own line, so it is affected only when something else, such as a +program's `Debug.Print ...;`, left a line open just before it. `openedUrls` counts a line only when what +follows `open ` holds no white space: a URL has none, so an ordinary line that happens to +start with the word is not taken for one. A probe that printed `open this line names no URL` +beside a real one got the real one alone. + +## The add-in test runner + +**`addin-test.bat` runs [scripts/addin_test.mjs](scripts/addin_test.mjs) over the lanes in +[test/addin/lanes.mjs](test/addin/lanes.mjs).** A lane is one scenario file, a `node:test` +file, and the runner starts each in a process of its own (`node --test`), a few at a time +(`--jobs`, default 2), handing it its lane in `TB_ADDIN_LANE`: a DevTools port (`--port`, +default 9560, plus the lane's index), a work folder at `%TEMP%\tbaddin\`, and the +install to copy. What the scenario does with that is +[scripts/lib/tb-lane.mjs](scripts/lib/tb-lane.mjs): it makes the lane's copy of the install +on first use, exports an install sample project (`addSample("Sample 15")`, the install only +read), builds an add-in and puts it in the copy (`addAddin`), and opens a project (`open`), +one IDE at a time, refusing one that does not compile. `close` ends the IDE and then fails +the lane if the compiler crashed meanwhile, or if a javascript dialog opened that the +scenario did not take out of `c.dialogs`, since the IDE opens one only on an error path; +then it deletes the copy. Run outside the runner, a scenario file skips itself, so a bare +`node --test` never starts an IDE. + +**A process per lane, not `node:test`'s own concurrency.** `node --test` can run files in +parallel, but it cannot hand each file an environment of its own, and the runner has to +choose which lanes may run together (below). A lane's output is held until it ends and then +printed whole, so two lanes' reports never interleave. `--timeout` (default 600 s) ends a +lane still running; its process's job takes the IDE with it. + +**The runner owns the registry, and checks it afterwards.** It calls `startTidy` with every +lane's folder before the first lane starts, so the lanes inherit `TB_REGISTRY_OWNER` and +leave the registry alone, and `finishTidy` once the last has ended. Then it checks rather +than trusts: no project-state or recent-list entry may name a lane's folder, a second sweep +of the remembered build targets must find none, and the add-ins' settings must be as +recorded. Any failure is exit code 2. + +**An add-in's own settings are the runner's too.** `SaveSetting` writes under +`HKCU\Software\VB and VBA Program Settings\`, the same key as any installed copy of the +add-in, so a lane names its add-ins' application names in `lanes.mjs` (`settings`). The +runner records those keys before the first lane starts, deletes them before each lane that +names them, so that the add-ins start from their defaults, and puts them back at the end. +Two lanes that name the same application never run at once, because each deletes the key +its add-in reads. An application key that appears during the run and that no lane named is +reported, since it is almost certainly an add-in whose settings will stay behind; the +report names it and leaves it. `settingsKey` refuses the IDE's own `twinBASIC_IDE`, which +holds all of the IDE's settings and which the tidy puts back only value by value. + +**It refuses to start while `%APPDATA%\twinBASIC\addins` holds a DLL**, until P6 says +whether the compiler loads from there: the page hands it that folder with +`RequestLoadAddins`. It never writes there. + +**Ctrl+C ends the lanes and still puts everything back.** The runner handles `SIGINT`: it +starts no more lanes, ends the running ones, waits, and tidies. Testing that took two tries. +Windows passes a process's "ignore Ctrl+C" setting on to the processes it starts, and every +process started from the session that ran the test had it, so the first `CTRL_C_EVENT`, +sent with `GenerateConsoleCtrlEvent` into the runner's own hidden console, reached nothing; +a bare Node listener started the same way did not see it either. Started through a +PowerShell that first cleared the setting (`SetConsoleCtrlHandler(NULL, FALSE)`), the +listener saw it, and so did the runner: 16 s in, with both lanes' IDEs open, it ended both +lanes and put everything back within two seconds. + +**The two scenarios**, the ones that finish Stage 1: + +- [test/addin/sample10.test.mjs](test/addin/sample10.test.mjs): the add-in loads and prints + its five `OnProjectLoaded` lines, naming the project; its image button's message box; its + tool window; the three-button message box answered `button2`, then the follow-up + answered `ok`; a notification; a DEBUG CONSOLE line, read from a mark. +- [test/addin/sample15.test.mjs](test/addin/sample15.test.mjs): the add-in loads; its tool + window opens with every option off, which also shows the runner deleted its saved + settings; a typed search lists all nine matches in both files of + [test/addin/host](test/addin/host), each with its `[line,column]`; a click on a match + opens `Haystack.twin` at 4:13; and Match case narrows the results to seven and is saved, + read back through `savedSettings`. + +**Measured on BETA 983:** + +- Both lanes pass, in about 25 s together: each is an add-in build of about 10 s, a host IDE + of about 9 s, and 2 s of scenario. +- Around a run, the whole registry comparison was identical: `ProjectState`, the recent + list, the association keys, all 13 `IDESettings` values compared through hashes, and the + remembered build targets. The run was repeated with the user's projects planted in the + recent list, two of them and then 21, identical both times. +- With Global Search settings planted beforehand (Match case on, and one extra value), the + lane began with every option off, and the key came back exactly, the extra value + included. With `settings` taken out of `lanes.mjs`, the run failed with exit code 2 and + named `GlobalSearchAddIn`. +- A DLL under the add-ins folder of `%APPDATA%` (a stand-in, with `APPDATA` pointed at a + scratch folder) and an `--only` that matches nothing were both refused with exit code 2. + `--timeout 8` ended both lanes mid-build, and left the registry identical and nothing + running. +- Exporting the samples left the install's `projects` folder as it was, mtimes included. diff --git a/WIP.HelpAddin.md b/WIP.HelpAddin.md index bad3a2cc..882f7104 100644 --- a/WIP.HelpAddin.md +++ b/WIP.HelpAddin.md @@ -4,7 +4,9 @@ See [WIP.md](WIP.md) for the maintenance guide. This file covers the planned twi add-in that shows the documentation for the symbol under the cursor, and the harness that tests IDE add-ins by machine, which the add-in is developed against. -**Status: planning. Nothing is built.** This file replaces the June draft, `add-in/PLAN.md` +**Status: Stage 1, the harness, is built** --- items 1 to 7 are done, and `addin-test.bat` +operates Samples 10 and 15 end to end and leaves the registry as it found it. The add-in +itself is not started; Stage 2's probes come next. This file replaces the June draft, `add-in/PLAN.md` in commit `d159acf8` ("Roughly plan the help add-in"). That commit is on no branch --- only the detached HEAD of an old worktree keeps it --- so everything in it worth keeping is here, corrected, and nothing depends on it surviving. [What changed from the June @@ -57,14 +59,33 @@ the compiler what a symbol is.** Each of those gaps shapes a stage below. install's IDE reports `GlobalSearchAddIn AddIn`, and a copy of it with empty `addins` folders reports none. - **The page creates `%APPDATA%\twinBASIC\addins\win32` and `...\win64`** at startup - (`CreateCommonFolders`, `main.js@961019`) *(reported)*. Whether the compiler also loads - from there is **P6**. If it does, a DLL placed there loads into every IDE the user starts. + (`CreateCommonFolders`, `main.js@961019`) *(reported)*, and hands the folder above them + to the compiler: `RequestStartCompiler` and `RequestLoadAddins` both send + `commonFolderRootPath`, the resolved `%APPDATA%\twinBASIC` (`main.js@1047705` and + `@1048667`). That makes it likely that the compiler loads add-ins from there too, and it + is still **P6**. If it does, a DLL placed there loads into every IDE the user starts. +- **An add-in runs inside the compiler's process.** Measured (P10): the process id an + add-in read with `GetCurrentProcessId` was that of `twinBASIC_win32_noDEP.exe`, which + `twinBASIC.exe` starts as a direct child, beside the page server + `twinBASIC_win32.exe --ide=`. So an add-in sees the environment the IDE was started + with. **A compiler restart is a new process**: the toolbar's restart button + (`#restartIcon`, bound to `tbCompiler_Restart`, which calls `root.forceTerminate()`) ended + the compiler, and the next one, with a new process id, loaded the add-in again and ran its + `OnProjectLoaded` a second time. The DEBUG CONSOLE was not cleared; the IDE added + `restarting from MEMORY []`. - **Load failures have their own messages** in the compiler's strings: `Failed to load addin. LoadLibrary() failed.`, `Entry point not found. Addin may have been compiled for a newer version of the twinBASIC IDE.`, `Entry point 'tbCreateCompilerAddin' call - failed.` and `...returned an object that does not implement interface IAddInV1.` Where - they are written is untested; the DEBUG CONSOLE is the likely place, and it is where a - harness would look. + failed.` and `...returned an object that does not implement interface IAddInV1.` **They + go to the DEBUG CONSOLE**, after the file's name in brackets --- measured with a 32-bit + add-in in `addins\win64`: `[InFolder_win64.dll] Failed to load addin. LoadLibrary() + failed.` **An add-in that failed to load is still in the compiler's list**, as `Unknown + Addin`, so a test looks for the name it expects rather than counting. +- **Holding Shift while a project opens skips the add-ins.** The page sends + `RequestLoadAddins` only when `shiftKeyDown` is false, and otherwise writes `[IDE] SHIFT + KEY DETECTED: DISABLED LOADING OF ADDINS` to the DEBUG CONSOLE (`main.js@1048443`). + `shiftKeyDown` follows the keymap's `tbMisc_ShiftKeyStateDown` and `...Up` actions, so a + test that presses Shift must not do it while a project is opening. - **The loader also looks for `tbCreateCompilerAddin_v2`**, and the linker knows a `tbCreateCompilerAddin_v3`. The tbIDE package declares neither, and what they take is unknown (**P14**). @@ -72,9 +93,19 @@ the compiler what a symbol is.** Each of those gaps shapes a stage below. and every item calls `notSupportedMenuOption()` *(reported)*. Restarting the compiler removes every add-in's UI and shortcuts (`removeAddinAlterations`) *(reported)*; whether it also reloads the DLLs from disk is **P9**. -- **Which `addins` folder is used follows the compiler's bitness**, and the bitness - probably follows the build target (Ctrl+F1 / Ctrl+F2), which the IDE remembers in the - shared registry as `targetArchitectureMemory` (**P7**). A shipped add-in needs both builds. +- **The build target picks the compiler, and the compiler picks the folder.** The IDE + remembers the target of each project in the shared registry, as one JSON object in + `IDESettings\targetArchitectureMemory` keyed by project path, and opens a project in the + target remembered for it --- or, with none, in the first on its list, win32. Measured with + a differently named DLL in each folder: a project with no memory got + `twinBASIC_win32_noDEP.exe`, which loaded `addins\win32` alone; a project remembered as + win64 got `twinBASIC_win64_noDEP.exe` with `twinBASIC_nativedbg_win64.exe`, which tried + `addins\win64` alone. Switching the target of an open project (Ctrl+F1 / Ctrl+F2) restarts + the compiler in the other bitness: `changedActiveBuildConfig` in `ide/main2.js` records the + new target and kills the compiler, and the one that replaced a `twinBASIC_win32_noDEP.exe` + on a switch to win64 was a `twinBASIC_win64_noDEP.exe` (measured for `--arch`, + [WIP.Harness.md](WIP.Harness.md#building-for-win64)). Which `addins` folder that one loads + was not looked at, and is the rest of **P7**. A shipped add-in needs both builds. ### Keyboard shortcuts @@ -112,8 +143,16 @@ Read at `main.js@608242`, `@610953` and `@611152`. Read at `main.js@1002292` (`toolWindowElementAddChild`) and `@1005960` (`toolWindowElementSetProperty`). -- **A tool window is part of the main document**, inside an open shadow root, not an iframe - *(reported)*. A harness reaches its content through `toolWindowsById[].bodyElement`. +- **A tool window is part of the main document**, inside an open shadow root, not an iframe. + Measured on Samples 10 and 15: `toolWindowsById` is keyed by the *second* argument the + add-in gave `ToolWindows.Add` (`"GlobalSearchAddInData"`, `"WaynesWindowData"`), its + `bodyElement` is in the shadow root, and the root's host is `#toolWindow` + (`#toolWindow900`). A window an add-in created and has not shown is there already, and + every element in it has no size. A window's content can be taller than the window: Sample + 10's eleventh button had a size and a place, but its place was under the window's bottom + edge, where a click lands on the resize handle. +- **An add-in's toolbar button is `#addinButton-`**, with the id the add-in gave + `AddButton`, inside `#rootMenu2`, and its caption as its `title`. - **`HtmlElements.Add(id, tagName)` accepts any tag.** The four IDE widget tags (`chartjs`, `monaco`, `listview`, `virtuallistview`) become a `div` with extra setup; every other name goes straight to `document.createElement`, so `iframe` is not refused. A parent is found @@ -125,7 +164,13 @@ Read at `main.js@1002292` (`toolWindowElementAddChild`) and `@1005960` - An inline handler inside `innerHTML` (``, `
`) is an attribute, not a property, so it is not dropped, and browsers run such handlers in the page's own JavaScript. That is the one way an add-in can call the IDE page's internals - (**P4**). See [Open decisions](#open-decisions) for where it may be used. + (**P4**). See [Open decisions](#open-decisions) for where it may be used. **Sample 15 + already does it:** each search result it gives its list view's `addItem` is HTML with an + inline `onclick='raiseEvent("onClickMatch", event, true, path, line, column)'`, and a + click on one runs it --- measured, since the click opened the right file at the right + line. The event travels only from the element that carries the handler: Sample 15's + `[line,col]` label sits beside the clickable line rather than inside it, so a click on the + label reaches the handler of the whole file's entry, which opens the file's first match. - Events: a known DOM event gets a real `addEventListener`, and a copy of the event goes back to the add-in over the compiler's root socket; an unknown name becomes a callback for `raiseEvent(...)` to call *(reported)*. `raiseEvent` finds its handler by climbing @@ -179,16 +224,23 @@ from the page's own origin (**P13**). Deferred. ### Dialogs -- `Host.ShowMessageBox` and `Host.ShowNotification` are drawn in the page --- a - `.modalDialogContainer` with `.msgBoxButton` buttons, and the fixed boxes `#msgBox1` to - `#msgBox3` --- not as native dialogs *(reported)*. A harness can read them and click them. -- **`main.js` calls `alert()` at 33 sites** *(reported)*. An `alert()` blocks the renderer, - so the harness must record and dismiss every one (`Page.javascriptDialogOpening`, then - `Page.handleJavaScriptDialog`). The ones an add-in test could reach: the rename provider +- `Host.ShowMessageBox` and `Host.ShowNotification` are drawn in the page, not as native + dialogs, and a harness reads them and clicks them (measured on Sample 10). A message box + is a `.modalDialogContainer` holding a `.modalTitleBar` (the title as a text node, then a + close button), a `.simpleMsgBox` with the message and a `.msgBoxButton` per button; the + add-in's call returns once one is clicked. A notification's text is the `.msgBoxText` of + one of the fixed boxes `#msgBox1` to `#msgBox3`. +- **The IDE calls `alert()` at 37 sites**, 33 in `main.js` and 4 in `main2.js`, and never + `confirm()` or `prompt()`. An `alert()` blocks the renderer, so the harness records and + dismisses every one (`Page.javascriptDialogOpening`, then `Page.handleJavaScriptDialog`), + which `attachIde` now does. The ones an add-in test could reach: the rename provider (`alert("need to massage workspace edits here...")`), Find with an invalid regular expression, and an unknown message on any of the compiler's sockets. A notification's "Copy to clipboard" link also calls `alert()`, but plain `ShowNotification` messages hide - that link. + that link. **An alert that opened before the harness attached cannot be dismissed over + CDP** (measured): the page then answers nothing, and the harness reports it as the likely + cause. The IDE's own candidates are its "IDE startup failure" alert and "Bad command line + syntax.", which `launchIde`'s single argument never provokes. ### IDE state outside the install @@ -219,7 +271,9 @@ from the page's own origin (**P13**). Deferred. compile session wrote nothing to it. - **`SaveSetting` from an add-in writes to the same tree**, under `VB and VBA Program Settings\`, so it is shared with any installed copy of the - same add-in. A test that changes an add-in-wide option changes it for the user too. + same add-in. A test that changes an add-in-wide option changes it for the user too, which + is why the add-in runner records and puts back every application a lane names (Stage 1, + item 7). - WebView2 profiles: `%LOCALAPPDATA%\twinBASIC\v0` for the IDE --- `tbbuild` replaces it per port with `WEBVIEW2_USER_DATA_FOLDER` --- and `%LOCALAPPDATA%\twinBASIC_WebPanel\v0` for the WEBPAGE panel *(reported)*. @@ -294,36 +348,93 @@ Everything after this stage is developed against it. [scripts/lib/tb-registry.mjs](scripts/lib/tb-registry.mjs), described in [WIP.Harness.md, What a run leaves in the registry](WIP.Harness.md#what-a-run-leaves-in-the-registry-and-putting-it-back). The 14 fixture cases, run one at a time, and a full `examples.bat` run leave the - registry identical, value for value, with every output line unchanged. The last two - bullets wait for the add-in runner (item 7). `snapshotKeys` takes any key, so the add-in's - `SaveSetting` key is one more entry in its list. The work also found an IDE bug, now in - [BUGS-TO-REPORT.md](BUGS-TO-REPORT.md): a recent list shorter than 21 entries gets its - empty slots filled with copies of the last entry. + registry identical, value for value, with every output line unchanged. The work also + found an IDE bug, now in [BUGS-TO-REPORT.md](BUGS-TO-REPORT.md): a recent list shorter + than 21 entries gets its empty slots filled with copies of the last entry. Item 4 added a + fourth thing to put back, the build target the IDE remembers for each project path, + because a lane that inherits `win64` builds and loads the wrong bitness. + + **The last two bullets are done in the runner (item 7).** It records the `SaveSetting` + keys a lane names and puts them back, and refuses to start while + `%APPDATA%\twinBASIC\addins` holds a DLL. **Item 7 also corrected the recent list.** The + sweep was exact only on an empty list, which is what the list was when it was verified: a + run that began with one entry ended with seventeen copies of it, because of the bug above, + and a full list loses its oldest entry for every project a run opens. The tidy now + records the whole list and puts it back as found ([WIP.Harness.md, What a run leaves in + the registry](WIP.Harness.md#what-a-run-leaves-in-the-registry-and-putting-it-back)). 4. **Build, then load.** The add-in is a Standard DLL. In a staged copy of its tree, pin `project.buildPath` to an explicit file in the lane IDE's `addins\\`, the way `tbrun` pins its exe path: the default `${SourcePath}\Build\...` template has already cost a run with an invisible Save dialog, and whether the samples' `${IdePath}` template behaves any better is untested. Build, end that IDE, then start the same lane IDE on a test project; its compiler loads the add-in as it starts. One project per IDE, as always. + + **Done, building into the work folder instead:** `buildAddin` in + [scripts/lib/tb-addin.mjs](scripts/lib/tb-addin.mjs) builds with the lane's copy into + `\out\`, and `addAddin` in `tb-ide-copy.mjs` then puts the DLL in the copy's + `addins\win32`. Built straight into `addins`, a rebuild would meet the previous build + loaded by the very IDE doing the building, and a loaded add-in cannot be overwritten + (P8). [WIP.Harness.md, Building an add-in and loading it](WIP.Harness.md#building-an-add-in-and-loading-it) + has the rest: how the build log is read, why only win32 for now, and what was measured. + Samples 10 and 15 both built and loaded, and the tree staging that `tbrun` did is now + [scripts/lib/tb-project.mjs](scripts/lib/tb-project.mjs), shared by both. 5. **Operating the IDE and reading it**, as library calls over CDP: - open a file: `fs.tree.resolvePath("twinbasic://Sources/")`, then - `openEditors.openFile(node,false,false,false,line,col)` *(reported)*; - - move the cursor or select: `window.editor` is the one Monaco code editor - (`setPosition`, `setSelection`, `getModel().getValue()`). Not - `monaco.editor.getEditors()`, which also returns editors that add-ins created - *(reported)*; + `openEditors.openFile(node,false,false,false,line,col)`, line and column counted from 1 + (measured); + - move the cursor or select: `window.editor` is the one Monaco code editor, given the + model of whichever file's tab is selected (`setPosition`, `setSelection`, + `getModel().getValue()`; measured). Not `monaco.editor.getEditors()`, which also + returns editors that add-ins created *(reported)*; - press keys: `Input.dispatchKeyEvent` key-down, then key-up, with real `key` and `code` values, less than 500 ms apart; - click: real `Input.dispatchMouseEvent` presses at the element's centre. The IDE's own controls ignore `element.click()` --- `tbrun` learned that on `#buildIcon`; - - ask which add-ins loaded: `loadedAddins(c)` in `tb-ide.mjs` (done for item 2); + - ask which add-ins loaded: `loadedAddins(c)` in `tb-ide.mjs` (done for item 2), which + lists a DLL that failed to load as `Unknown Addin`; + - build the open project: `buildProject(c)` in `tb-ide.mjs` (done for item 4); - read a tool window through `toolWindowsById[].bodyElement`; read the DEBUG CONSOLE's backing array, notifications and message boxes; dismiss any `alert()`; notice a compiler restart or crash, as `tbbuild`'s console check already does. + + **Done:** [scripts/lib/tb-operate.mjs](scripts/lib/tb-operate.mjs), with `readCrash` in + `tb-ide.mjs`, described in [WIP.Harness.md, Operating the IDE and reading + it](WIP.Harness.md#operating-the-ide-and-reading-it). Both acceptance scenarios were + carried out with it by hand on a lab IDE: **Sample 15** --- the toolbar button, the + search typed key by key, both files' results, and a click on one match that opened + `Haystack.twin` at line 4, column 13 --- and **Sample 10** --- its tool window, the + three-button message box answered `button2`, the follow-up answered `ok`, a notification + and a DEBUG CONSOLE line. The two gaps item 1 found are closed: every CDP call has a time + limit, and the connection records and dismisses dialogs, proved with an `alert()`; an + alert already open before the harness attached is the one case it cannot handle, and it + says so. Two dangers were closed on the way: `launchIde` refuses a DevTools port another + IDE holds, since the harness would otherwise operate that IDE, and the registry tidy no + longer puts back an association that pointed into the temp folder. The runner (item 7) + turns the two scenarios into tests. 6. **No real side effects.** The add-in's URL opener checks an environment variable (name to be chosen) and, when it is set, prints `open ` to the DEBUG CONSOLE instead of starting a browser. On a private desktop a real browser would start where nobody can see it and outlive the run. **P10** checks that the variable reaches the compiler process. + + **Done:** the variable is **`TB_ADDIN_TEST`**, and P10 answered yes (Stage 2 has the + measurement). An add-in treats it as set when it is not empty. `launchIde` in + [tb-ide.mjs](scripts/lib/tb-ide.mjs) sets it to `1` for **every** IDE the harness starts, + `tbbuild`'s, `tbrun`'s and `examples.bat`'s included, because each of them loads whatever + add-ins the user has installed, on a desktop nobody watches; a caller's `env` can set it + otherwise, or leave it out with the value `undefined`. `openedUrls(c, { since })` in + [tb-operate.mjs](scripts/lib/tb-operate.mjs) reads the `open ` lines back, and + `consoleMark(c)` in `tb-ide.mjs` takes the mark that `since` names, so a scenario asks what + was opened after the key it pressed. A line counts only when what follows `open ` has no + white space in it, as a URL has none, so an ordinary line that starts with the word is not + read as one. `PrintText` stores its text escaped (`` as `<b>`), so a URL comes + back exactly as printed, `&` included. + + **The probe stayed in scratch.** It was thirty lines: `Host_OnProjectLoaded` printing + `Environ$("TB_ADDIN_TEST")`, the same through `GetEnvironmentVariableW`, and its own process + id. What it measured matters only while the add-in depends on it, and the add-in checks it + itself from Stage 4 on (increment 1 below). `add-in/` holds the add-in's tree and nothing + else: `stageProject` copies the whole folder it is given and packs the copy, so a probe + kept inside it would be packed into the add-in's project. 7. **A runner.** Scenarios in JavaScript under `node:test`, one IDE per test project, lanes by `--port` as today. The add-in's pure twinBASIC logic --- word extraction, lookup --- is tested without loading any add-in: a test project holds those modules and a @@ -331,6 +442,22 @@ Everything after this stage is developed against it. the JavaScript side checks the lines. The wrapper is `addin-test.bat`, outside every gate and CI for the reason `examples.bat` is: it needs Windows and a twinBASIC install. + **Done:** [scripts/addin_test.mjs](scripts/addin_test.mjs), with the lanes in + [test/addin/](test/addin/) and what a scenario gets in + [scripts/lib/tb-lane.mjs](scripts/lib/tb-lane.mjs), described in [WIP.Harness.md, The + add-in test runner](WIP.Harness.md#the-add-in-test-runner). A lane is one scenario file, + run in a process of its own with its own port and copy of the install; the runner owns + the registry, the add-ins' saved settings included, and checks it afterwards. Ctrl+C and + a lane timeout both end the lanes and still put the registry back. The pure-logic tests + wait for Stage 4, which writes that logic. They belong in a lane too, built and run in + the lane's own copy rather than by `tbrun`: a `tbrun` started under the runner leaves its + registry entries to the runner, which sweeps only the lanes' folders. + + Two library changes came out of it: `click` waits up to five seconds for its target, + since a list view draws a row a moment after the row is in its data, and `removeTree` + retries a delete that an ending IDE still blocks, since on Node 24 `rmSync`'s own + `maxRetries` does not. + **Done when** the harness operates two shipped samples end to end, and the user's registry is unchanged afterwards: @@ -338,6 +465,10 @@ is unchanged afterwards: - **Sample 15:** type a search, see the results, click one, and the right file opens at the right line. +**Met on 2026-09-24, BETA 983:** both scenarios pass under `addin-test.bat`, and a +comparison of the whole registry around the run, `IDESettings` included through hashes, +was identical. + ### Stage 2: probes that decide the design Most probes are a small add-in plus a scenario. P5, P11 and P13 need only CDP and the file @@ -349,13 +480,13 @@ the build number it was measured on. | P1 | Do `{ctrl}` and `{alt}` add-in shortcuts ever fire? Register `{ctrl}{shift}d`, `{alt}f`, `{shift}d`, `d` and `f1`, and press each. | the bug report; which key the add-in uses; the NOTE on the KeyboardShortcuts page | | P2 | Does the add-in's `f1` fire with focus in the code editor, and what happens with signature help showing? | F1 or another key | | P3 | Does an `iframe` of a documentation page load and navigate inside a tool window? Size, scrolling, theme. | how pages are shown | -| P4 | Does `innerHTML` render, and do inline handlers in it run page script? | how summaries are drawn; whether the page-internals route exists | +| P4 | Does `innerHTML` render, and do inline handlers in it run page script? **Half answered, BETA 983:** HTML an add-in gives a list view's `addItem` renders, and its inline `onclick` runs the page's `raiseEvent` (Sample 15). `innerHTML` set as a property is untested. | how summaries are drawn; whether the page-internals route exists | | P5 | What does hover return for `MsgBox`, `Collection.Add`, `ToolWindows.Add` and a symbol declared in the project? What does definition return for a package symbol? | compiler-assisted context, or the add-in's own parser | | P6 | Does the compiler also load add-ins from `%APPDATA%\twinBASIC\addins\`? This needs a DLL placed there for a moment, and the user's own IDE would load it too --- **ask before running it.** | harness isolation | -| P7 | Which bitness does the compiler start in, and does switching the build target restart it in the other one and load the other `addins` folder? | building and testing both bitnesses | -| P8 | Is a loaded add-in DLL locked against being overwritten? | the rebuild loop | -| P9 | Does a compiler restart reload add-ins from disk? | a rebuild loop without restarting the IDE | -| P10 | Does an environment variable set by the harness reach the add-in (`Environ$`)? | the side-effect switch | +| P7 | Which bitness does the compiler start in, and does switching the build target restart it in the other one and load the other `addins` folder? **Half answered, BETA 983:** the target a project opens in picks the compiler --- win32 when the IDE remembers none, `twinBASIC_win64_noDEP.exe` for a project remembered as win64 --- and each compiler reads its own `addins` folder alone. Switching the target of an open project restarts the compiler in the other bitness --- `twinBASIC_win32_noDEP.exe` was replaced by `twinBASIC_win64_noDEP.exe` --- and which folder that one loads is untested. | building and testing both bitnesses | +| P8 | Is a loaded add-in DLL locked against being overwritten? **Answered, BETA 983: yes.** While its IDE runs, overwriting fails (`EBUSY`) and deleting fails (`EPERM`), though renaming works; the hold outlasts the compiler's exit by a few tens of milliseconds. | the rebuild loop --- the DLL is built outside `addins`, and copied in once the IDE has ended | +| P9 | Does a compiler restart reload add-ins from disk? **Half answered, BETA 983:** a restart ends the compiler and starts a new process, which loads every add-in again as it starts, so from disk. The loop itself is untested: rename the loaded DLL aside (P8 allows that), copy the new build in, restart. | a rebuild loop without restarting the IDE | +| P10 | Does an environment variable set by the harness reach the add-in (`Environ$`)? **Answered, BETA 983: yes**, through the launcher, the IDE and the compiler the IDE starts. With `TB_ADDIN_TEST=1` in `launchIde`'s environment, `Environ$` and `GetEnvironmentVariableW` both returned `1` in the add-in, and a compiler started by the restart button returned it too; left out, both said it was unset. `WEBVIEW2_USER_DATA_FOLDER`, which `launchIde` always sets, arrived with the lane's port in it. | the side-effect switch | | P11 | Does the IDE write into its own install folder during a session? **Answered, BETA 983: no.** A compile, a compiler crash and a `tbrun` build-and-run left all 233 files byte-identical, mtimes included. | hardlinks or copies --- copies, for safety, at 380 ms | | P12 | Does `raiseEvent` from plain tool-window HTML throw? | how the pane's events are written | | P13 | Does the compiler's HTTP server serve any file placed under `ide\`? | an offline route | @@ -430,6 +561,13 @@ Each increment is finished with its scenarios. 1. **F1 to a page.** A toolbar button; the key (from P1 and P2); the name under the cursor; index lookup; open the page in the browser or the pane. A miss says `No help for ''` through `ShowNotification`. + + **The URL opener honours the test switch.** It calls `ShellExecuteW`, except while + `Environ$("TB_ADDIN_TEST")` is not empty: then it prints `open ` to the DEBUG CONSOLE + and starts nothing (Stage 1, item 6). When the add-in loads it prints whether the switch + is on, and every scenario checks that line before it presses anything. An IDE build that + stopped passing the variable on to the compiler then fails the run, instead of starting a + browser on the private desktop. 2. **The help pane.** Search over the index, results, and a page view --- an iframe if P3 passes, otherwise a summary with a link to the browser. Theme: read `Host.Themes.ActiveThemeNameGroup` at start, handle `Host_OnChangedTheme` after, and pass @@ -522,12 +660,10 @@ Recommended, and not yet confirmed: - **Its code skeletons are not kept.** They were never compiled; Stage 4 writes the modules against the compiler, with tests. -## Rules for once Stage 1 exists - -Move these into WIP.md when the harness is built, because they will then bind every session: +## Rules -- **Never build or copy a test add-in into the real install's `addins\`, or into - `%APPDATA%\twinBASIC\addins\`.** Either way it loads into the user's own IDE. -- **A test never opens a real browser.** -- Kill by pid, never by image name, and one project per IDE --- both already rules in - [WIP.md](WIP.md#driving-the-twinbasic-compiler). +Stage 1 is built, so the rules for testing add-ins bind every session and are in +[WIP.md, Driving the twinBASIC compiler](WIP.md#driving-the-twinbasic-compiler): no test +add-in in the real install's `addins\` or in `%APPDATA%\twinBASIC\addins\`, no real browser +from a test, every `SaveSetting` application named in `lanes.mjs`, IDEs ended by pid, and +one project per IDE. diff --git a/WIP.md b/WIP.md index 3861ec4c..d18f1020 100644 --- a/WIP.md +++ b/WIP.md @@ -14,7 +14,7 @@ change. |---|---| | write or edit any page under `docs/` | [WIP.Authoring.md](WIP.Authoring.md) --- page template, frontmatter, cross-section linking tables, per-symbol workflow | | document a specific package | that package's own file, listed under [Package API notes](#package-api-notes) | -| run or change the twinBASIC compiler harness | [WIP.Harness.md](WIP.Harness.md) --- `export`, the attribute census, `tbbuild`, `tbrun` | +| run or change the twinBASIC compiler harness | [WIP.Harness.md](WIP.Harness.md) --- `export`, the attribute census, `tbbuild`, `tbrun`, the add-in test runner | | change `builder/`, `scripts/`, or any gate | [WIP.Build.md](WIP.Build.md) --- the pipeline and every gate's failure history | | touch fonts, diagrams, or the PDF's type | [WIP.Typography.md](WIP.Typography.md), then [WIP.Fonts.md](WIP.Fonts.md) for the generator | | change the accessibility scan | [WIP.A11y.md](WIP.A11y.md) --- the axe scan, the sample, the fingerprint gate | @@ -142,11 +142,27 @@ node scripts/tbrun.mjs # what does it print - **Keep a probe that might crash the compiler in a project of its own.** twinBASIC runs the compiler in the same process as user code, so one bad probe can take the run down and cost the other thirty their answer. - **`tbrun` takes an exported tree, not a `.twinproj`**, because it has to pin `project.buildPath` in its own staged copy --- a project still on the default template opens a native Save dialog that is invisible on the private desktop, and the build simply never happens while every health check says the IDE is fine. The probe is a module with a `[RunAfterBuild]` Sub, and must start with `Debug.Cls`. - **A census is evidence, not applicability.** The corpus not using an attribute somewhere does not mean the compiler refuses it there, and the reverse also holds. Only a probe settles that. +- **End an IDE by its pid, never by image name.** `taskkill /IM twinBASIC.exe` ends every other run's IDE, another session's included, and the user's own. `tbbuild --keep` prints the pid for this reason. Why each of those is true, what the WebView/CDP route costs, why the compiler's own websockets cannot be driven instead, and the seven ways a sweep of this corpus returns a wrong answer: [WIP.Harness.md](WIP.Harness.md). +**Testing an IDE add-in** is `addin-test.bat`, run by a person as `examples.bat` is. Each +lane in `test/addin/lanes.mjs` builds the add-ins it tests into a private copy of the +install and operates an IDE; the plan it serves is [WIP.HelpAddin.md](WIP.HelpAddin.md), and +how it works is [WIP.Harness.md, The add-in test +runner](WIP.Harness.md#the-add-in-test-runner). + +```sh +addin-test.bat # every lane +addin-test.bat --only sample15 # one lane; --port N moves the lanes' ports +``` + +- **Never build or copy a test add-in into the real install's `addins\`, or into `%APPDATA%\twinBASIC\addins\`.** Either way it loads into the user's own IDE. A test add-in goes only into a lane's copy of the install; `addAddin` refuses anywhere else, and the runner refuses to start while the `%APPDATA%` folder holds a DLL. +- **A test never opens a real browser.** Every IDE the harness starts has `TB_ADDIN_TEST=1`, and an add-in under test prints `open ` to the DEBUG CONSOLE instead. Never start a test IDE with the variable removed unless its add-in opens nothing either way. +- **Name in `lanes.mjs` every application an add-in under test passes to `SaveSetting`**, or its settings stay changed after the run: `SaveSetting` writes the key the user's own copy of the add-in reads. + ## Authoring a page **[WIP.Authoring.md](WIP.Authoring.md) is required reading before writing or @@ -440,6 +456,7 @@ Why the report separates the wedged task from the merely blocked ones, and why - `book.bat` — renders the PDF from `docs\_site-pdf\book.html` via `node book\render-book.mjs` into `docs\_pdf\twinBASIC Book.pdf`. Run `build.bat` first to populate `_site-pdf/`; `book.bat` refuses a tree older than its sources rather than rendering the previous book (see [The book refuses a stale source tree](WIP.Build.md#the-book-refuses-a-stale-source-tree)). - `examples.bat` — compiles the documentation's own twinBASIC code samples, every `tb` fence marked `check_build`, and reports the ones the compiler refuses against the line in the page they came from. Needs a twinBASIC install and Windows, so it is outside every gate and outside CI; ~110 s over the 1,119 samples marked today. Two modes need no compiler at all: `--census` classifies every fence and says how many classifiable ones are still unmarked, and `--report ` groups a saved `--propose --json` survey by diagnostic, section and unresolved name. `--propose` itself does compile. See [Compiling the reference's own code samples](#compiling-the-references-own-code-samples) and [WIP.ExamplesBuild.md](WIP.ExamplesBuild.md). +- `addin-test.bat` — tests IDE add-ins by operating an IDE: every lane in `test/addin/lanes.mjs` builds the add-ins it tests into a private copy of the install, opens a project and checks what the add-in does. Outside every gate and outside CI for the same reasons as `examples.bat`; ~25 s for the two lanes today, Samples 10 and 15. Exit 0 every lane passed and the registry is as it was found, 1 a lane failed, 2 the harness failed or could not put the registry back. See [Driving the twinBASIC compiler](#driving-the-twinbasic-compiler) for its rules. Two generators sit outside that loop and produce committed artifacts rather than build output — neither runs during a build, and neither is needed for one. `python scripts/build_fonts.py` rebuilds the subset webfaces under `docs/assets/fonts/` and needs a network connection; `node scripts/build_dot_metrics.mjs` regenerates `builder/inter-metrics.json` from those webfaces and needs only a browser. See [Typography](#typography). diff --git a/addin-test.bat b/addin-test.bat new file mode 100644 index 00000000..6afcf4db --- /dev/null +++ b/addin-test.bat @@ -0,0 +1,30 @@ +@echo off +@pushd "%~dp0" +rem Test twinBASIC IDE add-ins by machine: the scenarios under test\addin. +rem +rem Each lane in test\addin\lanes.mjs is a node:test file with its own copy of +rem the twinBASIC install, into which it builds the add-ins it tests, and its +rem own DevTools port. It opens a project, operates the IDE the way a person +rem does, and reads what the add-in did. The IDE's registry entries and the +rem add-ins' own saved settings are put back as they were found at the end. +rem WIP.HelpAddin.md is the plan this serves, and WIP.Harness.md says how. +rem +rem THIS IS NOT ONE OF THE GATES, for the reasons examples.bat is not: it +rem needs a twinBASIC install and Windows, with a private desktop and a +rem CDP-reachable WebView2, and neither is on the CI box. Keep it out of +rem build.bat, check.bat, test.bat and both CI workflows. +rem +rem Arguments are passed straight through: +rem +rem addin-test.bat --only sample15 just that lane +rem addin-test.bat --port 9600 lanes on ports 9600, 9601, ... +rem addin-test.bat --jobs 1 one lane at a time +rem +rem Exit: 0 every lane passed and the registry is as it was found, 1 a lane +rem failed, 2 the harness failed or could not put the registry back. +node scripts/addin_test.mjs %* +@rem popd resets ERRORLEVEL, so capture it first -- otherwise a failing lane +@rem would report success to whatever called addin-test.bat. +@set "ADDIN_TEST_ERR=%ERRORLEVEL%" +@popd +@exit /b %ADDIN_TEST_ERR% diff --git a/docs/Documentation/Tools.md b/docs/Documentation/Tools.md index b07d062c..8a6ca65b 100644 --- a/docs/Documentation/Tools.md +++ b/docs/Documentation/Tools.md @@ -8,14 +8,14 @@ permalink: /Documentation/Development/Tools # Tools and Scripts {: .no_toc } -One-line-per-tool reference for every executable in the documentation repository: the five Windows batch wrappers at the repository root, the Node and Python scripts under `scripts/` (cross-platform except for [`tbbuild.mjs`](#tbbuild), which drives the twinBASIC IDE), the `tbdocs` orchestrator and its CLI flags, [`census_attributes.mjs`](#census-attributes) under `builder/`, and the PDF render driver. If you are looking for the day-to-day workflow rather than a cheat sheet, the [Building and Deployment](Building) page is the gentler read; if you are modifying the build pipeline itself, the [tbdocs Internals](Builder) page goes one level deeper. +One-line-per-tool reference for every executable in the documentation repository: the seven Windows batch wrappers at the repository root, the Node and Python scripts under `scripts/` (cross-platform except for [`tbbuild.mjs`](#tbbuild), which drives the twinBASIC IDE), the `tbdocs` orchestrator and its CLI flags, [`census_attributes.mjs`](#census-attributes) under `builder/`, and the PDF render driver. If you are looking for the day-to-day workflow rather than a cheat sheet, the [Building and Deployment](Building) page is the gentler read; if you are modifying the build pipeline itself, the [tbdocs Internals](Builder) page goes one level deeper. * TOC goes here {:toc} ## Batch wrappers at the repository root {: #batch-wrappers } -All six sit at the repository root, beside `package.json` --- not under `docs/`. Each uses `@pushd "%~dp0"` to run from that root regardless of where it is invoked from, and each entry below gives the POSIX equivalent of what it runs. Those equivalents have no `pushd` in front of them, so **run them from the repository root** --- `tbdocs`'s `--src docs`, [`check_publish_policy.mjs`](#check-publish-policy)'s default source root, and every path handed to [`render-book.mjs`](#bookrender-bookmjs) are all resolved against the working directory. `examples.bat` is the exception to "each entry below gives the POSIX equivalent": it needs a twinBASIC install and drives the IDE, so it is Windows-only, and it is not part of the site build. Three other tools are Windows-specific for the same reason and are likewise not part of it: [`scripts/tbbuild.mjs`](#tbbuild) and [`scripts/tbrun.mjs`](#tbrun), which drive the twinBASIC IDE, and [`census_attributes.mjs`](#census-attributes), which runs the twinBASIC compiler's `export` verb --- though that one is cross-platform when given an already-exported tree with `--src`. Nothing else in the repository is: `tbdocs` and every gate in both wrappers is a Node script, and CI runs all of them on `ubuntu-latest` except [`check_tree_fresh.mjs`](#check-tree-fresh), which guards against a failure mode CI cannot have. +All seven sit at the repository root, beside `package.json` --- not under `docs/`. Each uses `@pushd "%~dp0"` to run from that root regardless of where it is invoked from, and each entry below gives the POSIX equivalent of what it runs. Those equivalents have no `pushd` in front of them, so **run them from the repository root** --- `tbdocs`'s `--src docs`, [`check_publish_policy.mjs`](#check-publish-policy)'s default source root, and every path handed to [`render-book.mjs`](#bookrender-bookmjs) are all resolved against the working directory. `examples.bat` and `addin-test.bat` are the exceptions to "each entry below gives the POSIX equivalent": they need a twinBASIC install and drive the IDE, so they are Windows-only, and neither is part of the site build. Three other tools are Windows-specific for the same reason and are likewise not part of it: [`scripts/tbbuild.mjs`](#tbbuild) and [`scripts/tbrun.mjs`](#tbrun), which drive the twinBASIC IDE, and [`census_attributes.mjs`](#census-attributes), which runs the twinBASIC compiler's `export` verb --- though that one is cross-platform when given an already-exported tree with `--src`. Nothing else in the repository is: `tbdocs` and every gate in both wrappers is a Node script, and CI runs all of them on `ubuntu-latest` except [`check_tree_fresh.mjs`](#check-tree-fresh), which guards against a failure mode CI cannot have. ### build.bat @@ -152,6 +152,21 @@ Compiles the documentation's own twinBASIC code samples --- every ` ```tb ` fenc Exit codes: **0** clean, **1** a sample does not compile, **2** the harness failed. +### addin-test.bat +{: #addin-testbat } + + addin-test.bat [flags] + +One invocation of [`addin_test.mjs`](#addin-test), with every flag passed straight through: + + node scripts/addin_test.mjs [flags] + +Tests twinBASIC IDE add-ins by machine: it builds each add-in under test, loads it into an IDE, operates the IDE the way a person would, and checks what the add-in did. + +**It is not one of the gates either**, and for the reasons `examples.bat` is not: it needs a twinBASIC install, and it needs Windows, a private desktop and a CDP-reachable WebView2. It is absent from `build.bat`, `check.bat`, `test.bat` and both CI workflows. + +Exit codes: **0** every lane passed and the registry is as it was found, **1** a lane failed, **2** the harness failed or could not put the registry back. + ## CLI tools ### tbdocs --- node builder/tbdocs.mjs @@ -506,7 +521,8 @@ Normalises literal en-dash / em-dash characters in markdown source under `docs/` {: #tbbuild } node scripts/tbbuild.mjs [--ide ] [--port N] - [--timeout S] [--json] [--keep] [--show|--hide] + [--arch win32|win64] [--timeout S] [--json] [--keep] + [--show|--hide] Compiles a `.twinproj` and prints its diagnostics, with no IDE window to click through. This is how a claim the documentation makes about the language gets checked against the compiler rather than against memory: write a one-module project that uses the construct in the position you are asking about, run this, and read what comes back. Windows only, and no part of the site build. @@ -515,9 +531,10 @@ twinBASIC has no command-line build. The compiler executable's whole surface is | Flag | Effect | |---|---| | `--ide ` | Path to `twinBASIC.exe`. Default: `$TB_IDE`, else the newest `twinBASIC_IDE_BETA_` folder on `%USERPROFILE%\Desktop`, which is where the IDE's own zip says to unpack it. **No install path is hardcoded anywhere in this tooling** --- an install path contains a username --- so an install kept elsewhere needs one of those two. | -| `--port ` | DevTools port. Default 9333. It also names the WebView2 user-data folder and the private desktop, which is what makes concurrent instances possible. | +| `--port ` | DevTools port. Default 9333. It also names the WebView2 user-data folder and the private desktop, which is what makes concurrent instances possible. A port another IDE already holds --- another run's, or another session's --- is refused after ten seconds, rather than attached to. | +| `--arch ` | The target to compile for, `win32` or `win64`. Default `win32`. The diagnostics can differ between the two, because `#If Win64` and the size of `LongPtr` change what compiles. The target is set on every run, because the IDE opens a project in whatever target it last used for that project. Switching restarts the compiler, which then compiles the project again, so a switch adds a few seconds. When the target is not `win32`, or the IDE remembered another one for the project, the report starts with a `target:` line. | | `--timeout ` | Give up waiting for the compile to settle. Default 180. | -| `--json` | Emit one JSON object --- counts, diagnostic rows, and any dialog text --- instead of lines of text. | +| `--json` | Emit one JSON object --- the target, counts, diagnostic rows, and the text of any alert the IDE opened, which is dismissed so the compile can go on --- instead of lines of text. | | `--keep` | Leave the IDE running afterwards. The IDE's registry entries for the project are then left as they are, because the IDE is still writing them. | | `--show` / `--hide` | Put the IDE on your own desktop where you can watch it, or on a private one where it cannot take focus. Hidden is the default unless `TBBUILD_SHOW` is set to something other than `0`, `false` or `no`; the two flags override that for one invocation. | @@ -529,15 +546,15 @@ Exit codes: **0** clean, **1** the project has errors, **2** the harness failed, **The IDE it starts ends with it.** The IDE runs inside a Windows job object, so when `tbbuild` ends --- finished, failed, or stopped with Ctrl+C --- every process the IDE started ends too. That includes a compiler the IDE was restarting after a crash, which a plain process-tree kill can miss and leave running. Two exceptions: under `--keep` the IDE runs outside the job and lives until you close it, and under `--show` it is started directly on your desktop, without the job. -**It leaves the IDE's own settings as it found them.** Every IDE it starts writes to the same registry keys as your own IDE: a saved state for the project (open tabs, watch expressions, Debug Console history) and a place at the top of the recent-projects list. Once the IDE has exited, `tbbuild` puts both back. An entry the run created is deleted, and a project that already had one --- one of your own --- gets its old state and its old place in the list back. The `.twinproj` file association is restored too, if the IDE changed it. When [`check_examples.mjs`](#check-examples) runs `tbbuild`, `check_examples` does this once for all its lanes instead. +**It leaves the IDE's own settings as it found them.** Every IDE it starts writes to the same registry keys as your own IDE: a saved state for the project (open tabs, watch expressions, Debug Console history), a place at the top of the recent-projects list, and, when the run switches the target, the target the IDE remembers for the project. Once the IDE has exited, `tbbuild` puts all three back. An entry the run created is deleted, and a project that already had one --- one of your own --- gets its old state, its old place in the list and its old target back. The `.twinproj` file association is restored too, if the IDE changed it. When [`check_examples.mjs`](#check-examples) runs `tbbuild`, `check_examples` does this once for all its lanes instead. -Four files under `scripts/lib/` belong to it and are never run directly. `tb-ide.mjs` holds the mechanics `tbbuild.mjs` and `tbrun.mjs` share: starting the IDE, attaching to it, waiting for the compile, and reading the diagnostics and the DEBUG CONSOLE. `tb-registry.mjs` records and restores the registry entries described above, through .NET's registry API by way of PowerShell, because `reg.exe` mangles any path holding a character outside the console code page; [`check_tb_registry.mjs`](#check-tb-registry) is its self-test. `tb-cdp.mjs` is a minimal CDP client over Node's global `WebSocket`, raw rather than puppeteer because a pending `alert()` blocks the renderer and puppeteer's `connect()` handshake talks to the renderer --- so it hangs on precisely the state you need to recover from. `tb-launch.ps1` holds the Win32 calls Node cannot make without a native FFI addon: `CreateDesktop` and `CreateProcess` with `STARTUPINFO.lpDesktop` for the private desktop, and the job object described above. It is the only PowerShell file under `scripts/`, and it is not executed as a file: `tb-ide.mjs` reads the text and passes it through `-EncodedCommand`, so the execution policy never comes into it and nobody has to be told to bypass one. +Four files under `scripts/lib/` belong to it and are never run directly. `tb-ide.mjs` holds the mechanics `tbbuild.mjs` and `tbrun.mjs` share: starting the IDE, attaching to it, waiting for the compile, and reading the diagnostics and the DEBUG CONSOLE. `tb-registry.mjs` records and restores the registry entries described above, through .NET's registry API by way of PowerShell, because `reg.exe` mangles any path holding a character outside the console code page; [`check_tb_registry.mjs`](#check-tb-registry) is its self-test. `tb-cdp.mjs` is a minimal CDP client over Node's global `WebSocket`, raw rather than puppeteer because a pending `alert()` blocks the renderer and puppeteer's `connect()` handshake talks to the renderer --- so it hangs on precisely the state you need to recover from. Every call it makes has a time limit, so a blocked page ends a run with a message rather than holding it forever. `tb-launch.ps1` holds the Win32 calls Node cannot make without a native FFI addon: `CreateDesktop` and `CreateProcess` with `STARTUPINFO.lpDesktop` for the private desktop, and the job object described above. It is the only PowerShell file under `scripts/`, and it is not executed as a file: `tb-ide.mjs` reads the text and passes it through `-EncodedCommand`, so the execution policy never comes into it and nobody has to be told to bypass one. ### tbrun.mjs {: #tbrun } - node scripts/tbrun.mjs [--port N] [--timeout S] [--quiet MS] - [--json] [--raw] [--keep] [--no-reap] + node scripts/tbrun.mjs [--port N] [--arch win32|win64] [--timeout S] + [--quiet MS] [--json] [--raw] [--keep] [--no-reap] [--reap-images a,b] [--show|--hide] Builds a probe project and captures what it writes to the IDE's @@ -548,7 +565,7 @@ measured with it. It takes an **exported source tree** (the folder holding `Sources/` and `Settings`), not a `.twinproj`, because it has to adjust the project before packing it. It stages a copy and -leaves your tree untouched. +leaves your tree untouched. The staging is in `scripts/lib/tb-project.mjs`. The probe is an ordinary module with a [`[RunAfterBuild]`](../../tB/Core/Attributes#runafterbuild) Sub, which the IDE runs once the exe is linked: @@ -580,13 +597,27 @@ list view holding only the rows that fit --- reading that instead returns the la lines of a long probe and looks no different from a full capture. `Debug.Cls` is what empties the array, which is the other reason to begin with it. +**A `win64` probe runs in the IDE's 64-bit compiler.** A `[RunAfterBuild]` Sub runs inside +the compiler that built it, not in the file that was built. For `win64` that compiler is +`twinBASIC_win64_noDEP.exe`, a 64-bit process, so the probe sees what 64-bit code sees: +`LenB` of a `LongPtr` is 8, [**ProcessorArchitecture**](../../tB/Modules/Compilation/ProcessorArchitecture) +returns **vbArchWin64**, and `Environ$("PROCESSOR_ARCHITECTURE")` is `AMD64`. Under `win32` +they are 4, **vbArchWin32** and `x86`. + +**Print a line whole when its characters matter.** Text that continues a line left open by +`Debug.Print ...;` comes back escaped: after `Debug.Print "A";`, a following +`Debug.Print "&"` shows in the Debug Console as `A&`, and `tbrun` captures what the +console shows. The IDE does this, not the probe; `Debug.Print "A"; "&"`, in one statement, +comes back as `A&`. + | Flag | Effect | |---|---| -| `--port ` | DevTools port for the IDE. Default 9346. Distinct ports let probes run concurrently --- the staging directory and the project id are keyed to it, so two runs never share a workspace. | +| `--port ` | DevTools port for the IDE. Default 9346. Distinct ports let probes run concurrently --- the staging directory and the project id are keyed to it, so two runs never share a workspace. A port another IDE holds is refused, as for `tbbuild`. | +| `--arch ` | The target to build for, `win32` or `win64`. Default `win32`, set on every run, as for `tbbuild`. A `win64` probe runs as a 64-bit process. | | `--timeout ` | Give up waiting for console output. Default 120. | | `--quiet ` | How long the console must stop changing before the output counts as complete. Default 2500. There is no sentinel string to match, so any probe works without telling the script anything. Raise it well above the default for a probe that drives an out-of-process server, which can take longer than that to start. | | `--raw` | Keep the console's timestamp column, which is otherwise stripped. | -| `--json` | One object with the built exe's path, the captured lines, the IDE pid and anything reaped. | +| `--json` | One object with the path of the built file, the target, the captured lines, the IDE pid and anything reaped. | | `--keep` | Leave the IDE running. Implies `--no-reap`, and leaves the IDE's registry entries for the probe as they are. | | `--no-reap` | Do not harvest automation servers the probe left behind. | | `--reap-images ` | Replace the harvested image list. Default is the Office suite. | @@ -614,12 +645,67 @@ and sweep once at the end. > *Save* dialog on build. Under `tbbuild` the IDE runs on a private desktop, so that dialog > is invisible, takes no input, and the build silently never happens --- the WebView2 > renderer stays responsive throughout, so even a health check says the IDE is fine. `tbrun` -> pins the path to a concrete file in its staged copy, which makes the trap unreachable. +> pins the path to a folder of its own in its staged copy, which makes the trap unreachable. +> The file keeps the IDE's own name, *project name*`_`*target*`.`*extension* --- for +> example `ArchProbe_win64.exe` --- so the name says what was built. Like `tbbuild`, it leaves the IDE's registry entries as it found them. Everything it opens is in its own temp folder, so it deletes every entry under that folder once the IDE has exited, and again at the start of a run, which removes what an earlier run on the same port left -behind. +behind. That includes the target the IDE remembers for each project, which a `win64` run +writes. **A probe builds for the target `--arch` names**, whatever the IDE remembers. Before +the option, a kept IDE switched to `win64` made every later run on the same port build 64-bit, +and nothing said so. + +### addin_test.mjs +{: #addin-test } + + node scripts/addin_test.mjs [--only ] [--port N] [--jobs N] [--timeout S] + [--ide ] [--show|--hide] + +Runs the add-in scenarios under `test/addin/`. A scenario file is a `node:test` file, and +[`addin-test.bat`](#addin-testbat) is the way to run it; run on its own, a scenario skips +itself. Each file is one **lane**, listed in `test/addin/lanes.mjs`. It runs in a process of +its own, with its own DevTools port and work folder, and with a private copy of the twinBASIC +install, whose add-in folders hold only what the lane puts there. A test add-in therefore +never loads into your own IDE, and two lanes never share one. The scenario builds the add-ins +it tests into its copy, opens a project, and operates the IDE: it clicks, presses keys, types, +and reads the add-ins' tool windows, message boxes, notifications, the code editor and the +Debug Console. The first two scenarios operate the IDE's own sample add-ins, Sample 10 and +Sample 15 (Global Search), end to end; the two lanes take about 25 seconds together. + +| Flag | Effect | +|---|---| +| `--only ` | Run only the lanes whose name matches. A lane's name is its file's name without `.test.mjs`. | +| `--port ` | Base DevTools port. Default 9560; the lanes get *n*, *n*+1 and so on, and their work folders are keyed to their ports. A port another IDE holds is refused, as for `tbbuild`. | +| `--jobs ` | Lanes at once. Default 2. | +| `--timeout ` | A lane still running after this long is ended and counted as failed. Default 600. | +| `--ide ` | The `twinBASIC.exe` to copy, found as for [`tbbuild.mjs`](#tbbuild). | +| `--show` / `--hide` | As for [`tbbuild.mjs`](#tbbuild). | + +Exit codes: **0** every lane passed and the registry is as it was found, **1** a lane +failed, **2** the harness failed or could not put the registry back. + +**It leaves the registry as it found it, and checks.** It puts back the IDE's own entries as +`tbbuild` does, and also the settings the add-ins under test save with `SaveSetting`. Those +are stored under `HKCU\Software\VB and VBA Program Settings\`, which any installed copy +of the same add-in shares, so a lane names its add-ins' application names in `lanes.mjs` +(`settings`). They are recorded before the first lane starts, deleted before each lane that +names them, so that its add-ins start from their defaults, and put back at the end. Two lanes +that name the same one never run at once. Afterwards it confirms that no entry names a lane's +folder and that the settings are as found, and reports any new application key that no lane +named. Pressing Ctrl+C ends the lanes and still puts everything back. + +**An add-in under test starts no browser.** Every IDE the harness starts, including those of +`tbbuild`, `tbrun` and `check_examples`, has the environment variable `TB_ADDIN_TEST` set to +`1`, and an add-in tested here is expected to check it. While it is set, the add-in prints +`open ` to the Debug Console instead of opening a page, and a scenario reads the line +there. A browser started on the harness's private desktop would open where nobody can see it +and keep running after the run. + +It refuses to start while `%APPDATA%\twinBASIC\addins` holds a DLL. The IDE passes that +folder to its compiler when it loads add-ins, so a DLL there may load into every test IDE, +as well as into your own. ### check_tb_registry.mjs {: #check-tb-registry } @@ -627,13 +713,20 @@ behind. node scripts/check_tb_registry.mjs The self-test for `scripts/lib/tb-registry.mjs`, the code that puts the IDE's registry -entries back after [`tbbuild.mjs`](#tbbuild), [`tbrun.mjs`](#tbrun) and -[`check_examples.mjs`](#check-examples). It plays out a run on a scratch copy of the IDE's -keys, under `HKCU\Software\tbharness-selftest`, and checks that everything comes back: a -project of yours that the run opened gets its saved state and its place in the recent list -back, the run's own entries go, the file association is restored, and a second restore -writes nothing. It also checks that the module refuses to sweep outside the temp folder or -restore a key near the root of the registry. It deletes the scratch key when it ends. +entries back after [`tbbuild.mjs`](#tbbuild), [`tbrun.mjs`](#tbrun), +[`addin_test.mjs`](#addin-test) and [`check_examples.mjs`](#check-examples). It plays out a +run on a scratch copy of the IDE's keys, under `HKCU\Software\tbharness-selftest`, and checks +that everything comes back: a project of yours that the run opened gets its saved state and +its place in the recent list back, the run's own entries go, the file association is +restored, and a second restore writes nothing. The recent list gets two more checks, because +the IDE changes it on its own while a run's projects are on it: it fills a short list's empty +slots with copies of the last entry, and a full list loses its oldest entry for each project +a run opens. The copies must go and the lost entries come back. The build targets the IDE remembers are checked the same way: those under +the run's folder go, and every other one stays, in its order and its exact text. So is the +rule that a file association pointing into the temp folder when a run began --- at another +run's private copy of the IDE --- is left as it is rather than put back. It also checks that +the module refuses to sweep outside the temp folder or restore a key near the root of the +registry. It deletes the scratch key when it ends. It is not a gate and is not in `test.bat`, because it needs Windows and a real registry and the CI runners have neither. Run it by hand after changing `tb-registry.mjs`. Exit code @@ -712,9 +805,14 @@ identifiers ending in a digit, which would also have declared `Var1`, `Arg1`, `L **A sample can take the compiler down**, and one in this corpus does. twinBASIC runs the compiler in the same process as user code, so in a batch of a hundred that costs the other -ninety-nine their result. `tbbuild` reports a crash as exit 4; this splits the batch and -recurses until the offending sample is alone, which is O(log n) extra builds paid only on -failure. The finding names the sample and points at `BUGS-TO-REPORT.md`. +ninety-nine their result. `tbbuild` reports a crash as exit 4 and names the file the +compiler was parsing when it died. The sample that file belongs to is built on its own and +the rest of the batch without it, so a crash usually costs two extra builds. When no sample +is named, the batch is split in half repeatedly until the offending sample is alone, which +is O(log n) extra builds. A crash can also need several samples at once, so that no part of +the batch crashes by itself. The samples it needs are then searched for as a set, and all +of them are reported. The cost is paid only on failure. The finding names the sample, or +the set, and points at `BUGS-TO-REPORT.md`. **A sample can be compiled against a file.** A fence carrying `resource=` --- in any language, typically ` ```json ` --- is written into the generated project at diff --git a/scripts/addin_test.mjs b/scripts/addin_test.mjs new file mode 100644 index 00000000..7f7de48a --- /dev/null +++ b/scripts/addin_test.mjs @@ -0,0 +1,283 @@ +#!/usr/bin/env node +// Run the IDE add-in scenarios: every lane listed in test/addin/lanes.mjs, each +// a node:test file run in a process of its own, with its own DevTools port, +// work folder and copy of the twinBASIC install. +// +// node scripts/addin_test.mjs [options] +// +// --only only the lanes whose name matches +// --port base DevTools port (default 9560); the lanes get n, n+1, ... +// --jobs lanes at once (default 2) +// --timeout a lane still running after this long is ended (default 600) +// --ide the twinBASIC.exe to copy (default: $TB_IDE, else the +// newest %USERPROFILE%/Desktop/twinBASIC_IDE_BETA_*) +// --show / --hide as tbbuild's +// +// Exit: 0 every lane passed and the registry is as it was found, 1 a lane +// failed, 2 the harness failed or could not put the registry back. +// +// Not a gate, for the reasons examples.bat is not one: it needs Windows and a +// twinBASIC install. addin-test.bat is the wrapper. +// +// ---------------------------------------------------------------- how +// +// A lane is one scenario file. The file builds the add-ins it tests into its +// lane's copy of the install and opens projects in it, one IDE at a time, +// through scripts/lib/tb-lane.mjs; this script only hands each file its lane +// (TB_ADDIN_LANE), runs the files a few at a time, and owns what they share: +// +// * THE REGISTRY. One process owns it per run (lib/tb-registry.mjs). This +// one records it before the first lane starts, and the lanes inherit +// TB_REGISTRY_OWNER and leave it alone. Once every lane has ended it +// puts back the IDE's lists, the .twinproj association and the build +// targets the IDE remembers, then checks that nothing under the lanes' +// folders is left. +// * THE ADD-INS' OWN SETTINGS. An add-in's SaveSetting writes under +// HKCU\Software\VB and VBA Program Settings\, the same key as any +// installed copy of that add-in. A lane names the applications its +// add-ins save settings for (`settings` in lanes.mjs); their keys are +// recorded before the first lane starts, deleted before each lane that +// names them, so that its add-ins start from their defaults, and put back +// at the end. Lanes that name the same application never run at once. +// * %APPDATA%\twinBASIC\addins. The page hands the compiler that folder with +// RequestLoadAddins, so a DLL in it may load into every test IDE, and +// whether it does is P6 in WIP.HelpAddin.md. The run refuses to start +// while the folder holds one. +// +// Ctrl+C ends the lanes and still puts the registry back. + +import { spawn } from "node:child_process"; +import { existsSync, mkdirSync, readdirSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { removeTree } from "./lib/tb-ide-copy.mjs"; +import { wantShow } from "./lib/tb-ide.mjs"; +import { buildNumber, findIde } from "./lib/tb-install.mjs"; +import { LANE_ENV } from "./lib/tb-lane.mjs"; +import { deleteSettings, finishTidy, ideLists, restoreKeys, SETTINGS_ROOT, settingsKey, snapshotKeys, + startTidy, subkeyNames, sweepArchitectureMemory } from "./lib/tb-registry.mjs"; + +const REPO = path.dirname(path.dirname(fileURLToPath(import.meta.url))); +const SUITE = path.join(REPO, "test", "addin"); + +const argv = process.argv.slice(2); +const flag = (n) => argv.includes(`--${n}`); +const opt = (n, d) => { const i = argv.indexOf(`--${n}`); return i >= 0 && argv[i + 1] ? argv[i + 1] : d; }; +const die = (code, msg) => { console.error(msg); process.exit(code); }; + +if (flag("help")) { + die(2, "usage: node scripts/addin_test.mjs [--only REGEX] [--port N] [--jobs N] " + + "[--timeout S] [--ide ] [--show|--hide]"); +} +const only = opt("only", null) ? new RegExp(opt("only")) : null; +const basePort = Number(opt("port", 9560)); +const jobs = Math.max(1, Number(opt("jobs", 2))); +const laneTimeout = Number(opt("timeout", 600)) * 1000; +const show = wantShow({ show: flag("show"), hide: flag("hide") }); + +// ---------------------------------------------------------------- refusals + +const appDataAddins = path.join(process.env.APPDATA ?? "", "twinBASIC", "addins"); +const strays = existsSync(appDataAddins) + ? readdirSync(appDataAddins, { recursive: true }).filter((f) => /\.dll$/i.test(f)) + : []; +if (strays.length) { + die(2, `${appDataAddins} holds ${strays.map((f) => `"${f}"`).join(", ")}.\n` + + "The IDE hands the compiler that folder when it asks it to load add-ins, so a DLL there " + + "may load into every test IDE (P6 in WIP.HelpAddin.md), and into your own IDE as well. " + + "Move it out of that folder to run the tests."); +} + +const ide = findIde(opt("ide", undefined)); +if (!ide || !existsSync(ide)) { + die(2, "no twinBASIC IDE found: pass --ide , set TB_IDE, " + + "or unpack a twinBASIC_IDE_BETA_ folder on your Desktop"); +} + +const manifest = (await import(pathToFileURL(path.join(SUITE, "lanes.mjs")).href)).default; +const lanes = manifest + .map((l) => ({ ...l, name: l.name ?? path.basename(l.file).replace(/\.test\.mjs$/, "") })) + .filter((l) => !only || only.test(l.name)) + .map((l, i) => ({ ...l, settings: l.settings ?? [], port: basePort + i, + work: path.join(tmpdir(), "tbaddin", String(basePort + i)) })); +if (!lanes.length) die(2, `no lane in ${path.join(SUITE, "lanes.mjs")} matches ${only}`); +for (const l of lanes) { + if (!existsSync(path.join(SUITE, l.file))) die(2, `lane ${l.name}: no file ${path.join(SUITE, l.file)}`); + try { l.settings.forEach(settingsKey); } catch (e) { die(2, `lane ${l.name}: ${e.message}`); } +} + +// ---------------------------------------------------------------- the registry + +// startTidy leaves the registry to a live owner that is not this process, and +// such an owner would tidy only its own folders, not the lanes'. +const alive = (pid) => { try { process.kill(pid, 0); return true; } catch { return false; } }; +const owner = Number(process.env.TB_REGISTRY_OWNER); +if (owner && owner !== process.pid && alive(owner)) { + die(2, `process ${owner} already owns the registry for a run (TB_REGISTRY_OWNER); ` + + "run the add-in tests on their own"); +} +for (const l of lanes) { + try { removeTree(l.work); } catch (e) { + die(2, `${l.work} could not be emptied (${e.code}): is a process from an earlier run still running?`); + } + mkdirSync(l.work, { recursive: true }); +} +const tidy = startTidy({ prefixes: lanes.map((l) => l.work) }); +if (!tidy) die(2, "could not record the registry, so the run could not put it back"); +const apps = [...new Set(lanes.flatMap((l) => l.settings))]; +const snapshotSettings = () => (apps.length ? [].concat(snapshotKeys(apps.map(settingsKey))) : []); +let settingsBefore, appsBefore; +try { + settingsBefore = snapshotSettings(); + // Only the names, to notice an application key the run creates that no lane + // named: an add-in saving settings nobody told the runner about. + appsBefore = subkeyNames(SETTINGS_ROOT).map((n) => n.toLowerCase()); +} catch (e) { + finishTidy(tidy); + die(2, `could not record the add-ins' settings: ${e.message}`); +} + +// ---------------------------------------------------------------- the lanes + +console.log(`addin-test: BETA ${buildNumber(ide) ?? "?"}, ${lanes.length} lane(s) on ports ` + + `${basePort}-${basePort + lanes.length - 1}, ${Math.min(jobs, lanes.length)} at a time`); + +let interrupted = false; +const children = new Set(); +process.on("SIGINT", () => { + if (interrupted) return; + interrupted = true; + console.error("\ninterrupted: ending the lanes, then putting the registry back"); + for (const child of children) child.kill(); +}); + +// A lane's output is held until it ends and then printed whole, so that two +// lanes' reports never interleave. +function runLane(l) { + return new Promise((resolve) => { + const t0 = Date.now(); + let finished = false; + const finish = (r) => { + if (finished) return; + finished = true; + const secs = ((Date.now() - t0) / 1000).toFixed(1); + const verdict = r.code === 0 ? "passed" : r.timedOut ? "TIMED OUT" : "FAILED"; + // node:test reports a test once it ends, so a lane ended in the middle + // of one, as a timeout or Ctrl+C ends it, has usually printed nothing. + const report = r.out.trimEnd() || "(the lane printed nothing before it ended)"; + process.stdout.write(`\n--- ${l.name} (port ${l.port}): ${verdict} in ${secs} s\n${report}\n`); + resolve({ ...r, lane: l }); + }; + try { + if (l.settings.length) deleteSettings(l.settings); + } catch (e) { + finish({ code: null, out: `could not clear the settings of ${l.settings.join(", ")}: ${e.message}` }); + return; + } + console.log(` ${l.name}: started on port ${l.port}`); + const child = spawn(process.execPath, ["--test", "--test-reporter=spec", path.join(SUITE, l.file)], { + cwd: REPO, stdio: ["ignore", "pipe", "pipe"], windowsHide: true, + env: { ...process.env, + [LANE_ENV]: JSON.stringify({ name: l.name, port: l.port, work: l.work, ide, show }) }, + }); + children.add(child); + let out = "", timedOut = false; + child.stdout.on("data", (d) => { out += d; }); + child.stderr.on("data", (d) => { out += d; }); + const timer = setTimeout(() => { timedOut = true; child.kill(); }, laneTimeout); + child.on("error", (e) => { + clearTimeout(timer); + children.delete(child); + finish({ code: null, out: `${out}\ncould not start the lane: ${e.message}`, timedOut }); + }); + child.on("close", (code) => { + clearTimeout(timer); + children.delete(child); + finish({ code, out, timedOut }); + }); + }); +} + +// A few lanes at a time, and never two that name the same application's +// settings: each deletes the key before it starts, and its add-ins read it. +async function runAll() { + const results = []; + const pending = [...lanes]; + const running = new Map(); + const clash = (a, b) => a.settings.some((s) => b.settings.includes(s)); + while ((pending.length && !interrupted) || running.size) { + for (let i = 0; i < pending.length && running.size < jobs && !interrupted;) { + const l = pending[i]; + if ([...running.keys()].some((r) => clash(l, r))) { i++; continue; } + pending.splice(i, 1); + running.set(l, runLane(l).then((r) => { results.push(r); running.delete(l); })); + } + if (running.size) await Promise.race(running.values()); + } + return results; +} + +const results = await runAll(); + +// ---------------------------------------------------------------- putting it back + +const problems = []; +const tidied = finishTidy(tidy); +if (!tidied) problems.push("the IDE's registry entries could not be put back (see the warning above)"); +try { + if (apps.length) restoreKeys(settingsBefore); +} catch (e) { + problems.push(`the add-ins' settings could not be put back: ${e.message}`); +} + +// The check that the run left nothing: an entry naming a lane's folder in the +// IDE's lists, a build target remembered for one, or an add-in's settings that +// differ from what was recorded. Another session's IDE that is open meanwhile +// can write its own copy of the recent list back, which is the one way an +// entry could return (WIP.Harness.md). +const norm = (p) => String(p).split("/").join("\\").toLowerCase(); +const folders = lanes.map((l) => norm(l.work) + "\\"); +try { + const lists = ideLists(); + const left = [...lists.projectState, ...lists.recentlyOpened] + .filter((p) => p && folders.some((f) => norm(p).startsWith(f))); + if (left.length) problems.push(`the IDE's lists still name the lanes' folders: ${left.join(", ")}`); + const targets = sweepArchitectureMemory(lanes.map((l) => l.work)); + if (targets) problems.push(`${targets} build target(s) were still remembered for the lanes' folders`); + const settingsAfter = snapshotSettings(); + for (let i = 0; i < apps.length; i++) { + if (JSON.stringify(settingsAfter[i]) !== JSON.stringify(settingsBefore[i])) { + problems.push(`the settings of ${apps[i]} are not as they were found`); + } + } + const named = apps.map((a) => a.toLowerCase()); + const created = subkeyNames(SETTINGS_ROOT) + .filter((n) => !appsBefore.includes(n.toLowerCase()) && !named.includes(n.toLowerCase())); + if (created.length) { + problems.push(`HKCU\\${SETTINGS_ROOT} gained ${created.map((n) => `"${n}"`).join(", ")} during the ` + + "run, which no lane names in test/addin/lanes.mjs. An add-in under test that saves settings " + + "must be named there, or its settings stay behind; the key is left as it is"); + } +} catch (e) { + problems.push(`could not check the registry: ${e.message}`); +} + +// A folder that will not delete is held open by a process that outlived its +// lane, which is worth hearing about (WIP.Harness.md, The IDE runs inside a job). +for (const l of lanes) { + try { removeTree(l.work); } + catch (e) { problems.push(`${l.work} could not be deleted (${e.code}): is a process of its lane still running?`); } +} + +const failed = results.filter((r) => r.code !== 0); +const settingsNote = apps.length ? `, and the settings of ${apps.join(", ")} as found` : ""; +console.log(""); +console.log(problems.length + ? `registry and work folders: ${problems.length} problem(s)\n ${problems.join("\n ")}` + : `registry: put back (${tidied.projectState} project-state, ${tidied.recentlyOpened} recent-list ` + + `and ${tidied.association ?? "no"} association writes); nothing names the lanes' folders${settingsNote}`); +console.log(`${results.length} of ${lanes.length} lane(s) ran: ${results.length - failed.length} passed` + + (failed.length ? `, ${failed.length} failed (${failed.map((r) => r.lane.name).join(", ")})` : "") + + (interrupted ? "; interrupted" : "")); +process.exit(problems.length ? 2 : failed.length || interrupted ? 1 : 0); diff --git a/scripts/check_examples.mjs b/scripts/check_examples.mjs index 07800b35..b04b6e16 100644 --- a/scripts/check_examples.mjs +++ b/scripts/check_examples.mjs @@ -61,8 +61,10 @@ // And one that does not: a sample can take the compiler down. twinBASIC runs it // in-process with user code, and a two-line syntax skeleton in Attributes.md // crashes it outright (BUGS-TO-REPORT.md). In a batch that costs every other -// sample its result, so a crash bisects: O(log n) extra builds, paid only on -// failure. +// sample its result, so a crash is isolated, paid for only on failure: the +// sample tbbuild names as the one the compiler died parsing is built on its own +// and the rest without it, a crash that names none bisects, O(log n) builds, and +// one that needs several samples at once is reported with all of them. import { spawn } from "node:child_process"; import { @@ -575,6 +577,26 @@ const COMPILER = IDE ? compilerExe(IDE) : null; // by the top-level catch, if main() dies in between. let tidy = null; +/** + * The samples of a staged batch that tbbuild's crash report says the compiler + * died parsing, as fence ids. + * + * tbbuild names them on its `last parsing:` line by the file's base name, which + * for a sample is its generated module's: `last parsing: tbx_df66b6fa33.twin` + * for a batch of nine holding the crash fixture. A file that is no sample of + * the batch -- the template's own source -- names nothing, and neither does a + * report without the line. + */ +function crashedIn(report, map) { + const ids = new Set(); + const line = /^last parsing: (.+)$/m.exec(report)?.[1] ?? ""; + for (const file of line.split(",")) { + const entry = map.get(file.trim().split(/[\\/]/).pop()); + if (entry) ids.add(entry.fence.id); + } + return ids; +} + /** Build one staged batch; returns per-fence errors, or a crash marker. */ async function buildStaged(staged, port) { const args = [path.join(REPO, "scripts", "tbbuild.mjs"), staged.proj, @@ -589,7 +611,7 @@ async function buildStaged(staged, port) { child.stderr.on("data", (d) => { err += d; }); const code = await new Promise((r) => child.on("exit", r)); - if (code === 4) return { crashed: true, detail: err.trim() }; + if (code === 4) return { crashed: true, detail: err.trim(), named: crashedIn(err, staged.map) }; if (code !== 0 && code !== 1) { throw new Error(`tbbuild exited ${code} on ${staged.proj}\n${err.trim() || out.trim()}`); } @@ -633,23 +655,20 @@ async function buildStaged(staged, port) { } /** - * Halve a batch WITHOUT cutting through anything that has to stay together. + * A batch's units, and the page context that travels with them. * - * The unit is what `makeBatches` made it: a `projname` group is one program, and - * a page's `hidden` context travels with every sample from that page. Halving - * the fence array instead would take a group's definitions away from its tests - * and then report the tests -- an isolation run that manufactures the failure it - * claims to have found. The hidden fences sit at the END of `batch.fences`, so a - * plain slice loses them for one half outright. - * - * Returns null when there is one unit left, which is the leaf: the smallest - * thing that can be blamed. + * Isolation cuts a batch by unit, never through one. The unit is what + * `makeBatches` made it: a `projname` group is one program, and a page's + * `hidden` context travels with every sample from that page. Cutting the fence + * array instead would take a group's definitions away from its tests and then + * report the tests -- an isolation run that manufactures the failure it claims + * to have found. The hidden fences sit at the END of `batch.fences`, so a plain + * slice loses them for one part outright. */ -function splitBatch(batch) { +function unitsOf(batch) { // Hidden fences and resource files are page context: they follow the samples // rather than being split between them. const travels = (f) => f.flags.has(HIDDEN_MARKER) || f.isResource; - const hidden = batch.fences.filter(travels); const units = new Map(); for (const f of batch.fences) { if (travels(f)) continue; @@ -657,14 +676,43 @@ function splitBatch(batch) { if (!units.has(key)) units.set(key, []); units.get(key).push(f); } - const list = [...units.values()]; + return { list: [...units.values()], hidden: batch.fences.filter(travels) }; +} + +/** A batch of some of another's units, with the page context those units need. */ +function batchOf(batch, units, hidden) { + const fences = units.flat(); + const pages = new Set(fences.map((f) => f.rel)); + return { ...batch, fences: [...fences, ...hidden.filter((h) => pages.has(h.rel))], pages }; +} + +/** + * Halve a batch WITHOUT cutting through anything that has to stay together. + * + * Returns null when there is one unit left, which is the leaf: the smallest + * thing that can be blamed. + */ +function splitBatch(batch) { + const { list, hidden } = unitsOf(batch); if (list.length < 2) return null; const half = Math.ceil(list.length / 2); - return [list.slice(0, half), list.slice(half)].map((part) => { - const fences = part.flat(); - const pages = new Set(fences.map((f) => f.rel)); - return { ...batch, fences: [...fences, ...hidden.filter((h) => pages.has(h.rel))], pages }; - }); + return [list.slice(0, half), list.slice(half)].map((part) => batchOf(batch, part, hidden)); +} + +/** + * Take the units holding these samples out of a batch: [those units, the rest]. + * + * Null when that divides nothing -- no sample of the batch named, or every unit + * named -- which is what keeps a recursion on either part smaller than the + * batch it came from. Hidden context is not a unit, so naming it takes nothing + * out; the samples it travels with are left for halving to find. + */ +function takeOut(batch, ids) { + const { list, hidden } = unitsOf(batch); + const named = list.filter((u) => u.some((f) => ids.has(f.id))); + if (!named.length || named.length === list.length) return null; + return [batchOf(batch, named, hidden), + batchOf(batch, list.filter((u) => !named.includes(u)), hidden)]; } /** The visible samples of a leaf batch, and how to describe it in a finding. */ @@ -692,17 +740,15 @@ const sameRow = (row) => row.replace(/[/\\]DocSamples\d+[/\\]/, "/"); * sample -- hundreds of IDE starts -- and then blame an arbitrary one. */ const templateOwnRows = new Map(); -function ownRowsOf(project, port, work) { +function ownRowsOf(project, lane) { if (!templateOwnRows.has(project)) { templateOwnRows.set(project, (async () => { - const staged = stageBatch({ project, fences: [] }, work); - const result = await buildStaged(staged, port); - if (!flag("keep")) rmSync(staged.dir, { recursive: true, force: true }); + const result = await lane.build({ project, fences: [] }); const rows = result.crashed ? [`the ${project} template crashes the compiler with no samples in it`] : [...(result.unattributed ?? []), ...(result.unreadable ?? [])]; if (rows.length) { - say(` note: template \`${project}\` does not build clean on its own; ` + + lane.note(` note: template \`${project}\` does not build clean on its own; ` + `${rows.length} row(s) are its own, not any sample's`); } return new Set(rows.map(sameRow)); @@ -712,53 +758,146 @@ function ownRowsOf(project, port, work) { } /** - * Build a batch, isolating a crash or an unattributable diagnostic. + * A lane: where a batch is built, and where what isolating it finds goes. * - * Both are attributed by halving until one unit is left. A crash is a compiler - * bug as well as a finding, and BUGS-TO-REPORT.md is where one goes. An - * unattributable diagnostic is the subtler of the two: the sample that caused it - * may have no diagnostic of its own at all -- a generic instantiated with a type - * the project does not have reports inside the PACKAGE's source, against the - * generic's own type parameter -- so before this the sample was counted as - * compiling while the run failed with a row naming no page. + * This one stages each batch into the lane's own workspace and builds it on the + * lane's port. The probes hand `runBatch` a fake, whose builds crash on sets of + * samples a probe chooses -- which is how isolation is tested without an IDE, + * and without a crash that needs two real samples to happen. */ -async function runBatch(batch, port, work) { - const staged = stageBatch(batch, work); - const result = await buildStaged(staged, port); - if (!flag("keep")) rmSync(staged.dir, { recursive: true, force: true }); - - // A function, not a shared object: spreading one would hand every caller the - // same arrays, and a recursion that pushes into them is a bug waiting. - const blank = () => ({ - perFence: new Map(), templateFaults: [], crashed: [], blamed: [], blamedRows: new Map(), - }); - const split = async (why) => { - const parts = splitBatch(batch); - if (!parts) return null; - say(` ${why} in ${batch.fences.length} sample(s) [${batch.project}]: splitting to find it`); - const merged = blank(); - for (const part of parts) { - const sub = await runBatch(part, port, work); - for (const [k, v] of sub.perFence ?? []) merged.perFence.set(k, v); - for (const [k, v] of sub.blamedRows ?? []) merged.blamedRows.set(k, v); - merged.templateFaults.push(...(sub.templateFaults ?? [])); - merged.crashed.push(...(sub.crashed ?? [])); - merged.blamed.push(...(sub.blamed ?? [])); - } - return merged; +function laneOf(port, work) { + return { + async build(batch) { + const staged = stageBatch(batch, work); + const result = await buildStaged(staged, port); + if (!flag("keep")) rmSync(staged.dir, { recursive: true, force: true }); + return result; + }, + finding: addFinding, + note: say, }; +} + +// A function, not a shared object: spreading one would hand every caller the +// same arrays, and a recursion that pushes into them is a bug waiting. +const blank = () => ({ + perFence: new Map(), templateFaults: [], crashed: [], blamed: [], blamedRows: new Map(), +}); + +/** Several parts' results, as one batch's. */ +function merge(subs) { + const merged = blank(); + for (const sub of subs) { + for (const [k, v] of sub.perFence ?? []) merged.perFence.set(k, v); + for (const [k, v] of sub.blamedRows ?? []) merged.blamedRows.set(k, v); + merged.templateFaults.push(...(sub.templateFaults ?? [])); + merged.crashed.push(...(sub.crashed ?? [])); + merged.blamed.push(...(sub.blamed ?? [])); + } + return merged; +} - if (result.crashed) { - const deeper = await split("a compiler crash"); - if (deeper) return deeper; +/** Build both halves of a batch; null when it is one unit, which cannot be halved. */ +async function split(batch, lane, why) { + const parts = splitBatch(batch); + if (!parts) return null; + lane.note(` ${why} in ${batch.fences.length} sample(s) [${batch.project}]: splitting to find it`); + const subs = []; + for (const part of parts) subs.push(await runBatch(part, lane)); + return merge(subs); +} + +/** + * Find what took the compiler down in a batch, and build everything else. + * + * Start where tbbuild says the compiler died: the sample it was parsing is + * built on its own and the rest without it, two builds where halving pays two + * for every level. With no sample of the batch named, halve. Either way a + * crash can need several samples at once, and then no part crashes by itself + * -- the named sample and the rest both build, or both halves do. That used to + * end with every sample of the batch counted as compiling; `together` finds the + * samples the crash needs instead. + * + * The result is marked `fromCrash`, because a part that crashed may come back + * with nothing in `crashed` -- its crash needed several samples too -- and + * whether a part crashed is what decides the next step. + */ +async function isolateCrash(batch, named, lane) { + const where = `a compiler crash in ${batch.fences.length} sample(s) [${batch.project}]`; + let parts = takeOut(batch, named); + if (parts) lane.note(` ${where}: building the sample it died parsing on its own`); + else if ((parts = splitBatch(batch))) lane.note(` ${where}: splitting to find it`); + else { const { rep, ids, rest } = leafOf(batch); - addFinding(rep, "crashes the twinBASIC compiler" + rest, + lane.finding(rep, "crashes the twinBASIC compiler" + rest, "the compiler dies parsing this sample; record it in BUGS-TO-REPORT.md"); // Named back to the caller, because a crashed sample produced no // diagnostics and would otherwise be counted as one that compiled -- the // same false-clean shape tbbuild's own crash check exists to close. - return { ...blank(), crashed: ids }; + return { ...blank(), crashed: ids, fromCrash: true }; } + const subs = []; + for (const part of parts) subs.push(await runBatch(part, lane)); + if (subs.some((s) => s.fromCrash)) return { ...merge(subs), fromCrash: true }; + lane.note(" neither part crashes on its own: looking for the samples it needs together"); + return { ...merge([...subs, await together(batch, ...parts, lane)]), fromCrash: true }; +} + +/** + * The samples a crash needs when it needs several: `a` and `b` each built + * clean, and together they crash. + * + * `partners` finds the smallest part of a pool that still crashes with what is + * held fixed. Whichever half of the pool crashes with it holds what the crash + * needs; when neither does, each half holds some of it, and each is searched + * with the other held. That assumes a crash follows from what a build holds -- + * more samples never prevent one -- and the last build checks it: a set that + * does not crash as found is reported whole instead. + * + * Every member built clean in `a` or `b`, so each has its own result. They are + * blamed rather than passed, because together they take the compiler down. + */ +async function together(batch, a, b, lane) { + const { hidden } = unitsOf(batch); + const crashes = async (units) => !!(await lane.build(batchOf(batch, units, hidden))).crashed; + const partners = async (fixed, pool) => { + if (pool.length === 1) return pool; + const half = Math.ceil(pool.length / 2); + const [x, y] = [pool.slice(0, half), pool.slice(half)]; + if (await crashes([...fixed, ...x])) return partners(fixed, x); + if (await crashes([...fixed, ...y])) return partners(fixed, y); + const inX = await partners([...fixed, ...y], x); + return [...inX, ...await partners([...fixed, ...inX], y)]; + }; + const inA = unitsOf(a).list, inB = unitsOf(b).list; + const needB = await partners(inA, inB); + const needA = await partners(needB, inA); + const found = await crashes([...needA, ...needB]); + const members = (found ? [...needA, ...needB] : [...inA, ...inB]).flat() + .filter((f) => !f.flags.has(HIDDEN_MARKER) && !f.isResource); + const [rep, ...others] = members; + lane.finding(rep, `crashes the twinBASIC compiler when built with ${others.length} other ` + + `sample(s), though none of the ${members.length} does on its own`, + `the others: ${others.map((f) => `${f.rel}:${f.line}`).join(", ")}` + + (found ? "" : "; no smaller set that crashes was found") + " -- record it in BUGS-TO-REPORT.md"); + return { ...blank(), blamed: members.map((f) => f.id) }; +} + +/** + * Build a batch, isolating a crash or an unattributable diagnostic. + * + * A crash goes to `isolateCrash`; an unattributable diagnostic is attributed by + * halving until one unit is left. A crash is a compiler bug as well as a + * finding, and BUGS-TO-REPORT.md is where one goes. An unattributable + * diagnostic is the subtler of the two: the sample that caused it may have no + * diagnostic of its own at all -- a generic instantiated with a type the + * project does not have reports inside the PACKAGE's source, against the + * generic's own type parameter -- so before this the sample was counted as + * compiling while the run failed with a row naming no page. + */ +async function runBatch(batch, lane) { + const result = await lane.build(batch); + if (result.crashed) return isolateCrash(batch, result.named, lane); const unreadable = result.unreadable ?? []; const rows = [...new Set((result.unattributed ?? []).map(sameRow))]; @@ -766,7 +905,7 @@ async function runBatch(batch, port, work) { return { perFence: result.perFence, templateFaults: unreadable, crashed: [], blamed: [] }; } - const own = await ownRowsOf(batch.project, port, work); + const own = await ownRowsOf(batch.project, lane); const mine = rows.filter((r) => !own.has(r)); if (!mine.length) { // Every row is the template's own. Reported as a template fault, which is @@ -774,7 +913,7 @@ async function runBatch(batch, port, work) { return { perFence: result.perFence, templateFaults: [...rows, ...unreadable], crashed: [], blamed: [] }; } - const deeper = await split("a diagnostic outside every sample"); + const deeper = await split(batch, lane, "a diagnostic outside every sample"); if (deeper) return { ...deeper, templateFaults: [...deeper.templateFaults, ...unreadable] }; // Blamed, not passed: the whole point is that such a sample can produce no @@ -792,16 +931,17 @@ async function runBatch(batch, port, work) { async function runAll(batches, work) { const queue = [...batches]; const results = []; - const lanes = Array.from({ length: Math.min(jobs, queue.length) }, async (_, lane) => { + const lanes = Array.from({ length: Math.min(jobs, queue.length) }, async (_, i) => { // A lane owns its port AND its workspace. Two IDEs pointed at one source // tree both wedge and neither ever returns -- distinct ports are not // enough, which cost two runs to learn. - const laneWork = path.join(work, `lane${lane}`); + const laneWork = path.join(work, `lane${i}`); mkdirSync(laneWork, { recursive: true }); + const lane = laneOf(basePort + i, laneWork); while (queue.length) { const batch = queue.shift(); - process.stderr.write(` building ${batch.fences.length} sample(s) [${batch.project}] on lane ${lane}\n`); - results.push(await runBatch(batch, basePort + lane, laneWork)); + process.stderr.write(` building ${batch.fences.length} sample(s) [${batch.project}] on lane ${i}\n`); + results.push(await runBatch(batch, lane)); } }); await Promise.all(lanes); @@ -1229,6 +1369,82 @@ async function runProbes() { if (withM2?.fences.some((f) => f.id === "mh")) { failures.push("split: a page's hidden context followed a page that never asked for it"); } + // Taking a crash's named sample out is the same kind of cut and can go wrong + // the same two ways. It also has to refuse a cut that divides nothing, or the + // recursion on the named part never gets any smaller. + const [outM2, restM1] = takeOut(mixed, new Set(["m2"])) ?? []; + if (outM2?.fences.map((f) => f.id).join() !== "m2") { + failures.push("take out: the named sample did not come out on its own"); + } + if (!restM1?.fences.some((f) => f.id === "mh")) { + failures.push("take out: a page's hidden context did not stay with its sample"); + } + const grouped = { project: "console", fences: [fake("t1", "t"), fake("t2", "t"), fake("u1", null)] }; + if (takeOut(grouped, new Set(["t2"]))?.[0].fences.map((f) => f.id).join() !== "t1,t2") { + failures.push("take out: a projname group was cut apart"); + } + if (takeOut(soleGroup, new Set(["g1"])) !== null) { + failures.push("take out: a cut that divides nothing was made"); + } + if (takeOut(mixed, new Set(["mh"])) !== null) { + failures.push("take out: a page's hidden context was taken out as a unit"); + } + // What is taken out is read off tbbuild's crash report, which names a sample + // by its generated module's file. This is a report as tbbuild printed it. + const staged9 = new Map([["tbx_df66b6fa33.twin", { fence: { id: "P.md#5" } }]]); + const report = "the compiler crashed 2x -- this project takes it down\n" + + "last parsing: tbx_df66b6fa33.twin\n" + + "(read the IDE's DEBUG CONSOLE with --keep for the exception detail)\n"; + if ([...crashedIn(report, staged9)].join() !== "P.md#5") { + failures.push("crash report: the sample tbbuild named was not read off it"); + } + if (crashedIn(report.replace("tbx_df66b6fa33", "tbxMain"), staged9).size) { + failures.push("crash report: the template's own file named a sample"); + } + if (crashedIn("the compiler crashed 1x -- this project takes it down\n", staged9).size) { + failures.push("crash report: a report naming no file named a sample"); + } + // Isolation itself, run by runBatch against a fake lane. `crash` gets the ids + // of a build's samples and says whether the compiler goes down, and which of + // them the report names. Each shape a crash can take has to end in exactly + // one finding, on the samples it needs -- never in a batch whose samples all + // count as compiling, which is what a crash that needs two of them used to + // become once halving had separated them. + const fakeLane = (crash) => { + const lane = { + found: [], builds: 0, + async build(b) { + lane.builds++; + const c = crash(new Set(b.fences.map((f) => f.id))); + return c ? { crashed: true, named: new Set(c.named ?? []) } + : { perFence: new Map(), unreadable: [], unattributed: [] }; + }, + finding: (fence) => lane.found.push(fence.id), + note: () => {}, + }; + return lane; + }; + const five = { project: "console", fences: ["a", "b", "c", "d", "e"].map((id) => fake(id, null)) }; + const when = (...need) => (ids) => need.every((id) => ids.has(id)); + const naming = (id) => (ids) => ({ named: ids.has(id) ? [id] : [] }); + for (const [shape, crashes, names, alone, set] of [ + ["a named sample that crashes alone", when("c"), naming("c"), ["c"], []], + ["an unnamed sample that crashes alone", when("d"), () => ({}), ["d"], []], + ["a named sample that crashes only with another", when("b", "d"), naming("d"), [], ["d", "b"]], + ["two unnamed samples that halving separates", when("a", "e"), () => ({}), [], ["a", "e"]], + ["an innocent sample named, the culprit elsewhere", when("e"), naming("b"), ["e"], []], + ["three samples that crash only together", when("a", "c", "e"), naming("c"), [], ["c", "a", "e"]], + ]) { + const lane = fakeLane((ids) => crashes(ids) && names(ids)); + const r = await runBatch(five, lane); + const got = `${[...r.crashed].sort()} / ${[...r.blamed].sort()} / ${lane.found}`; + const want = `${[...alone].sort()} / ${[...set].sort()} / ${alone[0] ?? set[0]}`; + if (got !== want) failures.push(`isolation: ${shape} -> ${got}, want ${want}`); + } + // ...and the common case stays cheap: the batch, the named sample, the rest. + const cheap = fakeLane((ids) => when("c")(ids) && naming("c")(ids)); + await runBatch(five, cheap); + if (cheap.builds !== 3) failures.push(`isolation: a named crash took ${cheap.builds} builds, want 3`); // Two builds of one template differ only by the stage index in every path, and // a template's own fault is recognised by comparing those rows. const row = (n) => `{ERROR} /DocSamples${n}/Packages/P/Sources/S.twin [10,20]: TB5079 x`; @@ -1356,9 +1572,11 @@ async function runProbes() { return false; } // 10 line-map (5 slots x 2 bases) + 4 wrapper container + 7 batching - // + 5 splitting + 9 concat + 10 resource + 8 report + 6 markup. - say(`ok ${CLASSIFIER_PROBES.length + INFO_PROBES.length + 59} probes: ` + - `classifier, markup, line mapping, batching, splitting, concat, resources and the report`); + // + 5 splitting + 5 taking out + 3 crash report + 7 isolation + 9 concat + // + 10 resource + 8 report + 6 markup. + say(`ok ${CLASSIFIER_PROBES.length + INFO_PROBES.length + 74} probes: ` + + "classifier, markup, line mapping, batching, splitting, crash isolation, concat, " + + "resources and the report"); return true; } @@ -1511,8 +1729,9 @@ async function main() { } continue; } - // A sample blamed as part of a group carries no rows of its own -- the - // finding above names the group -- but it is still not a pass. + // A sample blamed as part of a group, or with the samples it takes the + // compiler down together with, carries no rows of its own -- one finding + // names the rest -- but it is still not a pass. if (!diags.length) { if (!blamed.has(fence.id)) passed.push(fence); continue; diff --git a/scripts/check_tb_registry.mjs b/scripts/check_tb_registry.mjs index a9abf925..8d05f20a 100644 --- a/scripts/check_tb_registry.mjs +++ b/scripts/check_tb_registry.mjs @@ -21,9 +21,23 @@ // survive; // * a path with a character outside every console code page, which is why // the module goes through .NET rather than reg.exe; +// * what the IDE itself does to the recent list while a run is on it: a +// short list's empty slots filled with copies of its last entry, a full +// list's oldest entries pushed off the end, and, to be kept, a project the +// user opened meanwhile and copies that were there before; and another +// run's entry that its own tidy removed, which must not come back; // * the association keys: a value changed, one added, one deleted, a subkey // added, and a key that did not exist before created by the run; // * that a second restore writes nothing at all; +// * the build targets the IDE remembers: the entries under a harness temp +// folder, in both separators, deleted; the user's, the lookalike folder's +// and the rest kept, in their order and in the IDE's own JSON; a value +// that is not JSON left alone; and a named project's target that the run +// switched, saved under another spelling of its path, put back in its +// place, with an entry for a project that had none deleted; +// * an association that named the temp folder when the run began, which is +// another run's IDE copy's and is left alone, against one that did not, +// which is put back; // * the guards: a key near the root and a sweep outside the temp folder // refused, and an error raised inside PowerShell arriving as a sentence; // * the ownership rule: a dead owner does not block tidying, a live one makes @@ -46,10 +60,10 @@ const ABSENT = BASE + "\\Classes\\.notthere"; function ps(script, input) { const enc = Buffer.from("$ErrorActionPreference='Stop';$ProgressPreference='SilentlyContinue';" + "$u=New-Object System.Text.UTF8Encoding $false;[Console]::InputEncoding=$u;" + - "$in=[Console]::In.ReadToEnd()|ConvertFrom-Json;" + + "[Console]::OutputEncoding=$u;$in=[Console]::In.ReadToEnd()|ConvertFrom-Json;" + "$hk=[Microsoft.Win32.Registry]::CurrentUser;" + script, "utf16le").toString("base64"); - execFileSync("powershell", ["-NoProfile", "-NonInteractive", "-EncodedCommand", enc], - { input: JSON.stringify(input ?? {}), stdio: ["pipe", "ignore", "inherit"] }); + return execFileSync("powershell", ["-NoProfile", "-NonInteractive", "-EncodedCommand", enc], + { input: JSON.stringify(input ?? {}), encoding: "utf8", stdio: ["pipe", "pipe", "inherit"] }); } const wipe = () => ps("if ($hk.OpenSubKey($in.base)) { $hk.DeleteSubKeyTree($in.base) }", { base: BASE }); // Pairs rather than an object: PowerShell 5.1's ConvertFrom-Json refuses an @@ -61,6 +75,9 @@ const setValues = (key, values) => ps( const deleteValues = (key, names) => ps( "$k=$hk.OpenSubKey($in.key, $true); foreach ($n in $in.names) { $k.DeleteValue($n) }; $k.Close()", { key, names }); +const readValue = (key, name) => JSON.parse(ps( + "$k=$hk.OpenSubKey($in.key); $v=$k.GetValue($in.name); $k.Close(); " + + "ConvertTo-Json -InputObject $v -Compress", { key, name }).replace(/^/, "")); const USER = "D:\\work\\Real Project\\Mine.twinproj"; const OTHER = "D:\\work\\Other\\Other.twinproj"; @@ -125,6 +142,101 @@ try { assert.deepEqual(R.restoreProjects(snap, { prefixes: [TEMPDIR] }), { projectState: 0, recentlyOpened: 0 }); assert.equal(R.restoreKeys(keys), 0); + // ------------------------------------------------ what the IDE does to the recent list + // Each case: the list as found, the list after the run, the list put back. + const PROBE1 = TEMPDIR + "\\p1.twinproj", PROBE2 = TEMPDIR + "\\p2.twinproj"; + const OTHERRUN = path.join(tmpdir(), "tbharness-selftest-Łukasz", "tbrun", "9999", "o.twinproj"); + const full = Array.from({ length: 21 }, (_, i) => `D:\\full\\p${i}.twinproj`); + const recentCases = [ + ["a short list's empty slots filled with copies of its last entry", + ["D:\\x.twinproj"], [PROBE2, PROBE1, ...Array(19).fill("D:\\x.twinproj")], ["D:\\x.twinproj"]], + ["a full list's oldest entries pushed off the end", + full, [PROBE2, PROBE1, ...full.slice(0, 19)], full], + ["a project the user opened meanwhile kept on top, and moved one kept where it went", + ["D:\\a.twinproj", "D:\\b.twinproj", "D:\\c.twinproj"], + [PROBE1, "D:\\new.twinproj", "D:\\c.twinproj", "D:\\a.twinproj", "D:\\b.twinproj", + ...Array(16).fill("D:\\b.twinproj")], + ["D:\\new.twinproj", "D:\\c.twinproj", "D:\\a.twinproj", "D:\\b.twinproj"]], + ["copies that were there before kept, as many as there were", + ["D:\\a.twinproj", "D:\\b.twinproj", "D:\\b.twinproj"], + [PROBE1, "D:\\a.twinproj", ...Array(19).fill("D:\\b.twinproj")], + ["D:\\a.twinproj", "D:\\b.twinproj", "D:\\b.twinproj"]], + ["another run's entry that its own tidy removed meanwhile not brought back", + ["D:\\a.twinproj", OTHERRUN, "D:\\b.twinproj"], [PROBE1, "D:\\a.twinproj", "D:\\b.twinproj"], + ["D:\\a.twinproj", "D:\\b.twinproj"]], + ]; + for (const [what, found, afterRun, expected] of recentCases) { + setValues(ROOT + "\\RecentlyOpened", slots(found)); + const s = R.snapshotProjects([], { root: ROOT }); + setValues(ROOT + "\\RecentlyOpened", slots(afterRun)); + R.restoreProjects(s, { prefixes: [TEMPDIR] }); + assert.deepEqual(R.ideLists({ root: ROOT }).recentlyOpened, Object.values(slots(expected)), what); + assert.equal(R.restoreProjects(s, { prefixes: [TEMPDIR] }).recentlyOpened, 0, `${what}: idempotent`); + } + // A sweep with no list in hand, as startTidy makes first, only deletes. + setValues(ROOT + "\\RecentlyOpened", slots([PROBE1, "D:\\x.twinproj", "D:\\x.twinproj"])); + R.restoreProjects({ root: ROOT, entries: [] }, { prefixes: [TEMPDIR] }); + assert.deepEqual(R.ideLists({ root: ROOT }).recentlyOpened, + Object.values(slots(["D:\\x.twinproj", "D:\\x.twinproj"])), "a sweep with no snapshot only deletes"); + + // ------------------------------------------------ remembered build targets + const SETTINGS = ROOT + "\\IDESettings"; + const MEMORY = "targetArchitectureMemory"; + const PROBE = TEMPDIR + "\\tbrun-probe.twinproj"; + const PROBE_FWD = TEMPDIR.split("\\").join("/") + "/src/x.twinproj"; + const memory = { [USER]: "win64", [PROBE]: "win64", [LOOKALIKE]: "win64", [PROBE_FWD]: "win32", + [OTHER]: "win32" }; + setValues(SETTINGS, { [MEMORY]: JSON.stringify(memory) }); + assert.equal(R.sweepArchitectureMemory([TEMPDIR], { root: ROOT }), 2, + "both of the run's entries are deleted, whichever separator they use"); + assert.equal(readValue(SETTINGS, MEMORY), + JSON.stringify({ [USER]: "win64", [LOOKALIKE]: "win64", [OTHER]: "win32" }), + "every other entry is kept, in its order, written as the IDE writes it"); + assert.equal(R.sweepArchitectureMemory([TEMPDIR], { root: ROOT }), 0, "a second sweep deletes nothing"); + setValues(SETTINGS, { [MEMORY]: "{not json" }); + assert.equal(R.sweepArchitectureMemory([TEMPDIR], { root: ROOT }), 0); + assert.equal(readValue(SETTINGS, MEMORY), "{not json", "a value that is not JSON is left alone"); + deleteValues(SETTINGS, [MEMORY]); + assert.equal(R.sweepArchitectureMemory([TEMPDIR], { root: ROOT }), 0, "no value, nothing to do"); + assert.throws(() => R.sweepArchitectureMemory(["C:\\"], { root: ROOT }), /outside/); + + // A named project's target, which a run switches: tbbuild --arch on the + // user's own project. The IDE saves the switch under the path as it was + // given, which need not be spelled as the user's IDE spelled it. + const kept = { [OTHER]: "win32", [USER]: "win64", "D:\\z.twinproj": "win64" }; + setValues(SETTINGS, { [MEMORY]: JSON.stringify(kept) }); + const targets = R.snapshotArchitectureMemory([USER, NEWPROJ], { root: ROOT }); + setValues(SETTINGS, { [MEMORY]: JSON.stringify( + { ...kept, [USER]: "win32", [NEWPROJ]: "win64", [USER.toLowerCase()]: "win32" }) }); + assert.equal(R.restoreArchitectureMemory(targets), 3, + "the user's entry gets its value back; the run's other spelling of it, and its new project's, go"); + assert.equal(readValue(SETTINGS, MEMORY), JSON.stringify(kept), "every entry as it was, in its order"); + assert.equal(R.restoreArchitectureMemory(targets), 0, "a second restore writes nothing"); + setValues(SETTINGS, { [MEMORY]: JSON.stringify({ [OTHER]: "win32", "D:\\z.twinproj": "win64" }) }); + assert.equal(R.restoreArchitectureMemory(targets), 1); + assert.deepEqual(JSON.parse(readValue(SETTINGS, MEMORY)), kept, "an entry the run deleted comes back"); + // ...and the same through the whole tidy. + setValues(SETTINGS, { [MEMORY]: JSON.stringify(kept) }); + const named = R.startTidy({ root: ROOT, keys: [ASSOC], paths: [USER] }); + setValues(SETTINGS, { [MEMORY]: JSON.stringify({ ...kept, [USER]: "win32" }) }); + assert.equal(R.finishTidy(named).architecture, 1); + assert.equal(readValue(SETTINGS, MEMORY), JSON.stringify(kept), "finishTidy puts a named project's target back"); + + // ------------------------------------------------ an association another run's copy held + // startTidy and finishTidy, the whole tidy, on the scratch keys. + const COMMAND = ASSOC + "\\shell\\open\\command"; + const REAL = "\"C:\\IDE\\twinBASIC.exe\" \"%1\""; + const COPY = `"${path.join(tmpdir(), "tbaddin", "9870", "ide", "twinBASIC.exe")}" "%1"`; + setValues(COMMAND, { "": COPY }); // another run's copy has it + const dirty = R.startTidy({ root: ROOT, keys: [ASSOC] }); + setValues(COMMAND, { "": REAL }); // an IDE from a real install takes it back + assert.equal(R.finishTidy(dirty).association, null); + assert.equal(readValue(COMMAND, ""), REAL, "an association naming the temp folder is never put back"); + const clean = R.startTidy({ root: ROOT, keys: [ASSOC] }); + setValues(COMMAND, { "": COPY }); // this run's copy takes it + assert.ok(R.finishTidy(clean).association >= 1); + assert.equal(readValue(COMMAND, ""), REAL, "one that did not is put back"); + // ------------------------------------------------ the guards assert.throws(() => R.restoreKeys([{ path: "Software", snap: null }]), /close to the root/); assert.throws(() => R.snapshotKeys(["Software\\Classes"]), /close to the root/); diff --git a/scripts/lib/tb-addin.mjs b/scripts/lib/tb-addin.mjs new file mode 100644 index 00000000..17cb2481 --- /dev/null +++ b/scripts/lib/tb-addin.mjs @@ -0,0 +1,122 @@ +// Building a twinBASIC IDE add-in for a test, which WIP.HelpAddin.md (Stage 1, +// item 4) calls "build, then load". +// +// An add-in is a Standard DLL, and the compiler loads every DLL in its +// install's addins\win32 or addins\win64 folder as it starts. A test builds the +// add-in with the lane's private copy of the IDE (tb-ide-copy.mjs), ends that +// IDE, puts the DLL into the copy's addins folder with addAddin, and starts the +// copy again on the project it tests with. That IDE's compiler loads the add-in +// as it starts, and loadedAddins in tb-ide.mjs asks it which add-ins it loaded. +// One project per IDE, as everywhere in this harness. +// +// The DLL is built into the lane's work folder, not straight into the addins +// folder that the shipped add-in samples' own buildPath names. The IDE that +// builds is the lane's copy too, so on a rebuild its compiler would have the +// previous build loaded from that very folder while the linker tried to +// replace it. +// +// win32 only. A project path the IDE has no memory of opens in its first +// build target, win32, whose compiler is twinBASIC_win32_noDEP.exe and loads +// add-ins from addins\win32; the registry tidy deletes any memory the lane's +// paths have. A win64 project gets twinBASIC_win64_noDEP.exe, opened that way +// or switched to (setBuildTarget in tb-ide.mjs); which add-in folder a +// switched compiler loads is the rest of P7. + +import { mkdirSync, readFileSync } from "node:fs"; +import path from "node:path"; +import { compilerExe } from "./tb-install.mjs"; +import { attachIde, buildProject, compileOutcome, launchIde, normPath, shutdownIde, + summaryLine, waitForCompile } from "./tb-ide.mjs"; +import { laneProjectId, stageProject } from "./tb-project.mjs"; + +// An error carrying tbbuild's exit codes: 1 the project has compile errors, +// 2 the harness failed, 3 the compile never settled, 4 the compiler crashed. +function failure(exitCode, message) { + return Object.assign(new Error(message), { exitCode }); +} + +/** + * Build an add-in from its exported source tree. + * + * The run's registry tidy is the caller's: hold a startTidy({ prefixes: [work] }) + * from lib/tb-registry.mjs around this and the IDEs that follow it. + * + * @param {object} o + * @param {string} o.ide the twinBASIC.exe to build with: the lane's copy + * @param {string} o.src the add-in's exported tree, holding Settings and Sources + * @param {string} o.work the lane's work folder; the staged tree, the + * .twinproj and out\.dll go in it + * @param {number} o.port the lane's DevTools port + * @param {string} [o.arch] the build target; "win32" is the only one yet + * @param {boolean} [o.show] on the user's desktop instead of a private one + * @param {number} [o.timeout] milliseconds for the compile to settle, and again + * for the build (default 180000) + * @returns {Promise<{dll: string, arch: string, diagnostics: string[], log: string[]}>} + * `diagnostics` holds the warnings, hints and infos; `log` is the build log + * @throws an Error with an `exitCode` (see failure above); a compile error's + * message lists every diagnostic + */ +export async function buildAddin({ ide, src, work, port, arch = "win32", show = false, + timeout = 180 * 1000 }) { + if (arch !== "win32") { + throw failure(2, `cannot build an add-in for ${arch} yet: which compiler loads a ${arch} ` + + "add-in is P7 in WIP.HelpAddin.md"); + } + const project = path.join(work, "addin.twinproj"); + mkdirSync(path.join(work, "out"), { recursive: true }); + let dll; + let staged; + try { + staged = stageProject({ + src, stage: path.join(work, "addin-src"), project, compiler: compilerExe(ide), + settings: (original) => { + dll = path.join(work, "out", `${original["project.name"]}.dll`); + return { "project.buildPath": dll, "project.id": laneProjectId(1, port) }; + }, + }); + } catch (e) { + throw failure(2, e.message); + } + const type = staged.original["project.buildType"]; + if (type !== "Standard DLL") { + throw failure(2, `${src} builds a ${type ?? "project of no stated type"}, and an add-in ` + + "is a Standard DLL"); + } + + const run = await launchIde({ exe: ide, project, port, show }); + let built; + try { + const c = await attachIde(port); + if (!c) throw failure(2, "the IDE never exposed a debug port"); + try { + const outcome = compileOutcome(await waitForCompile(c, { project, timeout }), { name: project }); + if (!outcome.ok) throw failure(outcome.code, outcome.message); + if (outcome.counts[0] > 0) { + throw failure(1, [...outcome.rows, summaryLine(outcome.counts)].join("\n")); + } + const target = await c.evaluate( + "typeof buildConfigSelector === 'undefined' ? null : buildConfigSelector.value"); + if (target !== arch) { + throw failure(2, `the IDE opened the add-in to build for ${target}, not ${arch}. It ` + + "remembers a target for each project path, in IDESettings' targetArchitectureMemory; " + + `the run's registry tidy deletes the entries under ${work}.`); + } + built = await buildProject(c, { timeout }); + built.diagnostics = outcome.rows; + } finally { + c.close(); + } + } finally { + shutdownIde(run); + } + + if (!built.ok) throw failure(2, `the add-in did not build: ${built.message}\n${built.log.join("\n")}`); + if (normPath(built.file) !== normPath(dll)) { + throw failure(2, `the linker created ${built.file}, not ${dll}`); + } + // The linker's word, checked against the file: a DLL starts with "MZ". + let head = ""; + try { head = readFileSync(dll).subarray(0, 2).toString("latin1"); } catch { /* reported below */ } + if (head !== "MZ") throw failure(2, `the linker reported ${dll}, but it is missing or not a DLL`); + return { dll, arch, diagnostics: built.diagnostics, log: built.log }; +} diff --git a/scripts/lib/tb-cdp.mjs b/scripts/lib/tb-cdp.mjs index f36635e4..eed95aad 100644 --- a/scripts/lib/tb-cdp.mjs +++ b/scripts/lib/tb-cdp.mjs @@ -12,16 +12,31 @@ /** * Attach to a DevTools page target. * + * Every call has a time limit. A page blocked by a javascript dialog, or inside + * a synchronous host call, never answers `Runtime.evaluate` or an input event, + * and without a limit the caller waits forever, with nothing on screen to say + * why. A call that runs out rejects with a sentence saying so; a connection + * that closes rejects everything still waiting on it at once. + * * @param {number} port the --remote-debugging-port the target was started with * @param {string} match substring the target URL must contain + * @param {object} [o] + * @param {number} [o.timeout] milliseconds a call may take (default 30000); one + * call can pass its own */ -export async function attach(port, match = "main.htm") { - const list = await (await fetch(`http://127.0.0.1:${port}/json/list`)).json(); +export async function attach(port, match = "main.htm", { timeout = 30 * 1000 } = {}) { + const list = await (await fetch(`http://127.0.0.1:${port}/json/list`, + { signal: AbortSignal.timeout(10 * 1000) })).json(); const t = list.find((x) => x.type === "page" && x.url.includes(match)); if (!t) throw new Error(`no page target matching ${JSON.stringify(match)} on port ${port}`); const ws = new WebSocket(t.webSocketDebuggerUrl); - await new Promise((res, rej) => { ws.onopen = res; ws.onerror = rej; }); + await new Promise((res, rej) => { + const timer = setTimeout(() => rej(new Error(`the DevTools socket on port ${port} did not open`)), + 10 * 1000); + ws.onopen = () => { clearTimeout(timer); res(); }; + ws.onerror = (e) => { clearTimeout(timer); rej(e); }; + }); let id = 0; const pending = new Map(); @@ -36,16 +51,36 @@ export async function attach(port, match = "main.htm") { for (const l of listeners) l(m); } }; + ws.onclose = () => { + for (const { rej } of pending.values()) rej(new Error("the DevTools connection closed")); + pending.clear(); + }; - const send = (method, params = {}) => + const send = (method, params = {}, { timeout: ms = timeout } = {}) => new Promise((res, rej) => { const i = ++id; - pending.set(i, { res, rej }); - ws.send(JSON.stringify({ id: i, method, params })); + const timer = setTimeout(() => { + pending.delete(i); + rej(new Error(`${method} had no answer in ${ms / 1000} s -- the page may be blocked ` + + "by a javascript dialog or a synchronous host call")); + }, ms); + timer.unref?.(); + pending.set(i, { + res: (v) => { clearTimeout(timer); res(v); }, + rej: (e) => { clearTimeout(timer); rej(e); }, + }); + try { + ws.send(JSON.stringify({ id: i, method, params })); + } catch { + pending.delete(i); + clearTimeout(timer); + rej(new Error("the DevTools connection is closed")); + } }); - const evaluate = async (expression, { awaitPromise = false } = {}) => { - const r = await send("Runtime.evaluate", { expression, returnByValue: true, awaitPromise }); + const evaluate = async (expression, { awaitPromise = false, timeout: ms = timeout } = {}) => { + const r = await send("Runtime.evaluate", { expression, returnByValue: true, awaitPromise }, + { timeout: ms }); if (r.exceptionDetails) { throw new Error(r.exceptionDetails.exception?.description ?? JSON.stringify(r.exceptionDetails)); diff --git a/scripts/lib/tb-ide-copy.mjs b/scripts/lib/tb-ide-copy.mjs index 7ab933cc..98d0ac23 100644 --- a/scripts/lib/tb-ide-copy.mjs +++ b/scripts/lib/tb-ide-copy.mjs @@ -80,6 +80,46 @@ export function makeIdeCopy({ ide, dest, addins = {} }) { return path.join(root, "twinBASIC.exe"); } +/** + * Put an add-in DLL into a copy's addins\ folder, replacing one of the + * same name, so that the copy's next IDE loads it. + * + * Refuses anything that is not a copy made by makeIdeCopy: in the real install + * a test add-in would load into every IDE the user starts. End the copy's IDEs + * first. A compiler holds every add-in it loaded: Windows refuses to overwrite + * or delete the file while it runs (WIP.HelpAddin.md, P8), and for a moment + * after -- the first overwrite after shutdownIde failed and one 25 ms later + * worked, four times out of four -- so a refused copy is retried for two + * seconds before it counts. The refusal comes back as EIO from the copy here + * and as EBUSY from a plain write. + * + * @param {string} exe the copy's twinBASIC.exe, as makeIdeCopy returned it + * @param {string} dll the add-in + * @param {"win32" | "win64"} arch + * @returns {string} where the DLL now is + */ +export function addAddin(exe, dll, arch) { + const root = path.dirname(path.resolve(exe)); + if (!insideTemp(root) || !existsSync(path.join(root, MARKER))) { + throw new Error(`refusing to add an add-in to "${root}": it is not an IDE copy this module made`); + } + if (arch !== "win32" && arch !== "win64") throw new Error(`no such add-in folder: "${arch}"`); + const dest = path.join(root, "addins", arch, path.basename(dll)); + const cell = new Int32Array(new SharedArrayBuffer(4)); + for (let tries = 1; ; tries++) { + try { + cpSync(dll, dest); + return dest; + } catch (e) { + if (!["EIO", "EBUSY", "EPERM"].includes(e.code) || tries >= 20) { + throw new Error(`could not put the add-in in "${dest}" (${e.code}) -- ` + + "is an IDE started from this copy still running?"); + } + Atomics.wait(cell, 0, 0, 100); + } + } +} + /** * Delete a copy made by makeIdeCopy. Refuses anything without its marker, and * anything outside the temp folder. End the copy's IDEs first: a running IDE @@ -92,8 +132,32 @@ export function removeIdeCopy(exe) { if (!insideTemp(root) || !existsSync(path.join(root, MARKER))) { throw new Error(`refusing to delete "${root}": it is not an IDE copy this module made`); } - // Retries, because an IDE ended a moment ago can still be letting go of its - // files. Anything still holding them after that is a leaked process, and the - // EPERM is the right thing to report. - rmSync(root, { recursive: true, force: true, maxRetries: 10, retryDelay: 200 }); + removeTree(root); +} + +/** + * Delete a folder and everything in it, retrying for up to `timeout` + * milliseconds while something still holds a file there. + * + * An IDE ended a moment ago can still be letting go of its files: its compiler + * held a loaded add-in some tens of milliseconds after the process had gone + * (WIP.HelpAddin.md, P8). rmSync's own maxRetries does not cover that. On + * Node 24.13 it gave up at once, in a millisecond, with EPERM on a folder + * holding a file another process had open, with maxRetries 10 and retryDelay + * 200 exactly as with neither (measured). So the retrying is done here. + * Anything still holding a file after that is a process that outlived its + * IDE, and the EPERM is the right thing to report. + */ +export function removeTree(dir, { timeout = 5000 } = {}) { + const until = Date.now() + timeout; + const cell = new Int32Array(new SharedArrayBuffer(4)); + for (;;) { + try { + rmSync(dir, { recursive: true, force: true }); + return; + } catch (e) { + if (!["EPERM", "EBUSY", "ENOTEMPTY", "EACCES"].includes(e.code) || Date.now() >= until) throw e; + Atomics.wait(cell, 0, 0, 100); + } + } } diff --git a/scripts/lib/tb-ide.mjs b/scripts/lib/tb-ide.mjs index 22bcabfd..830e7226 100644 --- a/scripts/lib/tb-ide.mjs +++ b/scripts/lib/tb-ide.mjs @@ -1,6 +1,7 @@ -// Starting a twinBASIC IDE, reaching it over CDP, reading what it shows, and -// ending it. The mechanics scripts/tbbuild.mjs and scripts/tbrun.mjs share, and -// the ones the add-in harness in WIP.HelpAddin.md (Stage 1) is built on. +// Starting a twinBASIC IDE, reaching it over CDP, reading what it shows, +// building the project it has open, and ending it. The mechanics +// scripts/tbbuild.mjs and scripts/tbrun.mjs share, and the ones the add-in +// harness in WIP.HelpAddin.md (Stage 1) is built on. // // Each function here was once inline in one of those two scripts, and the // comments that explain it moved with it. See WIP.Harness.md, "Compiling a @@ -8,6 +9,7 @@ import { execFileSync, spawn } from "node:child_process"; import { readFileSync } from "node:fs"; +import net from "node:net"; import path from "node:path"; import { fileURLToPath } from "node:url"; import { attach } from "./tb-cdp.mjs"; @@ -17,6 +19,21 @@ export const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); // The IDE echoes projectFilePath back with whichever separators it was given. export const normPath = (p) => p.split("\\").join("/").toLowerCase(); +/** + * The environment variable that tells an add-in it is under test. While it is + * set, an add-in does nothing outside the IDE: it prints each such action to + * the DEBUG CONSOLE instead, as `open ` for a URL it would have opened in + * a browser (WIP.HelpAddin.md, Stage 1 item 6). A browser started from an IDE + * on a private desktop would open where nobody can see it and outlive the run. + * + * launchIde sets it to "1" for every IDE the harness starts, tbbuild's and + * tbrun's included: any of them can load an add-in the user has installed, + * and none has anybody watching it. Measured on BETA 983 (P10): it reaches the + * compiler, twinBASIC_win32_noDEP.exe, which the IDE starts as its own child, + * and every add-in that compiler loads, including after the compiler restarts. + */ +export const ADDIN_TEST_ENV = "TB_ADDIN_TEST"; + /** * Whether the IDE goes on the user's desktop or on a private one. * @@ -74,18 +91,41 @@ export function killTree(pid) { * folder and the private desktop's name * @param {boolean} [o.show] on the user's desktop instead of a private one * @param {boolean} [o.keep] the IDE is to outlive this Node process - * @param {object} [o.env] extra environment for the IDE + * A port something already listens on is refused. The harness attaches to + * whatever page answers on its port, so if another IDE already holds it, that + * IDE is the one the harness would read and operate --- and other sessions on + * the same machine run this harness too, on ports of their own choosing. + * An IDE ended a moment ago + * holds its port a little longer than it lives: 13 and 16 ms after + * shutdownIde, and once two seconds. So the port gets ten seconds to come + * free before the launch is refused. + * + * @param {object} [o.env] extra environment for the IDE. ADDIN_TEST_ENV is + * set to "1" unless this names it; a value of + * undefined leaves a variable out altogether * @returns {Promise<{pid: number, launcher: import("node:child_process").ChildProcess | null}>} - * `pid` is the IDE's own. Throws when a hidden launch fails. + * `pid` is the IDE's own. Throws when the port is taken, or when a hidden + * launch fails. */ export async function launchIde({ exe, project, port, show = false, keep = false, env = {} }) { + const waitFrom = Date.now(); + while (await portTaken(port)) { + if (Date.now() - waitFrom > 10 * 1000) { + throw new Error(`DevTools port ${port} is in use: another IDE has it, perhaps one another ` + + "session started, and this one could be mistaken for it. Pass a different --port."); + } + await sleep(100); + } const exeWin = exe.split("/").join("\\"); const target = path.resolve(project).split("/").join("\\"); + // Node leaves out of a child's environment any variable whose value is + // undefined, so `env` can remove one as well as set it. const fullEnv = { ...process.env, WEBVIEW2_ADDITIONAL_BROWSER_ARGUMENTS: `--remote-debugging-port=${port} --remote-allow-origins=*`, WEBVIEW2_USER_DATA_FOLDER: `${process.env.TEMP}/tbbuild-wv2-${port}`, + [ADDIN_TEST_ENV]: "1", ...env, }; @@ -184,15 +224,56 @@ export function waitForExit(pid, timeoutMs) { return false; } -/** Attach to the IDE's page once its DevTools port answers; null if it never does. */ +/** + * Attach to the IDE's page once its DevTools port answers; null if it never does. + * + * The connection records and dismisses every javascript dialog the page opens, + * in `c.dialogs` as `{type, message, at}`. The IDE calls `alert()` from 37 + * places in BETA 983, and never `confirm()` or `prompt()`. An alert blocks the + * page until it is answered, which on a private desktop nobody can do: every + * later call on the connection would time out. An alert is accepted, since it has only the + * one button; a confirm or a prompt, which only an add-in could open, is + * cancelled. CDP reports dialogs only once `Page.enable` has been sent, and + * until this function sent it, `tbbuild`'s list of dialogs could never fill. + * + * **A dialog that opened before this attached cannot be answered.** Measured: + * `Page.enable` got no answer while it was open, the new connection was told + * of no dialog, and `Page.handleJavaScriptDialog` replied "No dialog is + * showing" while the page stayed blocked. The IDE's own candidates are its + * "IDE startup failure" alert and "Bad command line syntax.", which + * launchIde's single argument never provokes. Such a page is marked + * `c.pageBlocked`, and waitForCompile passes that on, so the failure names the + * likely cause; `--show` puts the dialog where a person can read it. + */ export async function attachIde(port, { tries = 60 } = {}) { for (let i = 0; i < tries; i++) { await sleep(1000); - try { return await attach(port); } catch { /* still starting */ } + let c; + try { c = await attach(port); } catch { continue; } // still starting + c.dialogs = []; + c.pageBlocked = false; + c.on((m) => { + if (m.method !== "Page.javascriptDialogOpening") return; + const { type, message } = m.params; + c.dialogs.push({ type, message, at: Date.now() }); + c.send("Page.handleJavaScriptDialog", { accept: type === "alert" || type === "beforeunload" }) + .catch(() => { /* already answered, or the page is gone */ }); + }); + try { await c.send("Page.enable"); } catch { c.pageBlocked = true; } + return c; } return null; } +// Whether something already listens on a loopback port. A DevTools server +// binds 127.0.0.1, so binding it ourselves for a moment is the test. +const portTaken = (port) => new Promise((resolve) => { + const s = net.createServer(); + s.once("error", () => resolve(true)); + s.once("listening", () => s.close(() => resolve(false))); + s.listen(port, "127.0.0.1"); +}); + // Counts and rows are read in ONE evaluate. Read separately they raced: a run // reported two diagnostics beside a zero error count, because the background // compile finished between the two calls. @@ -209,14 +290,7 @@ export async function attachIde(port, { tries = 60 } = {}) { // crash-restart cycle takes about 1.3 s and the loop samples at 1 Hz, so // `drops` never reaches its threshold. The console is the only record that // cannot be missed by sampling: nothing removes an entry from it. -const BUILD_STATE_JS = `JSON.stringify({ - st: document.getElementById("compilerStatus")?.textContent ?? "", - e: document.getElementById("errorCount")?.textContent ?? "", - w: document.getElementById("warningCount")?.textContent ?? "", - h: document.getElementById("hintCount")?.textContent ?? "", - i: document.getElementById("infoCount")?.textContent ?? "", - p: typeof projectFilePath !== "undefined" ? projectFilePath : null, - crash: (() => { +const CRASH_JS = `(() => { if (typeof debugConsoleContent === "undefined" || !debugConsoleContent || !debugConsoleContent.dataNodes) return null; let n = 0; const files = []; @@ -226,7 +300,16 @@ const BUILD_STATE_JS = `JSON.stringify({ if (m && files.indexOf(m[1]) < 0) files.push(m[1]); } return n ? { n: n, files: files } : null; - })(), + })()`; + +const BUILD_STATE_JS = `JSON.stringify({ + st: document.getElementById("compilerStatus")?.textContent ?? "", + e: document.getElementById("errorCount")?.textContent ?? "", + w: document.getElementById("warningCount")?.textContent ?? "", + h: document.getElementById("hintCount")?.textContent ?? "", + i: document.getElementById("infoCount")?.textContent ?? "", + p: typeof projectFilePath !== "undefined" ? projectFilePath : null, + crash: ${CRASH_JS}, rows: (() => { const out = []; if (typeof problemsPanel === "undefined" || !problemsPanel) return out; @@ -287,6 +370,43 @@ const BUILD_STATE_JS = `JSON.stringify({ /** One sample of the status bar, the diagnostics and the crash record, as a JSON string. */ export const readBuildState = (c) => c.evaluate(BUILD_STATE_JS); +/** + * Whether the compiler has crashed since the DEBUG CONSOLE was last cleared: + * null, or `{ n, files }` --- how many "NATIVE EXCEPTION" entries, and the files + * the thread dumps name as being parsed. An add-in runs inside the compiler's + * process, so a crash ends whatever it was doing, and a scenario that saw one + * has not tested what it meant to. + */ +export const readCrash = (c) => c.evaluate(CRASH_JS); + +/** + * Wait for a crash record to name the file being parsed, and return it as it + * then stands: named, or as it was if no name came within `timeout` ms. + * + * The IDE's FIRST exception entry carries no thread dump, and the dump is what + * names the file. Only a compiler started in TRACE-MODE writes one, and the + * IDE switches that on in answer to the first exception, so the name arrives + * with the SECOND crash, once the restarted compiler reaches the same file. + * Reporting the crash without that name is a correct result nobody can act on. + * + * Poll, not a fixed sleep. Against the crash fixture the second crash came 1.6 + * to 1.9 s after the first on an idle machine, but up to 3.0 s with four IDEs + * compiling at once, as check_examples runs them; replayed over those runs, + * one re-read after a fixed 2 s missed the name 31% of the time. Five seconds + * covers the slowest one measured with 2 s to spare. + * + * @param {object} c a tb-cdp connection + * @param {{n: number, files: string[]}} crash a record readCrash returned + */ +export async function awaitCrashName(c, crash, { timeout = 5000 } = {}) { + const until = Date.now() + timeout; + while (!crash.files?.length && Date.now() < until) { + await sleep(250); + try { crash = (await readCrash(c)) ?? crash; } catch { /* keep what we have */ } + } + return crash; +} + /** * Wait for the project to open and its compile to settle. * @@ -304,8 +424,10 @@ export const readBuildState = (c) => c.evaluate(BUILD_STATE_JS); * given: compared as typed, a relative path never matched, and the IDE was * never reported open. * @param {number} o.timeout milliseconds - * @returns {Promise<{loaded: boolean, crash: object | null, drops: number, last: string | null}>} - * `last` is the final sample, as the JSON string readBuildState returned + * @returns {Promise<{loaded: boolean, crash: object | null, drops: number, last: string | null, + * blocked: boolean}>} + * `last` is the final sample, as the JSON string readBuildState returned; + * `blocked` is attachIde's `pageBlocked` */ export async function waitForCompile(c, { project, timeout }) { const want = normPath(path.resolve(project)); @@ -318,16 +440,9 @@ export async function waitForCompile(c, { project, timeout }) { const v = JSON.parse(s); if (!loaded) { if (v.p && normPath(v.p) === want) loaded = true; else continue; } if (v.crash) { - // The IDE's FIRST exception line carries no thread dump; the file being - // parsed is only named in the dump that comes with the restart about a - // second later. Catching the crash on sight and reporting it without that - // name is a correct result nobody can act on, so give the IDE one more - // moment and take whatever it has then. - crash = v.crash; - if (!crash.files?.length) { - await sleep(2000); - try { crash = JSON.parse(await readBuildState(c)).crash ?? crash; } catch { /* keep what we have */ } - } + // Caught on sight, but reported with the file the compiler died parsing, + // which only a later crash names -- see awaitCrashName. + crash = await awaitCrashName(c, v.crash); break; } const up = v.st === "tB Services: OPERATIONAL"; @@ -335,7 +450,7 @@ export async function waitForCompile(c, { project, timeout }) { if (up && s === last) { if (++stable >= 5) break; } else stable = 0; last = s; } - return { loaded, crash, drops, last }; + return { loaded, crash, drops, last, blocked: !!c.pageBlocked }; } /** @@ -348,7 +463,7 @@ export async function waitForCompile(c, { project, timeout }) { * `counts` is errors, warnings, hints, infos. `code` is 4 for a compiler * crash and 3 for a compile that never settled. */ -export function compileOutcome({ loaded, crash, drops, last }, { name }) { +export function compileOutcome({ loaded, crash, drops, last, blocked }, { name }) { // A crash is reported by the file the compiler died parsing, because in a batch // of generated probes that name is the whole answer: it says which sample to // take out, and a caller bisecting the batch has somewhere to start. @@ -366,7 +481,15 @@ export function compileOutcome({ loaded, crash, drops, last }, { name }) { message: `the compiler restarted ${drops}x -- this project crashes it\nlast status: ${last}`, }; } - if (!loaded) return { ok: false, code: 3, message: `the IDE never reported ${name} as open` }; + if (!loaded) { + return { + ok: false, code: 3, + message: `the IDE never reported ${name} as open` + (blocked + ? "\nIts page did not answer when the harness attached. A dialog it opened before then " + + "is the likely cause, and one cannot be answered over CDP; --show puts it on screen." + : ""), + }; + } const final = JSON.parse(last ?? "{}"); const rows = (final.rows ?? []).map((r) => @@ -397,6 +520,74 @@ export function compileOutcome({ loaded, crash, drops, last }, { name }) { export const summaryLine = (counts) => `--- ${counts[0]} error(s), ${counts[1]} warning(s), ${counts[2]} hint(s), ${counts[3]} info`; +/** The build targets the toolbar's build configuration box offers, besides safe mode. */ +export const TARGETS = ["win32", "win64"]; + +// The box, and the pid of the compiler the page is talking to: switching the +// target restarts the compiler, and a new pid is how to tell that it has. +const TARGET_JS = `(() => { + if (typeof buildConfigSelector === "undefined" || !buildConfigSelector) return null; + return JSON.stringify({ + value: buildConfigSelector.value, + options: Array.from(buildConfigSelector.options, (o) => o.value), + pid: typeof g_CurrentCompilerProcessId === "undefined" ? null : g_CurrentCompilerProcessId, + }); +})()`; + +/** + * Make the project's build target `arch`, win32 or win64, and wait for the + * compile under it to settle. + * + * A project opens in the target the IDE remembers for its path + * (IDESettings\targetArchitectureMemory, which lib/tb-registry.mjs tidies), + * and in win32, the box's first option, when it remembers none. So a caller + * that means a target sets it, even win32: otherwise an entry somebody left + * decides the build, and nothing says so. + * + * The switch is the IDE's own tbBuild_SwitchToWin64 and tbBuild_SwitchToWin32 + * commands: set the box, call its onchange. That handler, + * changedActiveBuildConfig, saves the target for the project's path and calls + * restartCompilerSafely, which kills the compiler; the new one is the target's + * own, twinBASIC_win64_noDEP.exe for win64, and it compiles the project again. + * Measured on BETA 983: the status bar stays OPERATIONAL for about 250 ms + * after the switch, and is down for about a second after that until the new + * compiler's pid appears. waitForCompile started straight away can count that + * as the compiler going down twice and report a crash, so the restart is waited + * for by the pid, not by the clock. + * + * @param {object} c a tb-cdp connection + * @param {string} arch "win32" or "win64" + * @param {object} o as for waitForCompile + * @returns {Promise<{from: string, waited: object | null}>} `from` is the target + * the project opened in; `waited` is waitForCompile's result for the compile + * under `arch`, or null when the project was in `arch` already + */ +export async function setBuildTarget(c, arch, { project, timeout }) { + const read = async () => JSON.parse((await c.evaluate(TARGET_JS)) ?? "null"); + const before = await read(); + if (!before) throw new Error("this IDE has no build configuration box (buildConfigSelector)"); + if (!before.options.includes(arch)) { + throw new Error(`the build configuration box offers ${before.options.join(", ")}, not ${arch}`); + } + if (before.value === arch) return { from: before.value, waited: null }; + if (!before.pid) throw new Error("the IDE has no compiler process to restart for another target"); + await c.evaluate(`buildConfigSelector.value = ${JSON.stringify(arch)}; buildConfigSelector.onchange()`); + const t0 = Date.now(); + for (;;) { + await sleep(250); + let now = null; + try { now = await read(); } catch { /* the page is busy with the restart */ } + if (now?.pid && now.pid !== before.pid) { + if (now.value !== arch) throw new Error(`the build target went back to ${now.value} after the switch`); + break; + } + if (Date.now() - t0 > 60 * 1000) { + throw new Error(`the compiler did not restart within 60 s of switching the build target to ${arch}`); + } + } + return { from: before.value, waited: await waitForCompile(c, { project, timeout }) }; +} + // Read the DEBUG CONSOLE's BACKING ARRAY, never the pane. `debugConsoleContent` // is a createListView(), which renders only the rows that fit -- so an // `.innerText` scrape returned the last ~11 lines of any longer probe and gave @@ -426,32 +617,152 @@ export const summaryLine = (counts) => // none, and the IDE's `substr(i + 7)` would then quietly eat six // characters of real output. No current addItem() path omits it; the guard // costs a comparison and removes a silent-corruption mode. -const consoleJs = (withTimestamps) => `(() => { +// +// An entry's text is stored escaped: an add-in's PrintText of "©=1" +// is stored as "<b>&copy=1" (measured, BETA 983), so decoding gives +// back exactly what was printed, markup and ampersands included. Except for +// the IDE's own bug (BUGS-TO-REPORT.md): text that continues a line left open +// by `Debug.Print ...;` is escaped twice, so the console shows "&" for +// "&", and so does this reader. It returns what the console shows. +// +// A mark (consoleMark) is checked in the same evaluate as the read, so a +// clear cannot fall between the two. The entry that was last when the mark +// was taken is read again as well, because the IDE can still add to it. All +// output from the compiler's process, a program's Debug.Print and an add-in's +// PrintText alike, goes through debugOutputPartial, which appends to the last +// entry in place while its line is open; output ending in a line break closes +// the line, and so does the IDE's own debugOutputLine, which starts a new +// entry. So what was appended comes first, as a line of its own, and then the +// entries after it. Reading on from the count alone missed that text +// (measured). +const consoleJs = (withTimestamps, from, mark) => `(() => { if (typeof debugConsoleContent === "undefined" || !debugConsoleContent || !debugConsoleContent.dataNodes) return null; + const nodes = debugConsoleContent.dataNodes; + const mark = ${JSON.stringify(mark ?? null)}; + const start = !mark ? ${from} + : nodes.length < mark.n || (mark.n > 0 && nodes[0] !== mark.first) ? 0 : mark.n; const decode = (html) => { const d = document.createElement("div"); // never attached; see above d.innerHTML = html; return d.textContent; }; - return debugConsoleContent.dataNodes.map(n => { + const text = (n) => { const i = n.indexOf(""); if (i < 0) return decode(n); return decode(${withTimestamps} ? n.substr(0, i + 7) + " " + n.substr(i + 7) : n.substr(i + 7)); - }).join("\\n"); + }; + const lines = nodes.slice(start).map(text); + // Appended text, if the last entry at the mark has grown since. Closing an + // open line adds only markup, so an unchanged text adds no line. + if (mark && "last" in mark && start === mark.n && mark.n > 0 && nodes[mark.n - 1] !== mark.last) { + const was = mark.last === null ? "" : text(mark.last), now = text(nodes[mark.n - 1]); + const added = now.startsWith(was) ? now.slice(was.length) : now; + if (added) lines.unshift(added); + } + return lines.join("\\n"); })()`; /** - * The whole DEBUG CONSOLE as text, one line per entry; null when this IDE has - * no `debugConsoleContent.dataNodes` to read. + * The DEBUG CONSOLE as text, one line per entry; null when this IDE has no + * `debugConsoleContent.dataNodes` to read. * * @param {object} c a tb-cdp connection * @param {object} [o] * @param {boolean} [o.timestamps] keep each entry's timestamp column + * @param {number} [o.from] start at this entry instead of the first + * @param {object} [o.since] a mark from consoleMark: only what was + * written after it --- text appended to the + * entry that was last then, and the entries + * after it --- or every entry if the + * console was cleared since. Wins over `from`. */ -export const readConsole = (c, { timestamps = false } = {}) => c.evaluate(consoleJs(timestamps)); +export const readConsole = (c, { timestamps = false, from = 0, since = null } = {}) => + c.evaluate(consoleJs(timestamps, Number(from), since)); + +// Where the console stands: how many entries it holds, and its first and last +// entries as stored, timestamp and all. Nothing removes an entry but a clear, +// so entries read later from index `n` on are new -- unless the first entry +// has changed or the count has fallen, which means the console was cleared in +// between and all of it is new. The last entry is kept because the IDE may +// still append to it (consoleJs says when). +const CONSOLE_MARK_JS = `(() => { + const d = typeof debugConsoleContent === "undefined" || !debugConsoleContent + ? null : debugConsoleContent.dataNodes; + return d ? { n: d.length, first: d.length ? d[0] : null, + last: d.length ? d[d.length - 1] : null } : null; +})()`; + +/** + * Where the DEBUG CONSOLE stands now, so that `readConsole(c, { since })` can + * later return only what was written after it. Null when this IDE has no + * console to read. + */ +export const consoleMark = (c) => c.evaluate(CONSOLE_MARK_JS); + +// The build log, in the compiler's own words (its strings, BETA 983). A build +// writes "[BUILD] Starting..." to the DEBUG CONSOLE, and a binary ends with +// "[LINKER] SUCCESS created output file ''" or with one of some twenty +// failure lines: "[LINKER] FAILED ...", "[BUILD] FAILED ...", "[BUILD] ERROR +// ...", "[BUILD] failed" and "[LINKER] compilation (codegen) error ...". +const BUILD_START = "[BUILD] Starting..."; +const BUILD_OK = /^\[LINKER\] SUCCESS created output file '(.+)'$/; +const BUILD_FAILED = /^\[(?:LINKER|BUILD)\] (?:FAILED|ERROR|failed)\b|^\[LINKER\] compilation \(codegen\) error/; + +/** + * Build the open project, as the toolbar's Build button does, and wait for the + * build log to say how it went. + * + * Only a binary is recognised, an EXE or a DLL, whose log ends with the + * linker's SUCCESS line. What a package build writes has not been looked at, + * and one would end in the timeout. + * + * @param {object} c a tb-cdp connection + * @param {object} [o] + * @param {number} [o.timeout] milliseconds (default 120000) + * @returns {Promise<{ok: boolean, file?: string, message?: string, log: string[]}>} + * `file` is the path the linker says it created; `log` is the console from + * the build's first line on + */ +export async function buildProject(c, { timeout = 120 * 1000 } = {}) { + const mark = await consoleMark(c); + if (!mark) { + return { ok: false, log: [], message: "no debugConsoleContent.dataNodes in this IDE, " + + "so the build log cannot be read" }; + } + if (!await clickCenter(c, "buildIcon")) { + return { ok: false, log: [], message: "no #buildIcon in the IDE page -- did the project load?" }; + } + const t0 = Date.now(); + let log = [], failedAt = 0; + while (Date.now() - t0 < timeout) { + await sleep(250); + const text = await readConsole(c, { since: mark }); + const lines = text ? text.split("\n").map((l) => l.trim()) : []; + const start = lines.indexOf(BUILD_START); + if (start < 0) continue; + log = lines.slice(start); + const ok = log.map((l) => BUILD_OK.exec(l)).find(Boolean); + if (ok) return { ok: true, file: ok[1], log }; + // Not every such line need end the build -- the strings include "[BUILD] + // failed to use project.iconForm setting", and whether a build goes on + // after that one has not been seen -- so one decides only after two + // seconds with no success line after it. + if (!failedAt && log.some((l) => BUILD_FAILED.test(l))) failedAt = Date.now(); + if (failedAt && Date.now() - failedAt > 2000) { + return { ok: false, message: log.find((l) => BUILD_FAILED.test(l)), log }; + } + } + return { + ok: false, log, + message: log.length + ? `the build started and reported nothing for ${timeout / 1000} s` + : `the build did not start in ${timeout / 1000} s -- is a dialog open? A template ` + + "buildPath opens a Save dialog, which the private desktop hides", + }; +} /** * The add-ins the IDE's compiler has loaded, as the Add-Ins menu lists them: @@ -461,6 +772,11 @@ export const readConsole = (c, { timestamps = false } = {}) => c.evaluate(consol * so this is the compiler's own answer rather than anything inferred from * files on disk. Add-ins load as the compiler starts, so ask once the project * has opened. + * + * A DLL the compiler found but could not load is listed too, as + * "Unknown Addin", so look for the name you expect rather than counting. The + * DEBUG CONSOLE says what went wrong, in a line that starts with the file's + * name in brackets: "[x.dll] Failed to load addin. LoadLibrary() failed." */ export const loadedAddins = (c) => c.evaluate( "new Promise((resolve) => root.getAddinsList(resolve))", { awaitPromise: true }); diff --git a/scripts/lib/tb-lane.mjs b/scripts/lib/tb-lane.mjs new file mode 100644 index 00000000..658d4bc2 --- /dev/null +++ b/scripts/lib/tb-lane.mjs @@ -0,0 +1,197 @@ +// One lane of the add-in test runner, as the scenario file running in it sees +// it. WIP.HelpAddin.md, Stage 1 item 7. +// +// scripts/addin_test.mjs runs every scenario file in a node:test process of +// its own, and hands it its lane in TB_ADDIN_LANE: a DevTools port, a work +// folder inside the temp folder, and the install to copy. The lane makes its +// own copy of that install (tb-ide-copy.mjs), builds the add-ins its scenarios +// test into the copy's addins\win32, and opens projects in the copy, one IDE at +// a time. The runner owns the registry for the whole run and puts it back once +// every lane has ended; a lane never tidies. +// +// A scenario file reads, in outline: +// +// const lane = addinLane(); +// describe("...", { skip: lane ? false : "run it with addin-test.bat" }, () => { +// let c; +// before(async () => { +// await lane.addSample("Sample 15"); +// c = await lane.open(HOST); +// }); +// after(() => lane.close()); +// test("...", async () => { ...tb-operate.mjs calls on c... }); +// }); +// +// Outside the runner addinLane() returns null and the suite is skipped, so a +// bare `node --test` never starts an IDE. + +import { existsSync, mkdirSync, readdirSync, rmSync } from "node:fs"; +import path from "node:path"; +import { buildAddin } from "./tb-addin.mjs"; +import { addAddin, makeIdeCopy, removeIdeCopy } from "./tb-ide-copy.mjs"; +import { attachIde, awaitCrashName, compileOutcome, launchIde, readCrash, shutdownIde, + summaryLine, waitForCompile } from "./tb-ide.mjs"; +import { compilerExe, runCompiler } from "./tb-install.mjs"; +import { laneProjectId, stageProject } from "./tb-project.mjs"; + +/** The environment variable the runner hands a lane to its scenario file in. */ +export const LANE_ENV = "TB_ADDIN_LANE"; + +/** The lane this process runs in, or null outside the runner. */ +export function addinLane() { + const raw = process.env[LANE_ENV]; + return raw ? new Lane(JSON.parse(raw)) : null; +} + +const winPath = (p) => path.resolve(p).split("/").join("\\"); + +export class Lane { + /** + * @param {object} o + * @param {string} o.name the lane's name, for messages + * @param {number} o.port its DevTools port + * @param {string} o.work its work folder, inside the temp folder + * @param {string} o.ide the install's twinBASIC.exe, which the lane copies + * @param {boolean} [o.show] IDEs on the user's desktop instead of a private one + */ + constructor({ name, port, work, ide, show = false }) { + Object.assign(this, { name, port, work, ide, show }); + this.exe = null; // the lane's copy of the install, made on first use + this.run = null; // the open IDE, from launchIde + this.c = null; // the connection to it + this.builds = 0; + } + + // The lane's copy of the install, made the first time anything needs it. A + // lane never starts the install itself: its compiler would load the add-ins + // in the install's own addins folders. + copy() { + if (!this.exe) this.exe = makeIdeCopy({ ide: this.ide, dest: path.join(this.work, "ide") }); + return this.exe; + } + + /** + * Export one of the install's sample projects and return the exported tree. + * The install is only read. The name is the folder's up to its dot: "Sample + * 15" is "Sample 15. twinBASIC IDE Addin (GlobalSearch)", and never + * "Sample 1a." or "Sample 150.". + */ + exportSample(sample) { + const projects = path.join(path.dirname(path.resolve(this.ide)), "projects"); + const dirs = readdirSync(projects).filter((d) => d.startsWith(`${sample}.`)); + if (dirs.length !== 1) { + throw new Error(`${projects} has ${dirs.length} folders named "${sample}.", where one was expected`); + } + const dir = path.join(projects, dirs[0]); + const files = readdirSync(dir).filter((f) => /\.twinproj$/i.test(f)); + if (files.length !== 1) throw new Error(`${dir} holds ${files.length} .twinproj files, not one`); + const out = path.join(this.work, "samples", sample.replace(/[^A-Za-z0-9]+/g, "-")); + rmSync(out, { recursive: true, force: true }); + mkdirSync(out, { recursive: true }); + // export wants backslashes, a full path to the project and a trailing + // separator on the folder (WIP.md, Driving the twinBASIC compiler). + const r = runCompiler(compilerExe(this.copy()), + ["export", winPath(path.join(dir, files[0])), `${winPath(out)}\\`, "--overwrite"]); + if (!r.done) throw new Error(`exporting ${sample} failed${r.why}:\n${r.tail}`); + return out; + } + + /** + * Build an add-in from its exported tree and put it in the copy's + * addins\win32, so that every IDE the lane opens after this loads it. + * + * @returns {Promise<{dll: string, arch: string, diagnostics: string[], log: string[]}>} + * buildAddin's result; it throws, with an exitCode, when the add-in does not build + */ + async addAddin(src) { + if (this.run) throw new Error(`lane ${this.name}: close the open project before building an add-in`); + const exe = this.copy(); + const built = await buildAddin({ ide: exe, src, work: path.join(this.work, `addin${++this.builds}`), + port: this.port, show: this.show }); + addAddin(exe, built.dll, "win32"); + return built; + } + + /** exportSample, then addAddin. */ + addSample(sample) { + return this.addAddin(this.exportSample(sample)); + } + + /** + * Open a project in the lane's copy and wait for its compile to settle: the + * tree is staged and packed as tbrun stages a probe (tb-project.mjs), with + * its build path pinned inside the work folder, and started on the lane's + * port. Refuses a project that does not compile, since a scenario on it + * would be testing something else. + * + * @param {string} src an exported tree: the folder holding Settings and Sources + * @param {object} [o] + * @param {number} [o.timeout] milliseconds for the compile to settle (default 180000) + * @returns {Promise} the connection (attachIde's), which the tb-operate.mjs calls take + */ + async open(src, { timeout = 180 * 1000 } = {}) { + if (this.run) throw new Error(`lane ${this.name} has a project open already: one IDE at a time`); + const exe = this.copy(); + const project = path.join(this.work, "project.twinproj"); + stageProject({ + src, stage: path.join(this.work, "project-src"), project, compiler: compilerExe(exe), + settings: (original) => ({ + "project.buildPath": path.join(this.work, "out", `${original["project.name"]}.exe`), + "project.id": laneProjectId(2, this.port), + }), + }); + this.run = await launchIde({ exe, project, port: this.port, show: this.show }); + this.c = await attachIde(this.port); + if (!this.c) throw new Error(`lane ${this.name}: the IDE never exposed a debug port`); + const outcome = compileOutcome(await waitForCompile(this.c, { project, timeout }), { name: project }); + if (!outcome.ok) throw new Error(`lane ${this.name}: ${outcome.message}`); + if (outcome.counts[0] > 0) { + throw new Error(`lane ${this.name}: ${src} does not compile\n` + + [...outcome.rows, summaryLine(outcome.counts)].join("\n")); + } + return this.c; + } + + /** + * End the open IDE, if there is one. Throws afterwards when the compiler + * crashed while it ran, since an add-in runs inside the compiler and a + * scenario that saw a crash has not tested what it meant to; and when a + * javascript dialog opened that the scenario did not take out of + * `c.dialogs`, since the IDE opens one only on an error path. + */ + async closeProject() { + const { c, run } = this; + this.c = this.run = null; + let crash = null, dialogs = []; + try { + if (c) { + // A crash the scenario ended soon after has not named its file yet: + // only a later crash does, so it gets the wait waitForCompile gives one. + crash = await readCrash(c).catch(() => null); + if (crash) crash = await awaitCrashName(c, crash); + dialogs = c.dialogs; + c.close(); + } + } finally { + shutdownIde(run); + } + const problems = []; + if (crash) problems.push(`the compiler crashed ${crash.n}x` + + (crash.files?.length ? `, parsing ${crash.files.join(", ")}` : "")); + if (dialogs.length) { + problems.push(`the IDE opened ${dialogs.length} javascript dialog(s): ` + + dialogs.map((d) => `${d.type} ${JSON.stringify(d.message)}`).join("; ")); + } + if (problems.length) throw new Error(`lane ${this.name}: ${problems.join("; ")}`); + } + + /** closeProject, then delete the lane's copy of the install. For after(). */ + async close() { + try { + await this.closeProject(); + } finally { + if (this.exe && existsSync(this.exe)) removeIdeCopy(this.exe); + this.exe = null; + } + } +} diff --git a/scripts/lib/tb-operate.mjs b/scripts/lib/tb-operate.mjs new file mode 100644 index 00000000..20c34075 --- /dev/null +++ b/scripts/lib/tb-operate.mjs @@ -0,0 +1,428 @@ +// Operating a running twinBASIC IDE the way a person does, and reading what it +// shows: the calls the add-in harness in WIP.HelpAddin.md (Stage 1, item 5) +// writes its scenarios with. Every call takes a connection from attachIde in +// tb-ide.mjs, which records and dismisses the page's javascript dialogs. +// +// Input goes in as real CDP input events, never as element.click() or a +// synthetic DOM event. The IDE's own controls ignore a JavaScript click +// (tbrun learned that on #buildIcon), and an add-in's shortcut is matched on +// the real key-down and key-up pair (main.js, document.onkeyup), so anything +// less than what a keyboard sends would test something the IDE never sees. +// +// Reading, on the other hand, goes straight to the page's own data where the +// view would lie: the DEBUG CONSOLE is a virtual list (readConsole in +// tb-ide.mjs says why), and a tool window lives in a shadow root that +// document.querySelector cannot see into. + +import { readConsole, sleep } from "./tb-ide.mjs"; + +// ------------------------------------------------------------------ finding + +// A target is an element id in the main document ("buildIcon", +// "addinButton-"), or an object: +// +// { css } the elements a selector finds in the main document +// { css, toolWindow } the same inside a tool window, named by the id its +// add-in gave ToolWindows.Add +// ..., text only those whose text, trimmed, is exactly this +// ..., last: true the last of them rather than the first +// +// Of the elements found, the first one on screen is the target, then the first +// at all. A list view keeps rows it drew before (its getCachedDomNode), and +// Sample 15's results held one file's entry twice while showing it once, so +// the first element a selector finds is not always the one a person sees. +// `last` is for stacked dialogs, where the one on top came last. +// The expression evaluates to the element or null. +function targetJs(target) { + if (typeof target === "string") return `document.getElementById(${JSON.stringify(target)})`; + const scope = target.toolWindow + ? `(() => { const w = typeof toolWindowsById === "undefined" ? null + : toolWindowsById[${JSON.stringify(target.toolWindow)}]; + return w && w.bodyElement ? w.bodyElement : null; })()` + : "document"; + return `(() => { + const scope = ${scope}; + if (!scope) return null; + let found = [...scope.querySelectorAll(${JSON.stringify(target.css)})]; + const text = ${JSON.stringify(target.text ?? null)}; + if (text !== null) found = found.filter((e) => e.textContent.trim() === text); + if (${!!target.last}) found.reverse(); + const shown = found.find((e) => { const r = e.getBoundingClientRect(); return r.width && r.height; }); + return shown || found[0] || null; + })()`; +} + +// A target as a message names it. +function named(target) { + if (typeof target === "string") return `#${target}`; + return [target.css, target.text !== undefined ? `with the text ${JSON.stringify(target.text)}` : "", + target.toolWindow ? `in tool window ${target.toolWindow}` : ""].filter(Boolean).join(" "); +} + +/** + * Where an element is on screen, in the viewport coordinates input events + * use; null when there is no such element or it has no size, which is what a + * hidden tool window's elements have. + */ +export async function elementRect(c, target) { + return c.evaluate(`(() => { + const e = ${targetJs(target)}; + if (!e) return null; + const r = e.getBoundingClientRect(); + return r.width && r.height ? { x: r.x, y: r.y, width: r.width, height: r.height } : null; + })()`); +} + +// ------------------------------------------------------------------ mouse + +/** A real left click at a point: the pointer moves there, presses and releases. */ +export async function clickAt(c, x, y, { clickCount = 1 } = {}) { + await c.send("Input.dispatchMouseEvent", { type: "mouseMoved", x, y }); + for (const type of ["mousePressed", "mouseReleased"]) { + await c.send("Input.dispatchMouseEvent", { type, x, y, button: "left", clickCount }); + } +} + +/** + * Click the centre of a target (see targetJs), as a person would: scrolled into + * view first, and only if the target is what is actually at that point. + * + * Both were learned on Sample 10, whose tool window is taller than it is + * shown: its eleventh button had a size and a place, but that place was under + * the window's own bottom edge, and the click went to the window's resize + * handle and did nothing. The element under the point is found through every + * shadow root, since a tool window is one. + * + * Waits up to `timeout` milliseconds for the target to be there, have a size + * and be uncovered, because what an add-in adds is drawn a moment after it is + * in the page's data: a list view that already held Sample 15's results had + * not yet drawn their rows when a click came under a millisecond later. + * + * Throws, naming the target, when it is still not clickable after that: there + * is no such element, it has no size (it is in a hidden tool window, say), or + * something else covers its centre. + * + * @param {object} [o] + * @param {number} [o.timeout] milliseconds to wait (default 5000) + * @param {number} [o.clickCount] 2 for a double click + */ +export async function click(c, target, { timeout = 5000, clickCount = 1 } = {}) { + const until = Date.now() + timeout; + for (;;) { + const p = await c.evaluate(`(() => { + const e = ${targetJs(target)}; + if (!e) return { error: "there is no such element" }; + e.scrollIntoView({ block: "center", inline: "center" }); + const r = e.getBoundingClientRect(); + if (!r.width || !r.height) return { error: "it has no size; is it in a hidden tool window?" }; + const x = r.x + r.width / 2, y = r.y + r.height / 2; + let hit = document.elementFromPoint(x, y); + while (hit && hit.shadowRoot) { + const inner = hit.shadowRoot.elementFromPoint(x, y); + if (!inner || inner === hit) break; + hit = inner; + } + if (hit && (hit === e || e.contains(hit))) return { x, y }; + const what = !hit ? "nothing" : hit.id ? "#" + hit.id + : hit.tagName.toLowerCase() + (hit.className ? "." + String(hit.className).trim().split(/\\s+/).join(".") : ""); + return { error: "its centre is covered by " + what }; + })()`); + if (!p.error) return clickAt(c, p.x, p.y, { clickCount }); + if (Date.now() >= until) throw new Error(`cannot click ${named(target)}: ${p.error}`); + await sleep(100); + } +} + +// ------------------------------------------------------------------ keyboard + +// What a US keyboard sends for each key: DOM key, DOM code and Windows virtual +// key code. The IDE names an add-in shortcut's letters from `code` and every +// other key from `key` (WIP.HelpAddin.md, Keyboard shortcuts), so both must +// be what a real keyboard gives. +const NAMED = { + Enter: ["Enter", 13, "\r"], Tab: ["Tab", 9, "\t"], Backspace: ["Backspace", 8], Escape: ["Escape", 27], + Delete: ["Delete", 46], Home: ["Home", 36], End: ["End", 35], PageUp: ["PageUp", 33], + PageDown: ["PageDown", 34], ArrowLeft: ["ArrowLeft", 37], ArrowUp: ["ArrowUp", 38], + ArrowRight: ["ArrowRight", 39], ArrowDown: ["ArrowDown", 40], Insert: ["Insert", 45], +}; +const PUNCTUATION = { // unshifted, shifted, code, virtual key + " ": [" ", " ", "Space", 32], "`": ["`", "~", "Backquote", 192], "-": ["-", "_", "Minus", 189], + "=": ["=", "+", "Equal", 187], "[": ["[", "{", "BracketLeft", 219], "]": ["]", "}", "BracketRight", 221], + "\\": ["\\", "|", "Backslash", 220], ";": [";", ":", "Semicolon", 186], "'": ["'", "\"", "Quote", 222], + ",": [",", "<", "Comma", 188], ".": [".", ">", "Period", 190], "/": ["/", "?", "Slash", 191], +}; +const DIGIT_SHIFTED = ")!@#$%^&*("; +const MODIFIERS = [ // name, key, code, virtual key, CDP modifier bit; pressed in this order + ["ctrl", "Control", "ControlLeft", 17, 2], ["shift", "Shift", "ShiftLeft", 16, 8], + ["alt", "Alt", "AltLeft", 18, 1], +]; + +// The key a character comes from, and whether it takes Shift. +function keyFor(name) { + if (NAMED[name]) { + const [code, vk, text] = NAMED[name]; + return { key: name, code, vk, text, shift: false }; + } + const f = /^F([1-9]|1[0-2])$/.exec(name); + if (f) return { key: name, code: name, vk: 111 + Number(f[1]), shift: false }; + if (name.length !== 1) throw new Error(`no such key: ${JSON.stringify(name)}`); + if (/[a-z]/.test(name)) return { key: name, code: `Key${name.toUpperCase()}`, vk: name.toUpperCase().charCodeAt(0), text: name, shift: false }; + if (/[A-Z]/.test(name)) return { key: name, code: `Key${name}`, vk: name.charCodeAt(0), text: name, shift: true }; + if (/[0-9]/.test(name)) return { key: name, code: `Digit${name}`, vk: name.charCodeAt(0), text: name, shift: false }; + const d = DIGIT_SHIFTED.indexOf(name); + if (d >= 0) return { key: name, code: `Digit${d}`, vk: 48 + d, text: name, shift: true }; + for (const [plain, shifted, code, vk] of Object.values(PUNCTUATION)) { + if (name === plain) return { key: name, code, vk, text: name, shift: false }; + if (name === shifted) return { key: name, code, vk, text: name, shift: true }; + } + throw new Error(`no key on a US keyboard types ${JSON.stringify(name)}`); +} + +/** + * Press and release one key, as a keyboard does: each modifier goes down, + * then the key goes down and up, then the modifiers come up in reverse. The + * whole press takes a few milliseconds, well inside the 500 ms the IDE allows + * between an add-in shortcut's key-down and key-up. + * + * @param {string} name a character ("d", "D", "!"), "F1" to "F12", or a + * named key: Enter, Tab, Backspace, Escape, Delete, + * Home, End, PageUp, PageDown, Insert and the arrows + * @param {object} [mods] { ctrl, shift, alt }; Shift is added by itself for a + * character that needs it + */ +export async function pressKey(c, name, { ctrl = false, shift = false, alt = false } = {}) { + const k = keyFor(name); + const held = MODIFIERS.filter(([m]) => ({ ctrl, shift: shift || k.shift, alt })[m]); + let bits = 0; + for (const [, key, code, vk, bit] of held) { + bits |= bit; + await c.send("Input.dispatchKeyEvent", { type: "rawKeyDown", key, code, windowsVirtualKeyCode: vk, + nativeVirtualKeyCode: vk, modifiers: bits }); + } + // A key that types something sends its text with the key-down, which is + // what makes it appear in an input; Ctrl or Alt held means a command, not text. + const text = k.text !== undefined && !ctrl && !alt ? k.text : undefined; + const base = { key: k.key, code: k.code, windowsVirtualKeyCode: k.vk, nativeVirtualKeyCode: k.vk, + modifiers: bits }; + await c.send("Input.dispatchKeyEvent", text !== undefined + ? { ...base, type: "keyDown", text, unmodifiedText: text } + : { ...base, type: "rawKeyDown" }); + await c.send("Input.dispatchKeyEvent", { ...base, type: "keyUp" }); + for (const [, key, code, vk, bit] of held.slice().reverse()) { + bits &= ~bit; + await c.send("Input.dispatchKeyEvent", { type: "keyUp", key, code, windowsVirtualKeyCode: vk, + nativeVirtualKeyCode: vk, modifiers: bits }); + } +} + +/** + * Type text into whatever has the focus, one key press per character, so that + * every key-down and key-up reaches the page --- an add-in listening for + * "keyup", as Sample 15's search box does, sees each one. Line breaks are + * pressed as Enter. Only what a US keyboard types: any other character throws. + */ +export async function typeText(c, text, { delay = 0 } = {}) { + for (const ch of text.replace(/\r\n/g, "\n")) { + await pressKey(c, ch === "\n" ? "Enter" : ch); + if (delay) await sleep(delay); + } +} + +// ------------------------------------------------------------------ tool windows + +const TOOL_WINDOWS_JS = `(() => { + if (typeof toolWindowsById === "undefined") return []; + return Object.entries(toolWindowsById).map(([id, w]) => { + const host = w.shadowDom && w.shadowDom.host; + const r = host ? host.getBoundingClientRect() : null; + return { + id, title: w.titleElement ? w.titleElement.textContent : null, + visible: !!(r && r.width && r.height), + text: w.bodyElement ? w.bodyElement.innerText : "", + }; + }); +})()`; + +/** + * The add-ins' tool windows: `{ id, title, visible, text }` each, where `id` + * is the one the add-in gave ToolWindows.Add and `text` is the body's text as + * rendered. A tool window an add-in created but has not shown is in the list, + * with `visible` false: its elements exist and have no size. + */ +export const toolWindows = (c) => c.evaluate(TOOL_WINDOWS_JS); + +/** One tool window by its id, as toolWindows describes it, or null. */ +export async function toolWindow(c, id) { + return (await toolWindows(c)).find((w) => w.id === id) ?? null; +} + +// ------------------------------------------------------------------ list views + +/** + * Every item of a list view, from its data rather than its rows: `{ text, + * html }` each, in order. A list view (an add-in's "listview" element, or the + * DEBUG CONSOLE) draws only the rows that fit, so reading its rows returns + * part of a long list and looks complete. The target is the list view's + * element, `{ toolWindow, css: "#resultsList" }` for Sample 15's results; null + * when it is not a list view. + */ +export async function listViewItems(c, target) { + return c.evaluate(`(() => { + const e = ${targetJs(target)}; + const lv = e && e.listview; + if (!lv || !Array.isArray(lv.dataNodes)) return null; + const d = document.createElement("div"); // never attached, so text is not re-laid out + return lv.dataNodes.slice(0, lv.itemCount).map((html) => { + d.innerHTML = html; + return { text: d.textContent, html }; + }); + })()`); +} + +// ------------------------------------------------------------------ message boxes + +// Host.ShowMessageBox is drawn in the page, not by Windows: a +// .modalDialogContainer holding a .modalTitleBar (the title, then a close +// button), a .simpleMsgBox with the message, and a .msgBoxButton per button. +const MESSAGE_BOXES_JS = `(() => [...document.querySelectorAll(".modalDialogContainer")] + .filter((m) => { const r = m.getBoundingClientRect(); return r.width && r.height; }) + .map((m) => { + const bar = m.querySelector(".modalTitleBar"); + const body = m.querySelector(".simpleMsgBox"); + return { + title: bar ? [...bar.childNodes].filter((n) => n.nodeType === 3).map((n) => n.textContent) + .join("").trim() : null, + text: body ? body.innerText : m.innerText, + buttons: [...m.querySelectorAll(".msgBoxButton")].map((b) => b.textContent.trim()), + }; + }))()`; + +/** + * The message boxes open in the IDE, the one on top last: `{ title, text, + * buttons }` each. An add-in's Host.ShowMessageBox is one, and so are the + * IDE's own modal dialogs, whose `text` is then the whole dialog's. + */ +export const messageBoxes = (c) => c.evaluate(MESSAGE_BOXES_JS); + +/** Click a button of the message box on top, by its caption. */ +export const answerMessageBox = (c, caption) => + click(c, { css: ".modalDialogContainer .msgBoxButton", text: caption, last: true }); + +/** + * The notifications showing now, as their text. Host.ShowNotification draws + * one in a box of the page's own, one of three fixed ones, #msgBox1 to + * #msgBox3, each with its text in a .msgBoxText. + */ +export const notifications = (c) => c.evaluate(`[...document.querySelectorAll(".msgBoxText")] + .filter((e) => { const r = e.getBoundingClientRect(); return r.width && r.height; }) + .map((e) => e.innerText)`); + +// ------------------------------------------------------------------ side effects + +/** + * The URLs the add-ins asked to open, in order. Under the harness an add-in + * starts no browser: ADDIN_TEST_ENV (tb-ide.mjs) is set, and the add-in prints + * `open ` to the DEBUG CONSOLE instead. A URL holds no white space, so a + * line that only begins with the word is not taken for one. + * + * @param {object} [o] + * @param {object} [o.since] a mark from consoleMark in tb-ide.mjs: only what + * was printed after it + */ +export async function openedUrls(c, { since = null } = {}) { + const text = await readConsole(c, { since }); + return (text ?? "").split("\n").map((l) => /^open (\S+)$/.exec(l.trim())).filter(Boolean) + .map((m) => m[1]); +} + +// ------------------------------------------------------------------ the code editor + +// The IDE has one Monaco code editor, window.editor, and gives it the model of +// whichever file's tab is selected; openEditors.selectedEditorNode is that +// tab, and its name is the file's path in the project. +// monaco.editor.getEditors() is no substitute: it also returns the editors +// add-ins create. +const EDITOR_JS = `(() => { + const ed = typeof editor === "undefined" ? null : editor; + const m = ed && ed.getModel ? ed.getModel() : null; + if (!m) return null; + const tab = typeof openEditors === "undefined" ? null : openEditors.selectedEditorNode; + const p = ed.getPosition(), s = ed.getSelection(); + return { + file: tab && tab.type === "CodeEditor" ? tab.name : null, + line: p.lineNumber, column: p.column, + selection: { startLine: s.startLineNumber, startColumn: s.startColumn, + endLine: s.endLineNumber, endColumn: s.endColumn }, + selectedText: m.getValueInRange(s), lineText: m.getLineContent(p.lineNumber), + lines: m.getLineCount(), focused: ed.hasTextFocus(), + }; +})()`; + +/** + * Where the code editor stands: `{ file, line, column, selection, selectedText, + * lineText, lines, focused }`, lines and columns counted from 1 as the IDE + * shows them, `file` the path in the project ("//Sources/") or + * null when no code file's tab is selected. Null when there is no editor. + */ +export const editorState = (c) => c.evaluate(EDITOR_JS); + +/** The whole text of the file in the code editor. */ +export const editorText = (c) => c.evaluate( + "typeof editor !== 'undefined' && editor.getModel() ? editor.getModel().getValue() : null"); + +/** + * Open a file of the project in the code editor, with the cursor at a place, + * the way the IDE's own Find in Files results do it. The path is the file's in + * the project, "//Sources/", with or without "twinbasic:" in + * front. Throws when the project has no such file. + */ +export async function openFile(c, file, { line = 1, column = 1 } = {}) { + const uri = file.startsWith("twinbasic:") ? file : `twinbasic:${file}`; + const ok = await c.evaluate(`(() => { + const node = fs.tree.resolvePath(${JSON.stringify(uri)}); + if (!node) return false; + openEditors.openFile(node, false, false, false, ${Number(line)}, ${Number(column)}); + return true; + })()`); + if (!ok) throw new Error(`the project has no file ${file}`); +} + +/** Put the code editor's cursor at a line and column, and give it the focus. */ +export async function setCursor(c, line, column) { + await c.evaluate(`(() => { + const p = { lineNumber: ${Number(line)}, column: ${Number(column)} }; + editor.setPosition(p); + editor.revealPositionInCenter(p); + editor.focus(); + })()`); +} + +/** + * Select a range in the code editor and give it the focus; the cursor ends at + * the range's end. + */ +export async function select(c, { startLine, startColumn, endLine, endColumn }) { + await c.evaluate(`(() => { + editor.setSelection({ startLineNumber: ${Number(startLine)}, startColumn: ${Number(startColumn)}, + endLineNumber: ${Number(endLine)}, endColumn: ${Number(endColumn)} }); + editor.revealLineInCenter(${Number(endLine)}); + editor.focus(); + })()`); +} + +// ------------------------------------------------------------------ waiting + +/** + * Wait until a condition over the page holds: `check` is called with the + * connection until it returns something truthy, which is returned. Null when + * the time runs out. + */ +export async function waitFor(c, check, { timeout = 10 * 1000, interval = 200 } = {}) { + const until = Date.now() + timeout; + for (;;) { + const v = await check(c); + if (v) return v; + if (Date.now() >= until) return null; + await sleep(interval); + } +} diff --git a/scripts/lib/tb-project.mjs b/scripts/lib/tb-project.mjs new file mode 100644 index 00000000..e9f193d8 --- /dev/null +++ b/scripts/lib/tb-project.mjs @@ -0,0 +1,61 @@ +// Staging an exported twinBASIC source tree as a project the harness can build. +// +// The harness never builds the tree it is given. It copies the tree, changes +// settings in the copy, and packs the copy into a .twinproj with the compiler +// executable's `import` verb. Two of those changes matter to every caller: +// +// * project.buildPath becomes an explicit file. The default +// `${SourcePath}\Build\...` template makes the IDE open a native Save +// dialog on the build, and on the private desktop the harness uses that +// dialog is invisible and takes no input, so the build never happens -- +// and nothing says so, because the IDE's page stays responsive. +// * project.id becomes one of the harness's own (laneProjectId), so that two +// projects open at once never share one: two tbrun probes that did +// confused the IDE's recent list. +// +// Rewriting the caller's own tree would do both, but a harness that edits the +// project it was pointed at is one nobody trusts with a real project. + +import { cpSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { runCompiler } from "./tb-install.mjs"; + +/** + * Copy an exported source tree, change its settings, and pack it. + * + * @param {object} o + * @param {string} o.src the exported tree: the folder holding Settings and Sources + * @param {string} o.stage where the copy goes; anything there is replaced + * @param {string} o.project the .twinproj to write + * @param {string} o.compiler the compiler executable (tb-install's compilerExe) + * @param {object | ((original: object) => object)} [o.settings] settings to + * set in the copy, or a function from the tree's own settings to them + * @returns {{original: object, settings: object}} the tree's settings, and the copy's + */ +export function stageProject({ src, stage, project, compiler, settings = {} }) { + rmSync(stage, { recursive: true, force: true }); + cpSync(src, stage, { recursive: true }); + const file = path.join(stage, "Settings"); + const original = JSON.parse(readFileSync(file, "utf8")); + const staged = { ...original, ...(typeof settings === "function" ? settings(original) : settings) }; + writeFileSync(file, JSON.stringify(staged, null, "\t"), "utf8"); + + // import's exit code does not say whether it worked -- 0 on the failures it + // reports, 999 on a tree holding an embedded package -- so runCompiler reads + // the output instead. + const pack = runCompiler(compiler, ["import", project, stage, "--overwrite"]); + if (!pack.done) throw new Error(`packing failed${pack.why}:\n${pack.tail}`); + return { original, settings: staged }; +} + +/** + * A project id that belongs to the harness, one per role and lane. + * + * Every harness id starts 7B247, and the digit after that is the role: 0 is + * tbrun's probe, 1 an add-in being built, 2 the project an add-in test opens, + * and 4 and 5 are check_examples' template and batches, which it numbers + * itself. The last six hex digits are the lane's DevTools port, which is what + * already has to differ between runs going on at once. + */ +export const laneProjectId = (role, port) => + `{7B247${role}00-0000-4000-9000-7B247${role}${port.toString(16).padStart(6, "0")}}`; diff --git a/scripts/lib/tb-registry.mjs b/scripts/lib/tb-registry.mjs index 11e45bce..52593b14 100644 --- a/scripts/lib/tb-registry.mjs +++ b/scripts/lib/tb-registry.mjs @@ -20,8 +20,18 @@ // * Everything under a folder the harness owns (its own temp work folders) // is deleted by prefix, which also catches what an earlier run left when // it died before tidying. +// * The recent list keeps no more copies of a path than it had, and gets +// back the entries that fell off its end while the run's projects were on +// it (restoreProjects says how the IDE does both). // * The association keys are put back value by value, and only where they -// differ, so an untouched key is never written. +// differ, so an untouched key is never written --- unless they named the +// temp folder when the run began, because then they were another run's +// IDE copy's, and putting them back would point at a deleted folder. +// * The build target the IDE remembers for each project path is deleted for +// every path under those folders, before the run and after it +// (sweepArchitectureMemory says why), and a named project's is put back +// as it was, since a run can switch the target of the project it opens +// (restoreArchitectureMemory). // // ONE PROCESS OWNS THIS PER RUN. check_examples starts many tbbuild processes // at once; each snapshotting and restoring on its own would put back whichever @@ -109,10 +119,10 @@ function SnapProjects([string]$root, $paths) { } if ($ps) { $ps.Close() } if ($ro) { $ro.Close() } - return [ordered]@{ root = $root; entries = $entries } + return [ordered]@{ root = $root; entries = $entries; recent = @($recent) } } -function RestoreProjects($snap, $prefixes) { +function RestoreProjects($snap, $prefixes, [string]$temp) { $root = [string]$snap.root $entries = @($snap.entries) $exact = @{} @@ -162,6 +172,29 @@ function RestoreProjects($snap, $prefixes) { foreach ($e in $back) { $list.Insert([Math]::Min([int]$e.recentIndex, $list.Count), [string]$e.recentValue) } + # The IDE fills a short list's empty slots with copies of its last entry, + # and a full list drops its oldest entry for each project a run opens + # (BUGS-TO-REPORT.md). With the list as it was found in hand, keep no more + # copies of a path than it had, and put back at the end what fell off -- + # but not a path in the temp folder, which belongs to some run, and that + # run may have tidied it away since. + if (@($snap.PSObject.Properties.Name) -contains 'recent') { + $count = @{} + foreach ($v in @($snap.recent)) { $n = Norm ([string]$v); if ($n) { $count[$n] = 1 + [int]$count[$n] } } + $have = @{} + $kept = New-Object System.Collections.ArrayList + foreach ($v in $list) { + $n = Norm ([string]$v) + if ([int]$have[$n] -lt [Math]::Max(1, [int]$count[$n])) { [void]$kept.Add($v); $have[$n] = 1 + [int]$have[$n] } + } + $tmp = Norm $temp + foreach ($v in @($snap.recent)) { + $n = Norm ([string]$v) + if (-not $n -or ($tmp -and $n.StartsWith($tmp))) { continue } + if ([int]$have[$n] -lt [int]$count[$n]) { [void]$kept.Add([string]$v); $have[$n] = 1 + [int]$have[$n] } + } + $list = $kept + } $changed = $false for ($i = 0; $i -lt $slots.Count; $i++) { $v = '' @@ -250,12 +283,41 @@ function RestoreKey([string]$path, $snap) { foreach ($s in @($snap.keys)) { RestoreKey ($path + $SEP + [string]$s.name) $s.snap } } +function ReadValue([string]$path, [string]$name) { + $out = [ordered]@{ exists = $false; data = $null } + $k = $hk.OpenSubKey($path) + if (-not $k) { return $out } + if (@($k.GetValueNames()) -contains $name) { $out.exists = $true; $out.data = [string]$k.GetValue($name) } + $k.Close() + return $out +} + +# Written only if the value still holds what the caller read, so that a value +# an IDE saved in the meantime is never overwritten with an older copy. +function WriteValueIf([string]$path, [string]$name, [string]$expected, [string]$data) { + $k = $hk.OpenSubKey($path, $true) + if (-not $k) { return [ordered]@{ written = $false } } + $same = (@($k.GetValueNames()) -contains $name) -and ([string]$k.GetValue($name) -ceq $expected) + if ($same) { $k.SetValue($name, $data, $String) } + $k.Close() + return [ordered]@{ written = $same } +} + try { $req = [Console]::In.ReadToEnd() | ConvertFrom-Json switch ([string]$req.op) { 'lists' { $result = Lists ([string]$req.root) } + 'readValue' { $result = ReadValue ([string]$req.key) ([string]$req.name) } + 'subkeys' { + $result = @() + $k = $hk.OpenSubKey([string]$req.key) + if ($k) { $result = @($k.GetSubKeyNames() | Sort-Object); $k.Close() } + } + 'writeValueIf' { + $result = WriteValueIf ([string]$req.key) ([string]$req.name) ([string]$req.expected) ([string]$req.data) + } 'snapshotProjects' { $result = SnapProjects ([string]$req.root) $req.paths } - 'restoreProjects' { $result = RestoreProjects $req.snapshot $req.prefixes } + 'restoreProjects' { $result = RestoreProjects $req.snapshot $req.prefixes ([string]$req.temp) } 'snapshotKeys' { $result = @(foreach ($p in @($req.keys)) { [ordered]@{ path = [string]$p; snap = (SnapKey ([string]$p)) } }) } @@ -299,8 +361,9 @@ export function ideLists({ root = IDE_SETTINGS_KEY } = {}) { } /** - * Record what the IDE holds for these projects now, so restoreProjects can put - * it back. A project with no entry is recorded as having none. + * Record what the IDE holds for these projects now, and its whole recent list, + * so restoreProjects can put it back. A project with no entry is recorded as + * having none. */ export function snapshotProjects(paths = [], { root = IDE_SETTINGS_KEY } = {}) { return request({ op: "snapshotProjects", root, paths: paths.map((p) => path.resolve(p)) }); @@ -310,13 +373,30 @@ export function snapshotProjects(paths = [], { root = IDE_SETTINGS_KEY } = {}) { * Put the snapshot's projects back as they were, and delete every entry under * the given folders. * + * The recent list is put back as the snapshot has it, with whatever else + * happened to it meanwhile kept: a project the user opened stays on top, but + * no path keeps more copies than the snapshot had, and the snapshot's entries + * that fell off the end come back there. Both are the IDE's doing. It fills a + * short list's empty slots with copies of its last entry, so a run that + * started on one entry ended with seventeen copies of it; and a full list + * drops its oldest entry for every project a run opens (BUGS-TO-REPORT.md). + * An entry in the temp folder is not brought back, since it belongs to some + * run, whose own tidy may have removed it meanwhile. A snapshot with no + * `recent`, such as `{ root, entries: [] }`, only deletes. + * * @param {string[]} [o.prefixes] folders inside the OS temp folder; anything * else is refused, so that a caller passing the wrong folder cannot sweep * away the state of the user's real projects * @returns {{projectState: number, recentlyOpened: number}} values written or deleted */ export function restoreProjects(snapshot, { prefixes = [] } = {}) { - return request({ op: "restoreProjects", snapshot, prefixes: prefixes.map(asTempFolder) }); + return request({ op: "restoreProjects", snapshot, prefixes: prefixes.map(asTempFolder), + temp: path.resolve(tmpdir()) + path.sep }); +} + +/** The names of a key's subkeys, sorted, and nothing else from under it; empty when there is no such key. */ +export function subkeyNames(key) { + return [].concat(request({ op: "subkeys", key }) ?? []); } /** Everything under these keys, values and subkeys, for restoreKeys. */ @@ -331,6 +411,158 @@ export function restoreKeys(snapshot) { return request({ op: "restoreKeys", snapshot }).changes; } +/** Where SaveSetting keeps every application's settings, an add-in's included. */ +export const SETTINGS_ROOT = "Software\\VB and VBA Program Settings"; + +/** + * The key SaveSetting writes an application's settings under. An add-in's + * SaveSetting writes here too, so its settings are shared with any installed + * copy of the same add-in, and a test that changes one changes the user's. + * + * The IDE's own application name is refused. That key holds all of the IDE's + * settings, and the parts of it a run changes are put back value by value by + * startTidy and finishTidy, never as a whole. + */ +export function settingsKey(app) { + const name = String(app ?? ""); + if (!name || /[\\/]/.test(name)) throw new Error(`not a SaveSetting application name: "${name}"`); + if (name.toLowerCase() === "twinbasic_ide") { + throw new Error("refusing the IDE's own settings key: the run's tidy puts back the parts of " + + "it the IDE changes, value by value"); + } + return `${SETTINGS_ROOT}\\${name}`; +} + +/** + * What SaveSetting has stored for an application, as `{ section: { name: value } }`, + * or null when it has stored nothing. + */ +export function savedSettings(app) { + const [entry] = snapshotKeys([settingsKey(app)]); + if (!entry?.snap) return null; + const out = {}; + for (const s of [].concat(entry.snap.keys ?? [])) { + out[s.name] = Object.fromEntries([].concat(s.snap?.values ?? []).map((v) => [v.name, v.data])); + } + return out; +} + +/** + * Delete what SaveSetting has stored for these applications, so that their + * next GetSetting returns its default. Snapshot the keys first + * (`snapshotKeys(apps.map(settingsKey))`) to put them back afterwards. + * + * @returns {number} the number of writes + */ +export function deleteSettings(apps) { + return restoreKeys(apps.map((a) => ({ path: settingsKey(a), snap: null }))); +} + +const ARCH_MEMORY = "targetArchitectureMemory"; +const norm = (p) => String(p).split("/").join("\\").toLowerCase(); + +// The build targets the IDE remembers, as an object, or null when there is no +// value or it is not a JSON object -- which is not the harness's to repair. +function readArchitectureMemory(root) { + const now = request({ op: "readValue", key: `${root}\\IDESettings`, name: ARCH_MEMORY }); + if (!now.exists) return null; + let memory; + try { memory = JSON.parse(now.data); } catch { return null; } + if (!memory || typeof memory !== "object" || Array.isArray(memory)) return null; + return { data: now.data, memory }; +} + +// Change the remembered build targets with `edit`, which is handed the object +// and returns how many entries it changed. The object is edited here rather +// than in PowerShell, and written back with JSON.stringify, which is how the +// IDE writes it, so the other entries keep their exact text and order. The +// write is refused if the value changed after it was read -- an IDE switching +// a target of its own meanwhile -- and the edit is then made again on what is +// there now. +function editArchitectureMemory(root, edit) { + for (let attempt = 0; attempt < 3; attempt++) { + const now = readArchitectureMemory(root); + if (!now) return 0; + const changes = edit(now.memory); + if (!changes) return 0; + const w = request({ op: "writeValueIf", key: `${root}\\IDESettings`, name: ARCH_MEMORY, + expected: now.data, data: JSON.stringify(now.memory) }); + if (w.written) return changes; + } + throw new Error(`the IDE's ${ARCH_MEMORY} kept changing while it was being tidied`); +} + +/** + * Delete the build target the IDE remembers for every project under these + * folders. + * + * The IDE keeps the target it last built each project for, win32 or win64, in + * one IDESettings value holding a JSON object keyed by project path, and a + * project it opens again starts in that target. A harness project's path is + * used run after run -- tbrun's work folder is keyed to its port -- so an + * entry one run leaves, when somebody switches a --keep IDE to win64, sets the + * target of every later run on that path, and nothing says so. On 2026-09-24 + * tbrun on ports 9372 and 9373 built 64-bit for that reason. + * + * Entries for any other path are left alone here, the user's own projects + * among them. Opening a project only reads its entry; one is written when the + * target of an open project changes, which `--arch` does and a person can. + * A named project's entry is put back by restoreArchitectureMemory instead. + * + * @param {string[]} prefixes folders inside the OS temp folder, as for restoreProjects + * @param {object} [o] + * @param {string} [o.root] the IDE's settings key + * @returns {number} entries deleted + */ +export function sweepArchitectureMemory(prefixes = [], { root = IDE_SETTINGS_KEY } = {}) { + const pre = prefixes.map((p) => norm(asTempFolder(p))); + if (!pre.length) return 0; + return editArchitectureMemory(root, (memory) => { + const drop = Object.keys(memory).filter((p) => pre.some((x) => norm(p).startsWith(x))); + for (const p of drop) delete memory[p]; + return drop.length; + }); +} + +/** + * Record the build targets the IDE remembers for these projects, for + * restoreArchitectureMemory: every entry whose key names one of them, however + * the key is spelled, with its value. A project with none is recorded as + * having none. + */ +export function snapshotArchitectureMemory(paths = [], { root = IDE_SETTINGS_KEY } = {}) { + const want = paths.map((p) => norm(path.resolve(p))); + const memory = want.length ? readArchitectureMemory(root)?.memory ?? {} : {}; + return { root, paths: want, entries: Object.entries(memory).filter(([p]) => want.includes(norm(p))) }; +} + +/** + * Put back the build targets a snapshot recorded, and delete every other entry + * for the same projects. + * + * A run that switches the target of a project it opens -- tbbuild --arch on + * the user's own project -- leaves the IDE's entry for it, saved under the + * path as the IDE was given it, which need not be spelled as the user's own + * IDE spelled it. So an entry the snapshot has is given its old value in its + * old place, and any other entry for the same project is deleted. + * + * @returns {number} entries written or deleted + */ +export function restoreArchitectureMemory(snap) { + if (!snap?.paths?.length) return 0; + return editArchitectureMemory(snap.root, (memory) => { + let changes = 0; + const had = new Map(snap.entries); + for (const p of Object.keys(memory)) { + if (!snap.paths.includes(norm(p))) continue; + if (!had.has(p)) { delete memory[p]; changes++; } + else if (memory[p] !== had.get(p)) { memory[p] = had.get(p); changes++; } + } + for (const [p, v] of had) if (!(p in memory)) { memory[p] = v; changes++; } + return changes; + }); +} + // Restoring a key deletes whatever the snapshot does not list, so a key near // the root -- `Software`, `Software\Classes` -- would take everything under it. // Three segments is the shallowest key this has any business restoring. @@ -374,33 +606,98 @@ function alive(pid) { * user's own, restored rather than deleted * @param {string[]} [o.prefixes] folders only the harness writes to; every * entry under them is deleted + * @param {string} [o.root] the IDE's settings key (default IDE_SETTINGS_KEY) + * @param {string[]} [o.keys] the association keys (default ASSOCIATION_KEYS); + * the self-test passes scratch keys for both */ -export function startTidy({ paths = [], prefixes = [] } = {}) { +export function startTidy({ paths = [], prefixes = [], root = IDE_SETTINGS_KEY, + keys = ASSOCIATION_KEYS } = {}) { const owner = Number(process.env.TB_REGISTRY_OWNER); if (owner && owner !== process.pid && alive(owner)) return null; process.env.TB_REGISTRY_OWNER = String(process.pid); + let tidy; try { - if (prefixes.length) restoreProjects({ root: IDE_SETTINGS_KEY, entries: [] }, { prefixes }); - return { projects: snapshotProjects(paths), keys: snapshotKeys(), prefixes }; + if (prefixes.length) restoreProjects({ root, entries: [] }, { prefixes }); + tidy = { projects: snapshotProjects(paths, { root }), keys: snapshotKeys(keys), prefixes, root }; + tidy.keysInTemp = namesTempFolder(tidy.keys); } catch (e) { console.error(`warning: the IDE's registry entries will not be tidied after this run: ${e.message}`); return null; } + sweepTargets(prefixes, root); + // After the sweep, so a named project inside a swept folder is not given back + // an entry the sweep has just deleted. + tidy.targets = snapshotTargets(paths, root); + return tidy; } /** * Put the registry back as startTidy found it. Call it only once every IDE of * the run has exited -- shutdownIde waits for that. * - * @returns {{projectState: number, recentlyOpened: number, association: number} | null} + * @returns {{projectState: number, recentlyOpened: number, association: number, + * architecture: number | null} | null} */ export function finishTidy(tidy) { if (!tidy) return null; + let done; try { const p = restoreProjects(tidy.projects, { prefixes: tidy.prefixes }); - return { ...p, association: restoreKeys(tidy.keys) }; + // An association that named the temp folder when the run began belonged to + // another run's copy of the IDE (tb-ide-copy.mjs), which will be deleted: + // putting it back would point .twinproj files at nothing. It is left as + // the IDEs set it, and the next IDE started from a real install points it + // back at that install. + if (tidy.keysInTemp) { + console.error("note: the .twinproj association pointed into the temp folder when this " + + "run began, at another run's copy of the IDE, so it is left as it is now"); + } + done = { ...p, association: tidy.keysInTemp ? null : restoreKeys(tidy.keys) }; } catch (e) { console.error(`warning: could not tidy the IDE's registry entries after this run: ${e.message}`); return null; } + const swept = sweepTargets(tidy.prefixes, tidy.root); + const restored = restoreTargets(tidy.targets); + return { ...done, architecture: swept === null || restored === null ? null : swept + restored }; +} + +// Whether any value in a key snapshot names a path inside the temp folder. +function namesTempFolder(snapshot) { + const tmp = norm(path.resolve(tmpdir())) + "\\"; + const named = (snap) => !!snap && ( + [].concat(snap.values ?? []).some((v) => [].concat(v?.data ?? []).some((d) => norm(d).includes(tmp))) || + [].concat(snap.keys ?? []).some((k) => named(k?.snap))); + return [].concat(snapshot ?? []).some((e) => named(e?.snap)); +} + +// Each with a warning of its own, so that failing here costs only this part of the tidy. +function sweepTargets(prefixes, root) { + try { + return sweepArchitectureMemory(prefixes, { root }); + } catch (e) { + console.error(`warning: could not tidy the build targets the IDE remembers: ${e.message}`); + return null; + } +} + +function snapshotTargets(paths, root) { + try { + return snapshotArchitectureMemory(paths, { root }); + } catch (e) { + console.error(`warning: the build targets the IDE remembers for this run's projects will ` + + `not be put back: ${e.message}`); + return null; + } +} + +// A snapshot that failed was warned about when it was taken; there is nothing to put back. +function restoreTargets(snap) { + if (!snap) return 0; + try { + return restoreArchitectureMemory(snap); + } catch (e) { + console.error(`warning: could not put back the build targets the IDE remembers: ${e.message}`); + return null; + } } diff --git a/scripts/tbbuild.mjs b/scripts/tbbuild.mjs index 6eb5c054..0b11029f 100644 --- a/scripts/tbbuild.mjs +++ b/scripts/tbbuild.mjs @@ -6,6 +6,11 @@ // --ide twinBASIC.exe (default: $TB_IDE, else the newest // %USERPROFILE%/Desktop/twinBASIC_IDE_BETA_*) // --port DevTools port to start the IDE on (default 9333) +// --arch win32 or win64 (default win32): the target to +// compile for. #If Win64 and LongPtr's size change +// what compiles, and a project opens in whatever +// target the IDE remembers for its path, so the +// target is set on every run, win32 included. // --timeout give up waiting for the compile (default 180) // --json emit one JSON object instead of text // --keep leave the IDE running afterwards. The IDE's pid is @@ -47,8 +52,8 @@ import { existsSync, statSync } from "node:fs"; import path from "node:path"; import { findIde } from "./lib/tb-install.mjs"; -import { attachIde, compileOutcome, launchIde, shutdownIde, summaryLine, - waitForCompile, wantShow } from "./lib/tb-ide.mjs"; +import { TARGETS, attachIde, compileOutcome, launchIde, setBuildTarget, shutdownIde, + summaryLine, waitForCompile, wantShow } from "./lib/tb-ide.mjs"; import { finishTidy, startTidy } from "./lib/tb-registry.mjs"; const args = process.argv.slice(2); @@ -61,15 +66,16 @@ const proj = args.find((a, i) => !a.startsWith("--") && !args[i - 1]?.startsWith // which is where the IDE's own zip tells people to unpack it. const IDE = findIde(opt("ide", undefined)); const port = Number(opt("port", 9333)); +const arch = opt("arch", TARGETS[0]); const timeout = Number(opt("timeout", 180)) * 1000; const asJson = flag("json"); const keep = flag("keep"); const show = wantShow({ show: flag("show"), hide: flag("hide") }); -if (!proj || flag("help")) { +if (!proj || flag("help") || !TARGETS.includes(arch)) { console.error("usage: node scripts/tbbuild.mjs " + - "[--ide ] [--port N] [--timeout S] [--json] [--keep] " + - "[--show|--hide]"); + "[--ide ] [--port N] [--arch win32|win64] [--timeout S] [--json] " + + "[--keep] [--show|--hide]"); process.exit(2); } // Refuse anything that is not a .twinproj, rather than discovering it two @@ -132,13 +138,27 @@ try { const c = await attachIde(port); if (!c) die(2, "the IDE never exposed a debug port"); -const dialogs = []; -c.on((m) => { - if (m.method === "Page.javascriptDialogOpening") dialogs.push(m.params.message); -}); +// Every alert the IDE opens is recorded and dismissed by the connection +// (attachIde), and reported with the diagnostics. +const dialogs = c.dialogs; -const outcome = compileOutcome(await waitForCompile(c, { project: proj, timeout }), { name: proj }); +let outcome = compileOutcome(await waitForCompile(c, { project: proj, timeout }), { name: proj }); if (!outcome.ok) die(outcome.code, outcome.message); + +// Set on every run, win32 included, and what is reported is the compile under +// it: see setBuildTarget. Switching restarts the compiler, which compiles the +// project again, so a run that switches takes a few seconds longer. +let openedIn; +try { + const target = await setBuildTarget(c, arch, { project: proj, timeout }); + openedIn = target.from; + if (target.waited) { + outcome = compileOutcome(target.waited, { name: proj }); + if (!outcome.ok) die(outcome.code, outcome.message); + } +} catch (e) { + die(2, e.message); +} const { rows, counts } = outcome; // The IDE's pid is reported so a caller can clean up precisely. It matters most @@ -147,15 +167,22 @@ const { rows, counts } = outcome; // and the user's own open IDE with it. if (asJson) { console.log(JSON.stringify({ - project: proj, + project: proj, arch, openedIn, errors: counts[0], warnings: counts[1], hints: counts[2], infos: counts[3], idePid: ide?.pid ?? null, kept: keep, - diagnostics: rows, dialogs, + diagnostics: rows, dialogs: dialogs.map((d) => d.message), }, null, 2)); } else { + // Said only when either target is not the default, so the usual report is + // unchanged, and the summary stays the last line. A project opens in win32 + // unless the IDE remembered another target for its path. + if (arch !== TARGETS[0] || openedIn !== TARGETS[0]) { + console.log(`target: ${arch}` + + (openedIn !== TARGETS[0] ? ` (the IDE remembered ${openedIn} for this project)` : "")); + } for (const r of rows) console.log(r); console.log(summaryLine(counts)); - if (dialogs.length) console.log("dialogs:", JSON.stringify(dialogs)); + if (dialogs.length) console.log("dialogs:", JSON.stringify(dialogs.map((d) => d.message))); // Only under --keep, where the pid is still alive and therefore actionable. if (keep && ide?.pid) console.log(`ide-pid: ${ide.pid}`); } diff --git a/scripts/tbrun.mjs b/scripts/tbrun.mjs index 3a5eba06..7a37e64e 100644 --- a/scripts/tbrun.mjs +++ b/scripts/tbrun.mjs @@ -3,6 +3,9 @@ // node scripts/tbrun.mjs [options] // // --port DevTools port to start the IDE on (default 9346) +// --arch win32 or win64 (default win32): the target to build +// for, and so the process the probe runs in -- a win64 +// probe runs in the IDE's 64-bit compiler // --timeout give up waiting for console output (default 120) // --quiet output is complete after this long with no change // (default 2500) @@ -41,14 +44,19 @@ // // Each of these cost an hour when the probe was first done by hand. // -// 1. THE BUILD PATH MUST BE AN EXPLICIT FILE. A project whose +// 1. THE BUILD PATH MUST BE IN AN EXPLICIT FOLDER. A project whose // `project.buildPath` is still the default `${SourcePath}\Build\...` // template opens a native Save dialog on the build -- and because the // IDE runs on a private desktop, that dialog is invisible, takes no // input, and the build simply never happens. Nothing reports it: the // WebView2 renderer stays responsive, so even a CDP health check says the -// IDE is fine. This script therefore owns the tree and pins buildPath to a -// concrete file before importing, which makes the trap unreachable. +// IDE is fine. This script therefore owns the tree and pins buildPath to +// its own out folder before importing, which makes the trap unreachable +// (lib/tb-project.mjs, shared with the add-in harness). The file name is +// the IDE's own `${ProjectName}_${Architecture}.${FileExtension}`, so it +// says what was built: measured on BETA 983, those variables in an +// explicit folder open no dialog, and give ArchProbe_win32.exe and +// ArchProbe_win64.exe. // 2. A JAVASCRIPT .click() ON THE BUILD BUTTON DOES NOTHING. `#buildIcon` is // a plain DIV wired through the IDE's own pointer handling; it needs real // CDP Input.dispatchMouseEvent presses at its centre (tb-ide's @@ -83,13 +91,13 @@ // a blunt enough instrument to need the guard rails in reapOrphans(). import { execFileSync } from "node:child_process"; -import { existsSync, readFileSync, writeFileSync, mkdirSync, statSync, readdirSync, - cpSync, rmSync } from "node:fs"; +import { existsSync, readFileSync, mkdirSync, statSync, readdirSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import path from "node:path"; -import { compilerExe, findIde, runCompiler } from "./lib/tb-install.mjs"; -import { attachIde, clickCenter, compileOutcome, killTree, launchIde, readConsole, - shutdownIde, summaryLine, waitForCompile, wantShow } from "./lib/tb-ide.mjs"; +import { compilerExe, findIde } from "./lib/tb-install.mjs"; +import { TARGETS, attachIde, clickCenter, compileOutcome, killTree, launchIde, readConsole, + setBuildTarget, shutdownIde, summaryLine, waitForCompile, wantShow } from "./lib/tb-ide.mjs"; +import { laneProjectId, stageProject } from "./lib/tb-project.mjs"; import { finishTidy, startTidy } from "./lib/tb-registry.mjs"; const argv = process.argv.slice(2); @@ -103,14 +111,15 @@ const die = (code, msg) => { console.error(msg); process.exit(code); }; // Every flag that TAKES A VALUE has to be named here, or its value is mistaken // for the source directory. -const VALUE_FLAGS = ["port", "timeout", "quiet", "ide", "reap-images"]; +const VALUE_FLAGS = ["port", "arch", "timeout", "quiet", "ide", "reap-images"]; const positional = argv.filter((a, i) => !a.startsWith("--") && !(i > 0 && VALUE_FLAGS.includes(argv[i - 1]?.replace(/^--/, "")))); +const arch = opt("arch", TARGETS[0]); -if (!positional.length || flag("help")) { - die(2, "usage: node scripts/tbrun.mjs [--port N] [--timeout S] " + - "[--quiet MS] [--json] [--raw] [--keep] [--no-reap] [--reap-images a,b] " + - "[--show|--hide]"); +if (!positional.length || flag("help") || !TARGETS.includes(arch)) { + die(2, "usage: node scripts/tbrun.mjs [--port N] [--arch win32|win64] " + + "[--timeout S] [--quiet MS] [--json] [--raw] [--keep] [--no-reap] " + + "[--reap-images a,b] [--show|--hide]"); } const srcDir = path.resolve(positional[0]); @@ -159,28 +168,18 @@ const work = path.join(tmpdir(), "tbrun", runKey); rmSync(work, { recursive: true, force: true }); mkdirSync(work, { recursive: true }); -// Staged into a temp copy rather than edited in place. Pinning buildPath is what -// makes the invisible Save dialog unreachable, but it is still a change to the -// caller's project, and a probe harness that rewrites the tree you pointed it at -// is one you stop trusting with a real project. +// Staged into a temp copy rather than edited in place, with buildPath pinned to +// the run's own out folder and a project.id keyed to the port +// (lib/tb-project.mjs). The file is named as the IDE's own template names it, +// so the build type and the target are in the name (1). const stage = path.join(work, "src"); -cpSync(srcDir, stage, { recursive: true }); -const stagedSettings = path.join(stage, "Settings"); -const exePath = path.join(work, "tbrun-probe.exe"); +const outDir = path.join(work, "out"); +mkdirSync(outDir, { recursive: true }); +const buildPath = path.join(outDir, "${ProjectName}_${Architecture}.${FileExtension}"); const projPath = path.join(work, "tbrun-probe.twinproj"); -const settings = JSON.parse(readFileSync(stagedSettings, "utf8")); -const wasTemplate = /\$\{/.test(settings["project.buildPath"] ?? ""); -settings["project.buildPath"] = exePath; -// Two probes sharing a project.id confuse the IDE's recents list -- so this is -// keyed to the port too, not a constant. The last group is 12 hex digits, of -// which the port fills the low six. -settings["project.id"] = - `{7B247000-0000-4000-9000-7B2470${port.toString(16).padStart(6, "0")}}`; -writeFileSync(stagedSettings, JSON.stringify(settings, null, "\t"), "utf8"); - const sourceText = (() => { - const dir = path.join(stage, "Sources"); + const dir = path.join(srcDir, "Sources"); if (!existsSync(dir)) return ""; return readdirSync(dir).filter((f) => f.endsWith(".twin")) .map((f) => readFileSync(path.join(dir, f), "utf8")).join(String.fromCharCode(10)); @@ -201,11 +200,30 @@ if (!hasHook) { // ------------------------------------------------------------------- pack -// import's exit code does not say whether it worked -- 0 on the failures it -// reports, 999 on a tree holding an embedded package -- so runCompiler reads -// the output, and a failure of either kind is the harness's, exit 2. -const pack = runCompiler(COMPILER, ["import", projPath, stage, "--overwrite"]); -if (!pack.done) die(2, `packing failed${pack.why}:\n${pack.tail}`); +// A packing failure is the harness's, exit 2. +let wasTemplate = false, projectName = ""; +try { + const staged = stageProject({ + src: srcDir, stage, project: projPath, compiler: COMPILER, + settings: { "project.buildPath": buildPath, "project.id": laneProjectId(0, port) }, + }); + wasTemplate = /\$\{/.test(staged.original["project.buildPath"] ?? ""); + projectName = String(staged.settings["project.name"] ?? ""); +} catch (e) { + die(2, e.message); +} + +// What the build wrote: the IDE expands the template, so the name is looked for +// rather than assumed -- ${FileExtension} follows the build type -- and the +// binary is told from anything written beside it by its "MZ" header. +function builtFile() { + const stem = `${projectName}_${arch}.`.toLowerCase(); + for (const f of readdirSync(outDir).filter((n) => n.toLowerCase().startsWith(stem))) { + const file = path.join(outDir, f); + try { if (readFileSync(file).subarray(0, 2).toString("latin1") === "MZ") return file; } catch { /* gone */ } + } + return null; +} // ---------------------------------------------------------------- compile @@ -243,9 +261,28 @@ try { const cdp = await attachIde(port); if (!cdp) failBuild(2, "the IDE never exposed a debug port"); -const outcome = compileOutcome( +let outcome = compileOutcome( await waitForCompile(cdp, { project: projPath, timeout: 180 * 1000 }), { name: projPath }); if (!outcome.ok) failBuild(2, outcome.message); + +// The target, set on every run, win32 included (setBuildTarget says why). The +// probe runs in the compiler that builds it, so under win64 it runs in the +// IDE's 64-bit compiler, twinBASIC_win64_noDEP.exe, as a 64-bit process: +// measured on BETA 983 with LenB of a LongPtr, ProcessorArchitecture(), +// PROCESSOR_ARCHITECTURE, IsWow64Process and the module path of the process. +try { + const target = await setBuildTarget(cdp, arch, { project: projPath, timeout: 180 * 1000 }); + // Only a target the IDE remembered is worth a word: a new path opens in win32. + if (target.from !== TARGETS[0]) { + console.error(`note: the IDE remembered ${target.from} for this path; the probe is built for ${arch}`); + } + if (target.waited) { + outcome = compileOutcome(target.waited, { name: projPath }); + if (!outcome.ok) failBuild(2, outcome.message); + } +} catch (e) { + failBuild(2, e.message); +} if (outcome.counts[0] > 0) { failBuild(1, [...outcome.rows, summaryLine(outcome.counts)].join("\n")); } @@ -299,8 +336,8 @@ if (!captured.length) { } if (flag("json")) { - console.log(JSON.stringify({ exe: exePath, lines: captured, idePid: ideRun?.pid ?? null, reaped }, - null, 2)); + console.log(JSON.stringify({ exe: builtFile(), arch, lines: captured, idePid: ideRun?.pid ?? null, + reaped }, null, 2)); } else { for (const l of captured) console.log(l); } diff --git a/test/README.md b/test/README.md index ba5ec693..c7bbc981 100644 --- a/test/README.md +++ b/test/README.md @@ -1,8 +1,16 @@ # Test fixtures -There is no test runner here and no `npm test` --- `package.json` declares no `scripts` -at all. Everything under `test/` is input for [`scripts/check_links_diff.mjs`](../scripts/check_links_diff.mjs), -the harness that proves the repository's two link-checker implementations agree. +There is no `npm test` --- `package.json` declares no `scripts` at all. What is under `test/` +is input for three tools: + +- `fixtures/` for [`scripts/check_links_diff.mjs`](../scripts/check_links_diff.mjs), the + harness that proves the repository's two link-checker implementations agree. The rest of + this file is about it. +- `example-projects/` for [`scripts/check_examples.mjs`](../scripts/check_examples.mjs): + the templates it builds the documentation's code samples into + ([WIP.ExamplesBuild.md](../WIP.ExamplesBuild.md)). +- `addin/` for [`scripts/addin_test.mjs`](../scripts/addin_test.mjs), which `addin-test.bat` + runs: the IDE add-in scenarios ([test/addin](#testaddin), at the end). If a fixture case has just gone red, start at [The invariant, and how it breaks](#the-invariant-and-how-it-breaks) --- the commit that broke it need not have touched @@ -99,3 +107,20 @@ That is why the fixture carries stub CSS, JS and font files. When a template cha introduces a new unconditional reference, the fix is almost always **to add the stub the template now expects**, not to raise the expected count: six accidental broken links dilute a category the fixture is supposed to hold at an exact, meaningful number. + +## test/addin + +The scenarios that test twinBASIC IDE add-ins by operating an IDE, run with +`addin-test.bat`. Each `*.test.mjs` is a `node:test` file and one lane of the run: it gets +its own DevTools port, work folder and private copy of the twinBASIC install from +[`scripts/lib/tb-lane.mjs`](../scripts/lib/tb-lane.mjs), builds the add-ins it tests into +that copy, and opens `host/`, the project a scenario works in. `host/` holds two modules +with the word `needle` in them, placed for Sample 15's search to find; change them and +that scenario's expected results change too. `lanes.mjs` lists the lanes, and names the +`SaveSetting` application whose settings each lane's add-ins change, so that the runner +can put them back. + +Never run a scenario with a bare `node --test`: it skips itself, since only the runner +gives it a lane, and only the runner puts the registry back afterwards. +[WIP.Harness.md, The add-in test runner](../WIP.Harness.md#the-add-in-test-runner) has +the design and what was measured. diff --git a/test/addin/host/Settings b/test/addin/host/Settings new file mode 100644 index 00000000..dc28e052 --- /dev/null +++ b/test/addin/host/Settings @@ -0,0 +1,61 @@ +{ + "configuration.inherits": "Defaults", + "project.appTitle": "AddinHost", + "project.buildPath": "${SourcePath}\\Build\\${ProjectName}_${Architecture}.${FileExtension}", + "project.buildType": "Standard EXE", + "project.description": "The project the add-in scenarios under test/addin open. scripts/lib/tb-lane.mjs stages a copy, with project.id and project.buildPath rewritten for the lane.", + "project.exportPathIsV2": true, + "project.id": "{7B247200-0000-4000-9000-7B2472000000}", + "project.name": "AddinHost", + "project.optionExplicit": true, + "project.references": [ + { + "id": "{00020430-0000-0000-C000-000000000046}", + "lcid": 0, + "name": "OLE Automation", + "path32": "C:\\Windows\\SysWOW64\\stdole2.tlb", + "path64": "C:\\Windows\\System32\\stdole2.tlb", + "symbolId": "stdole", + "versionMajor": 2, + "versionMinor": 0 + }, + { + "id": "{F50B82D0-DCAB-43FE-9631-11959D4A4728}", + "isCompilerPackage": true, + "licence": "MIT", + "name": "[COMPILER PACKAGE] twinBASIC - VB Compatibility Package (Forms)", + "path32": "", + "path64": "", + "publisher": "TWINBASIC-COMPILER", + "symbolId": "VB", + "versionBuild": 0, + "versionMajor": 0, + "versionMinor": 0, + "versionRevision": 31 + }, + { + "id": "{C192FB39-64CA-4D9B-B477-A5502F48EFCC}", + "isCompilerPackage": true, + "licence": "MIT", + "name": "[COMPILER PACKAGE] twinBASIC - App global class object", + "path32": "", + "path64": "", + "publisher": "TWINBASIC-COMPILER", + "symbolId": "AppGlobalClassProject", + "versionBuild": 0, + "versionMajor": 1, + "versionMinor": 0, + "versionRevision": 0 + } + ], + "project.settingsVersion": 1, + "project.startupObject": "Sub Main", + "project.warnings": { + "errors": [], + "hints": [], + "ignored": [], + "info": [], + "warnings": [] + }, + "runtime.useUnicodeStandardLibrary": true +} diff --git a/test/addin/host/Sources/Haystack.twin b/test/addin/host/Sources/Haystack.twin new file mode 100644 index 00000000..8a9a3da1 --- /dev/null +++ b/test/addin/host/Sources/Haystack.twin @@ -0,0 +1,8 @@ +Module Haystack + ' A haystack with a needle in it. + Public Function FindTheNeedle(ByVal n As Long) As Long + Dim needleCount As Long + needleCount = n * 2 + Return needleCount + End Function +End Module diff --git a/test/addin/host/Sources/Main.twin b/test/addin/host/Sources/Main.twin new file mode 100644 index 00000000..9af7e85c --- /dev/null +++ b/test/addin/host/Sources/Main.twin @@ -0,0 +1,7 @@ +Module Main + Public Sub Main() + Dim needle As Long + needle = FindTheNeedle(3) + Debug.Print needle + End Sub +End Module diff --git a/test/addin/lanes.mjs b/test/addin/lanes.mjs new file mode 100644 index 00000000..30f0ad09 --- /dev/null +++ b/test/addin/lanes.mjs @@ -0,0 +1,23 @@ +// The lanes scripts/addin_test.mjs runs (addin-test.bat). A lane is one +// scenario file in this folder, run with node:test in a process of its own, +// with its own DevTools port, work folder and copy of the twinBASIC install; +// scripts/lib/tb-lane.mjs is what the file gets. +// +// file the scenario file +// name what --only matches and the report calls the lane; by default +// the file's name without ".test.mjs" +// settings every application name the lane's add-ins pass to SaveSetting. +// SaveSetting writes under HKCU\Software\VB and VBA Program +// Settings\, the key any installed copy of the same add-in +// uses. The runner records those keys before the first lane +// starts, deletes them before this lane starts, so that its +// add-ins begin from their defaults, and puts them back as found +// once every lane has ended. Two lanes that name the same one +// never run at once. An add-in whose settings are not named here +// leaves them changed after the run. + +export default [ + { file: "sample10.test.mjs" }, + // Sample 15 saves all four of its option boxes whenever one is clicked. + { file: "sample15.test.mjs", settings: ["GlobalSearchAddIn"] }, +]; diff --git a/test/addin/sample10.test.mjs b/test/addin/sample10.test.mjs new file mode 100644 index 00000000..e61f0d15 --- /dev/null +++ b/test/addin/sample10.test.mjs @@ -0,0 +1,84 @@ +// Sample 10, the WaynesWorld add-in that ships with the IDE, operated end to +// end: its toolbar buttons, its tool window, a message box answered twice, a +// notification and a DEBUG CONSOLE line. One of the two scenarios that finish +// Stage 1 of WIP.HelpAddin.md. +// +// Run it with addin-test.bat, which gives it a lane; on its own it is skipped. + +import assert from "node:assert/strict"; +import path from "node:path"; +import { after, before, describe, test } from "node:test"; +import { fileURLToPath } from "node:url"; +import { consoleMark, loadedAddins, readConsole } from "../../scripts/lib/tb-ide.mjs"; +import { addinLane } from "../../scripts/lib/tb-lane.mjs"; +import { answerMessageBox, click, messageBoxes, notifications, toolWindow, + waitFor } from "../../scripts/lib/tb-operate.mjs"; + +const HOST = path.join(path.dirname(fileURLToPath(import.meta.url)), "host"); +const W = "WaynesWindowData"; // the id Sample 10 gives ToolWindows.Add +const lane = addinLane(); + +// The message box on top, with its text trimmed, or undefined. +async function topBox(c) { + const b = (await messageBoxes(c)).at(-1); + return b && { ...b, text: b.text.trim() }; +} +const noBoxes = (c) => waitFor(c, async (c) => (await messageBoxes(c)).length === 0); + +describe("Sample 10: WaynesWorld", { skip: lane ? false : "run it with addin-test.bat" }, () => { + let c; + before(async () => { + await lane.addSample("Sample 10"); + c = await lane.open(HOST); + }); + after(() => lane?.close()); + + test("the compiler loads the add-in, which reports the project", async () => { + const names = (await loadedAddins(c)).map((a) => a.name); + assert.ok(names.includes("WaynesWorld AddIn"), `loaded: ${JSON.stringify(names)}`); + const lines = ((await readConsole(c)) ?? "").split("\n").filter((l) => l.startsWith("[WaynesWorldAddin]")); + assert.equal(lines.length, 5, `its OnProjectLoaded lines: ${JSON.stringify(lines)}`); + assert.ok(lines.includes("[WaynesWorldAddin] ProjectName: AddinHost"), JSON.stringify(lines)); + }); + + test("its image button shows a message box", async () => { + await click(c, "addinButton-TestImageButton"); + const box = await waitFor(c, topBox); + assert.deepEqual(box, { title: "News alert...", text: "You clicked the image button!", buttons: ["OK"] }); + await answerMessageBox(c, "OK"); + assert.ok(await noBoxes(c), "the message box did not close"); + }); + + test("its other button opens its tool window", async () => { + await click(c, "addinButton-ShowToolWindow"); + const w = await waitFor(c, async (c) => { const t = await toolWindow(c, W); return t?.visible && t; }); + assert.ok(w, "the tool window did not appear"); + assert.match(w.text, /11\. ShowMessageBox/); + }); + + test("a three-button message box, answered with its second button", async () => { + await click(c, { toolWindow: W, css: "#myButton11" }); + const first = await waitFor(c, topBox); + assert.deepEqual(first, { title: "Choose an option", text: "Hello there from WaynesWorldAddIn!", + buttons: ["button1", "button2", "button3"] }); + await answerMessageBox(c, "button2"); + // The add-in's call returns the button's index, and it answers with a second box. + const second = await waitFor(c, async (c) => { const b = await topBox(c); return b?.title === "option" && b; }); + assert.deepEqual(second, { title: "option", text: "you selected button2", buttons: ["ok"] }); + await answerMessageBox(c, "ok"); + assert.ok(await noBoxes(c), "a message box is still open"); + }); + + test("a notification", async () => { + await click(c, { toolWindow: W, css: "#myButton10" }); + const shown = await waitFor(c, async (c) => (await notifications(c)).find((t) => t.trim() === "Hello there from WaynesWorldAddIn!")); + assert.ok(shown, `notifications: ${JSON.stringify(await notifications(c))}`); + }); + + test("a DEBUG CONSOLE line", async () => { + const mark = await consoleMark(c); + await click(c, { toolWindow: W, css: "#myButton7" }); + const text = await waitFor(c, async (c) => ((await readConsole(c, { since: mark })) ?? "").trim()); + assert.equal(text, "Hello there from WaynesWorldAddIn!"); + }); +}); diff --git a/test/addin/sample15.test.mjs b/test/addin/sample15.test.mjs new file mode 100644 index 00000000..ce2a1419 --- /dev/null +++ b/test/addin/sample15.test.mjs @@ -0,0 +1,119 @@ +// Sample 15, the Global Search add-in that ships with the IDE, operated end to +// end: its toolbar button, a search typed into its tool window, a click on one +// match, and an option that it saves with SaveSetting. One of the two +// scenarios that finish Stage 1 of WIP.HelpAddin.md. +// +// Run it with addin-test.bat, which gives it a lane; on its own it is skipped. + +import assert from "node:assert/strict"; +import path from "node:path"; +import { after, before, describe, test } from "node:test"; +import { fileURLToPath } from "node:url"; +import { loadedAddins } from "../../scripts/lib/tb-ide.mjs"; +import { addinLane } from "../../scripts/lib/tb-lane.mjs"; +import { click, editorState, toolWindow, typeText, waitFor } from "../../scripts/lib/tb-operate.mjs"; +import { savedSettings } from "../../scripts/lib/tb-registry.mjs"; + +const HOST = path.join(path.dirname(fileURLToPath(import.meta.url)), "host"); +const W = "GlobalSearchAddInData"; // the id Sample 15 gives ToolWindows.Add +const lane = addinLane(); + +// The search's results, read from the list view's data rather than its rows, +// since a list view draws only the rows that fit. One entry per file, sorted by +// path: its match count as the add-in words it, and each match's line of text +// with the [line,column] label the add-in puts beside it. +const results = (c) => c.evaluate(`(() => { + const w = toolWindowsById[${JSON.stringify(W)}]; + const e = w && w.bodyElement.querySelector("#resultsList"); + const lv = e && e.listview; + if (!lv) return null; + const d = document.createElement("div"); + return lv.dataNodes.slice(0, lv.itemCount).map((html) => { + d.innerHTML = html; + return { + path: d.querySelector(".colPath").textContent, + count: d.querySelector(".colMatchCount").textContent, + matches: [...d.querySelectorAll(".colMatchOuter")].map((m) => + [m.querySelector(".colMatch").textContent, m.querySelector(".lineInfo").textContent]), + }; + }).sort((a, b) => a.path < b.path ? -1 : a.path > b.path ? 1 : 0); +})()`); + +const matchCount = (r) => r.reduce((n, f) => n + f.matches.length, 0); + +// The option boxes, by id, checked or not. +const options = (c) => c.evaluate(`Object.fromEntries([...toolWindowsById[${JSON.stringify(W)}] + .bodyElement.querySelectorAll("input[type=checkbox]")].map((e) => [e.id, e.checked]))`); + +describe("Sample 15: Global Search", { skip: lane ? false : "run it with addin-test.bat" }, () => { + let c; + before(async () => { + await lane.addSample("Sample 15"); + c = await lane.open(HOST); + }); + after(() => lane?.close()); + + test("the compiler loads the add-in", async () => { + const names = (await loadedAddins(c)).map((a) => a.name); + assert.ok(names.includes("GlobalSearchAddIn AddIn"), `loaded: ${JSON.stringify(names)}`); + }); + + test("its toolbar button opens its tool window, with every option off", async () => { + await click(c, "addinButton-GlobalSearchAddInButton"); + const w = await waitFor(c, async (c) => { const t = await toolWindow(c, W); return t?.visible && t; }); + assert.ok(w, "the tool window did not appear"); + assert.equal(w.title, "GLOBAL SEARCH"); + // Off, because the runner deleted the add-in's saved settings before the lane started. + assert.deepEqual(await options(c), { + searchBarInsidePackages: false, searchBarMatchCase: false, + searchBarMatchWholeWordOnly: false, searchBarExcludeComments: false, + }); + }); + + test("a typed search lists every match in both files", async () => { + await click(c, { toolWindow: W, css: "#searchBarInput" }); + await typeText(c, "needle"); + // The add-in searches a second after the last key-up, and adds one file's + // entry at a time. + const found = await waitFor(c, async (c) => { const r = await results(c); return r?.length === 2 && r; }); + assert.deepEqual(found, [ + { path: "twinbasic:/AddinHost/Sources/Haystack.twin", count: "5 matches", matches: [ + ["' A haystack with a needle in it.", "[2,25]"], + ["Public Function FindTheNeedle(ByVal n As Long) As Long", "[3,28]"], + ["Dim needleCount As Long", "[4,13]"], + ["needleCount = n * 2", "[5,9]"], + ["Return needleCount", "[6,16]"], + ] }, + { path: "twinbasic:/AddinHost/Sources/Main.twin", count: "4 matches", matches: [ + ["Dim needle As Long", "[3,13]"], + ["needle = FindTheNeedle(3)", "[4,9]"], + ["needle = FindTheNeedle(3)", "[4,25]"], + ["Debug.Print needle", "[5,21]"], + ] }, + ]); + }); + + test("a click on a match opens its file at its line and column", async () => { + // The line itself: its [line,col] label is outside the element that + // carries the handler, and a click there opens the file's first match. + await click(c, { toolWindow: W, css: ".colMatch", text: "Dim needleCount As Long" }); + const ed = await waitFor(c, async (c) => { + const e = await editorState(c); + return e?.file === "/AddinHost/Sources/Haystack.twin" && e; + }); + assert.ok(ed, "Haystack.twin did not open in the code editor"); + assert.deepEqual([ed.line, ed.column, ed.lineText.trim()], [4, 13, "Dim needleCount As Long"]); + }); + + test("Match case narrows the search, and the add-in saves the option", async () => { + await click(c, { toolWindow: W, css: "#searchBarMatchCase" }); + const found = await waitFor(c, async (c) => { const r = await results(c); return r?.length === 2 && matchCount(r) === 7 && r; }); + assert.ok(found, `the results did not narrow to 7 matches: ${JSON.stringify(await results(c))}`); + assert.deepEqual(found.map((f) => f.count), ["4 matches", "3 matches"]); + assert.ok(!found.some((f) => f.matches.some(([line]) => line.includes("FindTheNeedle(ByVal"))), + "FindTheNeedle's declaration still matched"); + assert.deepEqual(savedSettings("GlobalSearchAddIn"), { Settings: { + insidePackages: "FALSE", matchCase: "TRUE", matchWholeWord: "FALSE", excludeComments: "FALSE", + } }); + }); +});