Skip to content

fix(named-controls, monkey): exit 2 means not measured, with the cause on the row (#660, #679) - #694

Merged
MendixMau merged 2 commits into
masterfrom
fix/honest-instrument-states-660
Nov 6, 2026
Merged

MendixMau merged 2 commits into
masterfrom
fix/honest-instrument-states-660

Conversation

@MendixMau

Copy link
Copy Markdown
Owner

What changed

#660 — project-tests/e2e/named-controls.js

  • The runner is spawned with stdio: ['inherit','pipe','pipe']. Its output is replayed to the caller after it exits. SETUP lines go to stdout and INSTRUMENT FAULT to stderr, so both streams are piped.
  • faultLine() keeps the last line that starts with INSTRUMENT FAULT or SETUP. The match is anchored, so the summary text "Fix the SETUP line(s) above" is not taken as the cause.
  • Runner status === 2 gives a FAULT row whose text includes that line, whether or not a findings file was written. judge(2, …) returns FAULT too.
  • Process exit codes: 2 if any row is FAULT, 1 if any is NOT PROVEN, 0 if all are PROVEN.

#679 — project-tests/e2e/monkey.js, config.js

  • config.js now has TEMPLATE_NAV and navIsTemplate(), the same pattern as TEMPLATE_PASSWORDS. The comparison ignores key order, and ALLOW_TEMPLATE_NAV=1 overrides it.
  • If NAV still equals the template sample, monkey prints a SETUP NAV is the unedited sample … line and exits 2 before H.launchBrowser(). No browser starts and no login is attempted.
  • A new exported exitCode() returns 1 on a crash-class finding, 2 when 0 of N targets were reached (this used to exit 0), and 0 otherwise. Code 3 still means an unknown --target.

Fixtures

Proof (scoped runs only)

Fixture This branch origin/master code
test-named-controls.sh 37/37 6 FAIL
test-monkey-template-nav.sh 10/10 6 FAIL (master launches Chromium and dies at page.goto in login)
test-monkey-oracle.sh 14/14 14/14
test-monkey-left-app.sh 22/22 —

Repo checks: check-scripts, check-portability, leak guard (LEAKGUARD_ALLOW_NO_DENYLIST=1 LEAKGUARD_BASE=origin/master) and check-pr-discipline are all clean.

Time cost (#687)

The new checks run per module run, inside existing rungs. Both make runs faster:

  • With a sample or stale NAV, monkey now ends before the browser launch in under 1 s. Before, it spent a Chromium launch, a login and up to the 1800 s rung budget fuzzing nothing.
  • named-controls adds no step; piping costs nothing measurable.
  • Can it turn red and force rework? Yes, but only where the run measured nothing. It turns that silent green into a FAULT that names its cause.

Regression risk after sync-project.sh

  • Exit codes:
    • monkey.js exits 2 (it used to exit 0) when NAV equals the template sample, or when 0 of N targets were reached.
    • named-controls.js exits 2 (it used to exit 1) when any row is FAULT. That includes the existing persona-mismatch and not-*.control.json rows.
  • Verdict flip: verify-module.sh maps rc 2 to FAULT (*:2 sits before info:*). A project still on the sample NAV, or whose NAV reaches no menu item, now shows the monkey rung as INSTRUMENT FAULT, and the module verdict becomes INCOMPLETE where it was green. This is intended: that run measured nothing. It is named in the CHANGELOG line. ALLOW_TEMPLATE_NAV=1 covers an app whose menu really is the sample's.
  • No caller of named-controls.js in bin/ or project-bin/ parses its exit code; it is run by hand per skills/journey-proof.md.
  • Its output is now replayed after the runner ends, not streamed live. The text is the same; only the timing changes.
  • Both layouts: no path logic changed. Git Bash: process.execPath is still spawned directly, with no bare wrapper name.
  • Baseline words: 0, because no skill text changed.

Fixes #660
Fixes #679

Review: pending independent review


Generated by Claude Code

…e on the row (#660, #679)

named-controls.js: pipe and replay the runner's stdout/stderr, keep its last
INSTRUMENT FAULT / SETUP line, score runner exit 2 as FAULT with that line, and
exit 2 when any row is FAULT (1 = NOT PROVEN, 0 = all PROVEN).

monkey.js: exit 2 before launching a browser when project.config.js's NAV still
equals the template sample (config.js TEMPLATE_NAV / navIsTemplate(),
ALLOW_TEMPLATE_NAV=1 overrides), and exit 2 when 0 of N targets were reached.
verify-module.sh maps rc 2 to FAULT, so a stale NAV now reads INCOMPLETE.

Fixtures: test-named-controls.sh (EXIT2 -> FAULT, cause line, CLI exit codes,
real-runner credentials abort), new test-monkey-template-nav.sh, and
test-monkey-oracle.sh's exit-code assertion follows exitCode(). Field run: the
scratch-project repros in #660/#679 (template NAV, TEST_PASS unset).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
Keep both sides of CHANGELOG.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013jQvaonym9V1wKKfAEPmRr
@MendixMau
MendixMau merged commit 911c9f8 into master Nov 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants