Skip to content

test-scratch: one boot per call and a warm scratch copy (3x faster test runs) - #690

Open
MendixMau wants to merge 3 commits into
masterfrom
feat/test-speed-one-boot
Open

MendixMau wants to merge 3 commits into
masterfrom
feat/test-speed-one-boot

Conversation

@MendixMau

Copy link
Copy Markdown
Owner

Why

In the MTA evaluation one test file cost 95 to 227 s while the tests themselves ran in 40 to 650 ms. The time goes into the scratch copy plus unit-index rebuild (30 to 450 s, mostly I/O) and mxbuild plus boot (50 to 120 s per boot), paid again for every file.

What

  • project-bin/test-scratch.sh: one boot per call. All given files (.test.mdl and .test.md) share one mxcli test call. A file with @cleanup none, or one where rollback-scan.sh (from Unit tests: inputs per decision, read committed rows back, scan what rollback will not undo #689, optional) reports an A line, gets its own boot so its rows do not reach the others. --per-file keeps the old shape.
  • Warm scratch copy. It persists with its own .mxcli and deployment/ and is re-synced from the live model each run (rsync -c for mprcontents/ and the .mpr). It is rebuilt cold on --cold, a new mxcli version, a different Mendix version, a changed userlib//javasource//widgets/ listing, an age over 7 days, a missing stamp, a leftover injection, or a run that did not finish. --no-cache gives the old behaviour.
  • Safety: a lock on the cache folder for the whole run (a second run waits, then falls back to a private cold copy). affected-tests.sh prints --cold for the deploy moment, so the gate never uses the warm copy.
  • Skills: unit-test.md §4a fast inner loop (one boot per call, set MXTK_SCRATCH_DIR to a short stable folder, watch/attach are not evidence) and a paragraph in testing-shape.md.

Proof (ProcureFlow, Mendix 11.15, shared heavily loaded machine)

Mode Wall
3 files, one per call, cold 3 boots, 193 to 227 s each
3 files, one call, cold 272 s
3 files, one call, warm 89 s
--watch, per change 26 to 42 s

Control: a wrong expect goes red in every mode. A @cleanup none writer and reader in one boot leaked (vendor count 7 instead of 6); with the split they pass. Load from other sessions made single runs vary 2 to 3 times; the phase split is in the batch notes.

Tests: test-test-scratch.sh 109/0 under /bin/bash 3.2 and bash 5 (new checks confirmed red when their fix is reverted), test-affected-tests.sh 152/0, portability and routing checks clean. test-app-restart.sh has the same 42 failures on master and is untouched.

Not included: a runtime test executor in mxcli (MTA-level 0.5 s per call). The gate needs one cold boot anyway; revisit if the inner loop is still too slow. Also found while measuring: --watch showed green after a failed rebuild, fixed upstream in mendixlabs/mxcli#1770.

Conflicts with #688 and #689 only in CHANGELOG.md.

🤖 Generated with Claude Code

MendixMau and others added 3 commits November 6, 2026 15:29
…opy, fast-loop section in unit-test

A run is copy + unit index, mxbuild + boot, restore; the tests are under 1 s. So:
- several test files in one call share one injection, build and boot; a file with
  @cleanup none gets its own boot (measured leak: a later file saw the row);
  --per-file opts out; --constant stays per boot
- the scratch copy persists and keeps its own .mxcli and deployment/, re-synced
  from the live model by rsync; dropped on --cold (the gate), a new mxcli, age,
  no stamp or a leftover test injection; --no-cache is the old throwaway copy
- unit-test.md 4a: the fast inner loop, and the measured limits of --watch
  (runs green against the old model when the rebuild fails) and --attach

Fixture tests/wave2/test-test-scratch.sh: 61 -> 85 checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- .test.md specs: collected from a directory, a named one is a test file (joins the boot, no double run)
- cache folder lock for the whole run; a second run waits, then uses a private cold copy
- in-progress marker: an unfinished run makes the next one cold
- stamp adds the Mendix version and a names+sizes hash of userlib/javasource/widgets
- mprcontents/ and the .mpr are synced by checksum
- own boot also for files flagged by project rollback-scan.sh (when present)
- affected-tests: the deploy moment prints --cold
- duplicated SPEED comment removed, constants line clarified, unit-test 4a notes on TMPDIR and per-call boot

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scan always exits 0 and prints a summary, so any output meant every file got its own boot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant