diff --git a/content/components/agent/_index.md b/content/components/agent/_index.md index cfd53a5..d8afd0e 100644 --- a/content/components/agent/_index.md +++ b/content/components/agent/_index.md @@ -47,7 +47,7 @@ cd agent **Version Management**: ```powershell # Check current version -.\version.ps1 +Get-Content .\agent\VERSION # Build with current version .\build.ps1 release diff --git a/content/development/auto-update-plan.md b/content/development/auto-update-plan.md index a8ec60a..649fd4d 100644 --- a/content/development/auto-update-plan.md +++ b/content/development/auto-update-plan.md @@ -89,28 +89,30 @@ pre-0.32.0 release pass, per the owner's 2026-10-19 sequencing decision. They remain a release gate, not accepted or resolved risk; this updater slice does not investigate or change dependencies. -### Recommended execution order: nine remaining items +### Recommended execution order: eight remaining items -Hosting and the application-policy/descriptor item are complete within their -documented scope. The remaining work is ordered by actual prerequisites, not -by the historical checklist numbering: +Hosting, application policy/descriptors, and the original installation-permission +gate are complete within their documented scopes. The permission gate closed +on 2026-10-20 with Linux/macOS separate-identity execution and an actual Windows +MSI/SYSTEM access-control check; see the +[permission qualification matrix](#permission-qualification-matrix). +The remaining work is ordered by actual prerequisites, not by the historical +checklist numbering: -1. **Implemented, pending platform qualification:** installation permissions, - safe paths and verification-to-use protection. -2. **Implemented, pending platform qualification:** failure-safe Windows +1. **Implemented, pending platform qualification:** failure-safe Windows binary/MSI helper handoffs, bounded deadlines and durable run-bound outcomes. Real MSI/service behavior and failure recovery still require qualification. -3. **Fixed helper/supervision and isolated DEB transaction tests implemented:** +2. **Fixed helper/supervision and isolated DEB transaction tests implemented:** finish Linux packaging/Manager/service wiring, failed-transaction recovery, real service and RPM qualification before enabling unattended installation. -4. Client trust-state lifecycle, clock/expiry diagnostics and explicit repair. -5. Truthful bounded update UX. -6. Supported 0.31.x, mixed-version and long-offline migration qualification. -7. Final-beta dependency vulnerability triage/fixes. -8. Explicitly approved production trust provisioning, verified independent +3. Client trust-state lifecycle, clock/expiry diagnostics and explicit repair. +4. Truthful bounded update UX. +5. Supported 0.31.x, mixed-version and long-offline migration qualification. +6. Final-beta dependency vulnerability triage/fixes. +7. Explicitly approved production trust provisioning, verified independent encrypted backups, public-root embedding and scoped credentials; activation initially remains off. -9. Final adversarial/platform/service/package/release qualification and go/no-go. +8. Final adversarial/platform/service/package/release qualification and go/no-go. Production activation requires separate explicit approval after the gates. Each item includes its own regression tests and documentation. Final @@ -889,19 +891,25 @@ remain separate gates. ### Installer permissions and safe-path implementation (unreleased) -Implementation: [Agent source commit](https://github.com/Printmaster-Org/printmaster/commit/da0f9e2c3714bea251512f3363328f9b8d21d22a). +Implementation: [permission qualification and installer fixes eac1f11](https://github.com/Printmaster-Org/printmaster/commit/eac1f11fbd289040a567b21d825c535792a5e9bb), +following the [initial safe-path implementation](https://github.com/Printmaster-Org/printmaster/commit/da0f9e2c3714bea251512f3363328f9b8d21d22a). Updater state creation now fails closed unless it can establish private permissions before metadata or update artifacts are used. Unix data directories are restricted to the service identity; updater state rejects symlinks, special files and multiply linked files. Installation targets must be ordinary files owned by root or the updater identity, with safe ancestors and no hard links. +Every Unix ancestor must also be owned by root or the updater identity, even +when its current mode is not writable: a foreign owner could grant itself +replacement rights later. Same-filesystem replacement remains an atomic rename; cross-filesystem copy-over is now rejected rather than truncating the installed executable. Windows updater state and publisher metadata receive protected ACLs for the running identity, SYSTEM and local Administrators; existing updater contents -are resecured and reparse points are rejected. Raw Windows service installs +are resecured and reparse points and multiply linked files are rejected. +Windows raw-service source/installed executables must also have exactly one +hard link before protection is applied. Raw Windows service installs register a protected copy under `Program Files\PrintMaster\Agent`, with ProgramData service directories restricted to trusted principals. The manual PowerShell update script now writes the Agent to that location and elevates @@ -909,22 +917,75 @@ before stopping services. A service still registered from an older writable location must be reinstalled with `--service update` before automatic updating is accepted from it. -These controls are implementation and targeted regression coverage, not -qualification of every supported installer. Windows ACL tests ran on Windows; -Unix execution on Linux/macOS, MSI-created paths, existing-install migration, -and service/helper failure outcomes remain to be qualified. The updater does -not defend against local administrators, a compromised service identity, or -arbitrary machine rollback. MSI execution and package-manager handoffs keep -their separate qualification gates; cross-filesystem raw-binary replacement -is unsupported. +The unreleased MSI now owns the exact canonical +`Program Files\PrintMaster\Agent\printmaster-agent.exe` rather than placing a +second executable one directory above the service's registered copy. Bootstrap +configuration is adjacent under `Agent\config`. The components whose key-file +paths moved have new component GUIDs. This fixes fresh-install ownership/layout; +it does **not** qualify an existing MSI's major upgrade, repair or service +re-registration. + +#### Permission qualification matrix + +Local evidence from the 2026-10-20 permission-gate pass: + +| Surface | Observed evidence | Status / limit | +| --- | --- | --- | +| Linux private updater/trust trees | Root launches disposable processes as UID/GID 60001 (service) and 60002 (outsider), without creating accounts. The service writes private artifacts and atomically updates its own raw-binary fixture; the outsider cannot read, overwrite, delete or inject private evidence. Both can write their own positive-control directories. | Passed three consecutive root runs under WSL Linux. No installed service modified. | +| Linux root-owned package executable | The unprivileged service can inspect the trusted executable but actual raw replacement returns OS permission denial; both processes leave package bytes unchanged. | Passed. Native package-manager integration remains gate 3. | +| Unix unsafe ownership / links | Reject foreign-owned 0755 ancestors, writable parents/targets, hard links and linked state/trust entries. | Passed on Linux and native hosted macOS, including root-only ownership cases. | +| Cross-filesystem raw replacement | Real `/dev/shm` versus temporary-filesystem rename is rejected; old and candidate bytes remain intact. | Passed on Linux. Copy-over remains unsupported. | +| Windows private state | A real restricted Windows token with BUILTIN Users as its restricting SID is denied read, write, delete, ACL-change and adjacent-file creation; an explicitly Users-writable positive control succeeds. | Passed on Windows without elevation. This is OS access enforcement, not a separate account or a SYSTEM-service installation. | +| Windows aliases / redirection / service copy | Real NTFS hard links are rejected for installed executable, state, trust, service-file protection and a separately launched source executable. A directory junction exercises reparse rejection without requiring symbolic-link privilege. Disposable raw-service copy tests verify the canonical path and protected DACL. | Passed locally and on hosted Windows. | +| MSI ownership/layout and actual permissions | Built the actual Agent MSI with release-workflow WiX 3.11.2 and inspected its compiled File, Component and Directory tables. After explicit owner approval, unregistered the disposable test Agent, archived its executable/data, and installed the MSI with exit 0. A SYSTEM task verified the running canonical SYSTEM service, protected executable and inherited service-created candidate/backup/descriptor/trust fixtures. A restricted Users token could not write/delete/change ACLs/inject files, or read private evidence; its Users-writable positive control passed. | Actual installation and opted-in `TestLiveWindowsInstallerPermissions` passed. A locked old data directory required preserving the remainder by copy, so this was a fresh MSI executable install with retained data directories, not a clean-machine data-volume test. State was initialized by the enrolled service, not repaired by the test. Existing ICE61 same-version-upgrade warning remains; major upgrade/repair/failure recovery are separate gates. | +| Native macOS | Hosted native permission/substitution tests pass. The separate root invocation actually launches UID/GID 60001 and 60002 and passes both positive controls, private-state denial, service-owned atomic replacement and foreign-owner cases. | Passed in [CI run 39481623764](https://github.com/Printmaster-Org/printmaster/actions/runs/39481623764), on test branch `test/installer-permissions-20261020` at `eac1f11`. Root-only cases skip in the ordinary-user invocation and pass in the explicit root step; they are not counted as covered by the skips. No claim of live launchd/service lifecycle qualification. | + +**The original gate 1 is closed within this permission/safe-path scope; eight +top-level gates remain.** The original gate 2 still owns MSI/helper transaction +failure/recovery; original gate 3 owns native package/service integration; +original gate 6 owns supported 0.31.x migration. Clean-machine installation, +live launchd service behavior, wider adversarial handoffs and final release +go/no-go remain in platform/release qualification. None is waived by these +tests. The CI Passed aggregate requires the reusable Windows/macOS permission +job, and the Linux root job includes the ownership/identity cases. + +The local Windows fixture initially used an unenrolled loopback configuration; +registration failed and the updater correctly did not initialize. Restoring +the protected test backup's original identity, configuration and credential +files allowed service-created updater initialization and the SYSTEM check to +pass. This is not a new onboarding or migration guarantee. The installed test +Agent is left running as SYSTEM at the canonical path. Its protected recovery +backup is `C:\ProgramData\PrintMaster-PermissionQualification-20261020`; +the temporary SYSTEM task was removed. RemoteSigned is unchanged and production +publisher trust/unattended activation remain off. + +The wider Agent regression reruns also exposed failures outside this permission +selection: WebSocket fixed-wait timing, a source-observation probe assertion, +a Windows helper fixture outcome and a short-budget SNMP fixture connection/ +cleanup panic. Local Jest also hit a Bash subprocess timeout/cleanup failure +in the Beta-release fixture (22 suites passed, one failed). The affected +permission/workflow Jest selection passes. Constrained Agent reruns pass the +complete updater package but fail the source-observation assertion. Hosted +Linux Common/Agent/Server, JavaScript/browser tests, native Windows/macOS, +and mock/real-Docker E2E all pass in the linked run, including the required +**CI Passed** aggregate at `eac1f11`. That is a green hosted CI baseline, not +proof that the earlier local failures did not occur. Those local failures +remain visible and must be reconciled before final release. +This slice does not alter unrelated tests or hide failures with retries/ +time-budget changes. + +The updater does not defend against local administrators, a compromised service +identity, or arbitrary machine rollback. Path +checks/rehashing are not a claim of race resistance against those identities. ### Remaining 0.32 release gates: security handoff reconciliation This checklist reconciles the security handoff with application revision `0d64d3d`, not the contributor fork implementation. These are required outcomes, not newly completed controls. **P0-06 (independent update trust) remains open.** -**P0-07 (privileged updater hardening) remains open for installer and platform -qualification** after the permission/safe-path implementation below. +**P0-07 (privileged updater hardening) remains open for transaction, migration +and final platform/service qualification**, although its permission/safe-path +subgate is now qualified above. The security handoff is a code-inspection assessment, not an exploit demonstration. @@ -948,9 +1009,10 @@ demonstration. updater/trust trees, rejects symlink/reparse redirection, binds final paths to the TUF descriptor and previous backup, and uses atomic Unix replacement without a copy-over fallback. Raw Windows service installation now registers - a protected Program Files copy. Verify these guarantees with actual Linux/ - macOS service identities, Windows installer-created ACLs, MSI/package paths - and supported 0.31.x migrations. Test adversarial substitution and failures + a protected Program Files copy. Separate actual Linux/macOS UIDs and the + Windows MSI/SYSTEM access-control boundary now pass the matrix above. + Continue qualification of native service lifecycles, MSI/package transaction + paths and supported 0.31.x migrations. Test adversarial substitution and failures at each platform handoff. A raw Windows service still registered outside the protected location must be migrated before auto-update; MSI and package transactions are not qualified by the raw-binary path. Do not treat path diff --git a/content/development/build-workflow.md b/content/development/build-workflow.md index 3e487df..fcd3ab2 100644 --- a/content/development/build-workflow.md +++ b/content/development/build-workflow.md @@ -6,40 +6,23 @@ sourceCommit: "565f4c0762e467f083f54a9e0e7f6bc23ada56c3" ## Quick Start (Development) -### Windows PowerShell Quick Launch - -Use the helper script to test, build, and launch the agent: - -```powershell -# From project root (where dev/ exists) -pwsh -NoProfile -ExecutionPolicy Bypass .\dev\launch.ps1 -``` - -This script will: -- Run `go test ./...` (exits if tests fail) -- Build agent into `./bin/printmaster-agent.exe` -- Start the built binary -- Open browser to `http://localhost:8080` - -### Manual Development Workflow - -If you prefer manual control: +### Windows PowerShell ```powershell -# Run tests -go test ./... - -# Build (from project root) -go build -o ./bin/printmaster-agent.exe ./agent +# From the application root: gates on lint, Go and syntax/Jest, then builds +.\build.ps1 agent # Run the agent -./bin/printmaster-agent.exe +.\agent\printmaster-agent.exe # Open UI Start-Process 'http://localhost:8080' ``` -**Note**: The launch script is intentionally conservative - tests must pass before build and server start. +On Unix/macOS use `./tests.sh` and `./build.sh agent`. The shared runner does not +launch or replace installed services. See [Testing](/development/testing/) for +prerequisites, optional Docker/docs suites and explicit skips. The legacy +`dev/launch.ps1` convenience launcher is not the complete test baseline. --- @@ -57,8 +40,15 @@ Start-Process 'http://localhost:8080' # Build both .\build.ps1 both +# Add browsers, Docker, docs and native fixtures +.\build.ps1 agent -AllTests + +# Explicit narrower gate or complete bypass +.\build.ps1 agent -TestArgs @('--suite','lint,go') +.\build.ps1 agent -SkipTests + # Run tests -.\build.ps1 test-all +.\tests.ps1 # Clean artifacts .\build.ps1 clean @@ -66,18 +56,22 @@ Start-Process 'http://localhost:8080' ### VS Code Tasks (Ctrl+Shift+B) -JavaScript validation uses separate runners: `npm run test:js` runs Jest unit -tests, and `npm run test:playwright` runs browser tests under -`common/web/__tests__/playwright`. CI runs the cross-browser variant, -`npm run test:playwright:all`. Device selection/details controls are checked by +The default **Test: Core** task and `tests.ps1`/`tests.sh` run the same shared +suite definitions used by CI: Go vet/staticcheck, every discovered Go module, +Jest and all six Playwright projects with no retries. `--skip-playwright` +explicitly omits browsers; it is not a complete baseline. Device selection/details controls are checked by Playwright with column pinning and horizontal scrolling; browser checks must not be placed under Jest's unit-test directories as standalone scripts. -This runner separation and device-control regression were verified against -[source commit cb88bee](https://github.com/Printmaster-Org/printmaster/commit/cb88bee07d0393563b250a2b9d00263363d3ff54). +CI installs prerequisites; the runner owns suite execution. Underlying npm +commands remain available for focused debugging. +CI launches six Playwright shards independently from Jest and the parallel +Go-module matrix, keeping two browser workers per runner. A normal local build +uses `--skip-playwright`; the standalone test command still includes browsers. - **Build: Agent (Dev)** - Default build task - **Build: Server (Dev)** - Build server - **Build: Both (Dev)** - Build both components +- **Test: Core** - Shared default suite selection - **Test: Agent (all)** - Run all agent tests - **Test: Server (all)** - Run all server tests - **Show Version** - Display current versions @@ -92,6 +86,15 @@ This runner separation and device-control regression were verified against ### Making Releases +Release scripts first call the build validation-only path, which invokes +`tests.ps1`/`tests.sh` with `--all`. This happens before version files, build +counters, tags, commits or pushes change. Docker, SQLite and sibling-docs/Hugo +prerequisites must be available; failures stop the release. Compilation after +the successful gate skips duplicate validation. Explicit `-TestArgs` / repeated +`--test-arg` overrides replace the default selection, while `-SkipTests` / +`--skip-tests` bypasses it visibly. See [Testing](/development/testing/#build-and-release-gates) +for options and qualification limits. + **Unreleased 0.32 trust safeguard:** tagged Stable/Beta Agent CI builds, local Agent `-Release`/`--release` builds and non-Dev Agent Docker builds now require a provisioned, self-authorized, unexpired public TUF root with the initial @@ -436,7 +439,7 @@ git checkout -b feature/new-scanner # 2. Make changes, test locally .\build.ps1 agent -.\build.ps1 test-all +.\tests.ps1 # 3. Commit work git add -A diff --git a/content/development/project-structure.md b/content/development/project-structure.md index 1632db2..3f36c70 100644 --- a/content/development/project-structure.md +++ b/content/development/project-structure.md @@ -66,14 +66,15 @@ and [SNMP reference downloads](/development/snmp-reference/#reference-downloads) for the Printer-MIB resources. ### Tooling & Automation -- `build.ps1`, `release.ps1`, `status.ps1`, `version.ps1`: primary automation entrypoints. +- `build.ps1`/`build.sh`, `release.ps1`/`release.sh`, `tests.ps1`/`tests.sh`: build, release and shared test entrypoints. - `dev/launch.ps1`: developer convenience launcher (tests + run agent). -- `package.json` + `playwright.config.js`: UI tests (Jest/Playwright). +- `scripts/run-tests.js`: shared suite orchestration used by local wrappers and CI; discovers top-level Go modules. +- `package.json` + `playwright.config.js`: underlying UI runners (Jest/Playwright). - `.vscode/tasks.json` (generated) + VS Code tasks listed in `BUILD_WORKFLOW.md`. ### Tests & Fixtures - Go unit tests live beside their packages. -- `tests/` (with supporting `test-results/`) will hold cross-component or UI regression suites. +- `tests/` contains mock and tagged Docker-backed cross-component suites; `test-results/` holds generated evidence. - `common/web/__tests__` contains Playwright specs invoked via `npm run test:playwright`. ## Architecture Principles (Current) diff --git a/content/development/testing-ci.md b/content/development/testing-ci.md index 4ee97c1..7309497 100644 --- a/content/development/testing-ci.md +++ b/content/development/testing-ci.md @@ -6,6 +6,11 @@ sourceCommit: "565f4c0762e467f083f54a9e0e7f6bc23ada56c3" This document describes how to write fast, deterministic tests for the PrintMaster agent and server. +Run `.\tests.ps1` (Windows) or `./tests.sh` (Unix/macOS) from the application +checkout for the shared default lint/Go/Jest/Playwright baseline. CI invokes +the same suite definitions. See [Testing](/development/testing/) for +prerequisites, selection flags and platform qualification boundaries. + ## Test Types PrintMaster has three levels of testing: @@ -65,20 +70,26 @@ For full-network tests that exercise the real SNMP stack, maintain separate manu E2E tests that verify agent-server communication are located in the `tests/` directory at the project root. These tests: -- Start actual server and agent instances +- Cover mock cross-component workflows by default, with actual Docker services in the opt-in tagged suite - Test full workflows (registration, heartbeat, WebSocket, proxy) -- Run in CI/CD after unit tests pass +- Run in CI/CD as shared runner suites - Use ephemeral ports and temp databases See [`../tests/E2E_TESTING.md`](/development/testing/e2e-testing/) for complete documentation. -**Running E2E tests:** +**Running cross-component tests from the repository root:** ```bash -cd tests -go test -v ./... +# Mock E2E (also included in the default Go suite) +./tests.sh --suite go --module tests + +# Actual isolated Docker services, offline SQLite seeding and tagged tests +./tests.sh --suite docker ``` -**Skipping E2E tests (fast unit tests only):** +**Focused component Go tests (not the complete baseline):** ```bash -go test -short ./... -``` \ No newline at end of file +./tests.sh --suite go --module agent,common,server +``` + +Use `.\tests.ps1` with the same arguments on Windows. `--with-docker` adds +both PostgreSQL integration and real Docker E2E to the default baseline. \ No newline at end of file diff --git a/content/development/testing/_index.md b/content/development/testing/_index.md index aba442f..e938d58 100644 --- a/content/development/testing/_index.md +++ b/content/development/testing/_index.md @@ -6,62 +6,194 @@ sourceCommit: "565f4c0762e467f083f54a9e0e7f6bc23ada56c3" This directory contains integration/E2E tests that test agent and server components together. -## Full repository baseline +## Shared repository test entrypoints -A green mock E2E run alone is not a full release baseline. Run the uncached Go -suites in **each** module (`common`, `agent`, `server`, `tests`): +From the application checkout use `.\tests.ps1` on Windows or `./tests.sh` +on Unix/macOS. Both thin wrappers and CI execute the same +`scripts/run-tests.js`; test commands, selection rules and module discovery +are not copied between shells. Direct Go/npm commands remain useful for +targeted diagnostics, not as competing definitions of the full baseline. -```sh -go test -count=1 -timeout=10m ./... -``` - -Run `go vet ./...` and the repository-compatible `staticcheck ./...` in the -Common, Agent and Server modules for the relevant platform targets. Also run -the Linux root/native-package qualification described in the -[auto-update plan](/development/auto-update-plan/). - -At the repository root, run `npm run test:js`, then the complete browser matrix. -For a constrained Windows host, use PowerShell environment assignment rather -than the POSIX assignment in `test:playwright:all`: +The shared entrypoints and CI wiring were introduced in +[source commit 6af4bf4](https://github.com/Printmaster-Org/printmaster/commit/6af4bf408806d6c995cdcb95fb4339d36f3d1b96). +The unused root `commit`, `update`, `version` and `status` script pairs were +removed after checking callers; build/release and their used internal helpers +remain supported. ```powershell -$env:PW_ALL_BROWSERS = '1' -$env:PW_WORKERS = '1' -npm run test:playwright +.\tests.ps1 # Core: lint, Go, Jest and all six browser projects +.\tests.ps1 --skip-playwright # Explicit partial run +.\tests.ps1 --suite go --module agent --package ./storage +.\tests.ps1 --with-docker # Core plus PostgreSQL and isolated Docker E2E +.\tests.ps1 --with-docs # Core plus sibling docs validation +.\tests.ps1 --all # Every nonprivileged suite +.\tests.ps1 --all --skip-docker # Also omits PostgreSQL (it requires Docker) +.\tests.ps1 --suite playwright --workers 1 +.\tests.ps1 --dry-run +.\tests.ps1 --help ``` -One worker is a local diagnostic option, not a change to CI's two-worker -configuration. Copy-control tests retain all font/width, truncation, control -geometry and clipboard assertions, but give each font/width case its own -test budget rather than combining thirty value interactions into one timeout. +| Suite | Coverage / prerequisites | +| --- | --- | +| `lint` | `go vet` and existing compatible `staticcheck` for every discovered module, including the mock E2E module. | +| `go` | Uncached verbose `go test` with a ten-minute per-package timeout in each top-level directory containing `go.mod`. New/removed modules are picked up automatically in local runs and CI. Mock E2E is included through the `tests` module. | +| `js` | Syntax-checks Agent/Server/common web JavaScript, then all Jest suites through the existing locked dependency. | +| `playwright` | Builds Agent/Server binaries, then all six configured projects with two workers by default and no retries. | +| `integration` | Docker preflight, then actual tagged PostgreSQL storage tests. Windows passes the active context's endpoint only to the testcontainers Go subprocess when not explicitly configured; later Docker CLI suites keep their original context. | +| `docker` | Builds an isolated unique Compose project with fresh temporary data, loopback-only dynamic ports, offline SQLite seeding and actual tagged host Go E2E tests. Saves container logs and removes only that project's containers/network/images and temporary data, including on test failure. | +| `native` | Bounded native permission, descriptor-substitution and recovery regression selection. | +| `docs` | Sibling `docs.printmaster.work` Node tests, fresh Hugo build and generated links/search checks. | +| `privileged` | Explicit Unix-only root permission/package fixtures via noninteractive `sudo -n`; includes isolated real Debian installation on Linux. Never modifies installed Agent services. | + +The default is **core only** by owner choice: Docker, docs and privileged +fixtures are not silently included. `--all` includes native/Docker/docs but +still requires `--privileged` to add root fixtures. `--suite` selects only +named suites (comma-separated or repeatable); `--module` limits Go/lint +modules and `--package` limits only Go tests. `--skip-NAME` omits a suite; +`--skip-docker` also removes integration. A request selecting no suites, +invalid module/flag or missing prerequisite fails rather than appearing green. +`--coverage` writes each module's `coverage.out` and the separate PostgreSQL +`coverage-integration.out`. `--dry-run` only prints selection and omissions; +it is not execution evidence. + +Install prerequisites separately: Go matching the module files, Node.js 26, +`npm ci`, compatible staticcheck, and +`npx playwright install --with-deps chromium firefox webkit` where appropriate. +The runner never installs missing tools or starts Docker Desktop automatically. +Docker E2E additionally needs host `sqlite3`; on Linux its fresh root-owned +database seeding/cleanup may need noninteractive sudo. Optional docs need the +sibling checkout's locked dependencies and its configured Hugo version. +Use `--staticcheck PATH`, `--hugo PATH`, or `--docs-dir PATH` for existing +non-PATH tools/checkouts. These do not change machine configuration. + +Staticcheck must also be **built with a Go version at least as new as the +modules it analyzes**. Updating its release alone is insufficient. If it reports +`application built with go1.26` against Go 1.27 modules, rebuild the CI-pinned +version with the matching toolchain, for example +`GOTOOLCHAIN=go1.27.0 go install honnef.co/go/tools/cmd/staticcheck@v0.8.1` +on Unix, or set `$env:GOTOOLCHAIN = 'go1.27.0'` in a PowerShell session before +that `go install` command. `go version -m` on the installed executable shows +its build toolchain. The runner does not silently replace tools. + +Execution stops at the first failed command by default, including within a +Go/lint module. Later selected suites are marked **NOT RUN**, not passed or +intentionally omitted. Use `--keep-going` to collect independent failures across +checks, modules and suites. Docker log capture and scoped cleanup still run on +failure in either mode. The runner prints PASS/FAIL and omitted-suite summaries +and returns nonzero on any selected failure. CI uses the same fail-fast default +within each job while independent jobs can still complete. +Internal platform/opt-in skips remain visible in verbose test output; they +are **not** converted into qualification evidence. Inherited +`PRINTMASTER_LIVE_*` opt-ins are stripped from subprocess environments so a +baseline cannot accidentally enable live service/installer replacement. Do not use retries, forced clicks, removed assertions or global timeout -increases as evidence that an unexplained browser failure is fixed. +increases to hide unexplained failures. `--workers 1` is a local resource +option, not evidence that a two-worker failure did not happen. + +CI selects these same suites, automatically discovering Go modules, while +running each Go module in its own matrix job. Jest, PostgreSQL, Docker E2E, +native platform checks and **six Playwright shards** start independently; +Playwright does not wait for Jest or Go tests. Each shard builds its own fixture +binaries and runs two workers, avoiding the documented contention from raising +workers on one small runner. `--shard INDEX/TOTAL` selects one shard without +changing projects, assertions or retries; local runs remain unsharded by default. +All shards/modules remain required by CI Passed, even if another fails. +Linux/macOS root fixtures are explicit jobs. CI owns tool/dependency +setup and caching, not a second set of test commands. The sibling docs site +retains its own CI; use `--with-docs` locally for paired changes. + +Observed CI qualification on 2026-10-21: + +| Layout | Run | End-to-end elapsed | +| --- | --- | --- | +| Original grouped workflow | [39481623764](https://github.com/Printmaster-Org/printmaster/actions/runs/39481623764) | 9m26s | +| Single browser job, sequential Go modules | [39489694968](https://github.com/Printmaster-Org/printmaster/actions/runs/39489694968) | 10m25s | +| Three browser shards, parallel Go modules | [39602215312](https://github.com/Printmaster-Org/printmaster/actions/runs/39602215312) | 6m01s | +| Six browser shards, parallel Go modules | [39603309582](https://github.com/Printmaster-Org/printmaster/actions/runs/39603309582) | 3m27s | + +The final run qualified [source e31370d](https://github.com/Printmaster-Org/printmaster/commit/e31370ddb420e6fdbe13189dc83018ba98ba5a37): +all required jobs passed, with 327 Jest tests and six browser shards of 114 +cases each (662 passed, 22 explicitly skipped overall). No browser assertions, +projects or retries were weakened. The real Windows default build also passed +lint/Go/JavaScript before compilation. Bash/PowerShell fixtures verify release +failure before version/tag mutation and one successful gate before both builds. +These timings include setup/queue time and reflect individual runs, not a +latency guarantee; extra runners trade setup/runner minutes for lower wall time. + +### Build and release gates + +Basic `build.ps1`/`build.sh` builds call the wrappers once with +`--skip-playwright`: lint, all discovered Go modules (including mock E2E), web +syntax and Jest. They do not start Docker or browsers by default. +`-AllTests` / `--all-tests` selects all nonprivileged suites. To replace the +selection, pass `-TestArgs @('--suite','lint,go')` in PowerShell or +`--test-arg --suite=lint,go` in Bash; repeat `--test-arg` for more arguments. +`-SkipTests` / `--skip-tests` is an explicit logged bypass. + +Release calls **build validation-only -> tests --all** before version changes, +build counters, tags, commits or pushes. A failed gate cannot start the release. +After a successful gate, compilation uses an explicit skip flag to avoid +repeating tests for each component. `-TestOnly` / `--test-only` exposes the +validation-only build path without compiling. Release supports `-TestArgs` / +repeated `--test-arg` for explicit selection/prerequisite paths, or `-SkipTests` / +`--skip-tests` as a deliberate bypass. Overrides replace the default `--all`, +so include `--all` in custom arguments when only supplying `--hugo PATH`. +Dry runs preview the gate without executing it. + +Missing Docker, SQLite, docs checkout/dependencies or Hugo fail the full gate; +there is no automatic installation, root elevation or silent downgrade. +Root fixtures still require explicit Unix `--privileged`. Local validation +does not replace the required cross-platform CI result or production trust +qualification. Legacy `PRINTMASTER_*_PASSED` and `PRINTMASTER_SKIP_*` environment +shortcuts no longer bypass builds; use explicit flags instead. + +A green core run is not a complete release baseline. Run `--all` and the +applicable explicitly privileged/platform qualification before go/no-go. +Report omitted suites and internal skips; passing one host or configuring CI +does not prove execution on another platform. + +## Installer permission qualification + +From the application repository root, run the bounded native selection: + +```sh +./tests.sh --suite native +``` + +Run the ownership/separate-identity cases as root on a disposable Unix host: -With Docker responsive, run `go test -count=1 -tags=integration ./storage/...` -from `server`, and the separately tagged -[real Docker E2E workflow](/development/testing/e2e-testing/). -On Windows, if testcontainers cannot discover Docker Desktop, set `DOCKER_HOST` -to the endpoint reported by the active Docker context for that process. -Check verbose output: an integration suite that skips for missing Docker is -**not** qualified. Do not start a shared Docker daemon without owner approval. +```sh +./tests.sh --suite privileged --privileged +``` -Finish with the canonical docs site's tests, fresh Hugo build and generated-site -link checks. Report failures and environmental skips separately from passing -tests; local success is not proof that the unpushed remote CI workflow ran. +On macOS the runner resolves its temporary executable directory and sets +`TMPDIR=/private/tmp` for native/root test processes rather than testing through +a `/tmp` or `/var` symbolic-link ancestor. Numeric fixture UIDs do not create +system accounts, and these tests never install or control the real Agent. +The Linux cross-filesystem case requires `/dev/shm` on a different filesystem; +report a skip explicitly if unavailable. + +On Windows, the restricted-token test uses actual OS access checks and a +positive control, not an elevated account or simulated ACL evaluator. Junction +tests avoid requiring symbolic-link privilege. The separate Windows/macOS CI +job is required by CI Passed; its native Windows/macOS execution passed in +[run 39481623764](https://github.com/Printmaster-Org/printmaster/actions/runs/39481623764). +Adding or compiling a test alone is not execution evidence. Actual MSI +permissions also passed an owner-approved local installation and opted-in +SYSTEM task (`TestLiveWindowsInstallerPermissions`). That destructive test is +skipped unless `PRINTMASTER_LIVE_INSTALLER_PERMISSIONS=qualify-test-agent-msi` +is explicitly set and it runs as SYSTEM against the installed test Agent. +It must not repair ACLs before checking them, and must never run on an +unapproved installation. For observed evidence and qualification limits, see the +[permission qualification matrix](/development/auto-update-plan/#permission-qualification-matrix). ## Running Tests ```bash -# Run all E2E tests -cd tests -go test -v ./... - -# Run with no cache -go test -v -count=1 ./... - -# Skip E2E tests (in other directories) -cd ../agent -go test -short ./... +# Mock E2E only (uncached by default) +./tests.sh --suite go --module tests +# Explicit component-only Go run +./tests.sh --suite go --module agent,server ``` ## Test Coverage diff --git a/content/development/testing/e2e-testing.md b/content/development/testing/e2e-testing.md index 41bc243..29330d5 100644 --- a/content/development/testing/e2e-testing.md +++ b/content/development/testing/e2e-testing.md @@ -18,39 +18,40 @@ SQLite databases with `server_data.sql` and `agent_data.sql`, then restarts and waits again. Do not seed a live database through a second SQLite implementation. Current data fixtures contain eight devices per database. -Run the tagged tests from the host's `tests` module, with explicit -`E2E_SERVER_URL`, `E2E_AGENT_URL` and `E2E_ADMIN_PASSWORD` environment variables, -using `go test -tags=e2e -v -count=1 ./...`. Defaults for the HTTP-only Compose -environment are `http://localhost:9090` and `http://localhost:8080`. -On a shared machine, use a private Compose project, container/network names, -loopback ports and disposable data mounts; do not adopt or delete existing test -databases or interrupt installed services. Tear down only that test project. - -The optional Compose `test-runner` service is **not** this qualification route: -its existing configuration still targets the old Server port and does not -reliably preserve the Go runner's exit status through its shell pipeline. -These runner-service issues remain open; do not treat that service's exit -status as qualification evidence. The actual CI workflow runs Go on the host. +The shared `docker` suite automates this route, then runs tagged Go tests on +the host with explicit `E2E_SERVER_URL`, `E2E_AGENT_URL` and +`E2E_ADMIN_PASSWORD`. It requires Docker Compose, host Go and `sqlite3`. +Each invocation uses a unique Compose project, temporary data mounts and +dynamic loopback-only ports. It captures logs in `test-results/` and attempts +project-scoped teardown on normal completion or command failure. It does not +adopt existing databases, prune Docker globally or interrupt installed services. + +The optional Compose `test-runner` service now targets Server port 9090 and +preserves Go's exit status when displaying its log. It is still **not** the +shared CI qualification route, which runs Go on the host. ### Run E2E Tests Locally ```bash # Linux/macOS -cd tests -./run-e2e.sh +./tests.sh --suite docker # Windows -cd tests -.\run-e2e.ps1 +.\tests.ps1 --suite docker ``` ### Options | Option | Description | |--------|-------------| -| `--build` / `-Build` | Force rebuild Docker containers | -| `--keep-up` / `-KeepUp` | Leave containers running after tests | -| `--verbose` / `-Verbose` | Show detailed output | +| `--suite docker` | Build and qualify isolated real services, then tear down that project | +| `--suite go --module tests` | Run mock E2E without Docker (also part of the default baseline) | +| `--with-docker` | Add real Docker E2E and PostgreSQL integration to the default baseline | +| `--dry-run` | Show selection only; does not execute or qualify anything | + +Run these commands from the application checkout. See [Testing](/development/testing/) +for the full suite/skip reference. Legacy `tests/run-e2e.*` helpers are not the +canonical CI baseline. ## Architecture @@ -223,70 +224,36 @@ curl http://localhost:8080/api/health ## Running Tests ```bash -# Run all E2E tests -cd tests -go test -v ./... - -# Run specific test -go test -v -run TestWebSocketProxy_AgentUI - -# Skip E2E tests (fast unit tests only) -cd agent -go test -short ./... -cd ../server -go test -short ./... +# Mock E2E +./tests.sh --suite go --module tests + +# Real Docker E2E +./tests.sh --suite docker + +# Focused component Go tests only (not a complete baseline) +./tests.sh --suite go --module agent,server ``` ## CI/CD Integration ```yaml -# .github/workflows/test.yml -- name: Run Unit Tests - run: | - cd agent && go test -short ./... - cd ../server && go test -short ./... - -- name: Build Binaries - run: | - ./build.ps1 both - -- name: Run E2E Tests - run: | - cd tests && go test -v -timeout 5m ./... +# Excerpts from .github/workflows/ci.yml; prerequisites are installed by CI. +- name: Go modules + run: bash tests.sh --suite lint,go --coverage +- name: Real Docker E2E + run: bash tests.sh --suite docker ``` -## Next Steps - -1. **Implement process management helpers** - - `startServer()` - Start server on ephemeral port - - `startAgent()` - Start agent connected to test server - - `cleanup()` - Ensure processes are killed - -2. **Add config generation** - - Minimal TOML for server - - Minimal TOML for agent - - Temp database paths - -3. **Implement first working test** - - TestAgentServerRegistration - - Verify agent can register and connect - -4. **Add WebSocket proxy tests** - - TestWebSocketProxy_AgentUI - - TestWebSocketProxy_DeviceUI - - TestWebSocketProxy_NoConnection - -5. **Add to CI/CD** - - Run after successful build - - Report test results - - Save logs on failure +The CI aggregate also requires JavaScript/browser, PostgreSQL and native +installer-permission jobs. Local selection flags must not weaken those required +jobs. Passing fixtures does not qualify live installer transactions or release +trust provisioning. ## Performance Considerations - E2E tests are slow (2-10 seconds each) - Use `t.Parallel()` where possible -- Don't run E2E tests by default (use `-short` flag) -- Keep E2E test count reasonable (<20 tests) +- Mock E2E is included by default; real Docker E2E is opt-in locally and required in CI - Focus on critical paths and new features ## Debugging Failed Tests