From 7f9616182fd67d10121b42647e507c69c1f75927 Mon Sep 17 00:00:00 2001 From: David Abram Date: Tue, 3 Nov 2026 17:42:43 +0100 Subject: [PATCH 1/6] plans: Define npm installer hardening workflow Specify the acceptance criteria and task sequence for fixing launcher permissions, validating signed manifest identity, replacing unrestricted tar extraction with bounded extraction, and atomically installing the native binary. Include packed-package smoke coverage, release wiring, and distribution documentation. Plan: harden-npm-installer Tasks: T01, T02, T03, T04, T05 Co-authored-by: SCE --- context/plans/harden-npm-installer.md | 116 ++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100644 context/plans/harden-npm-installer.md diff --git a/context/plans/harden-npm-installer.md b/context/plans/harden-npm-installer.md new file mode 100644 index 000000000..c87cc8258 --- /dev/null +++ b/context/plans/harden-npm-installer.md @@ -0,0 +1,116 @@ +# Plan: harden-npm-installer + +## Change summary + +Harden the `@crocoder-dev/sce` npm launcher package under `npm/` so that it is reliably executable after a real `npm pack` + `npm install`, only accepts a signed release manifest whose identity matches the package (version, binary `sce`, exactly one canonical target artifact with a valid SHA-256), and extracts the native binary through a bounded, entry-validating extractor instead of unrestricted `tar -xzf`, installing it atomically with mode `0755` while preserving an existing valid install on failure. + +This extends the existing architecture (npm launcher -> signed GitHub Release manifest -> SHA-256 verified archive -> native binary) rather than replacing it. The bundled public key, manifest signing flow, supported platform matrix (`darwin/arm64`, `linux/arm64`, `linux/x64`), and release archive layout (`sce-v-/{bin/sce,LICENSE,README.md}`) are preserved. It also adds a locally controlled packed-package smoke test to the flake check surface and to `release-npm-package`, and corrects `npm/README.md`, which currently lists GNU Linux targets while `npm/lib/platform.js` and the release pipeline use musl. + +Current-state facts this plan starts from: + +- `npm/bin/sce.js` and every other `npm/` file are tracked in Git as mode `100644`; `package.json` maps `bin.sce` to `./bin/sce.js`. +- `npm/lib/install.js` verifies the manifest signature, then trusts `artifact.archive` (falling back to the canonical name), requires only a truthy `checksum_sha256`, uses the first `target_triple` match, never compares `manifest.version`/`binary` to the package, extracts with `tar -xzf` into the download temp dir, then `copyFileSync` + `chmodSync` directly onto `runtime/sce` (non-atomic). +- `release-manifest` emits `{version, binary: "sce", artifacts: [...]}`; each artifact fragment carries `version`, `binary`, `archive`, `checksum_file`, `checksum_sha256`, `target_triple`, `os`, `arch`. +- `flake.nix` `npmSrc` is an explicit file list; new `npm/` source or test files must be added to it to reach `npm-bun-tests` / Biome checks. +- `release-npm-package` copies `npm/` with `cp -R` and runs `npm pack`; it performs no post-pack mode or execution check. + +## Acceptance criteria + +- [ ] AC1: A tarball produced by `npm pack` of `npm/` installs into an isolated prefix, and the installed `sce` bin (npm-generated link to `bin/sce.js`) is executable and successfully runs `sce --help` and `sce version` against a locally served fixture release, with no network access. + - Validate: `nix build .#checks.x86_64-linux.npm-pack-smoke --print-build-logs` (new check) passes; the smoke script asserts exit 0 and expected stdout for both commands. +- [ ] AC2: The root cause of the non-executable launcher is identified, fixed at its source, and recorded in the plan task evidence; the packed tarball entry `package/bin/sce.js` carries an executable mode. + - Validate: `git ls-files -s npm/bin/sce.js` shows the fixed mode (if Git mode is the cause) and the smoke check inspects the packed tarball entry mode. +- [ ] AC3: After signature verification and before any archive download, the installer rejects manifests whose `version` differs from the package version, whose `binary` is not `sce`, that have zero or more than one artifact for the current target, whose artifact `archive` is not `sce-v-.tar.gz`, or whose `checksum_sha256` is not 64 lowercase hex characters. + - Validate: `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'` — negative tests for each case assert the specific error and that the archive downloader was never called. +- [ ] AC4: Archive extraction no longer shells out to unrestricted `tar -xzf`; the installer parses the gzip+ustar stream itself with bounded sizes, rejects absolute paths, `..` traversal, symlinks, hardlinks, duplicate entries, unexpected entry types, entries outside the expected root, oversize payloads, and a missing/non-regular `bin/sce`, and writes only `bin/sce` into an isolated staging directory. + - Validate: `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'` — fixture archives for each malicious case are rejected and nothing is written outside staging; `nix shell nixpkgs#ripgrep -c rg -n '"-xzf"|spawnSync\("tar"' npm/lib` returns no match. +- [ ] AC5: Installation is atomic: the binary is staged next to `runtime/sce`, chmodded `0755`, and renamed into place; a checksum failure, extraction failure, or interruption leaves a previously installed valid `runtime/sce` byte-identical and executable, and leaves no partial `runtime/sce`. + - Validate: `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'` — interrupted-install and failure tests over a real temp `runtime/` directory. +- [ ] AC6: `nix run .#release-npm-package -- --version <.version> --out-dir ` fails if the packed tarball's `bin/sce.js` is not executable or the packed-package smoke install fails. + - Validate: run the app on a clean tree (passes); run it with `npm/bin/sce.js` temporarily set to `0644` in a scratch copy (fails with a permission diagnostic). Inspection of `.github/workflows/release-sce.yml` confirms it invokes this app before upload. +- [ ] AC7: `npm/README.md` lists `aarch64-unknown-linux-musl` and `x86_64-unknown-linux-musl`, and `context/sce/cli-npm-distribution-contract.md` describes the manifest identity checks, safe extraction, atomic install, and the packed-package smoke verification. + - Validate: `nix shell nixpkgs#ripgrep -c rg -n 'linux-gnu' npm/README.md` returns no match; inspect the contract file. + +### Full validation + +- `nix flake check` +- `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'` +- `nix run .#release-npm-package -- --version "$(cat .version)" --out-dir ` + +### Context sync + +- `context/sce/cli-npm-distribution-contract.md` (install flow, validation rules, verification baseline) +- `context/sce/cli-release-artifact-contract.md` (only if `release-npm-package` validation behavior is described there) +- `context/glossary.md` (`npm JS flake checks` entry gains the packed-package smoke check) +- `context/overview.md` (flake check list mention of the npm slice, if it enumerates checks) + +## Task context synchronization lifecycle + +- **Task context synchronization:** every task carries `pending | synced | blocked`. + A completed task must be `synced` before another task can start or the plan can + finish. +- For `blocked`, record **Blocker**, **Required action**, and **Retry condition** + beside the status. Never infer `synced` from conversation history; write every + lifecycle transition to the plan file. + +## Constraints and non-goals + +- **In scope:** `npm/bin/sce.js` (mode/shebang only), `npm/lib/install.js`, `npm/lib/platform.js`, new `npm/lib/` helper module(s) for archive reading, `npm/test/**`, a packed-package smoke script under `npm/test/` or `scripts/`, `npm/README.md`, `flake.nix` (`npmSrc`, new check, `release-npm-package`), `.github/workflows/release-sce.yml` only if it needs to surface the new validation, `context/sce/cli-npm-distribution-contract.md` and related context entries. +- **Out of scope:** changing the native build pipeline, release archive layout, manifest schema or signing key, supported platform matrix, publish workflow topology, or any npm/crates/GitHub publication. +- **Constraints:** no new npm runtime dependencies (use Node built-ins `zlib`, `fs`, `crypto`); Node `>=18` compatibility; Biome formatting/linting; all tooling via Nix; tests must not require network access; do not push or publish. +- **Non-goal:** a general-purpose tar implementation — the reader supports only the ustar/GNU subset the release pipeline emits (regular files and directories, optional pax/GNU long-name headers if GNU tar emits them) and rejects everything else. + +## Assumptions + +- The safe extractor is a small in-repo ustar reader over `zlib.gunzipSync`/streamed gunzip with explicit size caps (compressed archive, total uncompressed bytes, entry count, `bin/sce` size); no `tar` npm dependency is justified for this subset. +- Allowed archive entries are exactly the archive root directory, `bin/`, `bin/sce` (regular file), `LICENSE`, and `README.md` under `sce-v-/`; any other entry is rejected rather than ignored, matching what `release-artifacts` packs. +- The packed-package smoke test uses a locally controlled fixture: a stub executable `sce` (shell script answering `--help` and `version`) packed in the canonical archive layout, a test-generated signing key swapped into the staged fixture copy of the package (the production `release-manifest-public-key.pem` and trust code are unchanged), and a loopback HTTPS server with a generated self-signed certificate trusted via `NODE_EXTRA_CA_CERTS`, pointed at by the existing `SCE_NPM_RELEASE_BASE_URL` override. Real native-binary execution on every platform is not claimed; only the host platform running the check is exercised. +- The root cause of the permission failure is determined by reproduction in T01 (candidates: Git mode `100644` on `bin/sce.js` carried through `cp -R` + `npm pack`, and how npm/bin-links/other package managers treat bin targets); the fix targets whichever is reproduced. +- Manifest version comparison is exact string equality with `npm/package.json` `version`. + +## Task stack + +- [ ] T01: `Fix launcher executable permission and add packed-package smoke test` (status:todo) + - Task ID: T01 + - Scope: In — reproduce the non-executable launcher via real `npm pack` + `npm install --prefix `; identify and fix the root cause (e.g. Git mode `100755` for `npm/bin/sce.js`, packaging step); add a smoke script that packs `npm/`, builds the local fixture release (stub binary archive, signed manifest with a test key swapped into the staged package copy, loopback HTTPS server), installs into an isolated prefix with `SCE_NPM_RELEASE_BASE_URL`, asserts the packed `package/bin/sce.js` mode, and runs `sce --help` and `sce version`. Out — manifest validation and extractor changes (T02/T03), flake/release wiring (T04). + - Dependencies: none + - Done when: the smoke script fails on the pre-fix tree for the reproduced reason and passes after the fix; root cause is recorded in task evidence. + - Verify: `nix shell nixpkgs#nodejs nixpkgs#openssl -c bash ` before and after the fix; `git ls-files -s npm/bin/sce.js`; `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'`. + - Context synchronization: pending + +- [ ] T02: `Validate signed manifest identity before archive download` (status:todo) + - Task ID: T02 + - Scope: In — after `parseVerifiedReleaseManifest`, validate `version` equals package version, `binary === "sce"`, exactly one artifact for the target triple, `archive` equals `getArchiveName(version, target)`, `checksum_sha256` matches `/^[0-9a-f]{64}$/`; drop the `artifact.archive ??` fallback trust; update `selectReleaseArtifact` and existing fixtures (`abc123` checksum) accordingly; negative tests for version mismatch, wrong binary, missing/duplicate target, invalid archive name, invalid digest, each asserting no archive download. Out — extraction and install mechanics. + - Dependencies: T01 + - Done when: every inconsistency is rejected with a specific error before `downloadArchive` is invoked, valid manifests still install, and the bundled key/signature flow is unchanged. + - Verify: `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'`; `nix develop -c biome check npm`. + - Context synchronization: pending + +- [ ] T03: `Replace tar extraction with bounded validating extractor and atomic install` (status:todo) + - Task ID: T03 + - Scope: In — new `npm/lib/` archive module reading gzip+ustar with size/entry caps and the allowed-entry rules; extract only `bin/sce` into a fresh staging dir; stage into `runtime/` with a temp name, `chmod 0755`, `fsync`, and `rename` over `runtime/sce`; clean up staging on any failure; tests with generated fixture archives for absolute path, `..` traversal, symlink, hardlink, duplicate entry, unexpected type (e.g. FIFO/char device), entry outside root, oversize payload, missing or non-regular `bin/sce`, checksum failure, and an interrupted install (failure injected between staging and rename) preserving a prior valid `runtime/sce`. Out — manifest identity rules (T02), flake/release wiring. + - Dependencies: T02 + - Done when: `install.js` no longer spawns `tar`; all negative fixtures are rejected without writing outside staging; a prior valid install survives every failure path; the T01 smoke script still passes. + - Verify: `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'`; `nix shell nixpkgs#ripgrep -c rg -n 'spawnSync' npm/lib`; rerun the T01 smoke script. + - Context synchronization: pending + +- [ ] T04: `Wire packed-package smoke into flake checks and release-npm-package` (status:todo) + - Task ID: T04 + - Scope: In — add new `npm/` files to `npmSrc`; add a `npm-pack-smoke` flake check running the T01 smoke script in the sandbox (nodejs, openssl); make `release-npm-package` assert the packed tarball's `package/bin/sce.js` is executable and run the smoke against the produced tarball before reporting success; confirm `release-sce.yml` already invokes `release-npm-package` (edit only if needed). Out — publish workflow behavior changes, real network release testing. + - Dependencies: T03 + - Done when: `nix flake check` includes and passes the new check; `release-npm-package` fails on a `0644` launcher in a scratch copy and passes on the real tree. + - Verify: `nix build .#checks.x86_64-linux.npm-pack-smoke --print-build-logs`; `nix run .#release-npm-package -- --version "$(cat .version)" --out-dir `; `nix run nixpkgs#actionlint` if a workflow is touched. + - Context synchronization: pending + +- [ ] T05: `Correct npm README targets and update distribution contract` (status:todo) + - Task ID: T05 + - Scope: In — `npm/README.md` musl target triples and a short note on manifest identity/safe-extraction failures in troubleshooting; `context/sce/cli-npm-distribution-contract.md` install-flow, validation, atomic-install, and verification-baseline updates; glossary/overview entries for the new flake check. Out — documentation site changes. + - Dependencies: T04 + - Done when: README matches `npm/lib/platform.js`; contract describes implemented behavior and the new verification commands. + - Verify: `nix shell nixpkgs#ripgrep -c rg -n 'linux-gnu' npm/README.md` (no match); `nix develop -c biome check npm`. + - Context synchronization: pending + +## Open questions + +- Is the loopback HTTPS + self-signed certificate fixture acceptable inside the Nix build sandbox for `npm-pack-smoke`? If sandbox loopback networking proves unavailable on a CI runner, the fallback is to keep the smoke as a `nix run` app invoked by `release-npm-package` and PR CI instead of a pure flake check. +- Should the smoke additionally run the real `nix build .#default` binary inside the fixture archive on the host platform? This gives real-binary coverage for the host only, at the cost of making the check depend on the full CLI build; the plan defaults to a stub binary. From e1ddbb6fce1176b8dbf9d22380b8d0827c6e98d9 Mon Sep 17 00:00:00 2001 From: David Abram Date: Tue, 3 Nov 2026 17:48:11 +0100 Subject: [PATCH 2/6] npm: Harden packed launcher permissions with an end-to-end smoke test Commit the launcher with executable mode so npm pack preserves a runnable package/bin/sce.js, and add a loopback HTTPS fixture test that packs, installs, and exercises the launcher against a signed stub release. Record the completed installer hardening evidence in the plan and npm distribution contract. Plan: harden-npm-installer Task: T01 Co-authored-by: SCE --- context/context-map.md | 2 +- context/plans/harden-npm-installer.md | 17 +- context/sce/cli-npm-distribution-contract.md | 7 + npm/bin/sce.js | 0 npm/test/fixture-release-server.mjs | 97 +++++++++ npm/test/pack-smoke.sh | 200 +++++++++++++++++++ 6 files changed, 320 insertions(+), 3 deletions(-) mode change 100644 => 100755 npm/bin/sce.js create mode 100644 npm/test/fixture-release-server.mjs create mode 100755 npm/test/pack-smoke.sh diff --git a/context/context-map.md b/context/context-map.md index 920c2a033..f3c032d8c 100644 --- a/context/context-map.md +++ b/context/context-map.md @@ -113,7 +113,7 @@ Feature/domain context: - `context/sce/opencode-agent-trace-plugin-runtime.md` (current OpenCode agent-trace plugin runtime behavior, including captured `message.updated` handoff with `summary.diffs` branching: when diffs exist sends one `-patch` mixed batch containing a synthetic parent message plus per-diff `message.part` patch items, when no diffs sends the original `message.updated` payload; in-memory dedup `Set` keyed by `"${sessionID}:${messageID}"`; captured `message.part.updated` handoff to `sce hooks conversation-trace` for `text`/`reasoning` parts with non-empty text plus completed `question` tool parts emitted as `part_type: "question"` with JSON-stringified `{ question, answer }[]`; existing user-message diff extraction for `{ sessionID, diff, time, model_id }`; session-scoped OpenCode client version capture from `session.created`/`session.updated`; and CLI handoff to `sce hooks diff-trace` over STDIN JSON with required `tool_name="opencode"` plus required nullable `tool_version`; Rust hook parsing and AgentTraceDb insertion persist `oc_`-prefixed session IDs plus required payload fields including `model_id`) - `context/sce/cli-first-install-channels-contract.md` (current Nix/Cargo/npm/source-built Flatpak channel contract, release authority and workflow topology, Nix-owned Flatpak manifest/cargo-source generation and validation, reduced Flatpak app surface, and host-git bridge decision) - `context/sce/cli-release-artifact-contract.md` (shared `sce` binary release artifact naming, checksum/manifest outputs, pre-archive staged-binary preparation including macOS `libiconv` install-name sanitization/ad-hoc re-signing, native portability audit app/check for forbidden `/nix/store/` runtime references, GitHub Releases as the canonical artifact publication surface, manual dispatch `prerelease` flag behavior, the current three-target Linux/macOS release workflow topology including pre-upload extracted-archive smoke/audit validation in each native lane, implemented Flatpak source-manifest and source-built `.flatpak` bundle package assets uploaded by `.github/workflows/release-sce.yml`, and Flatpak's explicit source-built non-binary exception) -- `context/sce/cli-npm-distribution-contract.md` (implemented `sce` npm launcher package, release-manifest/checksum-verified native binary install flow, the supported darwin/arm64 plus linux x64+arm64 npm platform matrix, the prerequisite that npm-consumed native archives are portable and free of forbidden `/nix/store/` runtime references before upload, and dedicated `.github/workflows/publish-npm.yml` downstream npm publish-stage contract including prerelease publication via npm `next` dist-tag instead of `latest`) +- `context/sce/cli-npm-distribution-contract.md` (implemented `sce` npm launcher package, release-manifest/checksum-verified native binary install flow, the supported darwin/arm64 plus linux x64+arm64 npm platform matrix, the executable launcher-mode requirement and offline packed-package smoke test (`npm/test/pack-smoke.sh`), the prerequisite that npm-consumed native archives are portable and free of forbidden `/nix/store/` runtime references before upload, and dedicated `.github/workflows/publish-npm.yml` downstream npm publish-stage contract including prerelease publication via npm `next` dist-tag instead of `latest`) - `context/sce/cli-cargo-distribution-contract.md` (implemented `sce` Cargo publication posture plus supported crates.io and wrapper-driven local-checkout install guidance, explicit unsupported `cargo install --git` boundary, dedicated crates.io publish workflow with semver prerelease guard, canonical generated-input declaration/producer ownership consumed by the Cargo wrapper, generated-output check, package-fallback preparation, and Nix pre-Cargo derivation, plus the deterministic combined checksummed fallback used for Pkl-free downstream crate builds) - `context/sce/flake-build-performance.md` (current native-vs-release package split, release-only commit embedding, shared-producer-backed pre-Cargo generated-input derivation consumed by native/release/test/Clippy but excluded from dependency/format derivations, slim default/database devShell contract, normal-vs-expensive CI tiers, durable before/after benchmark summary, validation evidence, and remaining cold-build trade-offs) - `context/sce/config-schema-publication.md` (current embedded-only publication contract for the generated SCE config JSON Schema and the canonical SCE web application URL used in config declarations) diff --git a/context/plans/harden-npm-installer.md b/context/plans/harden-npm-installer.md index c87cc8258..8c7f23f7a 100644 --- a/context/plans/harden-npm-installer.md +++ b/context/plans/harden-npm-installer.md @@ -70,13 +70,26 @@ Current-state facts this plan starts from: ## Task stack -- [ ] T01: `Fix launcher executable permission and add packed-package smoke test` (status:todo) +- [x] T01: `Fix launcher executable permission and add packed-package smoke test` (status:done) - Task ID: T01 - Scope: In — reproduce the non-executable launcher via real `npm pack` + `npm install --prefix `; identify and fix the root cause (e.g. Git mode `100755` for `npm/bin/sce.js`, packaging step); add a smoke script that packs `npm/`, builds the local fixture release (stub binary archive, signed manifest with a test key swapped into the staged package copy, loopback HTTPS server), installs into an isolated prefix with `SCE_NPM_RELEASE_BASE_URL`, asserts the packed `package/bin/sce.js` mode, and runs `sce --help` and `sce version`. Out — manifest validation and extractor changes (T02/T03), flake/release wiring (T04). - Dependencies: none - Done when: the smoke script fails on the pre-fix tree for the reproduced reason and passes after the fix; root cause is recorded in task evidence. - Verify: `nix shell nixpkgs#nodejs nixpkgs#openssl -c bash ` before and after the fix; `git ls-files -s npm/bin/sce.js`; `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'`. - - Context synchronization: pending + - Completed: 2026-11-03 + - Files changed: `npm/bin/sce.js` (mode only, `100644` -> `100755`, staged in the index; content unchanged), `npm/test/pack-smoke.sh` (new, mode `0755`), `npm/test/fixture-release-server.mjs` (new). + - Root cause: `npm/bin/sce.js` was committed as Git mode `100644`; `release-npm-package` copies `npm/` with `cp -R` (preserving the non-executable filesystem mode) and `npm pack` records it verbatim, so the published tarball entry `package/bin/sce.js` was `-rw-r--r--`. Reproduction (npm 11.13.0 / Node 24.16.0): `tar -tvzf` of the packed tarball showed `-rw-r--r-- package/bin/sce.js`. npm 11 (`--prefix` and `-g`), bun 1.3, and pnpm all chmod bin targets while linking, so those installers mask the defect; any consumer that executes the packed `bin/sce.js` without bin-link repair gets a permission failure. Fixed at the source by committing the launcher as `100755`. + - Result: Added `npm/test/pack-smoke.sh`, which stages `npm/`, swaps a throwaway RSA-2048 manifest key into the staged copy only, builds a stub `sce` archive in the canonical `sce-v-/{bin/sce,LICENSE,README.md}` layout with a signed manifest (`openssl dgst -sha256 -sign`, base64), serves it from `npm/test/fixture-release-server.mjs` (loopback HTTPS, self-signed cert trusted via `NODE_EXTRA_CA_CERTS`), runs `npm pack`, asserts the packed `package/bin/sce.js` mode is executable, installs with `npm install --prefix --offline` using `SCE_NPM_RELEASE_BASE_URL` and an isolated npm cache/config, then asserts exact stdout for `sce --help` and `sce version`. Accepts `--package-dir ` for later reuse (T04). The production public key and trust code are unchanged. + - Verify outcomes: + - Pre-fix smoke: `nix shell nixpkgs#nodejs nixpkgs#openssl -c bash npm/test/pack-smoke.sh` -> exit 1, `Packed package/bin/sce.js has mode -rw-r--r--; expected an executable mode` (reproduced reason). + - Post-fix smoke: same command -> exit 0, `packed package/bin/sce.js mode -rwxr-xr-x`, `pack-smoke: OK (sce --help, sce version)`; temp work dir removed on exit. + - `git ls-files -s npm/bin/sce.js` -> `100755 13ef5b35... npm/bin/sce.js`. + - Bun tests: `nix shell nixpkgs#bun -c bun test ./test/install.test.js ./test/platform.test.js` (from `npm/`) -> 25 pass, 0 fail. The `nix develop -c sh -c '... bun test ...'` form was blocked by the repo bash policy `use-nix-for-bun`, so the nixpkgs Bun form was used. + - `nix develop -c biome check npm/bin npm/lib npm/test` -> no errors (one pre-existing warning: unused `getRepositorySlug` in `npm/lib/install.js`, untouched). + - `nix shell nixpkgs#shellcheck -c shellcheck npm/test/pack-smoke.sh` -> clean. + - Deviations: the new smoke files are not yet in `flake.nix` `npmSrc`, so flake Biome/test checks do not see them until T04 wires them in (as planned). + - Context impact: localized to the npm launcher packaging/verification surface; affects `context/sce/cli-npm-distribution-contract.md` (launcher mode requirement, new local smoke script). No flake check or glossary changes yet (T04/T05). + - Context synchronization: synced - [ ] T02: `Validate signed manifest identity before archive download` (status:todo) - Task ID: T02 diff --git a/context/sce/cli-npm-distribution-contract.md b/context/sce/cli-npm-distribution-contract.md index 3b0bbfc83..02e0b9280 100644 --- a/context/sce/cli-npm-distribution-contract.md +++ b/context/sce/cli-npm-distribution-contract.md @@ -8,6 +8,7 @@ This file captures the implemented npm distribution slice of the `sce` CLI insta - The committed package source lives under `npm/`. - `npm/README.md` documents the package as a thin launcher, lists the supported npm platform-to-target-triple matrix, and distinguishes npm `postinstall`/download/signature/checksum/permission failures from runtime loader failures caused by bad native release artifact linkage. - `npm/package.json` exposes the `sce` binary through `bin/sce.js`. +- `npm/bin/sce.js` is committed as Git mode `100755`. `release-npm-package` copies `npm/` with `cp -R` and `npm pack` records filesystem modes verbatim, so the launcher's executable bit in Git is what makes the packed `package/bin/sce.js` entry executable. npm, bun, and pnpm re-chmod bin targets while linking, which masks a non-executable entry for those installers but not for consumers that execute the packed file directly. - The npm package is a thin launcher package, not a separate build pipeline. ## Install/runtime behavior @@ -51,9 +52,15 @@ This file captures the implemented npm distribution slice of the `sce` CLI insta - The npm installer continues to trust GitHub Releases as the canonical host for signed manifest and native binary artifacts. - Native binary portability hygiene is owned by the shared release artifact surface before archive publication, including macOS `libiconv` install-name sanitization, Linux/macOS forbidden `/nix/store/` audits, and reusable-workflow extracted-binary smoke validation; npm continues to consume the signed release manifest and checksum-verified native archive without adding a separate npm-native build or dylib repair path. +## Packed-package smoke test + +- `npm/test/pack-smoke.sh` (with helper `npm/test/fixture-release-server.mjs`) verifies the real packed artifact end to end without network access: it stages a copy of the package (or `--package-dir `), swaps a throwaway RSA manifest-signing public key into the staged copy only, builds a stub `sce` archive in the canonical `sce-v-/{bin/sce,LICENSE,README.md}` layout with a matching signed manifest, serves it over loopback HTTPS (self-signed certificate trusted via `NODE_EXTRA_CA_CERTS`, selected via `SCE_NPM_RELEASE_BASE_URL`), runs `npm pack`, fails unless the packed `package/bin/sce.js` entry is executable, installs into an isolated `--prefix` with an isolated npm cache/config, and asserts exact output from `sce --help` and `sce version`. +- Only the host platform is exercised, with a stub binary; the production public key and manifest trust code are not modified. + ## Verification baseline - `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'` +- `nix shell nixpkgs#nodejs nixpkgs#openssl -c bash npm/test/pack-smoke.sh` - `nix develop -c sh -c 'cd npm && SCE_NPM_SKIP_DOWNLOAD=1 bun ./lib/install.js'` - `nix run .#release-manifest -- --version --artifacts-dir --out-dir --signing-key-file ` - `nix run .#release-npm-package -- --version --out-dir ` diff --git a/npm/bin/sce.js b/npm/bin/sce.js old mode 100644 new mode 100755 diff --git a/npm/test/fixture-release-server.mjs b/npm/test/fixture-release-server.mjs new file mode 100644 index 000000000..e49309ef2 --- /dev/null +++ b/npm/test/fixture-release-server.mjs @@ -0,0 +1,97 @@ +import { + createReadStream, + readFileSync, + statSync, + writeFileSync, +} from "node:fs"; +import { createServer } from "node:https"; +import path from "node:path"; +import process from "node:process"; + +const LOOPBACK_HOST = "127.0.0.1"; + +function readFlagValue(argv, index, flagName) { + const value = argv[index + 1]; + + if (!value || value.startsWith("-")) { + throw new Error(`Missing value for ${flagName}.`); + } + + return value; +} + +function parseArgs(argv) { + const options = { rootDir: "", certFile: "", keyFile: "", portFile: "" }; + const flags = new Map([ + ["--root", "rootDir"], + ["--cert", "certFile"], + ["--key", "keyFile"], + ["--port-file", "portFile"], + ]); + + for (let index = 0; index < argv.length; index += 2) { + const optionName = flags.get(argv[index]); + + if (!optionName) { + throw new Error(`Unknown argument: ${argv[index]}`); + } + + options[optionName] = readFlagValue(argv, index, argv[index]); + } + + for (const [flagName, optionName] of flags) { + if (!options[optionName]) { + throw new Error(`${flagName} is required.`); + } + } + + return options; +} + +function resolveRequestPath(rootDir, requestUrl) { + const { pathname } = new URL(requestUrl, `https://${LOOPBACK_HOST}`); + const fileName = decodeURIComponent(pathname).replace(/^\/+/, ""); + + if (fileName.length === 0 || fileName.includes("/")) { + return null; + } + + const filePath = path.join(rootDir, fileName); + + try { + return statSync(filePath).isFile() ? filePath : null; + } catch { + return null; + } +} + +const { rootDir, certFile, keyFile, portFile } = parseArgs( + process.argv.slice(2), +); + +const server = createServer( + { cert: readFileSync(certFile), key: readFileSync(keyFile) }, + (request, response) => { + const filePath = + request.method === "GET" + ? resolveRequestPath(rootDir, request.url ?? "/") + : null; + + if (!filePath) { + response.writeHead(404); + response.end(); + return; + } + + response.writeHead(200, { "content-type": "application/octet-stream" }); + createReadStream(filePath).pipe(response); + }, +); + +server.listen(0, LOOPBACK_HOST, () => { + writeFileSync(portFile, `${server.address().port}\n`, "utf8"); +}); + +process.on("SIGTERM", () => { + server.close(() => process.exit(0)); +}); diff --git a/npm/test/pack-smoke.sh b/npm/test/pack-smoke.sh new file mode 100755 index 000000000..7f236bcd4 --- /dev/null +++ b/npm/test/pack-smoke.sh @@ -0,0 +1,200 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Packed-package smoke test for the @crocoder-dev/sce npm launcher. +# +# Packs a staged copy of the npm package, serves a locally generated fixture +# release (stub `sce` binary, manifest signed with a throwaway key swapped into +# the staged copy) over loopback HTTPS, installs the tarball into an isolated +# prefix, and runs the installed `sce` bin. No network access is required. +# +# Requires: bash, coreutils, tar, gzip, node/npm (>=18), openssl. + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PACKAGE_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" +LAUNCHER_ENTRY="package/bin/sce.js" +SERVER_START_TIMEOUT_SECONDS=10 +EXPECTED_HELP_OUTPUT="sce fixture help" + +usage() { + cat <<'EOF' +Usage: pack-smoke.sh [--package-dir ] + +Packs the npm launcher package, installs it into an isolated prefix against a +loopback fixture release, and verifies `sce --help` and `sce version`. +EOF +} + +fail() { + printf 'pack-smoke: %s\n' "$1" >&2 + exit 1 +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --package-dir) + [[ -n "${2:-}" ]] || fail "Missing value for --package-dir." + PACKAGE_DIR="$(cd "$2" && pwd)" + shift 2 + ;; + --help | -h) + usage + exit 0 + ;; + *) + usage >&2 + fail "Unknown argument: $1" + ;; + esac +done + +for tool in node npm openssl tar gzip sha256sum; do + command -v "$tool" >/dev/null 2>&1 || + fail "Required tool '$tool' is not on PATH. Run via: nix shell nixpkgs#nodejs nixpkgs#openssl -c bash $0" +done + +[[ -f "$PACKAGE_DIR/package.json" ]] || fail "No package.json found in $PACKAGE_DIR." + +work_dir="$(mktemp -d "${TMPDIR:-/tmp}/sce-npm-pack-smoke.XXXXXX")" +server_pid="" +cleanup() { + if [[ -n "$server_pid" ]]; then + kill "$server_pid" 2>/dev/null || true + wait "$server_pid" 2>/dev/null || true + fi + rm -rf "$work_dir" +} +trap cleanup EXIT + +stage_dir="$work_dir/stage" +release_dir="$work_dir/release" +pack_dir="$work_dir/pack" +prefix_dir="$work_dir/prefix" +keys_dir="$work_dir/keys" +mkdir -p "$stage_dir" "$release_dir" "$pack_dir" "$prefix_dir" "$keys_dir" + +cp -R "$PACKAGE_DIR/." "$stage_dir/" +rm -rf "$stage_dir/runtime" "$stage_dir/node_modules" + +version="$(node -p "JSON.parse(require('fs').readFileSync(process.argv[1], 'utf8')).version" "$stage_dir/package.json")" +platform_json="$( + node --input-type=module -e ' + const { resolveSupportedPlatform } = await import(process.argv[1]); + const platform = resolveSupportedPlatform(); + if (!platform) process.exit(2); + console.log(JSON.stringify(platform)); + ' "$stage_dir/lib/platform.js" +)" || fail "Host platform $(node -p 'process.platform + "/" + process.arch') is not supported by the npm package." +target_triple="$(node -p "JSON.parse(process.argv[1]).targetTriple" "$platform_json")" +target_os="$(node -p "JSON.parse(process.argv[1]).os" "$platform_json")" +target_arch="$(node -p "JSON.parse(process.argv[1]).arch" "$platform_json")" + +printf 'pack-smoke: version %s, target %s\n' "$version" "$target_triple" + +# Throwaway manifest signing key; only the staged copy trusts it. +openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out "$keys_dir/signing.pem" 2>/dev/null +openssl pkey -in "$keys_dir/signing.pem" -pubout -out "$stage_dir/lib/release-manifest-public-key.pem" + +# Loopback TLS certificate trusted through NODE_EXTRA_CA_CERTS. +openssl req -x509 -newkey rsa:2048 -nodes -days 1 \ + -subj "/CN=127.0.0.1" \ + -addext "subjectAltName=IP:127.0.0.1" \ + -keyout "$keys_dir/tls-key.pem" -out "$keys_dir/tls-cert.pem" 2>/dev/null + +# Fixture release in the canonical archive layout. +archive_root="sce-v${version}-${target_triple}" +archive_name="${archive_root}.tar.gz" +manifest_name="sce-v${version}-release-manifest.json" +archive_src="$work_dir/archive-src" +mkdir -p "$archive_src/$archive_root/bin" +cat >"$archive_src/$archive_root/bin/sce" <&2; exit 64 ;; +esac +EOF +chmod 0755 "$archive_src/$archive_root/bin/sce" +printf 'fixture license\n' >"$archive_src/$archive_root/LICENSE" +printf 'fixture readme\n' >"$archive_src/$archive_root/README.md" +tar -C "$archive_src" --owner=0 --group=0 --numeric-owner -czf "$release_dir/$archive_name" "$archive_root" + +checksum="$(sha256sum "$release_dir/$archive_name" | cut -d ' ' -f 1)" +# shellcheck disable=SC2016 # JavaScript template literal, not a shell expansion. +node -e ' + const [version, archive, checksum, target_triple, os, arch, output] = process.argv.slice(1); + const manifest = { + version, + binary: "sce", + artifacts: [ + { + version, + binary: "sce", + archive, + checksum_file: `${archive}.sha256`, + checksum_sha256: checksum, + target_triple, + os, + arch, + }, + ], + }; + require("fs").writeFileSync(output, `${JSON.stringify(manifest, null, 2)}\n`); +' "$version" "$archive_name" "$checksum" "$target_triple" "$target_os" "$target_arch" "$release_dir/$manifest_name" +openssl dgst -sha256 -sign "$keys_dir/signing.pem" "$release_dir/$manifest_name" | + openssl base64 -A >"$release_dir/$manifest_name.sig" + +# Pack and verify the launcher entry mode inside the tarball. +packed_name="$(cd "$pack_dir" && npm pack --silent "$stage_dir")" +tarball="$pack_dir/$packed_name" +launcher_mode="$(tar -tvzf "$tarball" "$LAUNCHER_ENTRY" | cut -d ' ' -f 1)" +[[ -n "$launcher_mode" ]] || fail "Packed tarball is missing $LAUNCHER_ENTRY." +if [[ "${launcher_mode:3:1}" != "x" ]]; then + fail "Packed $LAUNCHER_ENTRY has mode $launcher_mode; expected an executable mode (e.g. -rwxr-xr-x). Ensure npm/bin/sce.js is committed as Git mode 100755." +fi +printf 'pack-smoke: packed %s mode %s\n' "$LAUNCHER_ENTRY" "$launcher_mode" + +# Serve the fixture release over loopback HTTPS. +port_file="$work_dir/server-port" +node "$SCRIPT_DIR/fixture-release-server.mjs" \ + --root "$release_dir" \ + --cert "$keys_dir/tls-cert.pem" \ + --key "$keys_dir/tls-key.pem" \ + --port-file "$port_file" & +server_pid=$! + +for ((attempt = 0; attempt < SERVER_START_TIMEOUT_SECONDS * 10; attempt++)); do + [[ -s "$port_file" ]] && break + kill -0 "$server_pid" 2>/dev/null || fail "Fixture release server exited before listening." + sleep 0.1 +done +[[ -s "$port_file" ]] || fail "Fixture release server did not start within ${SERVER_START_TIMEOUT_SECONDS}s." +port="$(tr -d '\n' <"$port_file")" + +# Install into an isolated prefix with an isolated npm cache/config. +SCE_NPM_RELEASE_BASE_URL="https://127.0.0.1:${port}" \ + NODE_EXTRA_CA_CERTS="$keys_dir/tls-cert.pem" \ + npm_config_cache="$work_dir/npm-cache" \ + npm_config_userconfig="$work_dir/npmrc" \ + npm install \ + --prefix "$prefix_dir" \ + --offline \ + --no-audit \ + --no-fund \ + --foreground-scripts \ + "$tarball" || + fail "npm install of the packed tarball failed." + +sce_bin="$prefix_dir/node_modules/.bin/sce" +[[ -x "$sce_bin" ]] || fail "Installed sce bin $sce_bin is not executable." + +help_output="$("$sce_bin" --help)" || fail "sce --help exited non-zero." +[[ "$help_output" == "$EXPECTED_HELP_OUTPUT" ]] || + fail "sce --help printed '$help_output'; expected '$EXPECTED_HELP_OUTPUT'." + +version_output="$("$sce_bin" version)" || fail "sce version exited non-zero." +[[ "$version_output" == "sce $version" ]] || + fail "sce version printed '$version_output'; expected 'sce $version'." + +printf 'pack-smoke: OK (sce --help, sce version)\n' From c051886e88effffdd5e3a719ce07dfb47b924876 Mon Sep 17 00:00:00 2001 From: David Abram Date: Tue, 3 Nov 2026 17:51:00 +0100 Subject: [PATCH 3/6] npm: Validate release manifest identity before download Ensure npm installation accepts only manifests bound to the requested package version and sce binary, with exactly one target artifact using its canonical archive name and lowercase SHA-256 digest. This removes fallback archive/checksum trust and rejects malformed identity metadata before temporary-directory creation or archive download. Add regression coverage for invalid identity fields, duplicate targets, and canonical archive URL selection. Plan: harden-npm-installer (T02) Co-authored-by: SCE --- context/plans/harden-npm-installer.md | 11 +- npm/lib/install.js | 16 +-- npm/lib/platform.js | 64 +++++++++- npm/test/install.test.js | 164 ++++++++++++++++++++++++-- npm/test/platform.test.js | 16 +++ 5 files changed, 242 insertions(+), 29 deletions(-) diff --git a/context/plans/harden-npm-installer.md b/context/plans/harden-npm-installer.md index 8c7f23f7a..defcc9f9d 100644 --- a/context/plans/harden-npm-installer.md +++ b/context/plans/harden-npm-installer.md @@ -91,12 +91,21 @@ Current-state facts this plan starts from: - Context impact: localized to the npm launcher packaging/verification surface; affects `context/sce/cli-npm-distribution-contract.md` (launcher mode requirement, new local smoke script). No flake check or glossary changes yet (T04/T05). - Context synchronization: synced -- [ ] T02: `Validate signed manifest identity before archive download` (status:todo) +- [x] T02: `Validate signed manifest identity before archive download` (status:done) - Task ID: T02 - Scope: In — after `parseVerifiedReleaseManifest`, validate `version` equals package version, `binary === "sce"`, exactly one artifact for the target triple, `archive` equals `getArchiveName(version, target)`, `checksum_sha256` matches `/^[0-9a-f]{64}$/`; drop the `artifact.archive ??` fallback trust; update `selectReleaseArtifact` and existing fixtures (`abc123` checksum) accordingly; negative tests for version mismatch, wrong binary, missing/duplicate target, invalid archive name, invalid digest, each asserting no archive download. Out — extraction and install mechanics. - Dependencies: T01 - Done when: every inconsistency is rejected with a specific error before `downloadArchive` is invoked, valid manifests still install, and the bundled key/signature flow is unchanged. - Verify: `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'`; `nix develop -c biome check npm`. + - Completed: 2026-11-03 + - Files changed: `npm/lib/platform.js`, `npm/lib/install.js`, `npm/test/install.test.js`, `npm/test/platform.test.js`. + - Result: Added `validateReleaseManifestIdentity(manifest, { version, targetTriple })` to `npm/lib/platform.js`; it requires a JSON object manifest, exact `version` equality with the package version, `binary === "sce"`, exactly one artifact for the target (via `selectReleaseArtifact`, which now rejects duplicates with `found N artifacts for target ; expected exactly one.`), `archive === getArchiveName(version, target)`, and `checksum_sha256` matching `/^[0-9a-f]{64}$/`, returning `{ artifact, archiveName, expectedChecksum }`. `installBinaryWithDependencies` calls it immediately after `loadReleaseManifest`, before creating the temp dir or downloading; the `artifact.archive ??` fallback and the truthy-only checksum check were removed, so the downloaded archive name is always canonical. Signature verification, the bundled public key, and `parseVerifiedReleaseManifest` are unchanged. Test fixtures now use full canonical manifests (`binary`, `archive`, 64-hex checksum) instead of `abc123`; new negative tests cover version mismatch, wrong/missing binary, missing target, duplicate target, non-canonical/missing archive, and short/uppercase/non-hex/missing checksum, each asserting zero archive downloads; a positive test asserts the canonical archive URL is requested. + - Verify outcomes: + - `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'` -> 38 pass, 0 fail. + - `nix develop -c biome check npm` -> no errors; one pre-existing warning (unused `getRepositorySlug` in `npm/lib/install.js`, untouched). + - Additional regression: `nix shell nixpkgs#nodejs nixpkgs#openssl -c bash npm/test/pack-smoke.sh` -> `pack-smoke: OK (sce --help, sce version)` (the T01 fixture manifest already carries canonical identity fields). + - Assumptions: per-artifact `version`/`binary` fields are not separately validated (AC3 covers top-level identity; the canonical archive name already encodes version and target). Version comparison is exact string equality. + - Context impact: localized to the npm installer trust flow; affects `context/sce/cli-npm-distribution-contract.md` (manifest identity validation rules and new error surface). No architecture, glossary, or flake check changes. - Context synchronization: pending - [ ] T03: `Replace tar extraction with bounded validating extractor and atomic install` (status:todo) diff --git a/npm/lib/install.js b/npm/lib/install.js index 1fba04ed7..c12664910 100644 --- a/npm/lib/install.js +++ b/npm/lib/install.js @@ -21,12 +21,11 @@ import { fileURLToPath } from "node:url"; import { formatUnsupportedPlatformMessage, - getArchiveName, getArchiveRoot, getInstalledBinaryPath, getReleaseManifestName, resolveSupportedPlatform, - selectReleaseArtifact, + validateReleaseManifestIdentity, } from "./platform.js"; const __dirname = path.dirname(fileURLToPath(import.meta.url)); @@ -343,19 +342,10 @@ export async function installBinaryWithDependencies({ } const releaseManifest = await loadReleaseManifest(version); - const artifact = selectReleaseArtifact( + const { archiveName, expectedChecksum } = validateReleaseManifestIdentity( releaseManifest, - supportedPlatform.targetTriple, + { version, targetTriple: supportedPlatform.targetTriple }, ); - const archiveName = - artifact.archive ?? getArchiveName(version, supportedPlatform.targetTriple); - const expectedChecksum = artifact.checksum_sha256; - - if (!expectedChecksum) { - throw new Error( - `Release artifact ${archiveName} is missing checksum_sha256 metadata.`, - ); - } const tempDir = createTempDir(); diff --git a/npm/lib/platform.js b/npm/lib/platform.js index 7bc431e09..8ab372fa2 100644 --- a/npm/lib/platform.js +++ b/npm/lib/platform.js @@ -1,5 +1,8 @@ import path from "node:path"; +const RELEASE_BINARY_NAME = "sce"; +const SHA256_HEX_PATTERN = /^[0-9a-f]{64}$/; + const SUPPORTED_TARGETS = new Map([ [ "darwin:arm64", @@ -54,15 +57,68 @@ export function selectReleaseArtifact(releaseManifest, targetTriple) { throw new Error("Invalid sce release manifest: missing artifacts array."); } - const artifact = releaseManifest.artifacts.find( - (candidate) => candidate.target_triple === targetTriple, + const matches = releaseManifest.artifacts.filter( + (candidate) => candidate?.target_triple === targetTriple, ); - if (!artifact) { + if (matches.length === 0) { throw new Error( `No sce release artifact found for target ${targetTriple}.`, ); } - return artifact; + if (matches.length > 1) { + throw new Error( + `Invalid sce release manifest: found ${matches.length} artifacts for target ${targetTriple}; expected exactly one.`, + ); + } + + return matches[0]; +} + +export function validateReleaseManifestIdentity( + releaseManifest, + { version, targetTriple }, +) { + if ( + !releaseManifest || + typeof releaseManifest !== "object" || + Array.isArray(releaseManifest) + ) { + throw new Error("Invalid sce release manifest: expected a JSON object."); + } + + if (releaseManifest.version !== version) { + throw new Error( + `Invalid sce release manifest: version ${JSON.stringify(releaseManifest.version)} does not match npm package version ${JSON.stringify(version)}.`, + ); + } + + if (releaseManifest.binary !== RELEASE_BINARY_NAME) { + throw new Error( + `Invalid sce release manifest: binary ${JSON.stringify(releaseManifest.binary)} is not ${JSON.stringify(RELEASE_BINARY_NAME)}.`, + ); + } + + const artifact = selectReleaseArtifact(releaseManifest, targetTriple); + const archiveName = getArchiveName(version, targetTriple); + + if (artifact.archive !== archiveName) { + throw new Error( + `Invalid sce release artifact for target ${targetTriple}: archive ${JSON.stringify(artifact.archive)} does not match expected ${JSON.stringify(archiveName)}.`, + ); + } + + const expectedChecksum = artifact.checksum_sha256; + + if ( + typeof expectedChecksum !== "string" || + !SHA256_HEX_PATTERN.test(expectedChecksum) + ) { + throw new Error( + `Invalid sce release artifact ${archiveName}: checksum_sha256 must be 64 lowercase hexadecimal characters.`, + ); + } + + return { artifact, archiveName, expectedChecksum }; } diff --git a/npm/test/install.test.js b/npm/test/install.test.js index a7961fdf8..8723605a5 100644 --- a/npm/test/install.test.js +++ b/npm/test/install.test.js @@ -18,12 +18,37 @@ import { } from "../lib/install.js"; import { getInstalledBinaryPath } from "../lib/platform.js"; +const FIXTURE_VERSION = "0.1.0"; +const FIXTURE_TARGET_TRIPLE = "x86_64-unknown-linux-musl"; +const FIXTURE_ARCHIVE_NAME = `sce-v${FIXTURE_VERSION}-${FIXTURE_TARGET_TRIPLE}.tar.gz`; +const FIXTURE_CHECKSUM = createHash("sha256") + .update("fixture-archive") + .digest("hex"); +const FIXTURE_PLATFORM = { + targetTriple: FIXTURE_TARGET_TRIPLE, + os: "linux", + arch: "x64", +}; + +function createArtifactFixture(overrides = {}) { + return { + version: FIXTURE_VERSION, + binary: "sce", + archive: FIXTURE_ARCHIVE_NAME, + checksum_file: `${FIXTURE_ARCHIVE_NAME}.sha256`, + checksum_sha256: FIXTURE_CHECKSUM, + target_triple: FIXTURE_TARGET_TRIPLE, + os: "linux", + arch: "x64", + ...overrides, + }; +} + function createManifestFixture(overrides = {}) { const manifest = { - version: "0.1.0", - artifacts: [ - { target_triple: "x86_64-unknown-linux-musl", checksum_sha256: "abc123" }, - ], + version: FIXTURE_VERSION, + binary: "sce", + artifacts: [createArtifactFixture()], ...overrides, }; const manifestPayload = JSON.stringify(manifest); @@ -308,12 +333,7 @@ describe("npm installer trust flow", () => { .update(archiveContents) .digest("hex"); const { manifestPayload, publicKey, signature } = createManifestFixture({ - artifacts: [ - { - target_triple: "x86_64-unknown-linux-musl", - checksum_sha256: expectedChecksum, - }, - ], + artifacts: [createArtifactFixture({ checksum_sha256: expectedChecksum })], }); let extractCalls = 0; @@ -350,7 +370,7 @@ describe("npm installer trust flow", () => { const archiveContents = "verified-archive"; const checksum = createHash("sha256").update(archiveContents).digest("hex"); const { manifestPayload, publicKey, signature } = createManifestFixture({ - artifacts: [{ target_triple: targetTriple, checksum_sha256: checksum }], + artifacts: [createArtifactFixture({ checksum_sha256: checksum })], }); const tempDir = path.join( tmpdir(), @@ -413,3 +433,125 @@ describe("npm installer trust flow", () => { }); }); }); + +describe("npm installer manifest identity validation", () => { + async function expectRejectedBeforeDownload(manifestOverrides, message) { + const { manifestPayload, publicKey, signature } = + createManifestFixture(manifestOverrides); + let archiveDownloads = 0; + + await expect( + installBinaryWithDependencies({ + supportedPlatform: FIXTURE_PLATFORM, + version: FIXTURE_VERSION, + loadReleaseManifest: () => + loadVerifiedReleaseManifest(FIXTURE_VERSION, { + downloadManifest: async () => manifestPayload, + downloadManifestSignature: async () => signature, + publicKeyPem: publicKey, + }), + createTempDir: () => { + throw new Error("temp dir must not be created"); + }, + downloadArchive: async () => { + archiveDownloads += 1; + }, + }), + ).rejects.toThrow(message); + + expect(archiveDownloads).toBe(0); + } + + test("rejects a manifest whose version differs from the package version", async () => { + await expectRejectedBeforeDownload( + { version: "0.1.1" }, + 'Invalid sce release manifest: version "0.1.1" does not match npm package version "0.1.0".', + ); + }); + + test("rejects a manifest whose binary is not sce", async () => { + await expectRejectedBeforeDownload( + { binary: "not-sce" }, + 'Invalid sce release manifest: binary "not-sce" is not "sce".', + ); + }); + + test("rejects a manifest with a missing binary field", async () => { + await expectRejectedBeforeDownload( + { binary: undefined }, + 'Invalid sce release manifest: binary undefined is not "sce".', + ); + }); + + test("rejects a manifest without an artifact for the current target", async () => { + await expectRejectedBeforeDownload( + { + artifacts: [ + createArtifactFixture({ target_triple: "aarch64-apple-darwin" }), + ], + }, + `No sce release artifact found for target ${FIXTURE_TARGET_TRIPLE}.`, + ); + }); + + test("rejects a manifest with duplicate artifacts for the current target", async () => { + await expectRejectedBeforeDownload( + { artifacts: [createArtifactFixture(), createArtifactFixture()] }, + `Invalid sce release manifest: found 2 artifacts for target ${FIXTURE_TARGET_TRIPLE}; expected exactly one.`, + ); + }); + + test("rejects an artifact whose archive name is not canonical", async () => { + await expectRejectedBeforeDownload( + { artifacts: [createArtifactFixture({ archive: "../evil.tar.gz" })] }, + `Invalid sce release artifact for target ${FIXTURE_TARGET_TRIPLE}: archive "../evil.tar.gz" does not match expected "${FIXTURE_ARCHIVE_NAME}".`, + ); + }); + + test("rejects an artifact with a missing archive name", async () => { + await expectRejectedBeforeDownload( + { artifacts: [createArtifactFixture({ archive: undefined })] }, + `Invalid sce release artifact for target ${FIXTURE_TARGET_TRIPLE}: archive undefined does not match expected "${FIXTURE_ARCHIVE_NAME}".`, + ); + }); + + for (const [label, checksum] of [ + ["short", "abc123"], + ["uppercase", FIXTURE_CHECKSUM.toUpperCase()], + ["non-hex", "g".repeat(64)], + ["missing", undefined], + ]) { + test(`rejects an artifact with a ${label} checksum_sha256`, async () => { + await expectRejectedBeforeDownload( + { artifacts: [createArtifactFixture({ checksum_sha256: checksum })] }, + `Invalid sce release artifact ${FIXTURE_ARCHIVE_NAME}: checksum_sha256 must be 64 lowercase hexadecimal characters.`, + ); + }); + } + + test("downloads the canonical archive name for a valid manifest", async () => { + const { manifestPayload, publicKey, signature } = createManifestFixture(); + const archiveUrls = []; + + await expect( + installBinaryWithDependencies({ + supportedPlatform: FIXTURE_PLATFORM, + version: FIXTURE_VERSION, + loadReleaseManifest: () => + loadVerifiedReleaseManifest(FIXTURE_VERSION, { + downloadManifest: async () => manifestPayload, + downloadManifestSignature: async () => signature, + publicKeyPem: publicKey, + }), + downloadArchive: async (url) => { + archiveUrls.push(url); + }, + checksumFile: async () => "mismatch", + }), + ).rejects.toThrow("Downloaded sce archive checksum mismatch"); + + expect(archiveUrls).toEqual([ + `${getReleaseBaseUrl(FIXTURE_VERSION)}/${FIXTURE_ARCHIVE_NAME}`, + ]); + }); +}); diff --git a/npm/test/platform.test.js b/npm/test/platform.test.js index 77e7d5059..2b0647e26 100644 --- a/npm/test/platform.test.js +++ b/npm/test/platform.test.js @@ -103,4 +103,20 @@ describe("selectReleaseArtifact", () => { "No sce release artifact found for target x86_64-unknown-linux-musl.", ); }); + + test("throws when target triple appears more than once", () => { + expect(() => + selectReleaseArtifact( + { + artifacts: [ + { target_triple: "x86_64-unknown-linux-musl", archive: "a.tgz" }, + { target_triple: "x86_64-unknown-linux-musl", archive: "b.tgz" }, + ], + }, + "x86_64-unknown-linux-musl", + ), + ).toThrow( + "Invalid sce release manifest: found 2 artifacts for target x86_64-unknown-linux-musl; expected exactly one.", + ); + }); }); From 1007b015c339e2730516814c6270af9e46a40f52 Mon Sep 17 00:00:00 2001 From: David Abram Date: Tue, 3 Nov 2026 17:58:35 +0100 Subject: [PATCH 4/6] npm: Harden release archive extraction and binary installation Replace host tar extraction with a bounded ustar/GNU archive reader that validates paths, entry types, sizes, and checksums before staging only bin/sce. Install staged binaries through durable temp-file writes and atomic rename so failed checksum, extraction, or interrupted installs preserve existing runtime state. Add fixture and installer tests covering malicious archives, GNU tar compatibility, size limits, and failure cleanup, and include the new archive sources in npm checks. Plan: harden-npm-installer.md (T02, T03) Co-authored-by: SCE --- context/architecture.md | 2 +- context/context-map.md | 2 +- context/plans/harden-npm-installer.md | 17 +- context/sce/cli-npm-distribution-contract.md | 6 +- flake.nix | 3 + npm/lib/archive.js | 423 +++++++++++++++++++ npm/lib/install.js | 65 +-- npm/test/archive.test.js | 370 ++++++++++++++++ npm/test/install.test.js | 228 +++++++--- npm/test/tar-fixture.js | 102 +++++ 10 files changed, 1095 insertions(+), 123 deletions(-) create mode 100644 npm/lib/archive.js create mode 100644 npm/test/archive.test.js create mode 100644 npm/test/tar-fixture.js diff --git a/context/architecture.md b/context/architecture.md index 1a7d00c93..3b117a9c1 100644 --- a/context/architecture.md +++ b/context/architecture.md @@ -43,7 +43,7 @@ The repository includes a new placeholder Rust binary crate at `cli/`. - `flake.nix` now exposes dedicated release-packaging apps for this rollout: `apps.release-artifacts` packages the current-platform `sce` archive/checksum/manifest-fragment set from the release output `packages.sce-release` (`nix build .#sce-release`), prepares/audits the staged `bin/sce` before archive creation (including macOS `libiconv` install-name rewriting plus ad-hoc re-signing when needed), and fails if `.version`, `cli/Cargo.toml`, `npm/package.json`, the built/prepared CLI version, or the native portability audit disagree; `apps.native-portability-audit` audits native binaries for forbidden `/nix/store/` runtime references using Linux ELF/string inspection or macOS `otool -L`; `apps.release-manifest` merges per-platform metadata fragments into release-level manifest/checksum outputs and signs the merged manifest with a non-repo private key provided via env/file input; `apps.release-npm-package` packs the checked-in npm launcher package into a release-ready `sce-v-npm.tgz` tarball plus npm metadata JSON while refusing mismatched checked-in version metadata; and Linux-only `apps.release-flatpak-package` emits the Flatpak source-manifest tarball/checksum/JSON metadata while refusing `.version`/Cargo/npm/AppStream version drift and non-git checkouts that cannot resolve a release commit; Linux-only `apps.release-flatpak-bundle` builds a source-built `.flatpak` bundle from the checkout and emits per-architecture bundle/checksum/JSON metadata. - CLI release automation: `.github/workflows/release-sce.yml` orchestrates `sce` GitHub release asset assembly, validates `.version`/Cargo/npm parity before tagging or releasing, calls three reusable per-platform workflow files (`release-sce-linux.yml`, `release-sce-linux-arm.yml`, `release-sce-macos-arm.yml`), assembles the signed native release manifest, builds npm and Flatpak source-manifest release packages, and uploads the CLI/npm/Flatpak asset set to the GitHub Release. Each native reusable workflow validates its generated archive before native artifact upload by extracting the target archive, smoke-running `bin/sce version --format json`, and running `nix run .#native-portability-audit` with the lane platform. Manual `workflow_dispatch` runs can set the GitHub Release-level `prerelease` flag through an explicit checkbox; tag-triggered releases keep the flag false and no tag-name prerelease inference is performed. - Downstream registry publication is now implemented in dedicated workflows: `.github/workflows/publish-crates.yml` publishes the checked-in crate version from a published release or manual dispatch after `.version`/tag/Cargo parity checks and requires semver prerelease metadata when a publish run is marked prerelease; `.github/workflows/publish-npm.yml` publishes the checked-in npm package version from a published release or manual dispatch after `.version`/tag/npm parity checks plus verification of the canonical `sce-v-npm.tgz` GitHub release asset, using the `next` npm dist-tag instead of `latest` for prerelease versions or prerelease-marked runs. -- The npm distribution implementation lives under `npm/`: `package.json` defines the `sce` package surface, `bin/sce.js` launches the package-local native binary, `lib/install.js` resolves the current package version against the release manifest, verifies `sce-v-release-manifest.json.sig` with the bundled public key before trusting manifest contents, and then installs the checksum-verified native archive for supported macOS/Linux targets, while `test/platform.test.js` and `test/install.test.js` cover platform selection plus signed-manifest installer behavior. +- The npm distribution implementation lives under `npm/`: `package.json` defines the `sce` package surface, `bin/sce.js` launches the package-local native binary, `lib/install.js` resolves the current package version against the release manifest, verifies `sce-v-release-manifest.json.sig` with the bundled public key before trusting manifest contents, validates manifest identity, and then installs the checksum-verified native archive for supported macOS/Linux targets through `lib/archive.js` (bounded, entry-validating gzip+tar reader plus atomic `runtime/sce` replacement, no host `tar`), while `test/platform.test.js`, `test/install.test.js`, and `test/archive.test.js` (with the `test/tar-fixture.js` archive builder) cover platform selection, signed-manifest installer behavior, and archive/atomic-install hardening. - `cli/src/main.rs` is the executable entrypoint (`sce`) and delegates to `app::run`. - `cli/src/cli_schema.rs` defines the clap-based CLI schema using derive macros for all top-level commands and subcommands, including the top-level `sync` command, and renders command-local help text for the `auth` command tree (`auth`, `auth login`, `auth logout`, `auth whoami`). diff --git a/context/context-map.md b/context/context-map.md index f3c032d8c..43d95b16f 100644 --- a/context/context-map.md +++ b/context/context-map.md @@ -113,7 +113,7 @@ Feature/domain context: - `context/sce/opencode-agent-trace-plugin-runtime.md` (current OpenCode agent-trace plugin runtime behavior, including captured `message.updated` handoff with `summary.diffs` branching: when diffs exist sends one `-patch` mixed batch containing a synthetic parent message plus per-diff `message.part` patch items, when no diffs sends the original `message.updated` payload; in-memory dedup `Set` keyed by `"${sessionID}:${messageID}"`; captured `message.part.updated` handoff to `sce hooks conversation-trace` for `text`/`reasoning` parts with non-empty text plus completed `question` tool parts emitted as `part_type: "question"` with JSON-stringified `{ question, answer }[]`; existing user-message diff extraction for `{ sessionID, diff, time, model_id }`; session-scoped OpenCode client version capture from `session.created`/`session.updated`; and CLI handoff to `sce hooks diff-trace` over STDIN JSON with required `tool_name="opencode"` plus required nullable `tool_version`; Rust hook parsing and AgentTraceDb insertion persist `oc_`-prefixed session IDs plus required payload fields including `model_id`) - `context/sce/cli-first-install-channels-contract.md` (current Nix/Cargo/npm/source-built Flatpak channel contract, release authority and workflow topology, Nix-owned Flatpak manifest/cargo-source generation and validation, reduced Flatpak app surface, and host-git bridge decision) - `context/sce/cli-release-artifact-contract.md` (shared `sce` binary release artifact naming, checksum/manifest outputs, pre-archive staged-binary preparation including macOS `libiconv` install-name sanitization/ad-hoc re-signing, native portability audit app/check for forbidden `/nix/store/` runtime references, GitHub Releases as the canonical artifact publication surface, manual dispatch `prerelease` flag behavior, the current three-target Linux/macOS release workflow topology including pre-upload extracted-archive smoke/audit validation in each native lane, implemented Flatpak source-manifest and source-built `.flatpak` bundle package assets uploaded by `.github/workflows/release-sce.yml`, and Flatpak's explicit source-built non-binary exception) -- `context/sce/cli-npm-distribution-contract.md` (implemented `sce` npm launcher package, release-manifest/checksum-verified native binary install flow, the supported darwin/arm64 plus linux x64+arm64 npm platform matrix, the executable launcher-mode requirement and offline packed-package smoke test (`npm/test/pack-smoke.sh`), the prerequisite that npm-consumed native archives are portable and free of forbidden `/nix/store/` runtime references before upload, and dedicated `.github/workflows/publish-npm.yml` downstream npm publish-stage contract including prerelease publication via npm `next` dist-tag instead of `latest`) +- `context/sce/cli-npm-distribution-contract.md` (implemented `sce` npm launcher package, release-manifest/checksum-verified native binary install flow with manifest identity checks, bounded validating archive extraction, and atomic `runtime/sce` install, the supported darwin/arm64 plus linux x64+arm64 npm platform matrix, the executable launcher-mode requirement and offline packed-package smoke test (`npm/test/pack-smoke.sh`), the prerequisite that npm-consumed native archives are portable and free of forbidden `/nix/store/` runtime references before upload, and dedicated `.github/workflows/publish-npm.yml` downstream npm publish-stage contract including prerelease publication via npm `next` dist-tag instead of `latest`) - `context/sce/cli-cargo-distribution-contract.md` (implemented `sce` Cargo publication posture plus supported crates.io and wrapper-driven local-checkout install guidance, explicit unsupported `cargo install --git` boundary, dedicated crates.io publish workflow with semver prerelease guard, canonical generated-input declaration/producer ownership consumed by the Cargo wrapper, generated-output check, package-fallback preparation, and Nix pre-Cargo derivation, plus the deterministic combined checksummed fallback used for Pkl-free downstream crate builds) - `context/sce/flake-build-performance.md` (current native-vs-release package split, release-only commit embedding, shared-producer-backed pre-Cargo generated-input derivation consumed by native/release/test/Clippy but excluded from dependency/format derivations, slim default/database devShell contract, normal-vs-expensive CI tiers, durable before/after benchmark summary, validation evidence, and remaining cold-build trade-offs) - `context/sce/config-schema-publication.md` (current embedded-only publication contract for the generated SCE config JSON Schema and the canonical SCE web application URL used in config declarations) diff --git a/context/plans/harden-npm-installer.md b/context/plans/harden-npm-installer.md index defcc9f9d..d6d3a5be1 100644 --- a/context/plans/harden-npm-installer.md +++ b/context/plans/harden-npm-installer.md @@ -106,15 +106,26 @@ Current-state facts this plan starts from: - Additional regression: `nix shell nixpkgs#nodejs nixpkgs#openssl -c bash npm/test/pack-smoke.sh` -> `pack-smoke: OK (sce --help, sce version)` (the T01 fixture manifest already carries canonical identity fields). - Assumptions: per-artifact `version`/`binary` fields are not separately validated (AC3 covers top-level identity; the canonical archive name already encodes version and target). Version comparison is exact string equality. - Context impact: localized to the npm installer trust flow; affects `context/sce/cli-npm-distribution-contract.md` (manifest identity validation rules and new error surface). No architecture, glossary, or flake check changes. - - Context synchronization: pending + - Context synchronization: synced -- [ ] T03: `Replace tar extraction with bounded validating extractor and atomic install` (status:todo) +- [x] T03: `Replace tar extraction with bounded validating extractor and atomic install` (status:done) - Task ID: T03 - Scope: In — new `npm/lib/` archive module reading gzip+ustar with size/entry caps and the allowed-entry rules; extract only `bin/sce` into a fresh staging dir; stage into `runtime/` with a temp name, `chmod 0755`, `fsync`, and `rename` over `runtime/sce`; clean up staging on any failure; tests with generated fixture archives for absolute path, `..` traversal, symlink, hardlink, duplicate entry, unexpected type (e.g. FIFO/char device), entry outside root, oversize payload, missing or non-regular `bin/sce`, checksum failure, and an interrupted install (failure injected between staging and rename) preserving a prior valid `runtime/sce`. Out — manifest identity rules (T02), flake/release wiring. - Dependencies: T02 - Done when: `install.js` no longer spawns `tar`; all negative fixtures are rejected without writing outside staging; a prior valid install survives every failure path; the T01 smoke script still passes. - Verify: `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'`; `nix shell nixpkgs#ripgrep -c rg -n 'spawnSync' npm/lib`; rerun the T01 smoke script. - - Context synchronization: pending + - Completed: 2026-11-03 + - Files changed: `npm/lib/archive.js` (new), `npm/lib/install.js`, `npm/test/archive.test.js` (new), `npm/test/tar-fixture.js` (new), `npm/test/install.test.js`, `flake.nix` (`npmSrc` only). + - Result: Added `npm/lib/archive.js` with `readReleaseBinaryFromArchive`, `extractReleaseBinary`, `installBinaryAtomically`, and `ARCHIVE_LIMITS` (64 MiB compressed, 256 MiB uncompressed via `gunzipSync` `maxOutputLength`, 192 MiB `bin/sce`, 16 entries, 4 KiB long names). The reader parses ustar/GNU headers itself (header checksum and magic verified, GNU `L` long names and pax `x` `path` supported, base-256 numeric fields and pax `size`/`linkpath` rejected) and accepts only `/`, `/bin/`, `/bin/sce` (regular), `/LICENSE`, `/README.md`; it rejects absolute paths, `..`/empty/`.` segments, backslashes, symlinks, hardlinks, devices, FIFOs and other types, entries outside the root, unexpected or duplicate entries, wrong entry kinds, oversize payloads, truncated streams, and a missing `bin/sce`. Only `bin/sce` is written, to `/staging/sce` (`wx`, `0600`). `installBinaryAtomically` writes a `.sce...tmp` file in `runtime/`, `fchmod 0755`, `fsync`, renames it over `runtime/sce`, fsyncs the directory best-effort, and removes the temp file on any failure. `installBinaryWithDependencies` no longer spawns `tar`; its `fileExists`/`createRuntimeDir`/`copyBinary`/`chmodBinary` injection points were replaced by `extractArchiveFn(archivePath, { archiveRoot, stagingDir })`, `runtimeDir`, and `installBinaryFile`. Tests build fixture archives with an in-repo ustar/GNU writer (`npm/test/tar-fixture.js`) and cover every malicious case, a real GNU tar archive, prefix-split and long-name headers, and checksum, extraction, and interrupted-rename failures over a real temp `runtime/` that preserve a prior `0755` binary byte-identical with no leftover temp files. + - Verify outcomes: + - The planned `nix develop -c sh -c 'cd npm && test ./test/*.test.js'` form was blocked by repo bash policy `use-nix-for-bun`; ran the nixpkgs Bun form over `./test/archive.test.js ./test/install.test.js ./test/platform.test.js` (from `npm/`) -> 69 pass, 0 fail. + - `nix build .#checks.x86_64-linux.npm-bun-tests .#checks.x86_64-linux.npm-biome-check .#checks.x86_64-linux.npm-biome-format` -> exit 0 (69 pass in the sandbox, including the GNU tar compatibility test). + - `nix shell nixpkgs#ripgrep -c rg -n 'spawnSync|"-xzf"|spawnSync\("tar"' npm/lib` -> no match (exit 1). + - T01 smoke: `nix shell nixpkgs#nodejs nixpkgs#openssl -c bash npm/test/pack-smoke.sh` -> `pack-smoke: OK (sce --help, sce version)` (a real GNU tar archive extracted by the new reader under Node). + - `nix develop -c biome check npm` -> no errors; one pre-existing warning (unused `getRepositorySlug`, untouched). + - Deviations: `flake.nix` `npmSrc` gained `npm/lib/archive.js`, `npm/test/archive.test.js`, and `npm/test/tar-fixture.js` now rather than in T04, because `install.js` imports the new module and the flake npm checks would otherwise break; the new files were registered with `git add -N` so the flake sees them. The T01 smoke files remain unwired until T04. Per user instruction during implementation, the new code carries no comments. + - Context impact: localized to the npm installer extraction/install surface; affects `context/sce/cli-npm-distribution-contract.md` (safe extraction rules, size limits, atomic install, removed host `tar` runtime requirement). No architecture or glossary changes. + - Context synchronization: synced - [ ] T04: `Wire packed-package smoke into flake checks and release-npm-package` (status:todo) - Task ID: T04 diff --git a/context/sce/cli-npm-distribution-contract.md b/context/sce/cli-npm-distribution-contract.md index 02e0b9280..9a7343170 100644 --- a/context/sce/cli-npm-distribution-contract.md +++ b/context/sce/cli-npm-distribution-contract.md @@ -16,8 +16,11 @@ This file captures the implemented npm distribution slice of the `sce` CLI insta - `npm/bin/sce.js` launches a package-local native `sce` binary from `npm/runtime/sce`. - `npm/lib/install.js` runs during `postinstall` and downloads the release manifest plus detached signature for the package version. - The npm package ships a built-in manifest-signing public key at `npm/lib/release-manifest-public-key.pem`; `npm/lib/install.js` uses Node built-in crypto to verify manifest signatures without relying on network-fetched trust anchors. -- The installer downloads `sce-v-release-manifest.json` and detached signature `sce-v-release-manifest.json.sig`, verifies the manifest before parsing/selecting artifacts, then selects the artifact whose `target_triple` matches the current supported platform, downloads `sce-v-.tar.gz`, verifies `checksum_sha256`, extracts `bin/sce`, and stores it in package-local runtime storage. +- The installer downloads `sce-v-release-manifest.json` and detached signature `sce-v-release-manifest.json.sig`, verifies the manifest before parsing/selecting artifacts, validates the signed manifest's identity (below), downloads the canonical `sce-v-.tar.gz`, verifies `checksum_sha256`, extracts only `bin/sce` through the bounded validating reader (below), and installs it atomically into package-local runtime storage. - Missing or invalid manifest signatures abort installation before archive download or extraction with an authenticity failure. +- Manifest identity validation (`validateReleaseManifestIdentity` in `npm/lib/platform.js`) runs after signature verification and before any temp directory or archive download. It rejects, with a specific `Invalid sce release manifest` / `Invalid sce release artifact` / `No sce release artifact found` error, a manifest that is not a JSON object, whose `version` is not exactly the npm package version, whose `binary` is not `sce`, that has zero or more than one artifact for the current `target_triple`, whose artifact `archive` is not exactly `sce-v-.tar.gz`, or whose `checksum_sha256` is not 64 lowercase hex characters. The download name is always the canonical archive name; the installer never falls back to or trusts a different manifest-supplied archive name. Per-artifact `version`/`binary` fields are not separately checked. +- Archive extraction (`npm/lib/archive.js`) does not shell out to `tar` or depend on a host `tar`. It gunzips and parses the ustar/GNU tar stream with Node built-ins under fixed caps (`ARCHIVE_LIMITS`: 64 MiB compressed, 256 MiB uncompressed, 192 MiB `bin/sce`, 16 entries, 4 KiB long names), verifies each header checksum and ustar/GNU magic, honors GNU `L` long names and pax `path` records, and accepts exactly `sce-v-/`, `.../bin/`, `.../bin/sce` (regular file), `.../LICENSE`, and `.../README.md`. Any other entry fails with an `Invalid sce release archive:` error: absolute paths, `..`/empty/`.` segments, backslashes, symlinks, hardlinks, devices, FIFOs or other types, entries outside the archive root, unexpected or duplicate entries, a wrong entry kind, oversize payloads, truncated or non-gzip streams, and a missing `bin/sce`. Only `bin/sce` is written, into an isolated `staging/` directory inside the per-install download temp directory, which is removed afterward. +- Installation into `npm/runtime/sce` is atomic (`installBinaryAtomically`): the staged binary is written to a temporary file next to `runtime/sce`, set to mode `0755`, fsynced, and renamed over `runtime/sce`; the temp file is removed on any failure. A checksum failure, extraction failure, or interruption before the rename leaves an existing `runtime/sce` byte-identical and executable, and a failed fresh install leaves no `runtime/sce`. - Supported npm launcher platforms currently match the implemented release automation targets: - `linux/x64` -> `x86_64-unknown-linux-musl` - `linux/arm64` -> `aarch64-unknown-linux-musl` @@ -55,6 +58,7 @@ This file captures the implemented npm distribution slice of the `sce` CLI insta ## Packed-package smoke test - `npm/test/pack-smoke.sh` (with helper `npm/test/fixture-release-server.mjs`) verifies the real packed artifact end to end without network access: it stages a copy of the package (or `--package-dir `), swaps a throwaway RSA manifest-signing public key into the staged copy only, builds a stub `sce` archive in the canonical `sce-v-/{bin/sce,LICENSE,README.md}` layout with a matching signed manifest, serves it over loopback HTTPS (self-signed certificate trusted via `NODE_EXTRA_CA_CERTS`, selected via `SCE_NPM_RELEASE_BASE_URL`), runs `npm pack`, fails unless the packed `package/bin/sce.js` entry is executable, installs into an isolated `--prefix` with an isolated npm cache/config, and asserts exact output from `sce --help` and `sce version`. +- The fixture archive is produced by real GNU `tar`, so the smoke also exercises the in-repo archive reader against the release pipeline's archive format under Node. - Only the host platform is exercised, with a stub binary; the production public key and manifest trust code are not modified. ## Verification baseline diff --git a/flake.nix b/flake.nix index 8335a6e26..e92e7add0 100644 --- a/flake.nix +++ b/flake.nix @@ -306,11 +306,14 @@ ./npm/README.md ./npm/package.json ./npm/bin/sce.js + ./npm/lib/archive.js ./npm/lib/install.js ./npm/lib/platform.js ./npm/lib/release-manifest-public-key.pem + ./npm/test/archive.test.js ./npm/test/install.test.js ./npm/test/platform.test.js + ./npm/test/tar-fixture.js ./scripts/lib/release-manifest-signing.mjs ]; }; diff --git a/npm/lib/archive.js b/npm/lib/archive.js new file mode 100644 index 000000000..0e2f69ad4 --- /dev/null +++ b/npm/lib/archive.js @@ -0,0 +1,423 @@ +import { + closeSync, + fchmodSync, + fsyncSync, + openSync, + readFileSync, + renameSync, + rmSync, + statSync, + writeSync, +} from "node:fs"; +import path from "node:path"; +import { gunzipSync } from "node:zlib"; + +const TAR_BLOCK_SIZE = 512; +const MEBIBYTE = 1024 * 1024; + +export const ARCHIVE_LIMITS = Object.freeze({ + maxCompressedBytes: 64 * MEBIBYTE, + maxUncompressedBytes: 256 * MEBIBYTE, + maxBinaryBytes: 192 * MEBIBYTE, + maxEntries: 16, + maxLongNameBytes: 4096, +}); + +const STAGED_BINARY_NAME = "sce"; +const INSTALLED_BINARY_MODE = 0o755; +const STAGED_FILE_MODE = 0o600; + +const ENTRY_KIND_DIRECTORY = "directory"; +const ENTRY_KIND_FILE = "file"; + +const TYPE_REGULAR = "0"; +const TYPE_REGULAR_LEGACY = "\0"; +const TYPE_HARDLINK = "1"; +const TYPE_SYMLINK = "2"; +const TYPE_DIRECTORY = "5"; +const TYPE_GNU_LONG_NAME = "L"; +const TYPE_PAX_HEADER = "x"; + +const USTAR_MAGIC = "ustar\u000000"; +const GNU_MAGIC = "ustar \u0000"; + +function archiveError(message) { + return new Error(`Invalid sce release archive: ${message}`); +} + +function getAllowedEntries(archiveRoot) { + return new Map([ + [archiveRoot, ENTRY_KIND_DIRECTORY], + [`${archiveRoot}/bin`, ENTRY_KIND_DIRECTORY], + [`${archiveRoot}/bin/sce`, ENTRY_KIND_FILE], + [`${archiveRoot}/LICENSE`, ENTRY_KIND_FILE], + [`${archiveRoot}/README.md`, ENTRY_KIND_FILE], + ]); +} + +function readCString(block, offset, length) { + const field = block.subarray(offset, offset + length); + const end = field.indexOf(0); + return field.subarray(0, end === -1 ? field.length : end).toString("utf8"); +} + +function readOctal(block, offset, length, fieldName) { + const field = block.subarray(offset, offset + length); + + if (field[0] & 0x80) { + throw archiveError(`unsupported base-256 ${fieldName} field.`); + } + + const text = readCString(block, offset, length).trim(); + + if (text.length === 0) { + return 0; + } + + if (!/^[0-7]+$/.test(text)) { + throw archiveError(`malformed ${fieldName} field.`); + } + + return Number.parseInt(text, 8); +} + +function isZeroBlock(block) { + return block.every((byte) => byte === 0); +} + +function verifyHeaderChecksum(block) { + const expected = readOctal(block, 148, 8, "checksum"); + let actual = 0; + + for (let index = 0; index < TAR_BLOCK_SIZE; index += 1) { + actual += index >= 148 && index < 156 ? 0x20 : block[index]; + } + + if (actual !== expected) { + throw archiveError("tar header checksum mismatch."); + } +} + +function parsePaxPath(payload) { + let offset = 0; + let paxPath; + + while (offset < payload.length) { + const spaceIndex = payload.indexOf(0x20, offset); + + if (spaceIndex === -1) { + throw archiveError("malformed pax header record."); + } + + const recordLength = Number.parseInt( + payload.subarray(offset, spaceIndex).toString("utf8"), + 10, + ); + + if ( + !Number.isSafeInteger(recordLength) || + recordLength <= spaceIndex - offset || + offset + recordLength > payload.length + ) { + throw archiveError("malformed pax header record."); + } + + const record = payload + .subarray(spaceIndex + 1, offset + recordLength - 1) + .toString("utf8"); + const equalsIndex = record.indexOf("="); + + if (equalsIndex === -1) { + throw archiveError("malformed pax header record."); + } + + const key = record.slice(0, equalsIndex); + const value = record.slice(equalsIndex + 1); + + if (key === "path") { + paxPath = value; + } else if (key === "linkpath" || key === "size") { + throw archiveError(`unsupported pax header key ${JSON.stringify(key)}.`); + } + + offset += recordLength; + } + + return paxPath; +} + +function normalizeEntryName(rawName) { + if (rawName.length === 0) { + throw archiveError("entry with an empty name."); + } + + if (rawName.startsWith("/") || /^[A-Za-z]:/.test(rawName)) { + throw archiveError( + `entry ${JSON.stringify(rawName)} has an absolute path.`, + ); + } + + if (rawName.includes("\\")) { + throw archiveError( + `entry ${JSON.stringify(rawName)} contains a backslash.`, + ); + } + + const name = rawName.endsWith("/") ? rawName.slice(0, -1) : rawName; + const segments = name.split("/"); + + if (segments.includes("..")) { + throw archiveError( + `entry ${JSON.stringify(rawName)} contains a ".." path segment.`, + ); + } + + if (segments.some((segment) => segment === "" || segment === ".")) { + throw archiveError( + `entry ${JSON.stringify(rawName)} contains an empty or "." path segment.`, + ); + } + + return name; +} + +function describeEntryType(typeFlag) { + switch (typeFlag) { + case TYPE_HARDLINK: + return "hardlink"; + case TYPE_SYMLINK: + return "symlink"; + case "3": + return "character device"; + case "4": + return "block device"; + case "6": + return "FIFO"; + default: + return `type ${JSON.stringify(typeFlag)}`; + } +} + +function classifyEntryType(typeFlag, name) { + if (typeFlag === TYPE_REGULAR || typeFlag === TYPE_REGULAR_LEGACY) { + return ENTRY_KIND_FILE; + } + + if (typeFlag === TYPE_DIRECTORY) { + return ENTRY_KIND_DIRECTORY; + } + + throw archiveError( + `entry ${JSON.stringify(name)} is an unsupported ${describeEntryType(typeFlag)}.`, + ); +} + +function gunzipArchive(archivePath, limits) { + const { size } = statSync(archivePath); + + if (size > limits.maxCompressedBytes) { + throw archiveError( + `compressed size ${size} exceeds limit ${limits.maxCompressedBytes} bytes.`, + ); + } + + try { + return gunzipSync(readFileSync(archivePath), { + maxOutputLength: limits.maxUncompressedBytes, + }); + } catch (error) { + if (error?.code === "ERR_BUFFER_TOO_LARGE") { + throw archiveError( + `uncompressed size exceeds limit ${limits.maxUncompressedBytes} bytes.`, + ); + } + + throw archiveError(`failed to decompress gzip stream: ${error.message}`); + } +} + +export function readReleaseBinaryFromArchive( + archivePath, + { archiveRoot, limits = ARCHIVE_LIMITS }, +) { + const tarBuffer = gunzipArchive(archivePath, limits); + const allowedEntries = getAllowedEntries(archiveRoot); + const binaryEntryName = `${archiveRoot}/bin/sce`; + const seenEntries = new Set(); + let binaryPayload; + let pendingLongName; + let entryCount = 0; + let offset = 0; + + while (true) { + if (offset + TAR_BLOCK_SIZE > tarBuffer.length) { + throw archiveError("truncated tar stream (missing end-of-archive)."); + } + + const header = tarBuffer.subarray(offset, offset + TAR_BLOCK_SIZE); + offset += TAR_BLOCK_SIZE; + + if (isZeroBlock(header)) { + if (pendingLongName !== undefined) { + throw archiveError("long-name header without a following entry."); + } + break; + } + + verifyHeaderChecksum(header); + + const magic = header.subarray(257, 265).toString("latin1"); + + if (magic !== USTAR_MAGIC && magic !== GNU_MAGIC) { + throw archiveError("entry header is not ustar or GNU tar format."); + } + + const typeFlag = String.fromCharCode(header[156]); + const size = readOctal(header, 124, 12, "size"); + const paddedSize = Math.ceil(size / TAR_BLOCK_SIZE) * TAR_BLOCK_SIZE; + + if (offset + paddedSize > tarBuffer.length) { + throw archiveError("truncated tar entry payload."); + } + + const payload = tarBuffer.subarray(offset, offset + size); + offset += paddedSize; + + if (typeFlag === TYPE_GNU_LONG_NAME || typeFlag === TYPE_PAX_HEADER) { + if (pendingLongName !== undefined) { + throw archiveError("consecutive long-name headers."); + } + + if (size > limits.maxLongNameBytes) { + throw archiveError("long-name header exceeds size limit."); + } + + pendingLongName = + typeFlag === TYPE_GNU_LONG_NAME + ? readCString(payload, 0, payload.length) + : (parsePaxPath(payload) ?? null); + continue; + } + + entryCount += 1; + + if (entryCount > limits.maxEntries) { + throw archiveError(`more than ${limits.maxEntries} entries.`); + } + + const prefix = magic === USTAR_MAGIC ? readCString(header, 345, 155) : ""; + const headerName = readCString(header, 0, 100); + const rawName = + pendingLongName ?? (prefix ? `${prefix}/${headerName}` : headerName); + pendingLongName = undefined; + + const name = normalizeEntryName(rawName); + const kind = classifyEntryType(typeFlag, name); + + if (name !== archiveRoot && !name.startsWith(`${archiveRoot}/`)) { + throw archiveError( + `entry ${JSON.stringify(name)} is outside the archive root ${JSON.stringify(archiveRoot)}.`, + ); + } + + const allowedKind = allowedEntries.get(name); + + if (allowedKind === undefined) { + throw archiveError(`unexpected entry ${JSON.stringify(name)}.`); + } + + if (seenEntries.has(name)) { + throw archiveError(`duplicate entry ${JSON.stringify(name)}.`); + } + seenEntries.add(name); + + if (kind !== allowedKind) { + throw archiveError( + `entry ${JSON.stringify(name)} must be a ${allowedKind === ENTRY_KIND_FILE ? "regular file" : "directory"}.`, + ); + } + + if (name === binaryEntryName) { + if (size > limits.maxBinaryBytes) { + throw archiveError( + `bin/sce size ${size} exceeds limit ${limits.maxBinaryBytes} bytes.`, + ); + } + + binaryPayload = payload; + } + } + + if (binaryPayload === undefined) { + throw archiveError(`missing regular file ${binaryEntryName}.`); + } + + return binaryPayload; +} + +function writeFileDurably(filePath, contents, mode) { + const fd = openSync(filePath, "wx", STAGED_FILE_MODE); + + try { + let written = 0; + while (written < contents.length) { + written += writeSync(fd, contents, written, contents.length - written); + } + fchmodSync(fd, mode); + fsyncSync(fd); + } finally { + closeSync(fd); + } +} + +export function extractReleaseBinary( + archivePath, + { archiveRoot, stagingDir, limits = ARCHIVE_LIMITS }, +) { + const binaryPayload = readReleaseBinaryFromArchive(archivePath, { + archiveRoot, + limits, + }); + const stagedBinaryPath = path.join(stagingDir, STAGED_BINARY_NAME); + + writeFileDurably(stagedBinaryPath, binaryPayload, STAGED_FILE_MODE); + + return stagedBinaryPath; +} + +function fsyncDirectoryBestEffort(directoryPath) { + let fd; + + try { + fd = openSync(directoryPath, "r"); + fsyncSync(fd); + } catch { + return; + } finally { + if (fd !== undefined) { + closeSync(fd); + } + } +} + +export function installBinaryAtomically( + sourcePath, + destinationPath, + { renameFile = renameSync } = {}, +) { + const destinationDir = path.dirname(destinationPath); + const tempPath = path.join( + destinationDir, + `.${path.basename(destinationPath)}.${process.pid}.${Date.now()}.tmp`, + ); + + try { + writeFileDurably(tempPath, readFileSync(sourcePath), INSTALLED_BINARY_MODE); + renameFile(tempPath, destinationPath); + } catch (error) { + rmSync(tempPath, { force: true }); + throw error; + } + + fsyncDirectoryBestEffort(destinationDir); +} diff --git a/npm/lib/install.js b/npm/lib/install.js index c12664910..6de2ad5bd 100644 --- a/npm/lib/install.js +++ b/npm/lib/install.js @@ -1,11 +1,7 @@ -import { spawnSync } from "node:child_process"; import { createHash, createVerify } from "node:crypto"; import { - chmodSync, - copyFileSync, createReadStream, createWriteStream, - existsSync, mkdirSync, mkdtempSync, readFileSync, @@ -19,10 +15,10 @@ import process from "node:process"; import { pipeline } from "node:stream/promises"; import { fileURLToPath } from "node:url"; +import { extractReleaseBinary, installBinaryAtomically } from "./archive.js"; import { formatUnsupportedPlatformMessage, getArchiveRoot, - getInstalledBinaryPath, getReleaseManifestName, resolveSupportedPlatform, validateReleaseManifestIdentity, @@ -287,31 +283,6 @@ function sha256File(filePath) { }); } -function extractArchive(archivePath, destinationDir) { - const tarResult = spawnSync( - "tar", - ["-xzf", archivePath, "-C", destinationDir], - { - stdio: "pipe", - encoding: "utf8", - }, - ); - - if (tarResult.error) { - throw new Error( - `Failed to extract sce release archive: ${tarResult.error.message}`, - ); - } - - if (tarResult.status !== 0) { - throw new Error( - tarResult.stderr?.trim() || - tarResult.error?.message || - "Failed to extract sce release archive.", - ); - } -} - export async function installBinaryWithDependencies({ supportedPlatform = resolveSupportedPlatform(), unsupportedMessage = formatUnsupportedPlatformMessage(), @@ -319,11 +290,9 @@ export async function installBinaryWithDependencies({ loadReleaseManifest = loadVerifiedReleaseManifest, downloadArchive = downloadToFile, checksumFile = sha256File, - extractArchiveFn = extractArchive, - fileExists = existsSync, - createRuntimeDir = mkdirSync, - copyBinary = copyFileSync, - chmodBinary = chmodSync, + extractArchiveFn = extractReleaseBinary, + runtimeDir = RUNTIME_DIR, + installBinaryFile = installBinaryAtomically, createTempDir = () => mkdtempSync(path.join(tmpdir(), "sce-npm-install-")), removeTempDir = (tempDir) => rmSync(tempDir, { recursive: true, force: true }), @@ -362,25 +331,15 @@ export async function installBinaryWithDependencies({ ); } - extractArchiveFn(archivePath, tempDir); - - const extractedBinaryPath = path.join( - tempDir, - getArchiveRoot(version, supportedPlatform.targetTriple), - "bin", - "sce", - ); - - if (!fileExists(extractedBinaryPath)) { - throw new Error( - `Extracted sce archive did not contain ${archiveName} -> bin/sce.`, - ); - } + const stagingDir = path.join(tempDir, "staging"); + mkdirSync(stagingDir); + const stagedBinaryPath = extractArchiveFn(archivePath, { + archiveRoot: getArchiveRoot(version, supportedPlatform.targetTriple), + stagingDir, + }); - createRuntimeDir(RUNTIME_DIR, { recursive: true }); - const installedBinaryPath = getInstalledBinaryPath(__dirname); - copyBinary(extractedBinaryPath, installedBinaryPath); - chmodBinary(installedBinaryPath, 0o755); + mkdirSync(runtimeDir, { recursive: true }); + installBinaryFile(stagedBinaryPath, path.join(runtimeDir, "sce")); } finally { removeTempDir(tempDir); } diff --git a/npm/test/archive.test.js b/npm/test/archive.test.js new file mode 100644 index 000000000..610b786fd --- /dev/null +++ b/npm/test/archive.test.js @@ -0,0 +1,370 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { spawnSync } from "node:child_process"; +import { + chmodSync, + mkdirSync, + mkdtempSync, + readdirSync, + readFileSync, + rmSync, + statSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { gzipSync } from "node:zlib"; +import { + ARCHIVE_LIMITS, + extractReleaseBinary, + installBinaryAtomically, + readReleaseBinaryFromArchive, +} from "../lib/archive.js"; +import { + createReleaseEntries, + createTarArchive, + createTarEntry, + createTarHeader, +} from "./tar-fixture.js"; + +const ARCHIVE_ROOT = "sce-v0.1.0-x86_64-unknown-linux-musl"; +const BINARY_CONTENT = "#!/bin/sh\necho fixture-sce\n"; + +const tempDirs = []; + +afterEach(() => { + for (const tempDir of tempDirs.splice(0)) { + rmSync(tempDir, { recursive: true, force: true }); + } +}); + +function createTempDir() { + const tempDir = mkdtempSync(path.join(tmpdir(), "sce-npm-archive-test-")); + tempDirs.push(tempDir); + return tempDir; +} + +function writeArchive(entries, options) { + const tempDir = createTempDir(); + const archivePath = path.join(tempDir, `${ARCHIVE_ROOT}.tar.gz`); + writeFileSync(archivePath, createTarArchive(entries, options)); + return { tempDir, archivePath }; +} + +function expectExtractionRejected(entries, message, limits) { + const { tempDir, archivePath } = writeArchive(entries); + const stagingDir = path.join(tempDir, "staging"); + mkdirSync(stagingDir); + + expect(() => + extractReleaseBinary(archivePath, { + archiveRoot: ARCHIVE_ROOT, + stagingDir, + limits, + }), + ).toThrow(message); + + expect(readdirSync(stagingDir)).toEqual([]); + expect(readdirSync(tempDir).sort()).toEqual( + [`${ARCHIVE_ROOT}.tar.gz`, "staging"].sort(), + ); +} + +function releaseEntriesWith(extraEntries) { + return [...createReleaseEntries(ARCHIVE_ROOT), ...extraEntries]; +} + +describe("release archive extraction", () => { + test("extracts only bin/sce from a canonical archive into staging", () => { + const { tempDir, archivePath } = writeArchive( + createReleaseEntries(ARCHIVE_ROOT, { binaryContent: BINARY_CONTENT }), + ); + const stagingDir = path.join(tempDir, "staging"); + mkdirSync(stagingDir); + + const stagedBinaryPath = extractReleaseBinary(archivePath, { + archiveRoot: ARCHIVE_ROOT, + stagingDir, + }); + + expect(stagedBinaryPath).toBe(path.join(stagingDir, "sce")); + expect(readFileSync(stagedBinaryPath, "utf8")).toBe(BINARY_CONTENT); + expect(readdirSync(stagingDir)).toEqual(["sce"]); + }); + + test("accepts an archive produced by GNU tar in the release layout", () => { + const sourceDir = createTempDir(); + const rootDir = path.join(sourceDir, ARCHIVE_ROOT); + mkdirSync(path.join(rootDir, "bin"), { recursive: true }); + writeFileSync(path.join(rootDir, "bin", "sce"), BINARY_CONTENT); + chmodSync(path.join(rootDir, "bin", "sce"), 0o755); + writeFileSync(path.join(rootDir, "LICENSE"), "MIT\n"); + writeFileSync(path.join(rootDir, "README.md"), "# sce\n"); + const archivePath = path.join(sourceDir, `${ARCHIVE_ROOT}.tar.gz`); + const tarResult = spawnSync( + "tar", + [ + "--sort=name", + "--owner=0", + "--group=0", + "--numeric-owner", + "-C", + sourceDir, + "-czf", + archivePath, + ARCHIVE_ROOT, + ], + { encoding: "utf8" }, + ); + + if (tarResult.error || tarResult.status !== 0) { + console.warn( + `Skipping GNU tar compatibility assertion: ${tarResult.error?.message ?? tarResult.stderr}`, + ); + return; + } + + expect( + readReleaseBinaryFromArchive(archivePath, { + archiveRoot: ARCHIVE_ROOT, + }).toString("utf8"), + ).toBe(BINARY_CONTENT); + }); + + test("accepts ustar prefix-split names and GNU long-name headers", () => { + const longNamePayload = `${ARCHIVE_ROOT}/README.md`; + const entries = [ + { name: `${ARCHIVE_ROOT}/`, type: "5", mode: 0o755 }, + { + name: "LICENSE", + prefix: ARCHIVE_ROOT, + format: "ustar", + content: "MIT\n", + }, + { name: "././@LongLink", type: "L", content: `${longNamePayload}\0` }, + { name: "truncated-name", content: "# sce\n" }, + { name: `${ARCHIVE_ROOT}/bin/`, type: "5", mode: 0o755 }, + { name: `${ARCHIVE_ROOT}/bin/sce`, content: BINARY_CONTENT }, + ]; + const { archivePath } = writeArchive(entries); + + expect( + readReleaseBinaryFromArchive(archivePath, { + archiveRoot: ARCHIVE_ROOT, + }).toString("utf8"), + ).toBe(BINARY_CONTENT); + }); + + test("rejects an absolute entry path", () => { + expectExtractionRejected( + releaseEntriesWith([{ name: "/etc/passwd", content: "x" }]), + 'Invalid sce release archive: entry "/etc/passwd" has an absolute path.', + ); + }); + + test("rejects a .. traversal entry", () => { + expectExtractionRejected( + releaseEntriesWith([{ name: `${ARCHIVE_ROOT}/../escape`, content: "x" }]), + `Invalid sce release archive: entry "${ARCHIVE_ROOT}/../escape" contains a ".." path segment.`, + ); + }); + + test("rejects a .. traversal hidden in a GNU long name", () => { + expectExtractionRejected( + [ + { name: "././@LongLink", type: "L", content: "../../escape\0" }, + { name: "innocent", content: "x" }, + ...createReleaseEntries(ARCHIVE_ROOT), + ], + 'Invalid sce release archive: entry "../../escape" contains a ".." path segment.', + ); + }); + + test("rejects a symlink entry", () => { + expectExtractionRejected( + [ + ...createReleaseEntries(ARCHIVE_ROOT).slice(0, 4), + { name: `${ARCHIVE_ROOT}/bin/sce`, type: "2", linkName: "/bin/sh" }, + ], + `Invalid sce release archive: entry "${ARCHIVE_ROOT}/bin/sce" is an unsupported symlink.`, + ); + }); + + test("rejects a hardlink entry", () => { + expectExtractionRejected( + [ + ...createReleaseEntries(ARCHIVE_ROOT).slice(0, 4), + { + name: `${ARCHIVE_ROOT}/bin/sce`, + type: "1", + linkName: `${ARCHIVE_ROOT}/LICENSE`, + }, + ], + `Invalid sce release archive: entry "${ARCHIVE_ROOT}/bin/sce" is an unsupported hardlink.`, + ); + }); + + test("rejects a duplicate entry", () => { + expectExtractionRejected( + releaseEntriesWith([ + { name: `${ARCHIVE_ROOT}/bin/sce`, content: "#!/bin/sh\nevil\n" }, + ]), + `Invalid sce release archive: duplicate entry "${ARCHIVE_ROOT}/bin/sce".`, + ); + }); + + for (const [label, type] of [ + ["FIFO", "6"], + ["character device", "3"], + ["block device", "4"], + ]) { + test(`rejects an unexpected ${label} entry`, () => { + expectExtractionRejected( + releaseEntriesWith([{ name: `${ARCHIVE_ROOT}/device`, type }]), + `Invalid sce release archive: entry "${ARCHIVE_ROOT}/device" is an unsupported ${label}.`, + ); + }); + } + + test("rejects an entry outside the archive root", () => { + expectExtractionRejected( + releaseEntriesWith([{ name: "other-root/bin/sce", content: "x" }]), + `Invalid sce release archive: entry "other-root/bin/sce" is outside the archive root "${ARCHIVE_ROOT}".`, + ); + }); + + test("rejects an unexpected entry inside the archive root", () => { + expectExtractionRejected( + releaseEntriesWith([{ name: `${ARCHIVE_ROOT}/bin/extra`, content: "x" }]), + `Invalid sce release archive: unexpected entry "${ARCHIVE_ROOT}/bin/extra".`, + ); + }); + + test("rejects an oversize bin/sce payload", () => { + expectExtractionRejected( + createReleaseEntries(ARCHIVE_ROOT, { binaryContent: "x".repeat(2048) }), + "Invalid sce release archive: bin/sce size 2048 exceeds limit 1024 bytes.", + { ...ARCHIVE_LIMITS, maxBinaryBytes: 1024 }, + ); + }); + + test("rejects an archive whose uncompressed size exceeds the limit", () => { + expectExtractionRejected( + createReleaseEntries(ARCHIVE_ROOT, { + binaryContent: "x".repeat(64 * 1024), + }), + "Invalid sce release archive: uncompressed size exceeds limit 16384 bytes.", + { ...ARCHIVE_LIMITS, maxUncompressedBytes: 16 * 1024 }, + ); + }); + + test("rejects an archive whose compressed size exceeds the limit", () => { + expectExtractionRejected( + createReleaseEntries(ARCHIVE_ROOT), + /Invalid sce release archive: compressed size \d+ exceeds limit 16 bytes\./, + { ...ARCHIVE_LIMITS, maxCompressedBytes: 16 }, + ); + }); + + test("rejects an archive with too many entries", () => { + expectExtractionRejected( + createReleaseEntries(ARCHIVE_ROOT), + "Invalid sce release archive: more than 3 entries.", + { ...ARCHIVE_LIMITS, maxEntries: 3 }, + ); + }); + + test("rejects an archive missing bin/sce", () => { + expectExtractionRejected( + createReleaseEntries(ARCHIVE_ROOT).slice(0, 4), + `Invalid sce release archive: missing regular file ${ARCHIVE_ROOT}/bin/sce.`, + ); + }); + + test("rejects a non-regular bin/sce", () => { + expectExtractionRejected( + [ + ...createReleaseEntries(ARCHIVE_ROOT).slice(0, 4), + { name: `${ARCHIVE_ROOT}/bin/sce/`, type: "5" }, + ], + `Invalid sce release archive: entry "${ARCHIVE_ROOT}/bin/sce" must be a regular file.`, + ); + }); + + test("rejects a header with a corrupted checksum", () => { + const header = createTarHeader({ name: `${ARCHIVE_ROOT}/`, type: "5" }); + header[0] ^= 0x01; + + expectExtractionRejected( + [header], + "Invalid sce release archive: tar header checksum mismatch.", + ); + }); + + test("rejects a truncated tar stream", () => { + const { tempDir } = writeArchive([]); + const archivePath = path.join(tempDir, "truncated.tar.gz"); + writeFileSync( + archivePath, + gzipSync( + createTarEntry({ name: `${ARCHIVE_ROOT}/`, type: "5" }).subarray( + 0, + 512, + ), + ), + ); + + expect(() => + readReleaseBinaryFromArchive(archivePath, { archiveRoot: ARCHIVE_ROOT }), + ).toThrow("Invalid sce release archive: truncated tar stream"); + }); + + test("rejects a non-gzip payload", () => { + const { tempDir } = writeArchive([]); + const archivePath = path.join(tempDir, "plain.tar.gz"); + writeFileSync(archivePath, "not a gzip stream"); + + expect(() => + readReleaseBinaryFromArchive(archivePath, { archiveRoot: ARCHIVE_ROOT }), + ).toThrow("Invalid sce release archive: failed to decompress gzip stream"); + }); +}); + +describe("atomic binary install", () => { + function setupRuntime() { + const tempDir = createTempDir(); + const runtimeDir = path.join(tempDir, "runtime"); + const sourcePath = path.join(tempDir, "staged-sce"); + const destinationPath = path.join(runtimeDir, "sce"); + mkdirSync(runtimeDir); + writeFileSync(sourcePath, "#!/bin/sh\necho new\n"); + writeFileSync(destinationPath, "#!/bin/sh\necho old\n"); + chmodSync(destinationPath, 0o755); + return { runtimeDir, sourcePath, destinationPath }; + } + + test("replaces the installed binary with mode 0755 and leaves no temp files", () => { + const { runtimeDir, sourcePath, destinationPath } = setupRuntime(); + + installBinaryAtomically(sourcePath, destinationPath); + + expect(readFileSync(destinationPath, "utf8")).toBe("#!/bin/sh\necho new\n"); + expect(statSync(destinationPath).mode & 0o777).toBe(0o755); + expect(readdirSync(runtimeDir)).toEqual(["sce"]); + }); + + test("preserves the previous binary when interrupted before rename", () => { + const { runtimeDir, sourcePath, destinationPath } = setupRuntime(); + + expect(() => + installBinaryAtomically(sourcePath, destinationPath, { + renameFile: () => { + throw new Error("simulated interruption"); + }, + }), + ).toThrow("simulated interruption"); + + expect(readFileSync(destinationPath, "utf8")).toBe("#!/bin/sh\necho old\n"); + expect(statSync(destinationPath).mode & 0o777).toBe(0o755); + expect(readdirSync(runtimeDir)).toEqual(["sce"]); + }); +}); diff --git a/npm/test/install.test.js b/npm/test/install.test.js index 8723605a5..4153d6b16 100644 --- a/npm/test/install.test.js +++ b/npm/test/install.test.js @@ -1,6 +1,16 @@ import { afterEach, describe, expect, test } from "bun:test"; import { createHash, generateKeyPairSync, sign } from "node:crypto"; -import { existsSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { + chmodSync, + existsSync, + mkdirSync, + mkdtempSync, + readdirSync, + readFileSync, + rmSync, + statSync, + writeFileSync, +} from "node:fs"; import { tmpdir } from "node:os"; import path from "node:path"; import { @@ -8,6 +18,10 @@ import { resolveReleaseManifestSigningKey, SIGNING_KEY_ENV_VAR, } from "../../scripts/lib/release-manifest-signing.mjs"; +import { + extractReleaseBinary, + installBinaryAtomically, +} from "../lib/archive.js"; import { getReleaseBaseUrl, installBinaryWithDependencies, @@ -16,7 +30,7 @@ import { readBundledReleaseManifestPublicKey, verifyReleaseManifestSignature, } from "../lib/install.js"; -import { getInstalledBinaryPath } from "../lib/platform.js"; +import { createReleaseEntries, createTarArchive } from "./tar-fixture.js"; const FIXTURE_VERSION = "0.1.0"; const FIXTURE_TARGET_TRIPLE = "x86_64-unknown-linux-musl"; @@ -74,6 +88,78 @@ afterEach(() => { } }); +const FIXTURE_ARCHIVE_ROOT = `sce-v${FIXTURE_VERSION}-${FIXTURE_TARGET_TRIPLE}`; +const FIXTURE_BINARY_CONTENT = "#!/bin/sh\necho fixture-sce\n"; +const PREVIOUS_BINARY_CONTENT = "#!/bin/sh\necho previous-sce\n"; + +function createScratchDir() { + const scratchDir = mkdtempSync(path.join(tmpdir(), "sce-npm-install-test-")); + tempDirs.push(scratchDir); + return scratchDir; +} + +function expectPreviousInstallPreserved(runtimeDir) { + const installedPath = path.join(runtimeDir, "sce"); + + expect(readFileSync(installedPath, "utf8")).toBe(PREVIOUS_BINARY_CONTENT); + expect(statSync(installedPath).mode & 0o777).toBe(0o755); + expect(readdirSync(runtimeDir)).toEqual(["sce"]); +} + +async function runFixtureInstall({ + archiveEntries = createReleaseEntries(FIXTURE_ARCHIVE_ROOT, { + binaryContent: FIXTURE_BINARY_CONTENT, + }), + previousBinary, + expectedError, + removeTempDir = false, + ...overrides +} = {}) { + const archiveBuffer = createTarArchive(archiveEntries); + const checksum = createHash("sha256").update(archiveBuffer).digest("hex"); + const { manifestPayload, publicKey, signature } = createManifestFixture({ + artifacts: [createArtifactFixture({ checksum_sha256: checksum })], + }); + const scratchDir = createScratchDir(); + const runtimeDir = path.join(scratchDir, "runtime"); + const tempDir = path.join(scratchDir, "download"); + + if (previousBinary !== undefined) { + mkdirSync(runtimeDir); + writeFileSync(path.join(runtimeDir, "sce"), previousBinary); + chmodSync(path.join(runtimeDir, "sce"), 0o755); + } + + const install = installBinaryWithDependencies({ + supportedPlatform: FIXTURE_PLATFORM, + version: FIXTURE_VERSION, + loadReleaseManifest: () => + loadVerifiedReleaseManifest(FIXTURE_VERSION, { + downloadManifest: async () => manifestPayload, + downloadManifestSignature: async () => signature, + publicKeyPem: publicKey, + }), + createTempDir: () => { + mkdirSync(tempDir); + return tempDir; + }, + ...(removeTempDir ? {} : { removeTempDir: () => {} }), + downloadArchive: async (_url, destinationPath) => { + writeFileSync(destinationPath, archiveBuffer); + }, + runtimeDir, + ...overrides, + }); + + if (expectedError === undefined) { + await expect(install).resolves.toBeUndefined(); + } else { + await expect(install).rejects.toThrow(expectedError); + } + + return { runtimeDir, tempDir }; +} + describe("release manifest verification primitives", () => { test("builds the default release base URL from the publisher repository slug", () => { const originalRepository = process.env.GITHUB_REPOSITORY; @@ -365,72 +451,86 @@ describe("npm installer trust flow", () => { }); test("installs after verifying the manifest signature and archive checksum", async () => { - const version = "0.1.0"; - const targetTriple = "x86_64-unknown-linux-musl"; - const archiveContents = "verified-archive"; - const checksum = createHash("sha256").update(archiveContents).digest("hex"); - const { manifestPayload, publicKey, signature } = createManifestFixture({ - artifacts: [createArtifactFixture({ checksum_sha256: checksum })], - }); - const tempDir = path.join( - tmpdir(), - `sce-npm-install-test-${Date.now()}-${Math.random()}`, + const { runtimeDir } = await runFixtureInstall(); + + expect(readFileSync(path.join(runtimeDir, "sce"), "utf8")).toBe( + FIXTURE_BINARY_CONTENT, ); - const installedBinaryPaths = []; - const chmodCalls = []; - tempDirs.push(tempDir); + expect(statSync(path.join(runtimeDir, "sce")).mode & 0o777).toBe(0o755); + expect(readdirSync(runtimeDir)).toEqual(["sce"]); + }); +}); - await expect( - installBinaryWithDependencies({ - supportedPlatform: { targetTriple, os: "linux", arch: "x64" }, - version, - loadReleaseManifest: () => - loadVerifiedReleaseManifest(version, { - downloadManifest: async () => manifestPayload, - downloadManifestSignature: async () => signature, - publicKeyPem: publicKey, - }), - createTempDir: () => { - mkdirSync(tempDir, { recursive: true }); - return tempDir; - }, - removeTempDir: () => {}, - downloadArchive: async (_url, destinationPath) => { - writeFileSync(destinationPath, archiveContents); - }, - checksumFile: async () => checksum, - extractArchiveFn: (_archivePath, destinationDir) => { - const extractedBinaryPath = path.join( - destinationDir, - `sce-v${version}-${targetTriple}`, - "bin", - "sce", - ); - mkdirSync(path.dirname(extractedBinaryPath), { recursive: true }); - writeFileSync(extractedBinaryPath, "#!/usr/bin/env bash\nexit 0\n"); - }, - fileExists: existsSync, - createRuntimeDir: () => {}, - copyBinary: (sourcePath, destinationPath) => { - installedBinaryPaths.push({ sourcePath, destinationPath }); - }, - chmodBinary: (destinationPath, mode) => { - chmodCalls.push({ destinationPath, mode }); - }, - }), - ).resolves.toBeUndefined(); +describe("npm installer extraction and atomic install", () => { + test("writes only bin/sce into the staging directory", async () => { + const stagingContents = []; - expect(installedBinaryPaths).toHaveLength(1); - expect(installedBinaryPaths[0].sourcePath).toBe( - path.join(tempDir, `sce-v${version}-${targetTriple}`, "bin", "sce"), - ); - expect(installedBinaryPaths[0].destinationPath).toBe( - getInstalledBinaryPath(path.join(process.cwd(), "lib")), - ); - expect(chmodCalls).toContainEqual({ - destinationPath: installedBinaryPaths[0].destinationPath, - mode: 0o755, + await runFixtureInstall({ + extractArchiveFn: (archivePath, options) => { + const stagedBinaryPath = extractReleaseBinary(archivePath, options); + stagingContents.push(readdirSync(options.stagingDir)); + return stagedBinaryPath; + }, + }); + + expect(stagingContents).toEqual([["sce"]]); + }); + + test("preserves a previous install when the archive checksum fails", async () => { + const { runtimeDir } = await runFixtureInstall({ + previousBinary: PREVIOUS_BINARY_CONTENT, + checksumFile: async () => "0".repeat(64), + expectedError: "Downloaded sce archive checksum mismatch", }); + + expectPreviousInstallPreserved(runtimeDir); + }); + + test("preserves a previous install when extraction rejects the archive", async () => { + const { runtimeDir } = await runFixtureInstall({ + previousBinary: PREVIOUS_BINARY_CONTENT, + archiveEntries: [ + ...createReleaseEntries(FIXTURE_ARCHIVE_ROOT), + { name: `${FIXTURE_ARCHIVE_ROOT}/../escape`, content: "x" }, + ], + expectedError: 'contains a ".." path segment', + }); + + expectPreviousInstallPreserved(runtimeDir); + }); + + test("preserves a previous install when interrupted between staging and rename", async () => { + const { runtimeDir } = await runFixtureInstall({ + previousBinary: PREVIOUS_BINARY_CONTENT, + installBinaryFile: (sourcePath, destinationPath) => + installBinaryAtomically(sourcePath, destinationPath, { + renameFile: () => { + throw new Error("simulated interruption"); + }, + }), + expectedError: "simulated interruption", + }); + + expectPreviousInstallPreserved(runtimeDir); + }); + + test("leaves no partial runtime/sce when a fresh install fails", async () => { + const { runtimeDir } = await runFixtureInstall({ + archiveEntries: createReleaseEntries(FIXTURE_ARCHIVE_ROOT).slice(0, 4), + expectedError: "missing regular file", + }); + + expect(existsSync(path.join(runtimeDir, "sce"))).toBe(false); + }); + + test("removes the download temp directory after a failed install", async () => { + const { tempDir } = await runFixtureInstall({ + archiveEntries: createReleaseEntries(FIXTURE_ARCHIVE_ROOT).slice(0, 4), + expectedError: "missing regular file", + removeTempDir: true, + }); + + expect(existsSync(tempDir)).toBe(false); }); }); diff --git a/npm/test/tar-fixture.js b/npm/test/tar-fixture.js new file mode 100644 index 000000000..1c8e89217 --- /dev/null +++ b/npm/test/tar-fixture.js @@ -0,0 +1,102 @@ +import { gzipSync } from "node:zlib"; + +const TAR_BLOCK_SIZE = 512; + +function writeString(block, offset, length, value) { + const bytes = Buffer.from(value, "utf8"); + + if (bytes.length > length) { + throw new Error(`tar fixture field too long: ${value}`); + } + + bytes.copy(block, offset); +} + +function writeOctal(block, offset, length, value) { + writeString( + block, + offset, + length, + `${value.toString(8).padStart(length - 1, "0")}\0`, + ); +} + +function padToBlock(buffer) { + const remainder = buffer.length % TAR_BLOCK_SIZE; + return remainder === 0 + ? buffer + : Buffer.concat([buffer, Buffer.alloc(TAR_BLOCK_SIZE - remainder)]); +} + +export function createTarHeader({ + name, + size = 0, + type = "0", + mode = 0o644, + linkName = "", + prefix = "", + format = "gnu", +}) { + const block = Buffer.alloc(TAR_BLOCK_SIZE); + + writeString(block, 0, 100, name); + writeOctal(block, 100, 8, mode); + writeOctal(block, 108, 8, 0); + writeOctal(block, 116, 8, 0); + writeOctal(block, 124, 12, size); + writeOctal(block, 136, 12, 0); + block.fill(0x20, 148, 156); + writeString(block, 156, 1, type); + writeString(block, 157, 100, linkName); + + if (format === "ustar") { + writeString(block, 257, 8, "ustar\u000000"); + writeString(block, 345, 155, prefix); + } else { + writeString(block, 257, 8, "ustar \u0000"); + } + + let checksum = 0; + for (const byte of block) { + checksum += byte; + } + writeString(block, 148, 8, `${checksum.toString(8).padStart(6, "0")}\0 `); + + return block; +} + +export function createTarEntry({ content = "", ...header }) { + const payload = Buffer.from(content); + return Buffer.concat([ + createTarHeader({ ...header, size: payload.length }), + padToBlock(payload), + ]); +} + +export function createTarArchive(entries, { gzip = true } = {}) { + const tarBuffer = Buffer.concat([ + ...entries.map((entry) => + Buffer.isBuffer(entry) ? entry : createTarEntry(entry), + ), + Buffer.alloc(TAR_BLOCK_SIZE * 2), + ]); + + return gzip ? gzipSync(tarBuffer) : tarBuffer; +} + +export function createReleaseEntries( + archiveRoot, + { binaryContent = "#!/bin/sh\necho sce\n" } = {}, +) { + return [ + { name: `${archiveRoot}/`, type: "5", mode: 0o755 }, + { name: `${archiveRoot}/LICENSE`, content: "MIT\n" }, + { name: `${archiveRoot}/README.md`, content: "# sce\n" }, + { name: `${archiveRoot}/bin/`, type: "5", mode: 0o755 }, + { + name: `${archiveRoot}/bin/sce`, + content: binaryContent, + mode: 0o755, + }, + ]; +} From 77a84b6023b88077e12ed340661e2b48fc76996a Mon Sep 17 00:00:00 2001 From: David Abram Date: Tue, 3 Nov 2026 18:02:41 +0100 Subject: [PATCH 5/6] CI+npm: Gate packed releases with offline smoke checks Validate the npm launcher after `npm pack` so release assets cannot be emitted with a non-executable `bin/sce.js` or a broken offline install. Add the packed-tarball smoke as a flake check, include its fixture inputs in the copied source, and run it from `release-npm-package` before writing metadata. Plan: context/plans/harden-npm-installer.md (T04) Co-authored-by: SCE --- context/architecture.md | 4 +- context/context-map.md | 2 +- context/glossary.md | 4 +- context/overview.md | 2 +- context/plans/harden-npm-installer.md | 16 ++++- context/sce/cli-npm-distribution-contract.md | 5 +- flake.nix | 36 ++++++++++- npm/test/pack-smoke.sh | 68 ++++++++++++-------- 8 files changed, 101 insertions(+), 36 deletions(-) diff --git a/context/architecture.md b/context/architecture.md index 3b117a9c1..97ac14da6 100644 --- a/context/architecture.md +++ b/context/architecture.md @@ -40,10 +40,10 @@ The repository includes a new placeholder Rust binary crate at `cli/`. - Cargo/crates.io and npm registry publication belong to separate downstream publish stages that consume already-versioned checked-in package metadata rather than inventing release versions during workflow execution. - Crates.io packaging is self-contained through ignored `cli/package-fallback/`: the preparation script delegates canonical Pkl generation and inventories to the shared producer, moves its validated payload and checksums into a temporary staged fallback, adds static hooks/schemas/migrations, and appends static checksums to the exact combined inventory. `cli/Cargo.toml` includes only that fallback and crate sources, and unpacked-crate `build.rs` validates then copies it into `OUT_DIR` without Pkl. Repository builds use `scripts/run-cli-cargo.sh`, which consumes the same producer before Cargo; `build.rs` requires the resulting `SCE_CLI_GENERATED_INPUT_DIR` handoff and copies it into `OUT_DIR` without invoking Pkl. - npm is a downstream consumer of the shared release artifact contract rather than a separate build owner. -- `flake.nix` now exposes dedicated release-packaging apps for this rollout: `apps.release-artifacts` packages the current-platform `sce` archive/checksum/manifest-fragment set from the release output `packages.sce-release` (`nix build .#sce-release`), prepares/audits the staged `bin/sce` before archive creation (including macOS `libiconv` install-name rewriting plus ad-hoc re-signing when needed), and fails if `.version`, `cli/Cargo.toml`, `npm/package.json`, the built/prepared CLI version, or the native portability audit disagree; `apps.native-portability-audit` audits native binaries for forbidden `/nix/store/` runtime references using Linux ELF/string inspection or macOS `otool -L`; `apps.release-manifest` merges per-platform metadata fragments into release-level manifest/checksum outputs and signs the merged manifest with a non-repo private key provided via env/file input; `apps.release-npm-package` packs the checked-in npm launcher package into a release-ready `sce-v-npm.tgz` tarball plus npm metadata JSON while refusing mismatched checked-in version metadata; and Linux-only `apps.release-flatpak-package` emits the Flatpak source-manifest tarball/checksum/JSON metadata while refusing `.version`/Cargo/npm/AppStream version drift and non-git checkouts that cannot resolve a release commit; Linux-only `apps.release-flatpak-bundle` builds a source-built `.flatpak` bundle from the checkout and emits per-architecture bundle/checksum/JSON metadata. +- `flake.nix` now exposes dedicated release-packaging apps for this rollout: `apps.release-artifacts` packages the current-platform `sce` archive/checksum/manifest-fragment set from the release output `packages.sce-release` (`nix build .#sce-release`), prepares/audits the staged `bin/sce` before archive creation (including macOS `libiconv` install-name rewriting plus ad-hoc re-signing when needed), and fails if `.version`, `cli/Cargo.toml`, `npm/package.json`, the built/prepared CLI version, or the native portability audit disagree; `apps.native-portability-audit` audits native binaries for forbidden `/nix/store/` runtime references using Linux ELF/string inspection or macOS `otool -L`; `apps.release-manifest` merges per-platform metadata fragments into release-level manifest/checksum outputs and signs the merged manifest with a non-repo private key provided via env/file input; `apps.release-npm-package` packs the checked-in npm launcher package into a release-ready `sce-v-npm.tgz` tarball plus npm metadata JSON while refusing mismatched checked-in version metadata and refusing to emit a tarball that fails the packed-package smoke (`npm/test/pack-smoke.sh --tarball`); and Linux-only `apps.release-flatpak-package` emits the Flatpak source-manifest tarball/checksum/JSON metadata while refusing `.version`/Cargo/npm/AppStream version drift and non-git checkouts that cannot resolve a release commit; Linux-only `apps.release-flatpak-bundle` builds a source-built `.flatpak` bundle from the checkout and emits per-architecture bundle/checksum/JSON metadata. - CLI release automation: `.github/workflows/release-sce.yml` orchestrates `sce` GitHub release asset assembly, validates `.version`/Cargo/npm parity before tagging or releasing, calls three reusable per-platform workflow files (`release-sce-linux.yml`, `release-sce-linux-arm.yml`, `release-sce-macos-arm.yml`), assembles the signed native release manifest, builds npm and Flatpak source-manifest release packages, and uploads the CLI/npm/Flatpak asset set to the GitHub Release. Each native reusable workflow validates its generated archive before native artifact upload by extracting the target archive, smoke-running `bin/sce version --format json`, and running `nix run .#native-portability-audit` with the lane platform. Manual `workflow_dispatch` runs can set the GitHub Release-level `prerelease` flag through an explicit checkbox; tag-triggered releases keep the flag false and no tag-name prerelease inference is performed. - Downstream registry publication is now implemented in dedicated workflows: `.github/workflows/publish-crates.yml` publishes the checked-in crate version from a published release or manual dispatch after `.version`/tag/Cargo parity checks and requires semver prerelease metadata when a publish run is marked prerelease; `.github/workflows/publish-npm.yml` publishes the checked-in npm package version from a published release or manual dispatch after `.version`/tag/npm parity checks plus verification of the canonical `sce-v-npm.tgz` GitHub release asset, using the `next` npm dist-tag instead of `latest` for prerelease versions or prerelease-marked runs. -- The npm distribution implementation lives under `npm/`: `package.json` defines the `sce` package surface, `bin/sce.js` launches the package-local native binary, `lib/install.js` resolves the current package version against the release manifest, verifies `sce-v-release-manifest.json.sig` with the bundled public key before trusting manifest contents, validates manifest identity, and then installs the checksum-verified native archive for supported macOS/Linux targets through `lib/archive.js` (bounded, entry-validating gzip+tar reader plus atomic `runtime/sce` replacement, no host `tar`), while `test/platform.test.js`, `test/install.test.js`, and `test/archive.test.js` (with the `test/tar-fixture.js` archive builder) cover platform selection, signed-manifest installer behavior, and archive/atomic-install hardening. +- The npm distribution implementation lives under `npm/`: `package.json` defines the `sce` package surface, `bin/sce.js` launches the package-local native binary, `lib/install.js` resolves the current package version against the release manifest, verifies `sce-v-release-manifest.json.sig` with the bundled public key before trusting manifest contents, validates manifest identity, and then installs the checksum-verified native archive for supported macOS/Linux targets through `lib/archive.js` (bounded, entry-validating gzip+tar reader plus atomic `runtime/sce` replacement, no host `tar`), while `test/platform.test.js`, `test/install.test.js`, and `test/archive.test.js` (with the `test/tar-fixture.js` archive builder) cover platform selection, signed-manifest installer behavior, and archive/atomic-install hardening, and `test/pack-smoke.sh` (flake check `npm-pack-smoke`) installs the real packed tarball offline against a loopback fixture release. - `cli/src/main.rs` is the executable entrypoint (`sce`) and delegates to `app::run`. - `cli/src/cli_schema.rs` defines the clap-based CLI schema using derive macros for all top-level commands and subcommands, including the top-level `sync` command, and renders command-local help text for the `auth` command tree (`auth`, `auth login`, `auth logout`, `auth whoami`). diff --git a/context/context-map.md b/context/context-map.md index 43d95b16f..64b24fa62 100644 --- a/context/context-map.md +++ b/context/context-map.md @@ -113,7 +113,7 @@ Feature/domain context: - `context/sce/opencode-agent-trace-plugin-runtime.md` (current OpenCode agent-trace plugin runtime behavior, including captured `message.updated` handoff with `summary.diffs` branching: when diffs exist sends one `-patch` mixed batch containing a synthetic parent message plus per-diff `message.part` patch items, when no diffs sends the original `message.updated` payload; in-memory dedup `Set` keyed by `"${sessionID}:${messageID}"`; captured `message.part.updated` handoff to `sce hooks conversation-trace` for `text`/`reasoning` parts with non-empty text plus completed `question` tool parts emitted as `part_type: "question"` with JSON-stringified `{ question, answer }[]`; existing user-message diff extraction for `{ sessionID, diff, time, model_id }`; session-scoped OpenCode client version capture from `session.created`/`session.updated`; and CLI handoff to `sce hooks diff-trace` over STDIN JSON with required `tool_name="opencode"` plus required nullable `tool_version`; Rust hook parsing and AgentTraceDb insertion persist `oc_`-prefixed session IDs plus required payload fields including `model_id`) - `context/sce/cli-first-install-channels-contract.md` (current Nix/Cargo/npm/source-built Flatpak channel contract, release authority and workflow topology, Nix-owned Flatpak manifest/cargo-source generation and validation, reduced Flatpak app surface, and host-git bridge decision) - `context/sce/cli-release-artifact-contract.md` (shared `sce` binary release artifact naming, checksum/manifest outputs, pre-archive staged-binary preparation including macOS `libiconv` install-name sanitization/ad-hoc re-signing, native portability audit app/check for forbidden `/nix/store/` runtime references, GitHub Releases as the canonical artifact publication surface, manual dispatch `prerelease` flag behavior, the current three-target Linux/macOS release workflow topology including pre-upload extracted-archive smoke/audit validation in each native lane, implemented Flatpak source-manifest and source-built `.flatpak` bundle package assets uploaded by `.github/workflows/release-sce.yml`, and Flatpak's explicit source-built non-binary exception) -- `context/sce/cli-npm-distribution-contract.md` (implemented `sce` npm launcher package, release-manifest/checksum-verified native binary install flow with manifest identity checks, bounded validating archive extraction, and atomic `runtime/sce` install, the supported darwin/arm64 plus linux x64+arm64 npm platform matrix, the executable launcher-mode requirement and offline packed-package smoke test (`npm/test/pack-smoke.sh`), the prerequisite that npm-consumed native archives are portable and free of forbidden `/nix/store/` runtime references before upload, and dedicated `.github/workflows/publish-npm.yml` downstream npm publish-stage contract including prerelease publication via npm `next` dist-tag instead of `latest`) +- `context/sce/cli-npm-distribution-contract.md` (implemented `sce` npm launcher package, release-manifest/checksum-verified native binary install flow with manifest identity checks, bounded validating archive extraction, and atomic `runtime/sce` install, the supported darwin/arm64 plus linux x64+arm64 npm platform matrix, the executable launcher-mode requirement and offline packed-package smoke test (`npm/test/pack-smoke.sh`, flake check `npm-pack-smoke`, and the `release-npm-package` post-pack gate), the prerequisite that npm-consumed native archives are portable and free of forbidden `/nix/store/` runtime references before upload, and dedicated `.github/workflows/publish-npm.yml` downstream npm publish-stage contract including prerelease publication via npm `next` dist-tag instead of `latest`) - `context/sce/cli-cargo-distribution-contract.md` (implemented `sce` Cargo publication posture plus supported crates.io and wrapper-driven local-checkout install guidance, explicit unsupported `cargo install --git` boundary, dedicated crates.io publish workflow with semver prerelease guard, canonical generated-input declaration/producer ownership consumed by the Cargo wrapper, generated-output check, package-fallback preparation, and Nix pre-Cargo derivation, plus the deterministic combined checksummed fallback used for Pkl-free downstream crate builds) - `context/sce/flake-build-performance.md` (current native-vs-release package split, release-only commit embedding, shared-producer-backed pre-Cargo generated-input derivation consumed by native/release/test/Clippy but excluded from dependency/format derivations, slim default/database devShell contract, normal-vs-expensive CI tiers, durable before/after benchmark summary, validation evidence, and remaining cold-build trade-offs) - `context/sce/config-schema-publication.md` (current embedded-only publication contract for the generated SCE config JSON Schema and the canonical SCE web application URL used in config declarations) diff --git a/context/glossary.md b/context/glossary.md index d2a53e81a..1f216848f 100644 --- a/context/glossary.md +++ b/context/glossary.md @@ -48,7 +48,7 @@ - `generated OpenCode plugin registration contract`: Current generated-config contract where `config/.opencode/opencode.json` serializes the OpenCode `plugin` field from canonical Pkl sources for SCE-managed plugins only; the current registered paths, in order, are `./plugins/sce-bash-policy.ts`, `./plugins/sce-agent-trace.ts`, and `./plugins/sce-mutation-scope.ts`. `sce-mutation-scope` must stay last; the OpenCode config merge appends the generated SCE entries after arbitrary user plugins and `sce doctor` (`inspect_opencode_plugin_ordering_health`) flags a non-last position. Claude does not use an OpenCode-style plugin manifest; Claude bash-policy enforcement is registered through generated `.claude/settings.json` as a `PreToolUse` `Bash` command hook running `sce policy bash`. - `root Biome contract`: Repository-root formatting/linting contract owned by `biome.json`, currently scoped only to `npm/**` and the shared `config/lib/**` plugin package root with package-local `node_modules/**` excluded; the canonical execution path is the root Nix dev shell (`nix develop -c biome ...`). - `cli flake checks`: Check derivations in root `flake.nix` (`checks..cli-tests`, `cli-clippy`, `cli-fmt`), plus `pkl-generated`, `workflow-actionlint`, split npm/config-lib JS checks, and Linux-only Flatpak checks; invoked via `nix flake check` at repo root. -- `npm JS flake checks`: The current `npm/` validation slice exposed by root `flake.nix`: `npm-bun-tests` runs only `bun test ./test/*.test.js`, `npm-biome-check` runs only Biome lint/check with formatter verification disabled, and `npm-biome-format` runs only Biome format verification with linter checks disabled. +- `npm JS flake checks`: The current `npm/` validation slice exposed by root `flake.nix`: `npm-bun-tests` runs only `bun test ./test/*.test.js`, `npm-biome-check` runs only Biome lint/check with formatter verification disabled, `npm-biome-format` runs only Biome format verification with linter checks disabled, and `npm-pack-smoke` runs `npm/test/pack-smoke.sh` (offline packed-package install against a loopback fixture release) in the build sandbox. - `config-lib JS flake checks`: The current shared `config/lib/` validation slice exposed by root `flake.nix`: `config-lib-bun-tests` runs Bun-discovered tests from the copied shared `config/lib/` package source (including bash-policy plugin wrapper tests and tracked agent-trace plugin tests), with dependencies resolved from `config/lib/package.json` and `config/lib/bun.lock`, while `config-lib-biome-check` and `config-lib-biome-format` run Biome lint/check and format verification over the copied shared package source with formatter/linter halves disabled respectively. - `config-lib repo-shaped test source`: Root-flake source-layout contract where `config-lib-bun-tests`, `config-lib-biome-check`, and `config-lib-biome-format` run from `config/lib/` while their copied Nix source preserves repo-relative shared fixtures, currently `cli/src/services/structured_patch/fixtures` for Claude agent-trace golden tests (fully Rust-owned; the Claude TypeScript Bun test was removed in T07). - `config-lib shared package root`: Shared Bun/TypeScript package root at `config/lib/` for repository-owned OpenCode plugin support code. It owns `package.json`, `bun.lock`, and `tsconfig.json`, pins `@opencode-ai/plugin` to `1.15.4`, includes `agent-trace-plugin/**/*.ts`, `bash-policy-plugin/**/*.ts`, `mutation-scope-plugin/**/*.ts`, and `pi-plugin/**/*.ts` in strict TypeScript coverage, and excludes package-local `node_modules/` from both TypeScript and root Biome coverage. @@ -212,7 +212,7 @@ - `sce npm launcher package`: Checked-in npm package under `npm/` named `@crocoder-dev/sce`; its `postinstall` flow downloads `sce-v-release-manifest.json` plus detached signature `sce-v-release-manifest.json.sig`, verifies that manifest with the bundled public key, then uses trusted `checksum_sha256` metadata to verify the matching GitHub release archive before installing the native binary into package-local runtime storage. The current supported npm install matrix is `darwin/arm64`, `linux/x64`, and `linux/arm64`, mapping to the current three release target triples. npm consumes portable native GitHub Release archives as-is; release packaging/workflow validation must remove or reject forbidden `/nix/store/` runtime references before upload. npm registry publication is a separate non-bumping downstream stage from GitHub release packaging. - `publish-npm workflow`: Dedicated npm registry publication workflow at `.github/workflows/publish-npm.yml` that triggers from a published GitHub release or manual dispatch, validates `.version`/`npm/package.json`/release-tag parity, downloads the canonical `sce-v-npm.tgz` GitHub release asset, verifies the tarball's embedded package metadata, and publishes that already-versioned package to npm. - `sce release authority contract`: Approved release topology where repo-root `.version` is the canonical checked-in release version source, GitHub Releases are the canonical publication surface for signed release artifacts, and Cargo/npm registry publication are separate downstream publish stages that consume already-versioned checked-in package metadata without workflow-side version bumping. -- `sce release-npm-package app`: Root-flake app exposed as `nix run .#release-npm-package`; stages the checked-in npm package, rewrites the requested version, runs `npm pack`, and emits `sce-v-npm.tgz` plus `sce-v-npm.json` for release publication. +- `sce release-npm-package app`: Root-flake app exposed as `nix run .#release-npm-package`; stages the checked-in npm package without rewriting its version (the requested version must already match `.version`, `cli/Cargo.toml`, and `npm/package.json`), runs `npm pack`, and emits `sce-v-npm.tgz` plus `sce-v-npm.json` for release publication only after the produced tarball passes `npm/test/pack-smoke.sh --tarball` (executable `package/bin/sce.js` plus a smoke install). - `sce release-flatpak-package app`: Linux-only root-flake app exposed as `nix run .#release-flatpak-package -- --version --out-dir `; runs the Nix-built `flatpak-version-parity-check` script (parity across `.version`, `cli/Cargo.toml`, `npm/package.json`, and Flatpak AppStream release metadata), requires a resolvable git release commit, stages `packaging/flatpak/` manifest/support files without mutating checked-in sources, uses the Nix manifest expression's commit-pinned flavor to emit the staged manifest, and produces deterministic Flatpak source-manifest tarball/checksum/JSON metadata. - `sce release-flatpak-bundle app`: Linux-only root-flake app exposed as `nix run .#release-flatpak-bundle -- --version --arch --out-dir `; runs the Nix-built `flatpak-version-parity-check` script, uses the Nix manifest expression's local-checkout-override flavor to produce a Flatpak `type: dir` manifest, runs imperative `flatpak-builder --force-clean --state-dir=/state --arch=` (state dir kept in the same temp root as the build dir so both share a filesystem) plus `flatpak build-bundle`, and produces SHA-256 checksum and JSON metadata (`asset_type: flatpak-bundle`); used by `.github/workflows/release-sce-linux.yml` (x86_64) and `.github/workflows/release-sce-linux-arm.yml` (aarch64). - `sce split platform release workflows`: CLI release automation topology where `.github/workflows/release-sce.yml` orchestrates reusable per-platform workflow files; the current reusable workflow set and active orchestrated release matrix are `release-sce-linux.yml`, `release-sce-linux-arm.yml`, and `release-sce-macos-arm.yml`, producing the current automated release target set `x86_64-unknown-linux-musl`, `aarch64-unknown-linux-musl`, and `aarch64-apple-darwin`. Each native reusable workflow validates the generated archive before native artifact upload by extracting it, smoke-running `bin/sce version --format json`, and invoking the native portability audit for the lane platform. diff --git a/context/overview.md b/context/overview.md index 3f4c0f691..77dd982bc 100644 --- a/context/overview.md +++ b/context/overview.md @@ -94,7 +94,7 @@ The root flake splits native and release outputs:`packages.sce`and`packages.defa Git-commit embedding is **release-only**: `SCE_GIT_COMMIT`is injected via a`releaseCommitArgs` fragment applied only to the release derivations (`scePackageMusl`on Linux,`sceReleasePackageNative`on Darwin), not to`commonCargoArgs`. So native `.#sce`/`.#default`and every`nix flake check` derivation (`cli-tests`, `cli-clippy`, `cli-fmt`) build without the commit in their inputs and stay cache-reusable across commits (native `sce version`reports`unknown`), while `.#sce-release`still reports the real commit via`sce version`. `cli/build.rs` `emit_git_commit`emits`SCE_GIT_COMMIT`only when the env var is explicitly set — no`git rev-parse`fallback and no`.git/HEAD`/`.git/packed-refs`rerun watches. On Darwin the release now uses a distinct native-toolchain derivation (native toolchain + commit), so it diverges from`.#sce`to carry the commit while native stays commit-independent. The default development shell is slimmed for fast iteration:`devShells.default`no longer includes`scePackage`or`tursoPackage`, so `nix develop`compiles neither the CLI package nor the Turso CLI — it provides only the Rust toolchain and JS/pkl tooling for`cargo`/`biome`/`pkl`work. Turso stays available as`packages..turso`and through a new opt-in`devShells..database`shell (default tools +`tursoPackage`), entered via `nix develop .#database`. Both shells share `defaultDevShellPackages`/`defaultDevShellHook` `let`bindings so they cannot drift. The CLI Cargo package metadata now includes crates.io publication-ready fields with crate-local install guidance in`cli/README.md`; supported Cargo install paths are `cargo install shared-context-engineering --locked`and local checkout installation through`./scripts/run-cli-cargo.sh install --path cli --locked`. Direct `cargo install --git`is unsupported because it cannot run the repository pre-Cargo producer. The published crate installs the`sce`binary. The crate also keeps`cargo clippy --manifest-path cli/Cargo.toml`warnings-denied through`cli/Cargo.toml`lint configuration, so an extra`-- -D warnings`flag is redundant. -The repository-root flake is the single Nix entrypoint for repo tooling and CLI packaging/checks, so root-level`nix flake check` evaluates the Crane-backed CLI checks (`cli-tests`, `cli-clippy`, `cli-fmt`), the ephemeral `pkl-generated`inventory check, the generated Codex `codex-hook-command` invocation check, Linux-only Flatpak checks,`workflow-actionlint`, and the split npm/config-lib JavaScript checks without nested-flake indirection. Repository Cargo builds copy a validated pre-Cargo generated payload into Cargo `OUT_DIR`; crates.io packaging prepares a self-contained Pkl-free fallback in a temporary clean workspace, and Flatpak helpers prepare the same payload beside generated manifests before the sandboxed source build. No general-purpose `cli/assets/generated/`mirror or committed generated target tree remains. +The repository-root flake is the single Nix entrypoint for repo tooling and CLI packaging/checks, so root-level`nix flake check` evaluates the Crane-backed CLI checks (`cli-tests`, `cli-clippy`, `cli-fmt`), the ephemeral `pkl-generated`inventory check, the generated Codex `codex-hook-command` invocation check, Linux-only Flatpak checks,`workflow-actionlint`, and the split npm/config-lib JavaScript checks plus the offline npm packed-package smoke (`npm-pack-smoke`) without nested-flake indirection. Repository Cargo builds copy a validated pre-Cargo generated payload into Cargo `OUT_DIR`; crates.io packaging prepares a self-contained Pkl-free fallback in a temporary clean workspace, and Flatpak helpers prepare the same payload beside generated manifests before the sandboxed source build. No general-purpose `cli/assets/generated/`mirror or committed generated target tree remains. Config-lib JS flake checks execute from`config/lib/`, but the copied Nix check source is repo-shaped when tests require shared repo fixtures; the current Claude agent-trace golden tests are fully Rust-owned in `cli/src/services/structured_patch/fixtures`(Claude TypeScript plugin test removed in T07). Local developer Nix tuning guidance now lives in`AGENTS.md`, including optional user-level `~/.config/nix/nix.conf`recommendations for`max-jobs`and`cores`plus an explicit system-level-only note for`auto-optimise-store`. The Pkl authoring layer owns generated OpenCode plugin registration for SCE-managed plugins: `config/pkl/base/opencode.pkl`defines the canonical plugin entries,`config/pkl/renderers/common.pkl`re-exports the shared plugin list for renderer use, and generated`config/.opencode/opencode.json`registers`./plugins/sce-bash-policy.ts`and`./plugins/sce-agent-trace.ts`through OpenCode's`plugin`field. Claude does not use an OpenCode-style plugin manifest; Claude bash-policy enforcement is registered through generated`.claude/settings.json`as a`PreToolUse` `Bash`command hook routed through`.claude/hooks/run-sce-or-show-install-guidance.sh`before running`sce policy bash`. diff --git a/context/plans/harden-npm-installer.md b/context/plans/harden-npm-installer.md index d6d3a5be1..f1ec14ea3 100644 --- a/context/plans/harden-npm-installer.md +++ b/context/plans/harden-npm-installer.md @@ -127,13 +127,25 @@ Current-state facts this plan starts from: - Context impact: localized to the npm installer extraction/install surface; affects `context/sce/cli-npm-distribution-contract.md` (safe extraction rules, size limits, atomic install, removed host `tar` runtime requirement). No architecture or glossary changes. - Context synchronization: synced -- [ ] T04: `Wire packed-package smoke into flake checks and release-npm-package` (status:todo) +- [x] T04: `Wire packed-package smoke into flake checks and release-npm-package` (status:done) - Task ID: T04 - Scope: In — add new `npm/` files to `npmSrc`; add a `npm-pack-smoke` flake check running the T01 smoke script in the sandbox (nodejs, openssl); make `release-npm-package` assert the packed tarball's `package/bin/sce.js` is executable and run the smoke against the produced tarball before reporting success; confirm `release-sce.yml` already invokes `release-npm-package` (edit only if needed). Out — publish workflow behavior changes, real network release testing. - Dependencies: T03 - Done when: `nix flake check` includes and passes the new check; `release-npm-package` fails on a `0644` launcher in a scratch copy and passes on the real tree. - Verify: `nix build .#checks.x86_64-linux.npm-pack-smoke --print-build-logs`; `nix run .#release-npm-package -- --version "$(cat .version)" --out-dir `; `nix run nixpkgs#actionlint` if a workflow is touched. - - Context synchronization: pending + - Completed: 2026-11-03 + - Files changed: `flake.nix`, `npm/test/pack-smoke.sh`. + - Result: `npmSrc` now includes `npm/test/fixture-release-server.mjs` and `npm/test/pack-smoke.sh`. New flake check `npm-pack-smoke` (`mkCopiedSourceCheck` over `npmSrc`, inputs bash/coreutils/gnutar/gzip/nodejs/openssl, `HOME` set inside the build dir) runs `bash ./test/pack-smoke.sh` in the sandbox and is registered in `checks`. `pack-smoke.sh` gained `--tarball `: it checks the launcher mode in that packed tarball, extracts its `package/` contents into the stage directory, and then runs the usual key-swap/repack/install/run flow. When `/usr/bin/env` is missing (Nix sandbox), the installed bin is run as `node ` after the existing `-x` check; on hosts it is still exec'd directly. `release-npm-package` adds bash/gnutar/gzip/openssl to its runtime inputs, runs `bash npm/test/pack-smoke.sh --tarball "$out_dir/$package_name"` after packing, and on failure deletes the tarball and exits 1 before writing metadata. `.github/workflows/release-sce.yml` already runs `nix run .#release-npm-package` in the "Build npm release package" step, before release creation/upload, so it was not edited. Per a user instruction during implementation, the descriptive comments in `pack-smoke.sh` were removed (only the shebangs and the functional `shellcheck disable` directive remain). + - Verify outcomes: + - `nix build .#checks.x86_64-linux.npm-pack-smoke --print-build-logs` -> exit 0; `packed package/bin/sce.js mode -rwxr-xr-x`, `pack-smoke: OK (sce --help, sce version)` (loopback HTTPS works inside the sandbox, which answers the plan's first open question). + - `nix run .#release-npm-package -- --version "$(cat .version)" --out-dir ` on the real tree -> exit 0; smoke OK; tarball and metadata written. + - Same app run from a scratch copy of the tracked tree with `npm/bin/sce.js` set to `0644` -> exit 1, `pack-smoke: Packed package/bin/sce.js has mode -rw-r--r--; expected an executable mode ...`, tarball removed, output dir left empty. + - `nix flake check` -> `all checks passed!`; `checks.x86_64-linux` includes `npm-pack-smoke`. + - `nix shell nixpkgs#shellcheck -c shellcheck npm/test/pack-smoke.sh` -> clean; `nix develop -c biome check npm` -> no errors (one pre-existing `getRepositorySlug` warning, untouched). + - actionlint not run: no workflow was touched. + - Deviations: the sandbox has no `/usr/bin/env`, so the launcher's `#!/usr/bin/env node` shebang cannot exec there. In that case the smoke runs the installed bin through `node` and relies on the `-x` check plus the packed-mode assertion. The direct-exec path is still exercised by `release-npm-package` and by host runs. + - Context impact: affects verification and release surfaces. `context/sce/cli-npm-distribution-contract.md` (new `npm-pack-smoke` check, `--tarball` smoke mode, `release-npm-package` post-pack gate), `context/glossary.md` `npm JS flake checks` entry, and any flake-check enumeration in `context/overview.md`; possibly `context/sce/cli-release-artifact-contract.md` if it describes `release-npm-package` validation. + - Context synchronization: synced - [ ] T05: `Correct npm README targets and update distribution contract` (status:todo) - Task ID: T05 diff --git a/context/sce/cli-npm-distribution-contract.md b/context/sce/cli-npm-distribution-contract.md index 9a7343170..64f810e62 100644 --- a/context/sce/cli-npm-distribution-contract.md +++ b/context/sce/cli-npm-distribution-contract.md @@ -37,6 +37,7 @@ This file captures the implemented npm distribution slice of the `sce` CLI insta - `sce-v-npm.tgz` - `sce-v-npm.json` - The helper also refuses to proceed when checked-in `.version`, `cli/Cargo.toml`, and `npm/package.json` are not already aligned. +- After packing, the helper runs `npm/test/pack-smoke.sh --tarball ` against the tarball it just produced. If the packed `package/bin/sce.js` is not executable or the smoke install fails, it deletes the tarball and exits non-zero before writing metadata, so `.github/workflows/release-sce.yml` (which runs the helper before creating the GitHub Release) never uploads an unverified npm package. - `.github/workflows/release-sce.yml` publishes those npm assets alongside the canonical CLI archives, `sce-v-release-manifest.json`, and `sce-v-release-manifest.json.sig`. - Native archive portability is a release-artifact prerequisite for every npm-supported target triple. Archives selected through the signed release manifest must have passed macOS/Linux native portability audit and extracted-binary `sce version --format json` smoke validation before upload. @@ -57,14 +58,16 @@ This file captures the implemented npm distribution slice of the `sce` CLI insta ## Packed-package smoke test -- `npm/test/pack-smoke.sh` (with helper `npm/test/fixture-release-server.mjs`) verifies the real packed artifact end to end without network access: it stages a copy of the package (or `--package-dir `), swaps a throwaway RSA manifest-signing public key into the staged copy only, builds a stub `sce` archive in the canonical `sce-v-/{bin/sce,LICENSE,README.md}` layout with a matching signed manifest, serves it over loopback HTTPS (self-signed certificate trusted via `NODE_EXTRA_CA_CERTS`, selected via `SCE_NPM_RELEASE_BASE_URL`), runs `npm pack`, fails unless the packed `package/bin/sce.js` entry is executable, installs into an isolated `--prefix` with an isolated npm cache/config, and asserts exact output from `sce --help` and `sce version`. +- `npm/test/pack-smoke.sh` (with helper `npm/test/fixture-release-server.mjs`) verifies the real packed artifact end to end without network access: it stages a copy of the package (or `--package-dir `, or the `package/` contents of an already packed `--tarball `, whose launcher mode is asserted first), swaps a throwaway RSA manifest-signing public key into the staged copy only, builds a stub `sce` archive in the canonical `sce-v-/{bin/sce,LICENSE,README.md}` layout with a matching signed manifest, serves it over loopback HTTPS (self-signed certificate trusted via `NODE_EXTRA_CA_CERTS`, selected via `SCE_NPM_RELEASE_BASE_URL`), runs `npm pack`, fails unless the packed `package/bin/sce.js` entry is executable, installs into an isolated `--prefix` with an isolated npm cache/config, and asserts exact output from `sce --help` and `sce version`. - The fixture archive is produced by real GNU `tar`, so the smoke also exercises the in-repo archive reader against the release pipeline's archive format under Node. - Only the host platform is exercised, with a stub binary; the production public key and manifest trust code are not modified. +- Root `flake.nix` runs the smoke in the build sandbox as `checks..npm-pack-smoke` (part of `nix flake check`; loopback HTTPS works inside the sandbox). The sandbox has no `/usr/bin/env`, so there the installed bin is run as `node ` after its `-x` check. Host runs, including the `release-npm-package` gate, exec the bin directly. ## Verification baseline - `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'` - `nix shell nixpkgs#nodejs nixpkgs#openssl -c bash npm/test/pack-smoke.sh` +- `nix build .#checks..npm-pack-smoke --print-build-logs` - `nix develop -c sh -c 'cd npm && SCE_NPM_SKIP_DOWNLOAD=1 bun ./lib/install.js'` - `nix run .#release-manifest -- --version --artifacts-dir --out-dir --signing-key-file ` - `nix run .#release-npm-package -- --version --out-dir ` diff --git a/flake.nix b/flake.nix index e92e7add0..431df1b53 100644 --- a/flake.nix +++ b/flake.nix @@ -311,7 +311,9 @@ ./npm/lib/platform.js ./npm/lib/release-manifest-public-key.pem ./npm/test/archive.test.js + ./npm/test/fixture-release-server.mjs ./npm/test/install.test.js + ./npm/test/pack-smoke.sh ./npm/test/platform.test.js ./npm/test/tar-fixture.js ./scripts/lib/release-manifest-signing.mjs @@ -1155,9 +1157,13 @@ releaseNpmPackageApp = pkgs.writeShellApplication { name = "release-npm-package"; runtimeInputs = [ + pkgs.bash pkgs.coreutils + pkgs.gnutar + pkgs.gzip pkgs.jq pkgs.nodejs + pkgs.openssl ]; text = '' set -euo pipefail @@ -1168,7 +1174,9 @@ Builds the canonical npm package tarball for `sce` using the checked-in npm launcher package and writes the packed tarball plus package metadata into the - output directory. + output directory. The packed tarball must carry an executable bin/sce.js and + pass the packed-package smoke install (npm/test/pack-smoke.sh) before any + asset is reported. EOF } @@ -1244,6 +1252,12 @@ mv "$packed_name" "$out_dir/$package_name" + if ! bash npm/test/pack-smoke.sh --tarball "$out_dir/$package_name"; then + rm -f "$out_dir/$package_name" + printf 'Packed npm package failed the packed-package smoke verification; removed %s\n' "$out_dir/$package_name" >&2 + exit 1 + fi + npm_pkg_name="$(${pkgs.nodejs}/bin/node -p "JSON.parse(require('fs').readFileSync('npm/package.json', 'utf8')).name")" jq \ @@ -1517,6 +1531,25 @@ testCommand = "bun test ./test/*.test.js"; }; + npmPackSmoke = mkCopiedSourceCheck { + name = "npm-pack-smoke"; + src = npmSrc; + workdir = "npm"; + nativeBuildInputs = [ + pkgs.bash + pkgs.coreutils + pkgs.gnutar + pkgs.gzip + pkgs.nodejs + pkgs.openssl + ]; + beforeCheck = '' + export HOME="$TMPDIR/home" + mkdir -p "$HOME" + ''; + checkCommand = "bash ./test/pack-smoke.sh"; + }; + npmBiomeCheck = mkBiomeCheck { name = "npm-biome-check"; src = npmSrc; @@ -1750,6 +1783,7 @@ codex-hook-command = codexHookCommandCheck; npm-bun-tests = npmTests; + npm-pack-smoke = npmPackSmoke; npm-biome-check = npmBiomeCheck; npm-biome-format = npmBiomeFormat; diff --git a/npm/test/pack-smoke.sh b/npm/test/pack-smoke.sh index 7f236bcd4..573155b1b 100755 --- a/npm/test/pack-smoke.sh +++ b/npm/test/pack-smoke.sh @@ -1,27 +1,22 @@ #!/usr/bin/env bash set -euo pipefail -# Packed-package smoke test for the @crocoder-dev/sce npm launcher. -# -# Packs a staged copy of the npm package, serves a locally generated fixture -# release (stub `sce` binary, manifest signed with a throwaway key swapped into -# the staged copy) over loopback HTTPS, installs the tarball into an isolated -# prefix, and runs the installed `sce` bin. No network access is required. -# -# Requires: bash, coreutils, tar, gzip, node/npm (>=18), openssl. - SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PACKAGE_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" LAUNCHER_ENTRY="package/bin/sce.js" SERVER_START_TIMEOUT_SECONDS=10 EXPECTED_HELP_OUTPUT="sce fixture help" +SOURCE_TARBALL="" usage() { cat <<'EOF' -Usage: pack-smoke.sh [--package-dir ] +Usage: pack-smoke.sh [--package-dir | --tarball ] Packs the npm launcher package, installs it into an isolated prefix against a loopback fixture release, and verifies `sce --help` and `sce version`. + +With --tarball, the launcher mode is asserted on the given packed tarball and +its package contents are used instead of a package directory. EOF } @@ -37,6 +32,12 @@ while [[ $# -gt 0 ]]; do PACKAGE_DIR="$(cd "$2" && pwd)" shift 2 ;; + --tarball) + [[ -n "${2:-}" ]] || fail "Missing value for --tarball." + [[ -f "$2" ]] || fail "Tarball $2 does not exist." + SOURCE_TARBALL="$(cd "$(dirname "$2")" && pwd)/$(basename "$2")" + shift 2 + ;; --help | -h) usage exit 0 @@ -53,7 +54,9 @@ for tool in node npm openssl tar gzip sha256sum; do fail "Required tool '$tool' is not on PATH. Run via: nix shell nixpkgs#nodejs nixpkgs#openssl -c bash $0" done -[[ -f "$PACKAGE_DIR/package.json" ]] || fail "No package.json found in $PACKAGE_DIR." +if [[ -z "$SOURCE_TARBALL" ]]; then + [[ -f "$PACKAGE_DIR/package.json" ]] || fail "No package.json found in $PACKAGE_DIR." +fi work_dir="$(mktemp -d "${TMPDIR:-/tmp}/sce-npm-pack-smoke.XXXXXX")" server_pid="" @@ -73,7 +76,25 @@ prefix_dir="$work_dir/prefix" keys_dir="$work_dir/keys" mkdir -p "$stage_dir" "$release_dir" "$pack_dir" "$prefix_dir" "$keys_dir" -cp -R "$PACKAGE_DIR/." "$stage_dir/" +assert_launcher_mode() { + local tarball="$1" + local launcher_mode + launcher_mode="$(tar -tvzf "$tarball" "$LAUNCHER_ENTRY" | cut -d ' ' -f 1)" || + fail "Packed tarball $tarball is missing $LAUNCHER_ENTRY." + [[ -n "$launcher_mode" ]] || fail "Packed tarball $tarball is missing $LAUNCHER_ENTRY." + if [[ "${launcher_mode:3:1}" != "x" ]]; then + fail "Packed $LAUNCHER_ENTRY has mode $launcher_mode; expected an executable mode (e.g. -rwxr-xr-x). Ensure npm/bin/sce.js is committed as Git mode 100755." + fi + printf 'pack-smoke: packed %s mode %s\n' "$LAUNCHER_ENTRY" "$launcher_mode" +} + +if [[ -n "$SOURCE_TARBALL" ]]; then + assert_launcher_mode "$SOURCE_TARBALL" + tar -xzf "$SOURCE_TARBALL" -C "$stage_dir" --strip-components=1 --no-same-owner + [[ -f "$stage_dir/package.json" ]] || fail "Tarball $SOURCE_TARBALL has no package/package.json." +else + cp -R "$PACKAGE_DIR/." "$stage_dir/" +fi rm -rf "$stage_dir/runtime" "$stage_dir/node_modules" version="$(node -p "JSON.parse(require('fs').readFileSync(process.argv[1], 'utf8')).version" "$stage_dir/package.json")" @@ -91,17 +112,14 @@ target_arch="$(node -p "JSON.parse(process.argv[1]).arch" "$platform_json")" printf 'pack-smoke: version %s, target %s\n' "$version" "$target_triple" -# Throwaway manifest signing key; only the staged copy trusts it. openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out "$keys_dir/signing.pem" 2>/dev/null openssl pkey -in "$keys_dir/signing.pem" -pubout -out "$stage_dir/lib/release-manifest-public-key.pem" -# Loopback TLS certificate trusted through NODE_EXTRA_CA_CERTS. openssl req -x509 -newkey rsa:2048 -nodes -days 1 \ -subj "/CN=127.0.0.1" \ -addext "subjectAltName=IP:127.0.0.1" \ -keyout "$keys_dir/tls-key.pem" -out "$keys_dir/tls-cert.pem" 2>/dev/null -# Fixture release in the canonical archive layout. archive_root="sce-v${version}-${target_triple}" archive_name="${archive_root}.tar.gz" manifest_name="sce-v${version}-release-manifest.json" @@ -145,17 +163,10 @@ node -e ' openssl dgst -sha256 -sign "$keys_dir/signing.pem" "$release_dir/$manifest_name" | openssl base64 -A >"$release_dir/$manifest_name.sig" -# Pack and verify the launcher entry mode inside the tarball. packed_name="$(cd "$pack_dir" && npm pack --silent "$stage_dir")" tarball="$pack_dir/$packed_name" -launcher_mode="$(tar -tvzf "$tarball" "$LAUNCHER_ENTRY" | cut -d ' ' -f 1)" -[[ -n "$launcher_mode" ]] || fail "Packed tarball is missing $LAUNCHER_ENTRY." -if [[ "${launcher_mode:3:1}" != "x" ]]; then - fail "Packed $LAUNCHER_ENTRY has mode $launcher_mode; expected an executable mode (e.g. -rwxr-xr-x). Ensure npm/bin/sce.js is committed as Git mode 100755." -fi -printf 'pack-smoke: packed %s mode %s\n' "$LAUNCHER_ENTRY" "$launcher_mode" +assert_launcher_mode "$tarball" -# Serve the fixture release over loopback HTTPS. port_file="$work_dir/server-port" node "$SCRIPT_DIR/fixture-release-server.mjs" \ --root "$release_dir" \ @@ -172,7 +183,6 @@ done [[ -s "$port_file" ]] || fail "Fixture release server did not start within ${SERVER_START_TIMEOUT_SECONDS}s." port="$(tr -d '\n' <"$port_file")" -# Install into an isolated prefix with an isolated npm cache/config. SCE_NPM_RELEASE_BASE_URL="https://127.0.0.1:${port}" \ NODE_EXTRA_CA_CERTS="$keys_dir/tls-cert.pem" \ npm_config_cache="$work_dir/npm-cache" \ @@ -189,11 +199,17 @@ SCE_NPM_RELEASE_BASE_URL="https://127.0.0.1:${port}" \ sce_bin="$prefix_dir/node_modules/.bin/sce" [[ -x "$sce_bin" ]] || fail "Installed sce bin $sce_bin is not executable." -help_output="$("$sce_bin" --help)" || fail "sce --help exited non-zero." +sce_cmd=("$sce_bin") +if [[ ! -x /usr/bin/env ]]; then + printf 'pack-smoke: /usr/bin/env unavailable; running installed bin through node\n' + sce_cmd=(node "$sce_bin") +fi + +help_output="$("${sce_cmd[@]}" --help)" || fail "sce --help exited non-zero." [[ "$help_output" == "$EXPECTED_HELP_OUTPUT" ]] || fail "sce --help printed '$help_output'; expected '$EXPECTED_HELP_OUTPUT'." -version_output="$("$sce_bin" version)" || fail "sce version exited non-zero." +version_output="$("${sce_cmd[@]}" version)" || fail "sce version exited non-zero." [[ "$version_output" == "sce $version" ]] || fail "sce version printed '$version_output'; expected 'sce $version'." From 93ae34cced7551ebae0a6cb8fb5e6cfc37178be3 Mon Sep 17 00:00:00 2001 From: David Abram Date: Tue, 3 Nov 2026 18:46:40 +0100 Subject: [PATCH 6/6] npm: Document hardened installer distribution contract Document the musl platform targets and the installer's manifest identity checks, bounded archive extraction, checksum verification, and atomic executable install. Record completion and validation evidence for the harden-npm-installer plan, including the completed T05 documentation task and all acceptance criteria. Plan: harden-npm-installer (T05) Co-authored-by: SCE --- context/glossary.md | 2 +- context/plans/harden-npm-installer.md | 60 +++++++++++++++++--- context/sce/cli-npm-distribution-contract.md | 2 +- npm/README.md | 7 ++- 4 files changed, 57 insertions(+), 14 deletions(-) diff --git a/context/glossary.md b/context/glossary.md index 1f216848f..63db3af6a 100644 --- a/context/glossary.md +++ b/context/glossary.md @@ -209,7 +209,7 @@ - `sce native portability audit app`: Root-flake app exposed as `nix run .#native-portability-audit`; audits a native `sce` binary for forbidden `/nix/store/` runtime references using macOS `otool -L` install-name inspection or Linux ELF dynamic metadata plus focused shared-object string scanning, and fails with offending references when found. The `native-portability-audit` flake check exercises deterministic clean/failing fixtures for both platform inspection modes. - `sce release-manifest app`: Root-flake app exposed as `nix run .#release-manifest`; merges per-platform `sce` metadata fragments into the release-level manifest JSON, emits detached signature `sce-v-release-manifest.json.sig`, and writes the combined `SHA256SUMS` file. - `SCE_RELEASE_MANIFEST_SIGNING_KEY`: Release-only private-key env var consumed by `nix run .#release-manifest` / `.github/workflows/release-sce.yml` to sign `sce-v-release-manifest.json`; it must never be committed or shipped in the npm package. -- `sce npm launcher package`: Checked-in npm package under `npm/` named `@crocoder-dev/sce`; its `postinstall` flow downloads `sce-v-release-manifest.json` plus detached signature `sce-v-release-manifest.json.sig`, verifies that manifest with the bundled public key, then uses trusted `checksum_sha256` metadata to verify the matching GitHub release archive before installing the native binary into package-local runtime storage. The current supported npm install matrix is `darwin/arm64`, `linux/x64`, and `linux/arm64`, mapping to the current three release target triples. npm consumes portable native GitHub Release archives as-is; release packaging/workflow validation must remove or reject forbidden `/nix/store/` runtime references before upload. npm registry publication is a separate non-bumping downstream stage from GitHub release packaging. +- `sce npm launcher package`: Checked-in npm package under `npm/` named `@crocoder-dev/sce`; its `postinstall` flow downloads `sce-v-release-manifest.json` plus detached signature `sce-v-release-manifest.json.sig`, verifies that manifest with the bundled public key, validates its identity (package version, binary `sce`, exactly one canonical artifact for the target, 64-hex `checksum_sha256`), checksum-verifies the canonical GitHub release archive, extracts only `bin/sce` through the bounded in-repo reader `npm/lib/archive.js` (no host `tar`), and atomically installs it with mode `0755` into package-local runtime storage. The current supported npm install matrix is `darwin/arm64`, `linux/x64`, and `linux/arm64`, mapping to the current three release target triples. npm consumes portable native GitHub Release archives as-is; release packaging/workflow validation must remove or reject forbidden `/nix/store/` runtime references before upload. npm registry publication is a separate non-bumping downstream stage from GitHub release packaging. - `publish-npm workflow`: Dedicated npm registry publication workflow at `.github/workflows/publish-npm.yml` that triggers from a published GitHub release or manual dispatch, validates `.version`/`npm/package.json`/release-tag parity, downloads the canonical `sce-v-npm.tgz` GitHub release asset, verifies the tarball's embedded package metadata, and publishes that already-versioned package to npm. - `sce release authority contract`: Approved release topology where repo-root `.version` is the canonical checked-in release version source, GitHub Releases are the canonical publication surface for signed release artifacts, and Cargo/npm registry publication are separate downstream publish stages that consume already-versioned checked-in package metadata without workflow-side version bumping. - `sce release-npm-package app`: Root-flake app exposed as `nix run .#release-npm-package`; stages the checked-in npm package without rewriting its version (the requested version must already match `.version`, `cli/Cargo.toml`, and `npm/package.json`), runs `npm pack`, and emits `sce-v-npm.tgz` plus `sce-v-npm.json` for release publication only after the produced tarball passes `npm/test/pack-smoke.sh --tarball` (executable `package/bin/sce.js` plus a smoke install). diff --git a/context/plans/harden-npm-installer.md b/context/plans/harden-npm-installer.md index f1ec14ea3..d9a530d1b 100644 --- a/context/plans/harden-npm-installer.md +++ b/context/plans/harden-npm-installer.md @@ -16,19 +16,19 @@ Current-state facts this plan starts from: ## Acceptance criteria -- [ ] AC1: A tarball produced by `npm pack` of `npm/` installs into an isolated prefix, and the installed `sce` bin (npm-generated link to `bin/sce.js`) is executable and successfully runs `sce --help` and `sce version` against a locally served fixture release, with no network access. +- [x] AC1: A tarball produced by `npm pack` of `npm/` installs into an isolated prefix, and the installed `sce` bin (npm-generated link to `bin/sce.js`) is executable and successfully runs `sce --help` and `sce version` against a locally served fixture release, with no network access. - Validate: `nix build .#checks.x86_64-linux.npm-pack-smoke --print-build-logs` (new check) passes; the smoke script asserts exit 0 and expected stdout for both commands. -- [ ] AC2: The root cause of the non-executable launcher is identified, fixed at its source, and recorded in the plan task evidence; the packed tarball entry `package/bin/sce.js` carries an executable mode. +- [x] AC2: The root cause of the non-executable launcher is identified, fixed at its source, and recorded in the plan task evidence; the packed tarball entry `package/bin/sce.js` carries an executable mode. - Validate: `git ls-files -s npm/bin/sce.js` shows the fixed mode (if Git mode is the cause) and the smoke check inspects the packed tarball entry mode. -- [ ] AC3: After signature verification and before any archive download, the installer rejects manifests whose `version` differs from the package version, whose `binary` is not `sce`, that have zero or more than one artifact for the current target, whose artifact `archive` is not `sce-v-.tar.gz`, or whose `checksum_sha256` is not 64 lowercase hex characters. +- [x] AC3: After signature verification and before any archive download, the installer rejects manifests whose `version` differs from the package version, whose `binary` is not `sce`, that have zero or more than one artifact for the current target, whose artifact `archive` is not `sce-v-.tar.gz`, or whose `checksum_sha256` is not 64 lowercase hex characters. - Validate: `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'` — negative tests for each case assert the specific error and that the archive downloader was never called. -- [ ] AC4: Archive extraction no longer shells out to unrestricted `tar -xzf`; the installer parses the gzip+ustar stream itself with bounded sizes, rejects absolute paths, `..` traversal, symlinks, hardlinks, duplicate entries, unexpected entry types, entries outside the expected root, oversize payloads, and a missing/non-regular `bin/sce`, and writes only `bin/sce` into an isolated staging directory. +- [x] AC4: Archive extraction no longer shells out to unrestricted `tar -xzf`; the installer parses the gzip+ustar stream itself with bounded sizes, rejects absolute paths, `..` traversal, symlinks, hardlinks, duplicate entries, unexpected entry types, entries outside the expected root, oversize payloads, and a missing/non-regular `bin/sce`, and writes only `bin/sce` into an isolated staging directory. - Validate: `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'` — fixture archives for each malicious case are rejected and nothing is written outside staging; `nix shell nixpkgs#ripgrep -c rg -n '"-xzf"|spawnSync\("tar"' npm/lib` returns no match. -- [ ] AC5: Installation is atomic: the binary is staged next to `runtime/sce`, chmodded `0755`, and renamed into place; a checksum failure, extraction failure, or interruption leaves a previously installed valid `runtime/sce` byte-identical and executable, and leaves no partial `runtime/sce`. +- [x] AC5: Installation is atomic: the binary is staged next to `runtime/sce`, chmodded `0755`, and renamed into place; a checksum failure, extraction failure, or interruption leaves a previously installed valid `runtime/sce` byte-identical and executable, and leaves no partial `runtime/sce`. - Validate: `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'` — interrupted-install and failure tests over a real temp `runtime/` directory. -- [ ] AC6: `nix run .#release-npm-package -- --version <.version> --out-dir ` fails if the packed tarball's `bin/sce.js` is not executable or the packed-package smoke install fails. +- [x] AC6: `nix run .#release-npm-package -- --version <.version> --out-dir ` fails if the packed tarball's `bin/sce.js` is not executable or the packed-package smoke install fails. - Validate: run the app on a clean tree (passes); run it with `npm/bin/sce.js` temporarily set to `0644` in a scratch copy (fails with a permission diagnostic). Inspection of `.github/workflows/release-sce.yml` confirms it invokes this app before upload. -- [ ] AC7: `npm/README.md` lists `aarch64-unknown-linux-musl` and `x86_64-unknown-linux-musl`, and `context/sce/cli-npm-distribution-contract.md` describes the manifest identity checks, safe extraction, atomic install, and the packed-package smoke verification. +- [x] AC7: `npm/README.md` lists `aarch64-unknown-linux-musl` and `x86_64-unknown-linux-musl`, and `context/sce/cli-npm-distribution-contract.md` describes the manifest identity checks, safe extraction, atomic install, and the packed-package smoke verification. - Validate: `nix shell nixpkgs#ripgrep -c rg -n 'linux-gnu' npm/README.md` returns no match; inspect the contract file. ### Full validation @@ -147,15 +147,57 @@ Current-state facts this plan starts from: - Context impact: affects verification and release surfaces. `context/sce/cli-npm-distribution-contract.md` (new `npm-pack-smoke` check, `--tarball` smoke mode, `release-npm-package` post-pack gate), `context/glossary.md` `npm JS flake checks` entry, and any flake-check enumeration in `context/overview.md`; possibly `context/sce/cli-release-artifact-contract.md` if it describes `release-npm-package` validation. - Context synchronization: synced -- [ ] T05: `Correct npm README targets and update distribution contract` (status:todo) +- [x] T05: `Correct npm README targets and update distribution contract` (status:done) - Task ID: T05 - Scope: In — `npm/README.md` musl target triples and a short note on manifest identity/safe-extraction failures in troubleshooting; `context/sce/cli-npm-distribution-contract.md` install-flow, validation, atomic-install, and verification-baseline updates; glossary/overview entries for the new flake check. Out — documentation site changes. - Dependencies: T04 - Done when: README matches `npm/lib/platform.js`; contract describes implemented behavior and the new verification commands. - Verify: `nix shell nixpkgs#ripgrep -c rg -n 'linux-gnu' npm/README.md` (no match); `nix develop -c biome check npm`. - - Context synchronization: pending + - Completed: 2026-11-03 + - Files changed: `npm/README.md`, `context/sce/cli-npm-distribution-contract.md`, `context/glossary.md`. + - Result: `npm/README.md` now lists `aarch64-unknown-linux-musl` and `x86_64-unknown-linux-musl` (matching `npm/lib/platform.js`). "How the installer works" now describes manifest identity validation, checksum verification of the canonical archive, bounded `bin/sce`-only extraction, atomic `0755` install, and the guarantee that a failed install preserves an existing binary. Troubleshooting gained a bullet for `Invalid sce release manifest` / `Invalid sce release artifact` / `No sce release artifact found` / `Invalid sce release archive` errors. Earlier task syncs had already brought the contract's install-flow, identity-validation, safe-extraction, atomic-install, smoke and verification-baseline sections up to date, so its only edit is the README-description bullet (musl matrix and new failure classes). The glossary `sce npm launcher package` entry now covers identity validation, the bounded reader, and the atomic install. `context/overview.md` and the glossary `npm JS flake checks` entry already listed `npm-pack-smoke` and were not changed. + - Verify outcomes: + - `nix shell nixpkgs#ripgrep -c rg -n 'linux-gnu' npm/README.md` -> no match (exit 1). + - `nix develop -c biome check npm` -> exit 0, no errors; one pre-existing warning (unused `getRepositorySlug` in `npm/lib/install.js`, untouched). + - Context impact: documentation-only; touches `context/sce/cli-npm-distribution-contract.md` and `context/glossary.md` directly as part of the task scope. No code, architecture, or flake changes. + - Context synchronization: synced ## Open questions - Is the loopback HTTPS + self-signed certificate fixture acceptable inside the Nix build sandbox for `npm-pack-smoke`? If sandbox loopback networking proves unavailable on a CI runner, the fallback is to keep the smoke as a `nix run` app invoked by `release-npm-package` and PR CI instead of a pure flake check. - Should the smoke additionally run the real `nix build .#default` binary inside the fixture archive on the host platform? This gives real-binary coverage for the host only, at the cost of making the check depend on the full CLI build; the plan defaults to a stub binary. + +## Validation Report + +**Status:** validated +**Date:** 2026-11-03 + +### Commands run + +- `nix flake check` -> exit 0 (all checks passed!; `checks.x86_64-linux` includes `npm-pack-smoke`, `npm-bun-tests`, `npm-biome-check`, `npm-biome-format`) +- `nix develop -c sh -c 'cd npm && bun test ./test/*.test.js'` -> not run (blocked by repo bash policy `use-nix-for-bun`); equivalent `nix run nixpkgs#bun -- test ./test/*.test.js` from `npm/` -> exit 0 (69 pass, 0 fail across 3 files) +- `nix run .#release-npm-package -- --version "$(cat .version)" --out-dir ` -> exit 0 (`pack-smoke: OK (sce --help, sce version)`; tarball and metadata JSON written) +- `nix build .#checks.x86_64-linux.npm-pack-smoke --print-build-logs --rebuild` -> exit 0 (`pack-smoke: OK (sce --help, sce version)`) +- `release-npm-package` from a scratch copy of the tracked tree with `npm/bin/sce.js` at `0644` -> exit 1 (`Packed package/bin/sce.js has mode -rw-r--r--; expected an executable mode`; tarball removed, out dir empty) +- `git ls-files -s npm/bin/sce.js` -> exit 0 (`100755`) +- `nix shell nixpkgs#ripgrep -c rg -n '"-xzf"|spawnSync\("tar"' npm/lib` -> exit 1 (no match) +- `nix shell nixpkgs#ripgrep -c rg -n 'linux-gnu' npm/README.md` -> exit 1 (no match) + +### Success-criteria verification + +- [x] AC1: Packed tarball installs into an isolated prefix and the `sce` bin runs `--help` and `version` against a local fixture release offline -> `npm-pack-smoke` rebuild passed with exact-output assertions. +- [x] AC2: Root cause identified, fixed at source, recorded; packed `package/bin/sce.js` is executable -> T01 evidence records Git mode `100644` root cause; `git ls-files -s` shows `100755`; smoke reports `packed package/bin/sce.js mode -rwxr-xr-x`. +- [x] AC3: Manifest identity rejected before archive download -> Bun suite 69/0 including negative tests for version, binary, missing/duplicate target, archive name, and checksum, each asserting zero downloads. +- [x] AC4: No unrestricted `tar -xzf`; bounded validating extractor -> Bun suite covers malicious fixture archives; ripgrep over `npm/lib` returns no match. +- [x] AC5: Atomic install preserving prior valid `runtime/sce` -> Bun suite interrupted-install and failure tests over a real temp `runtime/` pass. +- [x] AC6: `release-npm-package` fails on a non-executable launcher or failed smoke -> clean tree exit 0; `0644` scratch copy exit 1 with permission diagnostic; `.github/workflows/release-sce.yml` runs the app in "Build npm release package" before "Create GitHub release". +- [x] AC7: README lists musl triples and contract describes identity checks, safe extraction, atomic install, smoke verification -> no `linux-gnu` match; README lines 26-27 list both musl triples; `context/sce/cli-npm-distribution-contract.md` covers all four topics. + +### Failed checks and follow-ups + +- None. + +### Residual risks + +- The sandboxed `npm-pack-smoke` runs the installed bin through `node` (no `/usr/bin/env` in the sandbox); direct shebang execution is covered only by host runs such as `release-npm-package`. +- Only the host platform (x86_64-linux) was exercised; the smoke uses a stub binary, not the real native `sce`. diff --git a/context/sce/cli-npm-distribution-contract.md b/context/sce/cli-npm-distribution-contract.md index 64f810e62..be2732b7c 100644 --- a/context/sce/cli-npm-distribution-contract.md +++ b/context/sce/cli-npm-distribution-contract.md @@ -6,7 +6,7 @@ This file captures the implemented npm distribution slice of the `sce` CLI insta - The npm package name is `@crocoder-dev/sce`. - The committed package source lives under `npm/`. -- `npm/README.md` documents the package as a thin launcher, lists the supported npm platform-to-target-triple matrix, and distinguishes npm `postinstall`/download/signature/checksum/permission failures from runtime loader failures caused by bad native release artifact linkage. +- `npm/README.md` documents the package as a thin launcher, lists the supported npm platform-to-target-triple matrix (musl Linux triples, matching `npm/lib/platform.js`), summarizes manifest identity validation, bounded extraction, and atomic install, and distinguishes npm `postinstall`/download/signature/manifest-identity/checksum/archive-validation/permission failures from runtime loader failures caused by bad native release artifact linkage. - `npm/package.json` exposes the `sce` binary through `bin/sce.js`. - `npm/bin/sce.js` is committed as Git mode `100755`. `release-npm-package` copies `npm/` with `cp -R` and `npm pack` records filesystem modes verbatim, so the launcher's executable bit in Git is what makes the packed `package/bin/sce.js` entry executable. npm, bun, and pnpm re-chmod bin targets while linking, which masks a non-executable entry for those installers but not for consumers that execute the packed file directly. - The npm package is a thin launcher package, not a separate build pipeline. diff --git a/npm/README.md b/npm/README.md index 8ce64fdf7..fc49fc38c 100644 --- a/npm/README.md +++ b/npm/README.md @@ -23,20 +23,21 @@ npm install -g @crocoder-dev/sce ## Supported platforms - `darwin/arm64` → `aarch64-apple-darwin` -- `linux/arm64` → `aarch64-unknown-linux-gnu` -- `linux/x64` → `x86_64-unknown-linux-gnu` +- `linux/arm64` → `aarch64-unknown-linux-musl` +- `linux/x64` → `x86_64-unknown-linux-musl` Unsupported platforms fail with explicit guidance instead of attempting an alternate install channel inside the npm package. ## How the installer works -The npm package is a thin launcher. During `postinstall`, it downloads the signed GitHub Release manifest for the package version, verifies the bundled manifest signature, selects the checksum-pinned native archive for the current supported platform, extracts `bin/sce`, and stores it in package-local runtime storage. +The npm package is a thin launcher. During `postinstall`, it downloads the signed GitHub Release manifest for the package version, verifies the bundled manifest signature, checks that the signed manifest matches this package (same version, binary `sce`, exactly one canonical `sce-v-.tar.gz` artifact for the current platform with a valid SHA-256 checksum), downloads and checksum-verifies that archive, extracts only `bin/sce` with a bounded built-in reader that rejects unsafe or unexpected archive entries, and atomically installs it with mode `0755` into package-local runtime storage. A failed install leaves any previously installed binary in place. Native binary portability is handled before the GitHub Release archive is published. The npm installer does not rewrite macOS dynamic library install names, run `install_name_tool`, patch Linux ELF metadata, or build a separate npm-native binary. ## Troubleshooting install vs runtime failures - `postinstall` failures usually indicate npm lifecycle-script execution, network/download, signature, checksum, unsupported-platform, or package-local filesystem permission issues. +- `Invalid sce release manifest`, `Invalid sce release artifact`, or `No sce release artifact found` errors mean the signed manifest does not describe this package version and platform; `Invalid sce release archive` errors mean the downloaded archive contained unsafe, unexpected, or oversized entries. Both abort the install without replacing an existing binary and indicate a release publication problem, not a local configuration issue. - Runtime loader failures after a successful install, such as macOS `dyld` errors mentioning `/nix/store/.../*.dylib`, indicate a bad native release artifact. Fix those in the GitHub Release artifact pipeline before npm consumes the archive; do not add user-side npm workarounds or a separate npm build path. ## Other supported install channels