diff --git a/CHANGELOG.md b/CHANGELOG.md index 1c261cf8..aec6c862 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -678,14 +678,34 @@ into the new version's section — see docs/releasing.md. ### Added +- **`scan --max-new-patches ` rolls patches out gradually** + (env `SOCKET_MAX_NEW_PATCHES`; socket.yml `patches.maxNewPatches`). + Each run adds at most N patches to packages that had none, most severe first + (then by how many advisories a patch fixes), and defers the rest to the + next run; upgrades of packages that are already patched are never + capped, and `0` means upgrades only. Repeated scans on an unchanged repo + add the same packages in the same order and stop once everything is + patched. A patch that cannot land (not granted, refused by a preflight, + nothing in the lockfile to pin) never holds a slot, and a failed lookup + admits nothing new that run (`rollout_incomplete_lookup`). The project + directories of one scan share the budget. Works in hosted, vendored and + agent mode, `--dry-run` included; `scan --json` gains a top-level + `rollout` block (`maxNewPatches`, `counts`, ranked `deferred[]`) and + hosted mode lists deferred rows in `redirect.skipped[]` as + `rollout_deferred`. +- **The in-memory hosted engine paces rollouts too.** It (napi, + `hosted-bundle`) takes + `maxNewPatches`, `maxNewPatchesCap` and `inFlightPatches`, spends one + budget across every project root, and reports a session `rollout` block + and `ProjectResult.deferred[]`. - **socket.yml patch policy (staged rollout).** A `patches` block in the repo-root socket.yml narrows what `scan` patches: `enabled` (false = report only), `includePaths` / `ignorePaths` (gitignore patterns matched against each project's lockfiles, npm `ignore` semantics), `ecosystems`, `packages` / `ignorePackages` (`--package` specs), `minSeverity` (critical|high|medium|moderate|low, judged by the worst - advisory a patch fixes) and `maxNewPatches` (validated; the per-run cap - lands with `--max-new-patches`). List flags (`--ecosystems`, + advisory a patch fixes) and `maxNewPatches` (the per-run cap of + `--max-new-patches`). List flags (`--ecosystems`, `--package`, PATHs) only narrow further; `--min-severity` beats the file's floor and `--no-socket-yml` ignores the file. A package that already carries a patch is never removed, upgraded or replaced by @@ -1909,6 +1929,15 @@ into the new version's section — see docs/releasing.md. ### Changed +- **`scan` keeps a patch you already have unless the new one supersedes + it.** A package whose recorded patch (agent manifest, hosted lockfile + pin or vendor ledger) still ranks level with the top offer on every + meaningful rung (merged state, severity, a later publish date) keeps + its recorded patch instead of switching on the tier or uuid tiebreak, + so re-running `scan` never swaps patches. `updates[]` and the + `[UPDATE]` marker now use the per-package records the selection itself + uses, so they list exactly the upgrades the run applies; a JSON + report-only run still reads the batch records. - **Fewer downloads in vendored runs.** A vendored run now asks the patch service for all of its planned packages' download references in one request (in chunks of 500) from the first package it reaches, in place diff --git a/README.md b/README.md index 6cb2ba7c..6305f33b 100644 --- a/README.md +++ b/README.md @@ -257,7 +257,13 @@ the same way everywhere, from the patches your account can download: "Newest" is when the patch was published, not the package version. When a better patch appears for a package you already patched, the JSON `updates[]` array lists it and the -next `scan` in the same mode takes it. +next `scan` in the same mode takes it. A patch that only wins on the tier or UUID +tiebreak never replaces one you already have, so re-running `scan` never swaps patches. + +This order picks the patch *for* a package. When a capped scan +([`--max-new-patches`](#add-a-few-new-patches-per-run)) has to choose *which packages* +get their first patch, it takes the most severe first, then the ones fixing the most +advisories. ### State in `.socket/` @@ -416,6 +422,37 @@ hosted and vendored mode each PATH is a project directory, scanned as if it were To make the choice stick for everyone who runs `scan` in the repo (CI and the Socket autopatch bot included), put it in `socket.yml` instead — see [Roll out gradually](#roll-out-gradually-with-socketyml). +### Add a few new patches per run + +```bash +socket-patch scan --max-new-patches 5 # at most 5 packages get their first patch +socket-patch scan --max-new-patches 0 # only upgrade patches you already have +socket-patch scan --max-new-patches none # no cap this run +``` + +A capped scan patches the most critical packages first: by the severity of the patch, +then by how many advisories it fixes. Upgrades of packages that are already patched are +never capped. Commit the result and run `scan` again to add the next batch; repeated +runs on an unchanged repo add the same packages in the same order and stop once +everything is patched. `--dry-run` shows exactly what the run would add and defer, and +`--json` reports it under `rollout` (`jq '.rollout.counts.deferred'`). In hosted and +vendored mode, several project directories in one run (`scan apps/*`) share the cap, +visited in sorted order; `--json` takes one directory, so a CI job per directory gets +its own N. + +To drip patches in through a PR bot, set the cap once in the repo's `socket.yml` +(part of its [`patches:` policy](#roll-out-gradually-with-socketyml); the flag and +`SOCKET_MAX_NEW_PATCHES` override it, and `--no-socket-yml` ignores it): + +```yaml +# Weekly drip with the depscan autopatch PR +version: 2 +patches: + maxNewPatches: 5 # the PR keeps the same 5 until merged, then the next 5 +``` + +A cap only advances when the scan's changes are committed (or merged by a PR bot). In a +CI job that scans without committing, set no cap. ### Patch one specific CVE or advisory @@ -650,6 +687,7 @@ socket-patch scan [PATHS]... [options] | `--package ` | `SOCKET_SCAN_PACKAGES` | Only scan these packages: a name (`lodash`, `@scope/pkg`, `requests`; case-insensitive) or a purl with or without its version (`pkg:npm/lodash`, `pkg:pypi/requests@2.31.0`). Repeat the flag or separate with commas. | | `--prune` | — | Agent-mode garbage collection after the scan: remove manifest entries for packages no longer present in the crawl (installed trees + lockfiles — a wiped `node_modules` alone doesn't prune lockfile-listed entries) and delete orphan blob/diff-archive files (plus any legacy package archives). [Vendored](#vendor) packages are exempt from the crawl-based prune, but a vendored entry whose dependency has left the lockfile is reverted. Ignored, with a `redirect_prune_ignored` warning, in hosted mode; without a mode the scan is report-only. | | `--sync` | — | Shorthand for `--mode agent --prune`: the one-flag agent-mode auto-update run. | +| `--max-new-patches ` | `SOCKET_MAX_NEW_PATCHES` | Add at most N patches to packages that have none yet, most severe first; the rest are deferred to the next scan and listed in the output. Upgrades of already-patched packages are not capped. `0` adds no new patches, `none` means no cap (it also lifts a `socket.yml` `patches.maxNewPatches`). See [Add a few new patches per run](#add-a-few-new-patches-per-run). | | `--batch-size ` | `SOCKET_BATCH_SIZE` | Packages per API request (default: `500` on the authenticated API, `100` on the public proxy). A request whose body would exceed 256 KiB is split into smaller ones. | | `--min-severity ` | `SOCKET_MIN_SEVERITY` | Only patch packages whose patch fixes an advisory of at least `critical`, `high`, `medium` (or `moderate`) or `low`; `none` lifts the floor. Overrides `patches.minSeverity` in socket.yml. Patches of unknown severity are skipped whenever a floor is set. | | `--no-socket-yml` | `SOCKET_NO_SOCKET_YML` | Ignore the repo's socket.yml patch policy for this run (the built-in test/fixture directory ignores still apply). | @@ -678,6 +716,9 @@ socket-patch scan --package lodash # Two projects of a monorepo socket-patch scan apps/web apps/api +# Roll out gradually: at most 5 new patches, most critical first +socket-patch scan --max-new-patches 5 + # Vendored mode: build + commit every patched dependency socket-patch scan --json --mode vendored diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 6d2e797d..e9e7d099 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -87,6 +87,7 @@ Beyond the globals above, each subcommand defines a small set of local arguments | `scan` | `--package ` (repeatable or comma-separated) | `SOCKET_SCAN_PACKAGES` | (v5.0) Only scan these packages: a name (`lodash`, `@scope/pkg`, `requests`, `group:artifact`; matched against the full name or its last segment, case-insensitively) or a purl with or without a version (`pkg:npm/lodash` matches every version, `pkg:pypi/requests@2.31.0` only that one). Qualifiers are ignored. Filters the crawl like `--ecosystems`, after the prune universe is captured, so `--prune` still judges the full crawl | | `scan` | `--vendor` | — | Vendor every patched dependency instead of applying in place (`--vendor` == `--mode vendored`; conflicts with `--apply`/`--sync`, combines with `--prune`). Vendored mode is manifest-free (v5.0): the vendor ledger embeds the patch records and `.socket/manifest.json` is never written. The former opt-in for exactly that, `--detached`, is removed in v5.0 (unknown-flag usage error) | | `scan` | `--batch-size` | `SOCKET_BATCH_SIZE` | API batch chunk size. Unset (v5.0): `500` on the authenticated API (the server's per-request maximum), `100` on the public proxy; a given value applies on either endpoint (`0` is floored to `1`). A chunk whose request body would exceed 256 KiB (the public proxy's body cap) is split into consecutive smaller chunks, deterministically (greedy, in crawl order). A mid-run downgrade to the proxy keeps the chunks already formed | +| `scan` | `--max-new-patches ` | `SOCKET_MAX_NEW_PATCHES` | (v5.0) Per-run cap on NEW patches (packages with no recorded patch in the project), most severe first; the rest are deferred to the next scan. `0` admits upgrades only, `none` (case-insensitive) is unlimited, absent is unlimited unless socket.yml sets `patches.maxNewPatches`. Precedence: flag > env > socket.yml > unlimited (`--no-socket-yml` drops the socket.yml layer). The env value is read at run time (the `rollout` block reports `flag` vs `env`): empty is unset, malformed is a usage error (exit 2, before any network access). Upgrades and already-applied patches are never capped. See "Per-run limit on new patches" below | | `scan` | `--no-socket-yml` | `SOCKET_NO_SOCKET_YML` | (v5.0) Ignore the repository's socket.yml patch policy (its `patches` block and `projectIgnorePaths`) for this run; the built-in test/fixture ignores still apply. The `policy` block reports `source: "bypassed"`. See "socket.yml patch policy". | | `scan` | `--min-severity ` | `SOCKET_MIN_SEVERITY` | (v5.0) Severity floor for the patch a package may receive (worst advisory severity; unknown severity is skipped whenever a floor is set). Beats `patches.minSeverity`; the flag beats the env; `none` lifts the floor. A malformed value is exit 2. | | `get`, `scan` | `--all-releases` | `SOCKET_ALL_RELEASES` | Download patches for every release/distribution variant of a matched package — PyPI wheel/sdist (`artifact_id`), RubyGems (`platform`), Maven (`classifier`) — not just the one(s) matching the locally-installed distribution. On `scan` this makes the stored manifest portable across environments (e.g. cross-platform CI caches). On `get` (v3.6) it ALSO disables the coarse installed-**version** narrowing of CVE/GHSA fan-outs (see "get --mode and installed narrowing"): every found version's patch is fetched, installed or not | @@ -194,7 +195,7 @@ patches: packages: ["pkg:npm/lodash"] # allowlist in the --package grammar ignorePackages: ["pkg:npm/left-pad"] # denylist in the --package grammar minSeverity: high # critical|high|medium|moderate|low (moderate = medium) - maxNewPatches: 5 # integer 0..=4294967295; validated now, the per-run cap lands with `--max-new-patches` + maxNewPatches: 5 # integer 0..=4294967295; the per-run cap of `--max-new-patches` ``` - Deny wins: `ignorePackages` beats `packages`, ignore paths beat `includePaths`. An empty allowlist (`includePaths: []`, `ecosystems: []`, `packages: []`) is an error ("use `enabled: false`"), never "all". @@ -258,6 +259,49 @@ patches: - `filtered[]` and `retained[]` are sorted by project, then purl; purls use the canonical spelling (qualifiers stripped, percent-decoded). - Exit code is unchanged by filtering. +### Per-run limit on new patches (`scan --max-new-patches`, v5.0) + +`scan --max-new-patches ` (env `SOCKET_MAX_NEW_PATCHES`; socket.yml `patches.maxNewPatches`) paces a rollout: each run adds at most N patches to packages that had none, the most critical first, and defers the rest to the next run. It applies to `scan` in hosted, vendored and agent mode, wet and `--dry-run`, and to the in-memory engine (napi `maxNewPatches`, `hosted-bundle`); `get` is explicit intent and ignores it. Design: `docs/design/staged-rollout.md` §5. + +**Classification.** After per-package selection, each selected `(project, purl)` row is compared with the project's **recorded view** — the merged manifest > hosted lockfile pins > vendor ledger that `updates[]` reads (§5.1): + +| Class | Rule | Capped | The writer gets | +|---|---|---|---| +| ALREADY | the recorded uuid is the selected one, or the selection does not supersede it | no | the **recorded** uuid (re-confirmed idempotently, never swapped) | +| UPGRADE | the selection supersedes the recorded uuid (`ranking::search_result_supersedes`), or the recorded uuid is no longer offered | no | the selected uuid | +| NEW | nothing recorded for the base purl in this project | **yes** | the selected uuid, if admitted | + +Supersession is judged on the by-package records the selection itself uses, so a scan with or without a cap never replaces an applied patch with an equal sibling (the tier and uuid tiebreaks and a missing date never count). When the lockfiles of a project pin a purl to several uuids, the recorded uuid is the selected one if it is among them, else the smallest. A hosted pin on a patch server that discovery does not recognize (an origin missing from `--patch-server-url`) still counts as ALREADY when a lockfile names the selected uuid, so the cap can never stall on it. + +**Eligibility.** A NEW row is eligible only if every check the mode can decide without writing passes: the tier filter; the agent partition (vendor-owned and not-installed packages); the vendored Bun / vlt preflight; a `granted` hosted reference with a usable purl and url; the vlt artifact preflight; the vendored-to-hosted takeover refusals; wheel metadata; and a hosted rewrite whose confirmation probe shows a lockfile edit that pins it. Ineligible rows keep their own skip reasons and never hold a slot. References are fetched for every row before the budget is spent, and `--dry-run` fetches them too, so a dry run makes exactly the wet run's decisions. + +**Unit and order.** The budget counts distinct **base purls** (ecosystem + name + version, qualifiers stripped, percent-decoded; qualifier twins are one package): admitting one admits all of its eligible rows and costs one slot. Eligible NEW base purls are ranked by, ascending: in-flight first (in-memory `inFlightPatches` only); severity of the selected patch (critical, high, medium, low, unknown — the worst advisory it fixes); advisory count, descending; ecosystem name; base purl (bytewise); uuid. The order is total and has no time-dependent key (publish dates still pick the patch within a package, never the package order). + +**Budget scope.** Disk: one budget per invocation. The project directories a hosted or vendored scan's PATHs name are visited in sorted order; each spends what is left in rank order, and a base purl admitted in an earlier directory is admitted free in a later one. `scan --json` takes one directory, so a CI job per directory gets N per directory. In memory: one budget across every project root (roots are collected, planned once, then applied). The two therefore rank a multi-root repo differently, by design. + +**Incomplete data.** With a finite cap, a failed batch query, or a failed detail query for a package with no recorded patch, admits no NEW row that run (they are all deferred) and adds warning `rollout_incomplete_lookup`: a missing package must not let lower-ranked ones take its slot. ALREADY and UPGRADE rows proceed as usual. A failed hosted reference lookup that could only affect NEW rows of a capped run is warning `rollout_reference_failed` (the rows are deferred) instead of a run failure. + +**Convergence.** The limit is stateless: run k lands the top N, run k+1 finds them recorded and lands the next N, so M waiting patches take at most ceil(M/N) *committed* runs. A newly published or re-scored more severe patch moves ahead of the queue (intended: most critical first), and low-severity patches can wait while more severe ones keep arriving. A CI job that does not commit the scan's changes never advances: there a cap means "only the top N, every run" — commit the changes (or use a PR bot), or set no cap in such jobs. A write failure after admission still spends its slot (no backfill within a run). Known limits: a dependency that moves to a new version is NEW again (its hosted pin stays with the old lock entry); a qualifier twin that lands on a later run joins its package as ALREADY / UPGRADE, uncapped. + +**Output.** Every successful `scan --json` result carries an additive top-level `rollout` block (MINOR), zero counts when the run planned nothing (report-only and empty scans): + +```json +"rollout": { + "maxNewPatches": {"value": 5, "source": "flag"}, + "counts": {"new": 5, "deferred": 9, "upgrade": 1, "already": 12}, + "deferred": [ + {"purl": "pkg:npm/minimist@1.2.5", "uuids": ["…"], "severity": "critical", + "advisoryCount": 1, "projects": [""], "rank": 6} + ] +} +``` + +`maxNewPatches.value` is `null` for unlimited; `source` is `flag`, `env`, `file`, `default` or `cap` (the in-memory `maxNewPatchesCap` tightened it; the in-memory `maxNewPatches` option reports `flag`). `counts.new` / `counts.deferred` count base purls, `counts.upgrade` / `counts.already` count `(project, purl)` rows. `deferred[]` is in rank order: `purl` is the base purl, `uuids` the distinct selected uuids across its rows, `projects` the repo-relative project directories (`""` is the scanned directory), `rank` 1-based among eligible NEW base purls. Deferred rows are never written, downloaded or vendored: hosted mode mirrors each into `redirect.skipped[]` as `{purl, uuid, reason: "rollout_deferred", detail}`, the in-memory engine lists them in `ProjectResult.deferred[]` (`{purl, uuid, severity, rank}`) and `skipped[]`, and agent / vendored mode leave them out of `apply.patches[]` / `vendor`. Warnings (`rollout_incomplete_lookup`, `rollout_reference_failed`) go to the top-level `warnings[]`. Exit codes are unchanged: deferring is not a failure. + +Human output adds, when a cap is set, `Rollout: 3 of 9 new patches applied (maxNewPatches=3 from --max-new-patches); 0 upgrades, 0 already applied.` (with several project directories: `…, shared by this run's directories, 1 left)`) and next steps naming the deferred patches (`6 new patches deferred; commit these changes and run scan again to apply the next 3.`, `Next up: minimist@1.2.5 (critical), …`; a dry run says `would be deferred`, and an incomplete lookup says so instead). Hosted mode prints them on stdout after its own next steps, unindented; agent and vendored mode under a `Next steps:` heading. The `rollout` block is left out of error envelopes (`status: "error"`). + +CI recipe: `socket-patch scan --json --max-new-patches 5 | jq '.rollout.counts.deferred'`. + ### Embedded VEX (`apply --vex` / `scan --vex` / `vendor --vex`) `--vex ` folds OpenVEX 0.2.0 generation into `apply`, `scan`, and `vendor`: on a successful run the command writes the document to `` using the same engine as the standalone `vex` command. The `--vex-*` flags mirror `vex`'s `--product` / `--no-verify` / `--doc-id` / `--compact` knobs (namespaced to avoid colliding with the host command), and reuse the standalone env vars (`SOCKET_VEX_PRODUCT`, etc.). They are inert unless `--vex` is set. @@ -1011,6 +1055,7 @@ Empty string means unset at every layer: exported-but-empty flag-bound vars are | `SOCKET_FORCE` | `apply --force` / `-f`, `vendor --force` / `-f`, `--update --force` | `false` | Local to `apply`, `vendor` and `--update`. | | `SOCKET_PATCH_VERSION` | `--update ` | (latest) | Local to `--update`; the same pin `install.sh` and the gem launcher honor. | | `SOCKET_BATCH_SIZE` | `scan --batch-size` | `500` authenticated / `100` proxy | Local to `scan`. | +| `SOCKET_MAX_NEW_PATCHES` | `scan --max-new-patches` | (unlimited) | Local to `scan` (v5.0): a count or `none`; empty is unset, malformed exits 2. | | `SOCKET_SCAN_PACKAGES` | `scan --package` | (none) | Local to `scan` (v5.0); comma-separated names or purls. | | `SOCKET_NO_SOCKET_YML` | `scan --no-socket-yml` | `false` | Local to `scan` (v5.0); bool vocabulary, empty = unset. | | `SOCKET_MIN_SEVERITY` | `scan --min-severity` | (none) | Local to `scan` (v5.0); read by scan (not clap) so the `policy` block can say `source: "env"`; empty = unset, malformed = exit 2. | @@ -1452,10 +1497,24 @@ excluded before ranking for callers whose `canAccessPaidPatches` is false, so the winner is the best patch the account can download. `scan`'s `[UPDATE]` marker and `updates[]` use the same order -(`ranking::batch_supersedes`): a candidate supersedes the applied patch -only on a meaningful rung — merged over unmerged, higher severity between -unmerged patches, or a real, strictly later publish date. The tier and -uuid tiebreaks and a missing date never count. +(`ranking::search_result_supersedes`, v5.0): a candidate supersedes the +applied patch only on a meaningful rung — merged over unmerged, higher +severity between unmerged patches, or a real, strictly later publish +date. The tier and uuid tiebreaks and a missing date never count. Every +mode that fetches the by-package records (hosted, vendored, agent, and +every human run with a downloadable patch) judges this on those records, +so `updates[]` lists exactly the UPGRADE rows the run acts on; a package +the by-package lookup returns no offer for, and a JSON report-only run +(which fetches no by-package records), fall back to the batch records +(`ranking::batch_supersedes`). When the selection does not supersede the +recorded patch, scan keeps the recorded one (v5.0): a re-scan never swaps +an applied patch for an equal sibling. + +This order picks one patch **per package**. Which packages a capped scan +patches first is a separate, cross-package order (`rollout::rollout_cmp`, +see "Per-run limit on new patches"): severity of the selected patch, then +advisory count, then ecosystem, base purl and uuid — never the publish +date. #### Merge state is inferred, not reported @@ -1495,8 +1554,8 @@ supplies it (the public-proxy fallback path fills it in from the per-package results). > **Known gap — batch responses without `publishedAt`.** `scan`'s -> discovery (`packages[]`, the table, `updates[]`) is built from the -> **batch** endpoint, whose response shape currently omits `publishedAt`; +> discovery (`packages[]`, and `updates[]` on a JSON report-only run) is +> built from the **batch** endpoint, whose response shape currently omits `publishedAt`; > the selection that `--apply` performs is built from the **by-package** > endpoint, which carries it. The two diverge wherever the date decides — > between merged patches, or between unmerged patches of equal severity — @@ -1515,6 +1574,14 @@ per-package results). ### `jq` recipes for PR-comment bots +Deferred by the per-run cap (`scan --max-new-patches`), most urgent first: + +```bash +socket-patch scan --json --max-new-patches 5 | jq -r ' + .rollout.deferred[] | "\(.rank). \(.purl) (\(.severity))" +' +``` + Applied + updated patches (envelope shape): ```bash @@ -1557,7 +1624,7 @@ Exit `1` when `status` is `partialFailure` (any `events[*].action == "failed"`) |---|---| | `0` | Success | | `1` | Error (missing/invalid manifest, fetch failed, apply failed, selection cancelled in non-JSON mode, an invalid or ambiguous socket.yml on `scan` (v5.0), etc.) | -| `2` | Usage error: clap parse failures (unknown flag/value, missing required arg, an unknown subcommand such as the removed `setup`) and the conflicts the commands enforce themselves — `scan`'s cross-mode conflicts (`--mode` combined with a DIFFERENT mode's boolean spelling, rejected in `resolve_mode_flags`) and `--mode hosted` with `--global`/`--global-prefix` (same enforcement point); in hosted/vendored `scan` (bare `scan` included), a PATH that is not a directory, a PATH glob matching no directory, and `--json` with more than one project directory (`run_project_dirs`); `remove --preserve-state --skip-rollback` (the no-op quadrant; flag- or env-sourced alike), an unparseable path glob on `scan`/`rollback`, a `scan` PATH outside the repository root and a malformed `SOCKET_MIN_SEVERITY` (v5.0), `repair --offline --download-only`. `vex` also exits `2` on hard errors before document generation (see its tri-state table below). v5.0: `get`'s self-enforced conflicts exit `2` too (`--id`/`--cve`/`--ghsa`/`--package` multi-select, `--mode hosted\|vendored --save-only`, a malformed identifier for a forced `--id`/`--cve`/`--ghsa`) — previously `1` (MAJOR). The never-implemented `get --one-off` / `rollback --one-off` (and `SOCKET_ONE_OFF`) are removed in v5.0; `--one-off` is now an ordinary unknown-flag clap error. | +| `2` | Usage error: clap parse failures (unknown flag/value, missing required arg, an unknown subcommand such as the removed `setup`) and the conflicts the commands enforce themselves — `scan`'s cross-mode conflicts (`--mode` combined with a DIFFERENT mode's boolean spelling, rejected in `resolve_mode_flags`) and `--mode hosted` with `--global`/`--global-prefix` (same enforcement point); in hosted/vendored `scan` (bare `scan` included), a PATH that is not a directory, a PATH glob matching no directory, and `--json` with more than one project directory (`run_project_dirs`); `remove --preserve-state --skip-rollback` (the no-op quadrant; flag- or env-sourced alike), an unparseable path glob on `scan`/`rollback`, a `scan` PATH outside the repository root and a malformed `SOCKET_MIN_SEVERITY` or `SOCKET_MAX_NEW_PATCHES` (v5.0), `repair --offline --download-only`. `vex` also exits `2` on hard errors before document generation (see its tri-state table below). v5.0: `get`'s self-enforced conflicts exit `2` too (`--id`/`--cve`/`--ghsa`/`--package` multi-select, `--mode hosted\|vendored --save-only`, a malformed identifier for a forced `--id`/`--cve`/`--ghsa`) — previously `1` (MAJOR). The never-implemented `get --one-off` / `rollback --one-off` (and `SOCKET_ONE_OFF`) are removed in v5.0; `--one-off` is now an ordinary unknown-flag clap error. | `list` returns **`0`** for every project it can read, empty or not (**v5.0, BREAKING**: a project with no manifest and no ledger record — normal for hosted mode, which writes no manifest — used to exit `1` with `manifest_not_found`; it is now an empty list: `No patches in this project. Run \`socket-patch scan\`.` on stdout, and under `--json` the success envelope with `events: []`). Only an unreadable or invalid manifest (`manifest_unreadable` / `manifest_invalid`) exits `1`. Every lock-taking subcommand — including `scan`/`get --mode hosted` as of v5.0 — returns **`1`** with `errorCode: lock_held` when another live socket-patch process holds `<.socket>/apply.lock`. diff --git a/crates/socket-patch-cli/src/commands/get.rs b/crates/socket-patch-cli/src/commands/get.rs index e02f3908..daf8abbd 100644 --- a/crates/socket-patch-cli/src/commands/get.rs +++ b/crates/socket-patch-cli/src/commands/get.rs @@ -3590,6 +3590,8 @@ async fn run_get_hosted( &pairs, scan_result, None, + // `get` is explicit intent: the rollout cap never applies. + None, ) .await } diff --git a/crates/socket-patch-cli/src/commands/hosted_bundle.rs b/crates/socket-patch-cli/src/commands/hosted_bundle.rs index 277bc104..a054e7e7 100644 --- a/crates/socket-patch-cli/src/commands/hosted_bundle.rs +++ b/crates/socket-patch-cli/src/commands/hosted_bundle.rs @@ -10,7 +10,9 @@ //! Stdin: `{"files": {path: text}, "binaryFiles"?: {path: base64}, //! "presentOnly"?: [path], "symlinks"?: [path], "projectRoots"?: [dir], //! "pipenvMajor"?: n, "batchSize"?: n, "noSocketYml"?: bool, -//! "minSeverity"?: severity, "policyPaths"?: [path]}`. Stdout: the engine result +//! "minSeverity"?: severity, "policyPaths"?: [path], +//! "maxNewPatches"?: n | "none", "maxNewPatchesCap"?: n, +//! "inFlightPatches"?: [purl]}`. Stdout: the engine result //! (`HostedScanResult`, binary contents base64), or //! `{"status":"error","error":{"code","message"}}` with exit 2 for bad //! credentials/bundle input, or exit 1 for an engine failure. @@ -62,6 +64,12 @@ struct Bundle { policy_paths: Option>, #[serde(default)] policy_sha256: Option, + #[serde(default)] + max_new_patches: Option, + #[serde(default)] + max_new_patches_cap: Option, + #[serde(default)] + in_flight_patches: Option>, } fn print_error(code: &str, message: &str) { @@ -134,6 +142,9 @@ pub async fn run(args: HostedBundleArgs) -> i32 { min_severity: bundle.min_severity.clone(), policy_paths: bundle.policy_paths.clone(), policy_sha256: bundle.policy_sha256.clone(), + max_new_patches: bundle.max_new_patches, + max_new_patches_cap: bundle.max_new_patches_cap, + in_flight_patches: bundle.in_flight_patches.clone(), ..HostedScanOptions::default() }; let input = match build_input(bundle, options) { diff --git a/crates/socket-patch-cli/src/commands/scan/discovery.rs b/crates/socket-patch-cli/src/commands/scan/discovery.rs index ce5425e9..cd389626 100644 --- a/crates/socket-patch-cli/src/commands/scan/discovery.rs +++ b/crates/socket-patch-cli/src/commands/scan/discovery.rs @@ -13,7 +13,6 @@ use socket_patch_core::utils::concurrent::{api_concurrency_for, ordered_concurre use socket_patch_core::utils::purl::{normalize_purl, strip_purl_qualifiers}; use socket_patch_core::vendor::lock_inventory::LockfileEntry; use socket_patch_core::vendor::VendorState; -use std::borrow::Cow; use std::collections::{HashMap, HashSet}; use crate::args::GlobalArgs; @@ -365,58 +364,8 @@ pub(super) async fn preverify_vendor_baselines( (mismatched, views) } -/// Fold the hosted pins and the vendor ledger's patch records into the -/// manifest view update detection consults. Hosted mode records purl→uuid -/// ONLY in the lockfiles (`hosted_pins`, uuid only; v5 keeps no hosted -/// ledger) and vendored mode ONLY in `.socket/vendor/state.json`, so without -/// this fold a pure hosted or vendored project's `updates[]` would always -/// be empty. Precedence on a collision: manifest > hosted pins > vendor -/// ledger (the live lock over a possibly superseded vendored entry); the -/// vendor entries fold under the shared owner rule -/// ([`socket_patch_core::ledgers::Ledgers::owned`]), so one the manifest -/// claims (its key or base purl) stays behind the manifest's record. -/// Vendor entries are keyed by their ledger key (`detect_updates` bridges -/// the spellings); a legacy entry without an embedded record contributes -/// its uuid alone. Borrows the manifest untouched when nothing else -/// contributes. -pub(super) fn merge_ledger_records_for_updates<'a>( - manifest: Option<&'a PatchManifest>, - vendor: Option<&VendorState>, - hosted_pins: &[(String, String)], -) -> Option> { - use socket_patch_core::ledgers::{uuid_only_record, Ledgers, Store}; - let vendor = vendor.filter(|s| !s.entries.is_empty()); - if vendor.is_none() && hosted_pins.is_empty() { - return manifest.map(Cow::Borrowed); - } - let mut merged = manifest.cloned().unwrap_or_default(); - for (purl, uuid) in hosted_pins { - merged - .patches - .entry(purl.clone()) - .or_insert_with(|| uuid_only_record(uuid)); - } - let ledgers = Ledgers { - manifest, - vendor, - redirect: None, - }; - for owned in ledgers.owned() { - if owned.store == Store::Manifest { - continue; - } - merged - .patches - .entry(owned.key.to_string()) - .or_insert_with(|| { - owned - .record - .cloned() - .unwrap_or_else(|| uuid_only_record(owned.uuid)) - }); - } - Some(Cow::Owned(merged)) -} +pub(super) use socket_patch_core::ledgers::merge_ledger_records_for_updates; + /// Cross-reference an existing manifest against discovery results to find /// PURLs whose newest available patch UUID differs from the locally-recorded @@ -566,6 +515,7 @@ pub(super) fn severity_order(s: &str) -> u8 { #[cfg(test)] mod tests { use super::*; + use std::borrow::Cow; use socket_patch_core::api::types::BatchPatchInfo; use crate::commands::scan::tests::manifest_with; diff --git a/crates/socket-patch-cli/src/commands/scan/hosted.rs b/crates/socket-patch-cli/src/commands/scan/hosted.rs index 078df2b8..b5c939e1 100644 --- a/crates/socket-patch-cli/src/commands/scan/hosted.rs +++ b/crates/socket-patch-cli/src/commands/scan/hosted.rs @@ -78,6 +78,10 @@ fn emit_json_error_with_code( let mut result = scan_result.unwrap_or_else(|| serde_json::json!({ "status": "error" })); result["status"] = serde_json::json!("error"); result["error"] = serde_json::json!(message); + // The rollout block describes a successful run only. + if let Some(obj) = result.as_object_mut() { + obj.remove("rollout"); + } if let Some(code) = code { result["errorCode"] = serde_json::json!(code); } @@ -539,21 +543,28 @@ pub(super) async fn run_redirect( // handed to the VEX step so it does not walk the tree for the npm // roots again. npm_prior: Option<&crate::ecosystem_dispatch::NpmCrawlSnapshot>, + // The merged recorded view (manifest > hosted pins > vendor ledger) the + // rollout classifies against, whether a batch failed, and the stage + // that holds this directory's budget. + recorded: &super::rollout::RecordedState<'_>, + batch_failed: bool, + stage: &mut super::rollout::Stage, ) -> i32 { // Same discovery/selection as `--apply`/`--vendor`. - let selected: Vec = match discover_selected( + let discovered = match discover_selected( api_client, all_packages_with_patches, can_access_paid_patches, policy, false, false, + false, telemetry, scan_result.as_mut(), ) .await { - Ok(offers) => offers.selected.into_values().collect(), + Ok(d) => d, // Hosted mode has no discovery envelope to fold the message into at // this point (it builds its `redirect` result further down). // `discover_selected` already printed the message to stderr; a @@ -570,13 +581,22 @@ pub(super) async fn run_redirect( return code; } }; + let rows = super::classified_rows( + stage, + &discovered, + recorded, + batch_failed, + all_packages_with_patches, + scan_result.as_mut(), + ); // The redirect body consumes the selection only as (purl, uuid) pairs — // the seam `get --mode hosted` injects its advisory-pinned selection - // through (see `run_redirect_selected`). - let pairs: Vec<(String, String)> = selected + // through (see `run_redirect_selected`). ALREADY rows carry the + // recorded uuid, so a re-scan re-confirms the pin instead of swapping it. + let pairs: Vec<(String, String)> = rows .iter() - .map(|s| (s.purl.clone(), s.uuid.clone())) + .map(|r| (r.writer.purl.clone(), r.writer.uuid.clone())) .collect(); run_redirect_selected( &args.common, @@ -586,6 +606,7 @@ pub(super) async fn run_redirect( &pairs, scan_result, npm_prior, + Some(super::rollout::Gate::new(stage, rows)), ) .await } @@ -614,6 +635,7 @@ pub(super) async fn run_redirect( /// human/JSON split keys on `common.json`; a `--json` caller passing `None` /// would get a minimal envelope that drops its own keys). `prune_requested` /// only feeds the `redirect_prune_ignored` warning — `get` passes `false`. +#[allow(clippy::too_many_arguments)] pub(crate) async fn run_redirect_selected( common: &crate::args::GlobalArgs, vex: &crate::commands::vex::VexEmbedArgs, @@ -622,6 +644,9 @@ pub(crate) async fn run_redirect_selected( selected: &[(String, String)], mut scan_result: Option, npm_prior: Option<&crate::ecosystem_dispatch::NpmCrawlSnapshot>, + // `scan`'s rollout gate: NEW rows past the budget are deferred after + // every write-free eligibility check below (§5.2). `get` passes `None`. + mut rollout: Option>, ) -> i32 { use socket_patch_core::hosted::engine::{ self, Candidate, CandidateFiles, RewriteOptions, SkippedPatch, @@ -647,6 +672,24 @@ pub(crate) async fn run_redirect_selected( status.finish(); let references = match fetched { Ok(r) => r, + // A capped run whose every row is NEW: the failure affects only + // rows the incomplete lookup defers anyway (§5.2), so it becomes + // a warning instead of failing the run. + Err(e) + if rollout.as_ref().is_some_and(|gate| { + let new = gate.new_keys(); + gate.stage.capped() + && selected + .iter() + .all(|(p, u)| new.contains(&(p.clone(), u.clone()))) + }) => + { + if let Some(gate) = rollout.as_mut() { + gate.stage.incomplete = true; + gate.stage.reference_failed = Some(e.to_string()); + } + std::collections::HashMap::new() + } Err(e) => { let message = format!("failed to resolve patch references: {e}"); eprintln!( @@ -659,8 +702,27 @@ pub(crate) async fn run_redirect_selected( return 1; } }; - candidates = engine::build_candidates(selected, &references, &mut skipped); + // Deferred rows need no reference: a failed lookup that only hit + // them builds nothing (and records no `not_found` for them). + if rollout + .as_ref() + .is_none_or(|gate| gate.stage.reference_failed.is_none()) + { + candidates = engine::build_candidates(selected, &references, &mut skipped); + } } + // The rollout rows key on the selection's purl spelling; the server's + // reference may spell a candidate's `purl` differently. Uuids are unique + // across the selection. + let sel_purl_of: std::collections::HashMap<&str, &str> = selected + .iter() + .map(|(purl, uuid)| (uuid.as_str(), purl.as_str())) + .collect(); + let sel_purl = |c: &Candidate| -> String { + sel_purl_of + .get(c.dep.patch_uuid.as_str()) + .map_or_else(|| c.purl.clone(), |p| (*p).to_string()) + }; // Check binary lock symlinks before a mode takeover changes any wiring // (the takeover reverts rewrite locks in place, never create or remove @@ -699,7 +761,15 @@ pub(crate) async fn run_redirect_selected( // preview must not create `.socket/`, flip to `lock_held` under a // concurrent wet run, or fail on a read-only checkout). Held to the end // of the function. - let _lock: Option = if !common.dry_run && !candidates.is_empty() { + // A capped run whose only candidates are NEW rows it cannot admit + // (budget 0, or incomplete data) writes nothing: take no lock either. + let may_write = rollout.as_ref().is_none_or(|gate| { + let new = gate.new_keys(); + candidates.iter().any(|c| { + !new.contains(&(sel_purl(c), c.dep.patch_uuid.clone())) || gate.may_admit(&sel_purl(c)) + }) + }); + let mut lock: Option = if !common.dry_run && !candidates.is_empty() && may_write { match acquire_hosted_lock(common, &mut scan_result) { Ok(guard) => Some(guard), Err(code) => return code, @@ -864,28 +934,83 @@ pub(crate) async fn run_redirect_selected( socket_patch_core::utils::fs::read_regular_to_string_sync(path).ok() }) }; - let done = engine::rewrite( + let rewrite_options = || RewriteOptions { + dry_run: common.dry_run, + targets_pipenv_lock, + pipenv_major, + pipenv_unknown_detail: format!( + "Pipenv was not found on PATH, so the Pipfile.lock references use the modern `file` form (Pipenv 2018 and later). A project installed with Pipenv 7–11 needs `path` references instead: put that pipenv on PATH or set {}= and re-run `scan --mode hosted`.", + socket_patch_core::utils::pipenv::MAJOR_OVERRIDE_ENV + ), + trust_lockfile_config: !common.no_trust_lockfile_config, + npm_allow_remote_config: !common.no_npm_allow_remote_config, + npm_outer: &npm_outer, + blocking: true, + }; + // The rollout gate plans again without its deferred rows: keep what + // the second pass needs. + let second_pass = rollout + .is_some() + .then(|| (read.clone(), python_metadata.clone())); + let mut done = engine::rewrite( &view, read, &candidates, python_metadata, &vlt_preflight.withheld_from_vlt, &dry_run_takeover_urls, - RewriteOptions { - dry_run: common.dry_run, - targets_pipenv_lock, - pipenv_major, - pipenv_unknown_detail: format!( - "Pipenv was not found on PATH, so the Pipfile.lock references use the modern `file` form (Pipenv 2018 and later). A project installed with Pipenv 7–11 needs `path` references instead: put that pipenv on PATH or set {}= and re-run `scan --mode hosted`.", - socket_patch_core::utils::pipenv::MAJOR_OVERRIDE_ENV - ), - trust_lockfile_config: !common.no_trust_lockfile_config, - npm_allow_remote_config: !common.no_npm_allow_remote_config, - npm_outer: &npm_outer, - blocking: true, - }, + rewrite_options(), ) .await; + + // The rollout gate (§5.2): every write-free check has run — grants, + // purl/url, vlt preflight, takeover refusals, wheel metadata, and the + // rewrite above, whose confirmation probe proves a NEW row would be + // pinned. Only then is the budget spent; deferred rows leave the + // rewrite set, which is rewritten again without them. + if let (Some(gate), Some((read, python_metadata))) = (rollout.as_mut(), second_pass) { + let eligible: std::collections::HashSet<(String, String)> = done + .confirmed + .iter() + .map(|(purl, uuid)| { + let purl = sel_purl_of.get(uuid.as_str()).map_or(purl.as_str(), |p| p); + (purl.to_string(), uuid.clone()) + }) + .collect(); + let texts: Vec<&str> = done.files.values().map(String::as_str).collect(); + super::rollout::mark_pinned(&mut gate.rows, &texts); + let unknown = gate.stage.reference_failed.is_some(); + gate.stage.plan(&gate.rows, |row| { + unknown || eligible.contains(&(row.writer.purl.clone(), row.writer.uuid.clone())) + }); + let deferred = gate.stage.deferred_keys(); + skipped.extend(gate.stage.deferred_skips()); + let before = candidates.len(); + candidates.retain(|c| !deferred.contains(&(sel_purl(c), c.dep.patch_uuid.clone()))); + if candidates.len() != before { + done = engine::rewrite( + &view, + read, + &candidates, + python_metadata, + &vlt_preflight.withheld_from_vlt, + &dry_run_takeover_urls, + rewrite_options(), + ) + .await; + } + // A row that turned out to be pinned already (`mark_pinned`: a pin + // discovery did not recognize) still gets written: take the lock + // skipped above. Only NEW rows ran without it, so no takeover did. + if lock.is_none() && !common.dry_run && !candidates.is_empty() { + match acquire_hosted_lock(common, &mut scan_result) { + Ok(guard) => lock = Some(guard), + Err(code) => return code, + } + } + } + // Held to the end of the function. + let _lock = lock; // Dry-run mode-takeover previews were withheld from the rewriters (their // lock fragments still carry the vendored wiring the wet run reverts // first), so the presence probe cannot see them: the wet run reverts @@ -1188,6 +1313,9 @@ pub(crate) async fn run_redirect_selected( common.dry_run, ); let mut result = build_redirect_json_envelope(scan_result.take(), redirect); + if let Some(gate) = &rollout { + super::finish_rollout_json(gate.stage, &mut result); + } if let Some(statements) = vex_statements { result["vex"] = serde_json::json!({ "path": vex.vex.as_ref().expect("vex_statements is Some only when --vex was given").display().to_string(), @@ -1228,8 +1356,9 @@ pub(crate) async fn run_redirect_selected( human_files.extend(takeover_files.iter().cloned()); human_files.sort(); human_files.dedup(); - // The one stdout line: scripts read it, so it stays on stdout; - // everything below is on stderr and names its package itself. + // The summary line: scripts read it, so it stays on stdout, as do + // the rollout line and the next steps; the warnings below are on + // stderr and name their package themselves. println!( "{}", format_redirect_summary(confirmed.len(), human_files.len(), common.dry_run) @@ -1250,8 +1379,10 @@ pub(crate) async fn run_redirect_selected( .collect(); // Human output prints the bare strings — `Value`'s `Display` // would JSON-quote them. + // Deferred rows are summed up by the rollout lines instead. let skipped_pairs: Vec<(String, String)> = skipped .iter() + .filter(|s| s.reason != super::rollout::ROLLOUT_DEFERRED) .map(|s| (s.purl.clone(), s.reason.clone())) .collect(); // Granted, but nothing in the project pins it (no lock entry, @@ -1311,12 +1442,26 @@ pub(crate) async fn run_redirect_selected( crate::commands::vex::format_vex_dry_run_skip("rewritten") ); } - if !common.dry_run { - for line in - format_next_steps(&human_files, !takeover_migrated.is_empty()) - { - println!("{line}"); + let (rollout_line, deferred_steps) = match &rollout { + Some(gate) => { + for (code, detail) in gate.stage.warnings() { + eprintln!("{}", format_warning(code, &detail, width)); + } + super::rollout::human(gate.stage, common.dry_run) } + None => (None, Vec::new()), + }; + if let Some(line) = rollout_line { + println!("{line}"); + } + let mut next_steps = if common.dry_run { + Vec::new() + } else { + format_next_steps(&human_files, !takeover_migrated.is_empty()) + }; + next_steps.extend(deferred_steps); + for line in next_steps { + println!("{line}"); } } // Errors print even under --silent ("errors only", never @@ -2060,6 +2205,7 @@ fn format_next_steps(files: &[String], vendored_removed: bool) -> Vec { /// future embeds the whole hosted engine, and callers outside scan (`get /// --mode hosted`) must not materialize it in their own poll frame (Windows /// 1 MiB main-thread stack; same rationale as scan's `boxed_*` family). +#[allow(clippy::too_many_arguments)] pub(crate) fn boxed_run_redirect_selected<'a>( common: &'a crate::args::GlobalArgs, vex: &'a crate::commands::vex::VexEmbedArgs, @@ -2068,6 +2214,7 @@ pub(crate) fn boxed_run_redirect_selected<'a>( selected: &'a [(String, String)], scan_result: Option, npm_prior: Option<&'a crate::ecosystem_dispatch::NpmCrawlSnapshot>, + rollout: Option>, ) -> std::pin::Pin + 'a>> { Box::pin(run_redirect_selected( common, @@ -2077,6 +2224,7 @@ pub(crate) fn boxed_run_redirect_selected<'a>( selected, scan_result, npm_prior, + rollout, )) } diff --git a/crates/socket-patch-cli/src/commands/scan/mod.rs b/crates/socket-patch-cli/src/commands/scan/mod.rs index 39dece92..f1714644 100644 --- a/crates/socket-patch-cli/src/commands/scan/mod.rs +++ b/crates/socket-patch-cli/src/commands/scan/mod.rs @@ -43,6 +43,8 @@ pub(crate) mod hosted; pub(crate) mod policy; mod socket_yml_args; pub(crate) mod render; +pub(crate) mod rollout; +pub mod rollout_args; pub(crate) mod vendor_flow; use self::discovery::{ @@ -338,6 +340,9 @@ pub struct ScanArgs { #[command(flatten)] pub vex: VexEmbedArgs, + #[command(flatten)] + pub rollout: rollout_args::RolloutArgs, + #[command(flatten)] pub socket_yml: SocketYmlArgs, } @@ -480,7 +485,7 @@ async fn embed_vex_human( /// are the human-only knobs of [`fetch_patch_details`] (JSON callers pass /// `false, false`). `json_warnings` is the JSON callers' envelope: a /// partial failure adds one [`PATCH_DETAILS_FAILED`] warning per failed -/// package to it. +/// package to it, and [`Discovered::failed`] lists each failed purl. #[allow(clippy::too_many_arguments)] async fn discover_selected( api_client: &socket_patch_core::api::client::ApiClient, @@ -489,9 +494,10 @@ async fn discover_selected( policy: &ScanPolicy, show_progress: bool, warn: bool, + detail_error_line: bool, telemetry: &mut PendingTelemetry, json_warnings: Option<&mut serde_json::Value>, -) -> Result { +) -> Result { let (all_search_results, failures) = fetch_patch_details(api_client, packages, show_progress, warn).await; // The scan event's send overlapped the detail fetches; every caller's @@ -501,16 +507,20 @@ async fn discover_selected( let error_count = failures.len(); if error_count > 0 && error_count == packages.len() { let err = failures - .into_iter() .last() - .map_or_else(|| "all patch-detail queries failed".to_string(), |(_, e)| e); + .map_or_else(|| "all patch-detail queries failed".to_string(), |(_, e)| e.clone()); let message = format!("all {error_count} patch-detail queries failed: {err}"); - eprintln!("Error: {message}"); + if detail_error_line { + eprintln!("{}", render::fetch_details_failed(&failures)); + } else { + eprintln!("Error: {message}"); + } return Err((1, message)); } // Some queries failed, some succeeded: a `--json` run has no stderr // warning (`warn` is human-only), so each failed package becomes a // run-level `warnings[]` entry — never a silent drop from the envelope. + let fetched = all_search_results.len(); let offers = select_accessible(all_search_results, can_access_paid_patches, policy); if let Some(result) = json_warnings { for (purl, e) in &failures { @@ -522,7 +532,140 @@ async fn discover_selected( } policy.fold_into_json(result); } - Ok(offers) + Ok(Discovered { + offers, + fetched, + failed: failures, + }) +} + +/// [`discover_selected`]'s result: the offers, how many records came back +/// (before the tier filter), and each failed detail query as `(purl, +/// error)` (a failure for a package with no recorded patch makes a capped +/// run's data incomplete). +struct Discovered { + offers: rollout::Offers, + fetched: usize, + failed: Vec<(String, String)>, +} + +/// The `updates[]` JSON array. +fn updates_json(updates: &[discovery::UpdateInfo]) -> Vec { + updates + .iter() + .map(|u| { + serde_json::json!({ + "purl": u.purl, + "oldUuid": u.old_uuid, + "newUuid": u.new_uuid, + }) + }) + .collect() +} + +/// Classify the discovered offers against the recorded view (§5.1), note +/// whether a capped run's data is incomplete, and (JSON) replace the +/// batch-derived `updates[]` with the by-package UPGRADE rows. +fn classified_rows( + stage: &mut rollout::Stage, + discovered: &Discovered, + recorded: &rollout::RecordedState<'_>, + batch_failed: bool, + packages: &[BatchPackagePatches], + result: Option<&mut serde_json::Value>, +) -> Vec { + let failed: Vec = discovered.failed.iter().map(|(purl, _)| purl.clone()).collect(); + stage.incomplete = rollout::lookup_incomplete(&recorded.index, &failed, batch_failed); + let rows = rollout::classify(&discovered.offers, &recorded.index, &stage.project); + if let Some(result) = result { + let updates = offer_updates(&rows, discovered, recorded, packages); + result["updates"] = serde_json::Value::Array(updates_json(&updates)); + } + rows +} + +/// `common` for `select_patches`: scan never prompts, so it always takes +/// the top-ranked patch, and with `json` off it never gets +/// `selection_required` (scan has no "re-run with the chosen UUID" path). +pub(crate) fn selection_args(common: &GlobalArgs) -> GlobalArgs { + GlobalArgs { + json: false, + yes: true, + ..common.clone() + } +} + +/// `updates[]` from the by-package records (see [`rollout::merge_updates`]). +fn offer_updates( + rows: &[rollout::Row], + discovered: &Discovered, + recorded: &rollout::RecordedState<'_>, + packages: &[BatchPackagePatches], +) -> Vec { + let purls: Vec = packages.iter().map(|p| p.purl.clone()).collect(); + rollout::merge_updates( + rows, + &discovered.offers, + &purls, + detect_updates(recorded.manifest, packages), + ) +} + +/// The offers the writers would receive, one per row. +fn writers_of(rows: &[rollout::Row]) -> Vec { + rows.iter().map(|r| r.writer.clone()).collect() +} + +/// Plan the rows whose writer survived the mode's own partition (`kept`; +/// the rest cannot land and hold no slot), then return `kept` without the +/// deferred rows. +fn plan_kept_rows( + stage: &mut rollout::Stage, + rows: Vec, + kept: Vec, +) -> Vec { + let kept_keys: HashSet<(&str, &str)> = kept + .iter() + .map(|p| (p.purl.as_str(), p.uuid.as_str())) + .collect(); + let kept_rows: Vec = rows + .into_iter() + .filter(|r| kept_keys.contains(&(r.writer.purl.as_str(), r.writer.uuid.as_str()))) + .collect(); + stage.plan(&kept_rows, |_| true); + let deferred = stage.deferred_keys(); + kept.into_iter() + .filter(|p| !deferred.contains(&(p.purl.clone(), p.uuid.clone()))) + .collect() +} + +/// Fold the stage's `rollout` block and warnings into a JSON result. +pub(super) fn finish_rollout_json(stage: &rollout::Stage, result: &mut serde_json::Value) { + result["rollout"] = stage.json(); + for (code, detail) in stage.warnings() { + push_scan_json_warning(result, code, &detail); + } +} + +/// The human `Rollout:` line and the Next-steps lines about deferred +/// patches, for the agent and vendored summaries. +fn print_rollout_human(stage: &rollout::Stage, dry_run: bool, silent: bool) { + if silent { + return; + } + for (code, detail) in stage.warnings() { + eprintln!("Warning ({code}): {detail}"); + } + let (line, next) = rollout::human(stage, dry_run); + if let Some(line) = line { + println!("\n{line}"); + } + if !next.is_empty() { + println!("Next steps:"); + for step in next { + println!(" {step}"); + } + } } /// The tier filter, then the policy's per-package selection (see @@ -540,17 +683,6 @@ fn select_accessible( policy.select(accessible) } -/// `common` for `select_patches`: scan never prompts, so it always takes -/// the top-ranked patch, and with `json` off it never gets -/// `selection_required` (scan has no "re-run with the chosen UUID" path). -pub(crate) fn selection_args(common: &GlobalArgs) -> GlobalArgs { - GlobalArgs { - json: false, - yes: true, - ..common.clone() - } -} - /// Print the blank stdout line that opens a paragraph, once: `opened` /// flips on the first call. fn open_paragraph(opened: &mut bool) { @@ -626,6 +758,9 @@ async fn fetch_patch_details( fn emit_discovery_error_json(result: &mut serde_json::Value, message: &str) { result["status"] = serde_json::json!("error"); result["error"] = serde_json::json!(message); + if let Some(obj) = result.as_object_mut() { + obj.remove("rollout"); + } print_json(result); } @@ -1250,6 +1385,17 @@ async fn run_project_dirs( ); return 2; } + // One budget per invocation (§5.2): the directories spend it in sorted + // order, and a package admitted in one is admitted free in the next. + let configured = match args.rollout.resolve_from_env(invocation.policy.max_new_patches()) { + Ok(max) => max, + Err(message) => { + eprintln!("Error: {message}"); + return 2; + } + }; + let root = std::fs::canonicalize(&args.common.cwd).unwrap_or_else(|_| args.common.cwd.clone()); + let carry = rollout_args::RolloutCarry::new(configured, root); let mut code = 0; for (dir, explicit) in &dirs { if dirs.len() > 1 && !args.common.silent { @@ -1259,6 +1405,7 @@ async fn run_project_dirs( let mut child = args.clone(); child.paths.clear(); child.common.cwd = dir.clone(); + child.rollout.carry = Some(carry.clone()); code = code.max(Box::pin(run_scan(child, telemetry, Some(invocation), *explicit)).await); } code @@ -1336,6 +1483,25 @@ async fn run_scan( } }; + // The per-run cap on NEW patches (`--max-new-patches` > env > the + // socket.yml `patches.maxNewPatches`). A malformed env value is a usage + // error. + let configured_cap = match args.rollout.carry.as_ref() { + Some(carry) => carry.lock().configured, + None => match args.rollout.resolve_from_env(invocation.policy.max_new_patches()) { + Ok(max) => max, + Err(message) => { + eprintln!("Error: {message}"); + return 2; + } + }, + }; + let mut stage = rollout::Stage::new( + configured_cap, + args.rollout.carry.clone(), + &args.common.cwd, + ); + // Strict airgap (CLI_CONTRACT.md `--offline`): scan's patch discovery // is remote data, so refuse before the crawl and before the API client // is built (org auto-resolve is itself a network call). @@ -1623,6 +1789,7 @@ async fn run_scan( "packages": [], "updates": [], "paths": path_scope.raw(), + "rollout": stage.json(), }); // Layout refusals: additive top-level `warnings` (omitted when // empty) so a consumer can tell an unscannable project from an @@ -1916,8 +2083,12 @@ async fn run_scan( telemetry_org.as_deref(), ); - let updates = detect_updates(update_manifest.as_deref(), &all_packages_with_patches); + let mut updates = detect_updates(update_manifest.as_deref(), &all_packages_with_patches); policy.set_update_purls(updates.iter().map(|u| u.purl.as_str())); + let recorded = rollout::RecordedState { + manifest: update_manifest.as_deref(), + index: rollout::RecordedIndex::new(update_manifest.as_deref(), &hosted_pins), + }; // The hosted-wiring probes below take `all_purls` (POST-filter: only // packages this run covered), unlike the PRE-filter `scanned_purls` @@ -1935,11 +2106,8 @@ async fn run_scan( "canAccessPaidPatches": can_access_paid_patches, "packages": all_packages_with_patches, "paths": path_scope.raw(), - "updates": updates.iter().map(|u| serde_json::json!({ - "purl": u.purl, - "oldUuid": u.old_uuid, - "newUuid": u.new_uuid, - })).collect::>(), + "updates": updates_json(&updates), + "rollout": stage.json(), }); // Layout refusals ride the non-empty envelope too (additive, // omitted when empty). @@ -1981,6 +2149,9 @@ async fn run_scan( Some(result), telemetry, npm_crawl.as_ref(), + &recorded, + batch_error_count > 0, + &mut stage, ) .await; } @@ -2015,6 +2186,7 @@ async fn run_scan( &policy, false, false, + false, telemetry, Some(&mut result), ) @@ -2027,33 +2199,44 @@ async fn run_scan( // --- Apply path (if requested) ----------------------------------- if apply { - let selected: Vec = match discover_selected( + let discovered = match discover_selected( &api_client, &all_packages_with_patches, can_access_paid_patches, &policy, false, false, + false, telemetry, Some(&mut result), ) .await { - Ok(offers) => offers.selected.into_values().collect(), + Ok(d) => d, Err((code, message)) => { emit_discovery_error_json(&mut result, &message); return code; } }; + let rows = classified_rows( + &mut stage, + &discovered, + &recorded, + batch_error_count > 0, + &all_packages_with_patches, + Some(&mut result), + ); // Vendor-owned and lockfile-only purls leave the selection as - // skip records BEFORE download (see `partition_agent_selection`). + // skip records BEFORE download (see `partition_agent_selection`); + // they cannot land, so they hold no rollout slot either. let AgentSelection { - kept: selected, + kept, skip_records: vendored_records, vendored_purls: vendored_skip_purls, .. - } = partition_agent_selection(selected, &vendored_purls, &lockfile_only); + } = partition_agent_selection(writers_of(&rows), &vendored_purls, &lockfile_only); + let selected = plan_kept_rows(&mut stage, rows, kept); if dry { // Synthesize the per-patch outcome without touching disk. @@ -2151,6 +2334,9 @@ async fn run_scan( use_public_proxy, &all_packages_with_patches, can_access_paid_patches, + &recorded, + batch_error_count > 0, + &mut stage, &policy, &mut result, &manifest_path, @@ -2184,6 +2370,7 @@ async fn run_scan( .await; } + finish_rollout_json(&stage, &mut result); let final_code = embed_vex_into_json( &args.common, &args.vex, @@ -2233,6 +2420,73 @@ async fn run_scan( return finish_human(0).await; } + // Count downloadable patches: a free-tier org whose every offer is + // paid-tier has nothing any mode could select, so every human arm stops + // below the table with the same paid-subscription line. + let downloadable_count = if can_access_paid_patches { + all_packages_with_patches.len() + } else { + all_packages_with_patches + .iter() + .filter(|pkg| pkg.patches.iter().any(|p| p.tier == "free")) + .count() + }; + + // The by-package records every arm selects from, fetched before the + // table so its `[UPDATE]` markers are the same UPGRADE rows the + // selection acts on (§5.1). Discovery said these packages HAVE + // patches, so an empty merged set is a fetch failure. + // A failed discovery still prints the table first; its exit code is + // returned below it. + let mut discovery_failure: Option = None; + let rows: Vec = if downloadable_count == 0 { + Vec::new() + } else { + match discover_selected( + &api_client, + &all_packages_with_patches, + can_access_paid_patches, + &policy, + human, + !silent, + !hosted, + telemetry, + None, + ) + .await + { + // The agent / vendored / report-only arms need records to show: + // an empty merged set is a fetch failure there. + Ok(discovered) if !hosted && discovered.fetched == 0 => { + eprintln!("{}", render::fetch_details_failed(&discovered.failed)); + discovery_failure = Some(1); + Vec::new() + } + Ok(discovered) => { + let rows = classified_rows( + &mut stage, + &discovered, + &recorded, + batch_error_count > 0, + &all_packages_with_patches, + None, + ); + updates = offer_updates( + &rows, + &discovered, + &recorded, + &all_packages_with_patches, + ); + rows + } + // `discover_selected` already printed the failure to stderr. + Err((code, _)) => { + discovery_failure = Some(code); + Vec::new() + } + } + }; + // Presentational only, so `--silent` skips it wholesale. if !silent { let mut updates_available = 0usize; @@ -2354,52 +2608,25 @@ async fn run_scan( } } - // Count downloadable patches: a free-tier org whose every offer is - // paid-tier has nothing any mode could select, so every human arm stops - // here with the same paid-subscription line. - let downloadable_count = if can_access_paid_patches { - all_packages_with_patches.len() - } else { - all_packages_with_patches - .iter() - .filter(|pkg| pkg.patches.iter().any(|p| p.tier == "free")) - .count() - }; - if downloadable_count == 0 { if !silent { println!("\nNo downloadable patches: every patch found requires a paid Socket plan."); } return finish_human(0).await; } + if let Some(code) = discovery_failure { + return code; + } + policy.print_human(silent, verbose); // Hosted mode is a self-contained flow: it reuses the discovery, table // and update detection above, then hands the selection to the redirect // engine (the same entry as `get --mode hosted`) — it must NOT fall // through to the apply/vendor branches. if hosted { - let selected: Vec = match discover_selected( - &api_client, - &all_packages_with_patches, - can_access_paid_patches, - &policy, - human, - !silent, - telemetry, - None, - ) - .await - { - Ok(offers) => offers.selected.into_values().collect(), - // `discover_selected` already printed the failure to stderr. - Err((code, _)) => { - return code; - } - }; - policy.print_human(silent, verbose); - let pairs: Vec<(String, String)> = selected + let pairs: Vec<(String, String)> = rows .iter() - .map(|s| (s.purl.clone(), s.uuid.clone())) + .map(|r| (r.writer.purl.clone(), r.writer.uuid.clone())) .collect(); return boxed_run_redirect_selected( &args.common, @@ -2409,32 +2636,15 @@ async fn run_scan( &pairs, None, npm_crawl.as_ref(), + Some(rollout::Gate::new(&mut stage, rows)), ) .await; } - // Fetch the full per-package patch lists — the same loop the JSON arms - // run through `discover_selected`, here with progress + per-package - // warnings. Discovery said these packages HAVE patches, so an empty - // merged set is a fetch failure. - let (all_search_results, detail_failures) = - fetch_patch_details(&api_client, &all_packages_with_patches, human, !silent).await; - if all_search_results.is_empty() { - eprintln!("{}", render::fetch_details_failed(&detail_failures)); - return 1; - } - // A scan left without a mode (`--prune` or global; see // `resolve_mode_flags`) only reports, plus the `--prune` GC. let report_only = args.mode.is_none(); - - // Scan always takes the top-ranked patch the policy admits. - let selected: Vec = - select_accessible(all_search_results, can_access_paid_patches, &policy) - .selected - .into_values() - .collect(); - policy.print_human(silent, verbose); + let selected: Vec = writers_of(&rows); // The skip / already-recorded lines below open their own paragraph // under the table's Summary: one blank line before the first of them. @@ -2459,6 +2669,23 @@ async fn run_scan( split.kept }; + // The rollout plan (§5.2): deferred NEW rows leave the selection here. + // Vendored eligibility is the wet run's Bun / vlt preflight; the agent + // partition above already removed what cannot land in place. + let selected = if report_only { + selected + } else if vendor { + let refused = vendor_flow::preflight_refused_purls(&args.common.cwd, &selected).await; + stage.plan(&rows, |r| !refused.contains(&r.writer.purl)); + let deferred = stage.deferred_keys(); + selected + .into_iter() + .filter(|p| !deferred.contains(&(p.purl.clone(), p.uuid.clone()))) + .collect() + } else { + plan_kept_rows(&mut stage, rows, selected) + }; + // Drop selections the manifest already records at the same uuid. // Agent mode only: vendored mode never reads the manifest. let recorded = |p: &PatchSearchResult| { @@ -2485,12 +2712,19 @@ async fn run_scan( if selected.is_empty() { if !silent { open_paragraph(&mut skip_paragraph); - if already_recorded.is_empty() { + if !stage.deferred_keys().is_empty() { + if args.common.dry_run { + println!("No new patches would be added this run."); + } else { + println!("No new patches added this run."); + } + } else if already_recorded.is_empty() { println!("No patches selected."); } else { println!("{}", render::ALL_ALREADY_RECORDED); } } + print_rollout_human(&stage, args.common.dry_run, silent); return finish_human(0).await; } @@ -2569,6 +2803,7 @@ async fn run_scan( print_dry_run_refusals(preview); } } + print_rollout_human(&stage, true, silent); return finish_human(0).await; } @@ -2668,6 +2903,9 @@ async fn run_scan( } } + // The deferred next steps assume a run that landed. + print_rollout_human(&stage, false, silent || code != 0); + // Post-apply GC: only with `--prune` or `--sync`; otherwise an agent // apply leaves every other manifest entry alone (`socket-patch repair` // cleans up explicitly). Vendor mode runs its own GC in `vendor_flow`. diff --git a/crates/socket-patch-cli/src/commands/scan/rollout.rs b/crates/socket-patch-cli/src/commands/scan/rollout.rs new file mode 100644 index 00000000..c2256945 --- /dev/null +++ b/crates/socket-patch-cli/src/commands/scan/rollout.rs @@ -0,0 +1,649 @@ +//! Scan's side of the rollout stage (`docs/design/staged-rollout.md` §5, +//! §9.2): `updates[]`, the hosted gate and the human lines. The stage +//! itself is [`socket_patch_core::rollout::stage`]. + +use std::collections::{BTreeMap, BTreeSet, HashSet}; + +use socket_patch_core::rollout::{canonical_base_purl, severity_label, MaxNew, MaxNewSource, Recorded, RolloutPlan}; +pub(crate) use socket_patch_core::rollout::stage::*; + +use super::discovery::UpdateInfo; + +/// `updates[]`: the UPGRADE rows, reported under the batch package's purl +/// spelling when one names the same base purl (one entry per package). +pub(super) fn upgrades(rows: &[Row], package_purls: &[String]) -> Vec { + let by_base: BTreeMap = package_purls + .iter() + .map(|p| (canonical_base_purl(p), p)) + .collect(); + let mut seen: HashSet = HashSet::new(); + let mut out = Vec::new(); + for row in rows { + let Recorded::Superseded { old_uuid } = &row.candidate.recorded else { + continue; + }; + let purl = by_base + .get(&row.candidate.base_purl) + .map_or_else(|| row.candidate.purl.clone(), |p| (*p).clone()); + if seen.insert(purl.clone()) { + out.push(UpdateInfo { + purl, + old_uuid: old_uuid.clone(), + new_uuid: row.candidate.uuid.clone(), + }); + } + } + out.sort_by(|a, b| a.purl.cmp(&b.purl)); + out +} + +/// The rows the hosted engine plans (§9.0 step 7 inside the engine, after +/// its eligibility checks) and the stage that records the plan. +pub(crate) struct Gate<'a> { + pub(crate) stage: &'a mut Stage, + pub(crate) rows: Vec, +} + +impl<'a> Gate<'a> { + pub(crate) fn new(stage: &'a mut Stage, rows: Vec) -> Self { + Gate { stage, rows } + } + + /// `(purl, uuid)` of every NEW row, for O(1) [`Self::is_new`] checks. + pub(crate) fn new_keys(&self) -> HashSet<(String, String)> { + self.rows + .iter() + .filter(|r| r.candidate.recorded.is_new()) + .map(|r| (r.writer.purl.clone(), r.writer.uuid.clone())) + .collect() + } + + /// Whether a NEW `(purl, uuid)` row could be admitted: budget left, or + /// its package already admitted by an earlier directory. + pub(crate) fn may_admit(&self, purl: &str) -> bool { + self.stage.may_admit_new() + || (!(self.stage.capped() && self.stage.incomplete) + && self + .stage + .already_admitted + .contains(&canonical_base_purl(purl))) + } +} + +/// `updates[]` for a run that fetched by-package records: the UPGRADE rows, +/// plus the batch-derived entries for packages the by-package lookup +/// returned no offer for (nothing was selected there to disagree with). +pub(super) fn merge_updates( + rows: &[Row], + offers: &Offers, + package_purls: &[String], + batch: Vec, +) -> Vec { + let offered: BTreeSet = offers + .unfiltered + .keys() + .map(|p| canonical_base_purl(p)) + .collect(); + let mut out = upgrades(rows, package_purls); + out.extend( + batch + .into_iter() + .filter(|u| !offered.contains(&canonical_base_purl(&u.purl))), + ); + out.sort_by(|a, b| a.purl.cmp(&b.purl)); + out.dedup_by(|a, b| a.purl == b.purl); + out +} + +/// The human `Rollout:` line (only when a cap is set) and the +/// Next-steps lines about deferred patches. +pub(crate) fn human(stage: &Stage, dry_run: bool) -> (Option, Vec) { + let Some(plan) = stage.plan.as_ref() else { + return (None, Vec::new()); + }; + let ctx = HumanContext { + dry_run, + incomplete: stage.incomplete && stage.capped(), + shared: stage.carry.is_some(), + }; + human_lines(&stage.configured, plan, ctx) +} + +/// `pkg:npm/@scope/x@1.0.0` → `@scope/x@1.0.0`. +fn short_name(base_purl: &str) -> &str { + base_purl + .strip_prefix("pkg:") + .and_then(|rest| rest.split_once('/')) + .map_or(base_purl, |(_, name)| name) +} + +/// How [`human_lines`] words a run. +#[derive(Debug, Clone, Copy, Default)] +pub(crate) struct HumanContext { + pub(crate) dry_run: bool, + /// A lookup failed, so no new patch could be admitted. + pub(crate) incomplete: bool, + /// The budget is shared with other project directories of this run. + pub(crate) shared: bool, +} + +pub(crate) fn human_lines( + configured: &MaxNew, + plan: &RolloutPlan, + ctx: HumanContext, +) -> (Option, Vec) { + let c = plan.counts; + let line = configured.value.map(|cap| { + let verb = if ctx.dry_run { + "would be applied" + } else { + "applied" + }; + let shared = match (ctx.shared, plan.remaining) { + (true, Some(left)) => format!(", shared by this run's directories, {left} left"), + _ => String::new(), + }; + format!( + "Rollout: {} of {} {verb} (maxNewPatches={cap} from {}{shared}); {}, {} already applied.", + c.new, + crate::ui::plural((c.new + c.deferred) as usize, "new patch", "new patches"), + source_label(configured), + crate::ui::plural(c.upgrade as usize, "upgrade", "upgrades"), + c.already, + ) + }); + let groups = deferred_groups(plan); + if groups.is_empty() { + return (line, Vec::new()); + } + let deferred = crate::ui::plural(groups.len(), "new patch", "new patches"); + let deferred = if ctx.dry_run { + format!("{deferred} would be deferred") + } else { + format!("{deferred} deferred") + }; + let first = match configured.value { + _ if ctx.incomplete => format!( + "{deferred}: a patch lookup failed, so no new patch was added this run; run scan \ + again once the patch API answers." + ), + Some(0) => format!( + "{deferred}: maxNewPatches=0 adds no new patches; {} to add them.", + match configured.source { + MaxNewSource::Flag => "pass a larger --max-new-patches", + MaxNewSource::Env => "raise SOCKET_MAX_NEW_PATCHES", + MaxNewSource::File => "raise patches.maxNewPatches in socket.yml", + MaxNewSource::Cap | MaxNewSource::Default => "raise the cap", + } + ), + Some(cap) if ctx.dry_run => format!( + "{deferred}; the wet run adds the top {}, and each later committed scan the next ones.", + (cap as usize).min(c.new as usize + groups.len()) + ), + Some(cap) => format!( + "{deferred}; commit these changes and run scan again to apply the next {}.", + (cap as usize).min(groups.len()) + ), + None => format!("{deferred}; run scan again to add them."), + }; + let shown: Vec = groups + .iter() + .take(3) + .map(|g| { + format!( + "{} ({})", + short_name(&g.base_purl), + severity_label(g.severity_order) + ) + }) + .collect(); + let more = if groups.len() > 3 { ", …" } else { "" }; + ( + line, + vec![first, format!("Next up: {}{more}", shown.join(", "))], + ) +} + +#[cfg(test)] +mod tests { + use super::*; + use socket_patch_core::api::types::PatchSearchResult; + use socket_patch_core::manifest::schema::PatchManifest; + use std::path::Path; + use socket_patch_core::api::types::VulnerabilityResponse; + use socket_patch_core::manifest::schema::PatchRecord; + use std::collections::HashMap; + + fn offer(purl: &str, uuid: &str, published: &str, severities: &[&str]) -> PatchSearchResult { + PatchSearchResult { + uuid: uuid.to_string(), + purl: purl.to_string(), + published_at: published.to_string(), + description: String::new(), + license: "MIT".to_string(), + tier: "free".to_string(), + vulnerabilities: severities + .iter() + .enumerate() + .map(|(i, s)| { + ( + format!("GHSA-{uuid}-{i}"), + VulnerabilityResponse { + cves: Vec::new(), + summary: String::new(), + severity: (*s).to_string(), + description: String::new(), + }, + ) + }) + .collect(), + } + } + + fn manifest(entries: &[(&str, &str)]) -> PatchManifest { + let mut m = PatchManifest::new(); + for (purl, uuid) in entries { + m.patches.insert( + (*purl).to_string(), + PatchRecord { + uuid: (*uuid).to_string(), + exported_at: String::new(), + files: HashMap::new(), + vulnerabilities: HashMap::new(), + description: String::new(), + license: String::new(), + tier: String::new(), + }, + ); + } + m + } + + fn classes(rows: &[Row]) -> Vec<(String, Recorded, String)> { + rows.iter() + .map(|r| { + ( + r.candidate.purl.clone(), + r.candidate.recorded.clone(), + r.writer.uuid.clone(), + ) + }) + .collect() + } + + #[test] + fn offers_drop_inaccessible_paid_patches_and_pick_the_top_one() { + let mut paid = offer("pkg:npm/a@1", "p", "2026-01-01T00:00:00Z", &["critical"]); + paid.tier = "paid".into(); + let results = vec![ + offer("pkg:npm/a@1", "old", "2020-01-01T00:00:00Z", &["high"]), + paid, + offer("pkg:npm/a@1", "new", "2026-01-01T00:00:00Z", &["high"]), + ]; + let free = offers_from_results(&results, false); + assert_eq!(free.selected["pkg:npm/a@1"].uuid, "new"); + assert_eq!(free.unfiltered["pkg:npm/a@1"].len(), 2); + let all = offers_from_results(&results, true); + assert_eq!(all.selected["pkg:npm/a@1"].uuid, "p"); + } + + #[test] + fn classification_rows() { + let results = vec![ + // NEW + offer("pkg:npm/new@1", "n1", "2026-01-01T00:00:00Z", &["high"]), + // ALREADY: recorded == selected + offer("pkg:npm/same@1", "s1", "2026-01-01T00:00:00Z", &["high"]), + // UPGRADE: the selection is newer than the recorded patch + offer("pkg:npm/up@1", "u-new", "2026-06-01T00:00:00Z", &["high"]), + offer("pkg:npm/up@1", "u-old", "2026-01-01T00:00:00Z", &["high"]), + // ALREADY (kept): the selection only wins on the uuid tiebreak + offer("pkg:npm/tie@1", "a-sel", "", &["high"]), + offer("pkg:npm/tie@1", "z-rec", "", &["high"]), + // UPGRADE: the recorded uuid is no longer offered + offer("pkg:npm/gone@1", "g2", "2026-01-01T00:00:00Z", &["low"]), + ]; + let offers = offers_from_results(&results, true); + let recorded = manifest(&[ + ("pkg:npm/same@1", "s1"), + ("pkg:npm/up@1", "u-old"), + ("pkg:npm/tie@1", "z-rec"), + ("pkg:npm/gone@1", "g1"), + ]); + let rows = classify(&offers, &RecordedIndex::new(Some(&recorded), &[]), ""); + assert_eq!( + classes(&rows), + vec![ + ( + "pkg:npm/gone@1".into(), + Recorded::Superseded { + old_uuid: "g1".into() + }, + "g2".into() + ), + ("pkg:npm/new@1".into(), Recorded::None, "n1".into()), + ("pkg:npm/same@1".into(), Recorded::Same, "s1".into()), + ( + "pkg:npm/tie@1".into(), + Recorded::Kept { + uuid: "z-rec".into() + }, + "z-rec".into() + ), + ( + "pkg:npm/up@1".into(), + Recorded::Superseded { + old_uuid: "u-old".into() + }, + "u-new".into() + ), + ] + ); + let updates = upgrades(&rows, &["pkg:npm/up@1".to_string()]); + assert_eq!( + updates + .iter() + .map(|u| (u.purl.as_str(), u.old_uuid.as_str(), u.new_uuid.as_str())) + .collect::>(), + [ + ("pkg:npm/gone@1", "g1", "g2"), + ("pkg:npm/up@1", "u-old", "u-new") + ] + ); + } + + #[test] + fn recorded_matches_percent_encoding_and_qualifier_twins() { + let results = vec![ + offer("pkg:npm/%40s/x@1", "e1", "", &["high"]), + offer("pkg:pypi/w@1?artifact_id=b", "w2", "", &["high"]), + ]; + let offers = offers_from_results(&results, true); + let recorded = manifest(&[ + ("pkg:npm/@s/x@1", "e1"), + ("pkg:pypi/w@1?artifact_id=a", "w1"), + ]); + let rows = classify(&offers, &RecordedIndex::new(Some(&recorded), &[]), ""); + assert_eq!(rows[0].candidate.recorded, Recorded::Same); + // The twin's recorded uuid is not offered for this twin: the late + // twin lands uncapped as an UPGRADE. + assert_eq!( + rows[1].candidate.recorded, + Recorded::Superseded { + old_uuid: "w1".into() + } + ); + } + + #[test] + fn several_recorded_uuids_prefer_the_selected_one_else_the_smallest() { + let results = vec![offer("pkg:pypi/w@1", "b", "", &["high"])]; + let offers = offers_from_results(&results, true); + let recorded = manifest(&[ + ("pkg:pypi/w@1?artifact_id=1", "c"), + ("pkg:pypi/w@1?artifact_id=2", "b"), + ]); + let rows = classify(&offers, &RecordedIndex::new(Some(&recorded), &[]), ""); + assert_eq!(rows[0].candidate.recorded, Recorded::Same); + let recorded = manifest(&[ + ("pkg:pypi/w@1?artifact_id=1", "d"), + ("pkg:pypi/w@1?artifact_id=2", "c"), + ]); + let rows = classify(&offers, &RecordedIndex::new(Some(&recorded), &[]), ""); + assert_eq!( + rows[0].candidate.recorded, + Recorded::Superseded { + old_uuid: "c".into() + } + ); + } + + #[test] + fn mentioned_uuids_finds_every_canonical_shape_once() { + let mut out = HashSet::new(); + mentioned_uuids( + "https://h/p/22222222-2222-4222-8222-222222222222/AAAAAAAA-1111-4111-8111-00000000000A/x.tgz \ + not-a-uuid 1234 socket-patch-bbbbbbbb-1111-4111-8111-00000000000b", + &mut out, + ); + let mut got: Vec<&str> = out.iter().map(String::as_str).collect(); + got.sort(); + assert_eq!( + got, + [ + "22222222-2222-4222-8222-222222222222", + "aaaaaaaa-1111-4111-8111-00000000000a", + "bbbbbbbb-1111-4111-8111-00000000000b", + ] + ); + } + + #[test] + fn a_lock_naming_the_selected_uuid_marks_the_row_already() { + let results = vec![ + offer( + "pkg:npm/a@1", + "aaaaaaaa-1111-4111-8111-00000000000a", + "", + &["high"], + ), + offer( + "pkg:npm/b@1", + "bbbbbbbb-1111-4111-8111-00000000000b", + "", + &["high"], + ), + ]; + let offers = offers_from_results(&results, true); + let mut rows = classify(&offers, &RecordedIndex::default(), ""); + mark_pinned( + &mut rows, + &["resolved: https://x/AAAAAAAA-1111-4111-8111-00000000000A/a.tgz"], + ); + assert_eq!(rows[0].candidate.recorded, Recorded::Same); + assert_eq!(rows[1].candidate.recorded, Recorded::None); + } + + #[test] + fn case_folded_pins_match_the_selection_spelling() { + // Discovery keys a nuget pin by the lowercased name; the API and + // the lockfile keep the original case. + let results = vec![ + offer("pkg:nuget/Newtonsoft.Json@13.0.3", "a-sel", "", &["high"]), + offer("pkg:nuget/Newtonsoft.Json@13.0.3", "z-pin", "", &["high"]), + ]; + let offers = offers_from_results(&results, true); + let index = RecordedIndex::new( + None, + &[("pkg:nuget/newtonsoft.json@13.0.3".into(), "z-pin".into())], + ); + let rows = classify(&offers, &index, ""); + assert_eq!( + rows[0].candidate.recorded, + Recorded::Kept { + uuid: "z-pin".into() + }, + "an equal sibling never replaces the pinned patch" + ); + assert_eq!(rows[0].writer.uuid, "z-pin"); + } + + #[test] + fn both_pinned_qualifier_twins_are_already() { + // Hosted pins are keyed by base purl; each twin carries its own + // patch. Neither may read as an UPGRADE on a converged repo. + let results = vec![ + offer( + "pkg:pypi/foo@1.0?artifact_id=sdist", + "u-sdist", + "", + &["high"], + ), + offer("pkg:pypi/foo@1.0?artifact_id=whl", "u-whl", "", &["high"]), + ]; + let offers = offers_from_results(&results, true); + let pins = vec![ + ("pkg:pypi/foo@1.0".to_string(), "u-sdist".to_string()), + ("pkg:pypi/foo@1.0".to_string(), "u-whl".to_string()), + ]; + let mut merged = PatchManifest::new(); + merged.patches.insert( + "pkg:pypi/foo@1.0".into(), + manifest(&[("x", "u-sdist")]).patches.remove("x").unwrap(), + ); + let rows = classify(&offers, &RecordedIndex::new(Some(&merged), &pins), ""); + assert!( + rows.iter().all(|r| r.candidate.recorded == Recorded::Same), + "{rows:?}" + ); + } + + #[test] + fn lookup_incomplete_only_when_a_new_package_could_be_missing() { + let index = RecordedIndex::new(Some(&manifest(&[("pkg:npm/rec@1", "u")])), &[]); + assert!(!lookup_incomplete(&index, &[], false)); + assert!( + lookup_incomplete(&index, &[], true), + "a failed batch hides unknown packages" + ); + assert!( + !lookup_incomplete(&index, &["pkg:npm/rec@1".into()], false), + "a recorded package's failed lookup cannot hide a NEW row" + ); + assert!(lookup_incomplete( + &index, + &["pkg:npm/other@1".into()], + false + )); + } + + #[test] + fn stage_carries_the_budget_and_renders_the_block() { + let configured = MaxNew { + value: Some(2), + source: MaxNewSource::Flag, + }; + let carry = RolloutCarry::new(configured, "/repo".into()); + let results = vec![ + offer("pkg:npm/a@1", "ua", "", &["critical"]), + offer("pkg:npm/b@1", "ub", "", &["low"]), + ]; + let offers = offers_from_results(&results, true); + let mut first = Stage::new(configured, Some(carry.clone()), Path::new("/repo/x")); + assert_eq!(first.project, "x"); + let rows = classify(&offers, &RecordedIndex::default(), &first.project); + first.plan(&rows, |_| true); + assert_eq!(carry.lock().remaining, Some(0)); + let results = vec![ + offer("pkg:npm/a@1", "ua", "", &["critical"]), + offer("pkg:npm/c@1", "uc", "", &["high"]), + ]; + let offers = offers_from_results(&results, true); + let mut second = Stage::new(configured, Some(carry.clone()), Path::new("/repo/y")); + let rows = classify(&offers, &RecordedIndex::default(), &second.project); + second.plan(&rows, |_| true); + let json = second.json(); + assert_eq!( + json["maxNewPatches"], + serde_json::json!({"value": 2, "source": "flag"}) + ); + assert_eq!( + json["counts"], + serde_json::json!({"new": 1, "deferred": 1, "upgrade": 0, "already": 0}) + ); + assert_eq!( + json["deferred"], + serde_json::json!([{ + "purl": "pkg:npm/c@1", "uuids": ["uc"], "severity": "high", + "advisoryCount": 1, "projects": ["y"], "rank": 2 + }]) + ); + assert_eq!( + serde_json::to_value(second.deferred_skips()).unwrap(), + serde_json::json!([{ + "purl": "pkg:npm/c@1", "uuid": "uc", "reason": "rollout_deferred", + "detail": "rank 2 in the rollout queue; a later scan adds it" + }]) + ); + let (line, next) = human(&second, false); + assert_eq!( + line.as_deref(), + Some( + "Rollout: 1 of 2 new patches applied (maxNewPatches=2 from --max-new-patches, \ + shared by this run's directories, 0 left); 0 upgrades, 0 already applied." + ) + ); + assert_eq!( + next, + [ + "1 new patch deferred; commit these changes and run scan again to apply the next 1.", + "Next up: c@1 (high)" + ] + ); + } + + #[test] + fn unlimited_runs_print_no_rollout_line() { + let results = vec![offer("pkg:npm/a@1", "ua", "", &["critical"])]; + let offers = offers_from_results(&results, true); + let mut stage = Stage::new(MaxNew::UNLIMITED, None, Path::new("/repo")); + let rows = classify(&offers, &RecordedIndex::default(), ""); + stage.plan(&rows, |_| true); + assert_eq!(human(&stage, false), (None, Vec::new())); + assert_eq!( + stage.json()["maxNewPatches"]["value"], + serde_json::Value::Null + ); + assert_eq!(stage.json()["counts"]["new"], 1); + } + + #[test] + fn a_zero_cap_says_how_to_add_the_deferred_patches() { + let results: Vec = ["a", "b", "c", "d"] + .iter() + .map(|n| offer(&format!("pkg:npm/{n}@1"), n, "", &["high"])) + .collect(); + let offers = offers_from_results(&results, true); + let zero = MaxNew { + value: Some(0), + source: MaxNewSource::File, + }; + let mut stage = Stage::new(zero, None, Path::new("/repo")); + let rows = classify(&offers, &RecordedIndex::default(), ""); + stage.plan(&rows, |_| true); + let (line, next) = human(&stage, true); + assert_eq!( + line.as_deref(), + Some( + "Rollout: 0 of 4 new patches would be applied (maxNewPatches=0 from socket.yml); \ + 0 upgrades, 0 already applied." + ) + ); + assert!( + next[0].starts_with("4 new patches would be deferred: maxNewPatches=0"), + "{next:?}" + ); + assert_eq!(next[1], "Next up: a@1 (high), b@1 (high), c@1 (high), …"); + } + + #[test] + fn incomplete_lookups_warn_only_when_they_deferred_something() { + let results = vec![offer("pkg:npm/a@1", "ua", "", &["critical"])]; + let offers = offers_from_results(&results, true); + let capped = MaxNew { + value: Some(3), + source: MaxNewSource::Flag, + }; + let mut stage = Stage::new(capped, None, Path::new("/repo")); + stage.incomplete = true; + assert!(!stage.may_admit_new()); + stage.plan(&classify(&offers, &RecordedIndex::default(), ""), |_| true); + let codes: Vec<&str> = stage.warnings().iter().map(|(c, _)| *c).collect(); + assert_eq!(codes, [ROLLOUT_INCOMPLETE_LOOKUP]); + let mut unlimited = Stage::new(MaxNew::UNLIMITED, None, Path::new("/repo")); + unlimited.incomplete = true; + assert!(unlimited.may_admit_new()); + unlimited.plan(&classify(&offers, &RecordedIndex::default(), ""), |_| true); + assert!(unlimited.warnings().is_empty()); + } +} diff --git a/crates/socket-patch-cli/src/commands/scan/rollout_args.rs b/crates/socket-patch-cli/src/commands/scan/rollout_args.rs new file mode 100644 index 00000000..47b7b7ca --- /dev/null +++ b/crates/socket-patch-cli/src/commands/scan/rollout_args.rs @@ -0,0 +1,128 @@ +//! `scan --max-new-patches` (work item B of the staged-rollout design, +//! `docs/design/staged-rollout.md` §5). + + +use clap::Args; +pub(crate) use socket_patch_core::rollout::stage::RolloutCarry; +use socket_patch_core::rollout::{resolve_max_new, MaxNew}; + +/// The env binding of `--max-new-patches`. Read by [`RolloutArgs::resolve`] +/// rather than clap's `env =`, because the rollout block reports whether +/// the value came from the flag or the environment. +pub use socket_patch_core::rollout::stage::MAX_NEW_PATCHES_ENV; + +/// A parsed `--max-new-patches` value; `None` is `none` (no cap). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct MaxNewPatches(pub Option); + +/// `N` (0..=4294967295) or `none`, case-insensitive. +pub fn parse_max_new_patches(s: &str) -> Result { + let s = s.trim(); + if s.is_empty() { + return Err("a number of patches (0 to 4294967295) or `none` is required".to_string()); + } + if s.eq_ignore_ascii_case("none") { + return Ok(MaxNewPatches(None)); + } + s.parse::() + .map(|n| MaxNewPatches(Some(n))) + .map_err(|_| format!("`{s}` is not a number of patches (0 to 4294967295) or `none`")) +} + +#[derive(Args, Clone, Default)] +pub struct RolloutArgs { + /// Add at most N patches to packages that have none yet, most severe + /// first; the rest are deferred to the next scan. Upgrades of patched + /// packages are not capped, and `0` adds only upgrades. `none` means no + /// cap. Also read from SOCKET_MAX_NEW_PATCHES + #[arg( + long = "max-new-patches", + value_name = "N|none", + value_parser = parse_max_new_patches + )] + pub max_new_patches: Option, + + /// The budget shared by the project directories of one invocation. + #[arg(skip)] + pub(crate) carry: Option, +} + +impl RolloutArgs { + /// The configured cap: the flag, then `env` (the + /// [`MAX_NEW_PATCHES_ENV`] value; empty is unset), then the socket.yml + /// value, then unlimited. A malformed env value is a usage error. + pub fn resolve(&self, env: Option<&str>, file: Option) -> Result { + // The flag wins outright: an env value it overrides is never read. + let env = env.filter(|_| self.max_new_patches.is_none()); + let env = match env.filter(|v| !v.trim().is_empty()) { + Some(raw) => Some( + parse_max_new_patches(raw) + .map_err(|e| format!("{MAX_NEW_PATCHES_ENV}: {e}"))? + .0, + ), + None => None, + }; + Ok(resolve_max_new( + self.max_new_patches.map(|v| v.0), + env, + file, + None, + )) + } + + /// [`Self::resolve`] against the process environment. + pub fn resolve_from_env(&self, file: Option) -> Result { + let env = std::env::var(MAX_NEW_PATCHES_ENV).ok(); + self.resolve(env.as_deref(), file) + } +} + + +#[cfg(test)] +mod tests { + use super::*; + use socket_patch_core::rollout::MaxNewSource; + + #[test] + fn parses_numbers_and_none() { + assert_eq!(parse_max_new_patches("5"), Ok(MaxNewPatches(Some(5)))); + assert_eq!(parse_max_new_patches("0"), Ok(MaxNewPatches(Some(0)))); + assert_eq!( + parse_max_new_patches("4294967295"), + Ok(MaxNewPatches(Some(u32::MAX))) + ); + assert_eq!(parse_max_new_patches("NONE"), Ok(MaxNewPatches(None))); + for bad in ["", "-1", "4294967296", "five", "1.5", "all"] { + assert!(parse_max_new_patches(bad).is_err(), "{bad:?}"); + } + } + + #[test] + fn flag_beats_env_beats_file() { + let flag = RolloutArgs { + max_new_patches: Some(MaxNewPatches(Some(1))), + carry: None, + }; + let none = RolloutArgs::default(); + let got = flag.resolve(Some("2"), Some(3)).unwrap(); + assert_eq!((got.value, got.source), (Some(1), MaxNewSource::Flag)); + let got = none.resolve(Some("2"), Some(3)).unwrap(); + assert_eq!((got.value, got.source), (Some(2), MaxNewSource::Env)); + let got = none.resolve(Some("none"), Some(3)).unwrap(); + assert_eq!((got.value, got.source), (None, MaxNewSource::Env)); + let got = none.resolve(Some(""), Some(3)).unwrap(); + assert_eq!((got.value, got.source), (Some(3), MaxNewSource::File)); + let got = none.resolve(None, None).unwrap(); + assert_eq!((got.value, got.source), (None, MaxNewSource::Default)); + let err = none.resolve(Some("lots"), None).unwrap_err(); + assert!(err.starts_with("SOCKET_MAX_NEW_PATCHES: "), "{err}"); + let got = none.resolve(Some(" "), Some(3)).unwrap(); + assert_eq!(got.source, MaxNewSource::File, "whitespace is unset"); + let got = flag.resolve(Some("lots"), None).unwrap(); + assert_eq!( + got.source, + MaxNewSource::Flag, + "an overridden env value is not parsed" + ); + } +} diff --git a/crates/socket-patch-cli/src/commands/scan/vendor_flow.rs b/crates/socket-patch-cli/src/commands/scan/vendor_flow.rs index 9b2e11e8..2f388504 100644 --- a/crates/socket-patch-cli/src/commands/scan/vendor_flow.rs +++ b/crates/socket-patch-cli/src/commands/scan/vendor_flow.rs @@ -41,9 +41,10 @@ use crate::json_envelope::{Command as EnvelopeCommand, Envelope}; use crate::ui::{plural, print_json}; use super::gc::{gc_json, print_gc_vendored_line, run_apply_gc}; +use super::rollout::Stage; use super::{ - discover_selected, download_params, embed_vex_into_json, emit_discovery_error_json, - push_run_warning, ScanArgs, + classified_rows, discover_selected, download_params, embed_vex_into_json, + emit_discovery_error_json, finish_rollout_json, push_run_warning, writers_of, ScanArgs, }; /// Run-level warning: a `.socket/manifest.json` record for a purl the @@ -124,6 +125,29 @@ pub(crate) async fn preview_vendor_json( serde_json::json!({ "dryRun": true, "patches": patches }) } +/// The purls of `selected` the wet run's Bun or vlt preflight would refuse +/// before any download (the `would_refuse` rows of +/// [`preview_vendor_json`]): the vendored planning pass, so a refused NEW +/// patch holds no rollout slot. +pub(super) async fn preflight_refused_purls( + cwd: &Path, + selected: &[PatchSearchResult], +) -> HashSet { + let state = load_state(cwd).await; + let refusal = + bun_vendor_preflight_with_ledger(cwd, selected, state.as_ref().map(|s| &s.entries)).await; + let vlt_refusals = + vlt_vendor_preflight_selected(cwd, selected, state.as_ref().map(|s| &s.entries)).await; + selected + .iter() + .filter(|p| { + refusal.as_ref().is_some_and(|r| r.applies_to(&p.purl)) + || vlt_refusal_for(&vlt_refusals, &p.purl).is_some() + }) + .map(|p| p.purl.clone()) + .collect() +} + /// Human rendering of the vendored dry-run preview's `would_refuse` records /// (see [`preview_vendor_json`]): the count line above it still says /// "would download and vendor", so name what the wet run would refuse and @@ -452,6 +476,9 @@ async fn run_vendor_json_path( use_public_proxy: bool, all_packages_with_patches: &[BatchPackagePatches], can_access_paid_patches: bool, + recorded: &super::rollout::RecordedState<'_>, + batch_failed: bool, + stage: &mut Stage, policy: &super::policy::ScanPolicy, result: &mut serde_json::Value, manifest_path: &Path, @@ -470,24 +497,44 @@ async fn run_vendor_json_path( // Same discovery as `--apply`. Vendored purls are NOT filtered here — // re-vendoring a stale uuid is the point of the flag (same-uuid re-runs // land on the backend's `already_vendored` skip). - let selected: Vec = match discover_selected( + let discovered = match discover_selected( api_client, all_packages_with_patches, can_access_paid_patches, policy, false, false, + false, telemetry, Some(&mut *result), ) .await { - Ok(offers) => offers.selected.into_values().collect(), + Ok(d) => d, Err((code, message)) => { emit_discovery_error_json(result, &message); return code; } }; + let rows = classified_rows( + stage, + &discovered, + recorded, + batch_failed, + all_packages_with_patches, + Some(&mut *result), + ); + // The planning pass: a patch the preflight refuses holds no slot (it + // still reaches the engine, which reports the refusal). + let writers = writers_of(&rows); + let refused = preflight_refused_purls(&args.common.cwd, &writers).await; + stage.plan(&rows, |r| !refused.contains(&r.writer.purl)); + let deferred = stage.deferred_keys(); + let selected: Vec = writers + .into_iter() + .filter(|p| !deferred.contains(&(p.purl.clone(), p.uuid.clone()))) + .collect(); + finish_rollout_json(stage, result); if args.common.dry_run { // No downloads, no backends: classify against the ledger @@ -557,6 +604,9 @@ async fn run_vendor_json_path( "code": code, "message": message, }); + if let Some(obj) = result.as_object_mut() { + obj.remove("rollout"); + } print_json(result); return 1; } @@ -781,6 +831,9 @@ pub(super) fn boxed_vendor_json_path<'a>( use_public_proxy: bool, all_packages_with_patches: &'a [BatchPackagePatches], can_access_paid_patches: bool, + recorded: &'a super::rollout::RecordedState<'a>, + batch_failed: bool, + stage: &'a mut Stage, policy: &'a super::policy::ScanPolicy, result: &'a mut serde_json::Value, manifest_path: &'a Path, @@ -799,6 +852,9 @@ pub(super) fn boxed_vendor_json_path<'a>( use_public_proxy, all_packages_with_patches, can_access_paid_patches, + recorded, + batch_failed, + stage, policy, result, manifest_path, diff --git a/crates/socket-patch-cli/src/lib.rs b/crates/socket-patch-cli/src/lib.rs index 297fb9f3..32e01e9e 100644 --- a/crates/socket-patch-cli/src/lib.rs +++ b/crates/socket-patch-cli/src/lib.rs @@ -182,6 +182,7 @@ fn short_help_hidden_own(sub: &str) -> &'static [&'static str] { "vex_compact", "no_socket_yml", "min_severity", + "max_new_patches", ], "get" => &["id", "cve", "ghsa", "package", "save_only", "all_releases"], "vex" => &["doc_id", "compact"], diff --git a/crates/socket-patch-cli/tests/cli_parse_scan.rs b/crates/socket-patch-cli/tests/cli_parse_scan.rs index d6af0472..ab81fa6e 100644 --- a/crates/socket-patch-cli/tests/cli_parse_scan.rs +++ b/crates/socket-patch-cli/tests/cli_parse_scan.rs @@ -45,6 +45,7 @@ const SCAN_ENV_VARS: &[&str] = &[ "SOCKET_JSON", "SOCKET_LOCK_TIMEOUT", "SOCKET_MANIFEST_PATH", + "SOCKET_MAX_NEW_PATCHES", "SOCKET_MIN_SEVERITY", "SOCKET_NO_SOCKET_YML", "SOCKET_NO_TRUST_LOCKFILE_CONFIG", @@ -521,6 +522,11 @@ fn scan_json_empty_cwd_emits_updates_key() { // v4 duality rework: the positional PATH globs are echoed on every // scan envelope, always present (empty when no scoping was given). "paths": [], + "rollout": { + "maxNewPatches": { "value": null, "source": "default" }, + "counts": { "new": 0, "deferred": 0, "upgrade": 0, "already": 0 }, + "deferred": [], + }, // v5: a bare scan runs hosted mode, so its result nests here. "redirect": { "mode": "hosted", @@ -871,6 +877,61 @@ fn no_vlt_install_cleanup_flag_and_env_parse() { } } +// ── --max-new-patches (staged rollout) ─────────────────────────────────── + +#[test] +#[serial_test::serial] +fn max_new_patches_defaults_to_unset() { + let args = parse_scan(&[]); + assert_eq!(args.rollout.max_new_patches, None); +} + +#[test] +#[serial_test::serial] +fn max_new_patches_takes_a_count_or_none() { + use socket_patch_cli::commands::scan::rollout_args::MaxNewPatches; + for (raw, want) in [ + ("5", Some(5)), + ("0", Some(0)), + ("4294967295", Some(u32::MAX)), + ("none", None), + ("NONE", None), + ] { + let args = parse_scan(&["--max-new-patches", raw]); + assert_eq!(args.rollout.max_new_patches, Some(MaxNewPatches(want)), "{raw}"); + } +} + +#[test] +#[serial_test::serial] +fn max_new_patches_rejects_malformed_values() { + for raw in ["-1", "4294967296", "five", "", "all"] { + let err = match try_parse_scan(&[&format!("--max-new-patches={raw}")]) { + Ok(_) => panic!("{raw:?} must not parse"), + Err(e) => e, + }; + assert_eq!(err.exit_code(), 2, "{raw:?}: usage error"); + } +} + +#[test] +fn max_new_patches_env_is_read_at_run_time() { + // The env binding is resolved by `RolloutArgs::resolve` (the rollout + // block reports flag vs env), not by clap: empty is unset, malformed is + // a usage error, and the flag wins. + use socket_patch_cli::commands::scan::rollout_args::{MaxNewPatches, RolloutArgs}; + use socket_patch_core::rollout::MaxNewSource; + let unset = RolloutArgs::default(); + let got = unset.resolve(Some("7"), None).unwrap(); + assert_eq!((got.value, got.source), (Some(7), MaxNewSource::Env)); + let got = unset.resolve(Some(""), None).unwrap(); + assert_eq!((got.value, got.source), (None, MaxNewSource::Default)); + assert!(unset.resolve(Some("x"), None).is_err()); + let mut flag = RolloutArgs::default(); + flag.max_new_patches = Some(MaxNewPatches(Some(1))); + let got = flag.resolve(Some("7"), None).unwrap(); + assert_eq!((got.value, got.source), (Some(1), MaxNewSource::Flag)); +} /// Parse `scan` under a clean env plus `env`, restoring it afterwards. fn parse_scan_with_env(extra: &[&str], env: &[(&str, &str)]) -> Result { with_clean_env(|| { diff --git a/crates/socket-patch-cli/tests/hosted_memory_common/mod.rs b/crates/socket-patch-cli/tests/hosted_memory_common/mod.rs index 6d8b2427..cecd456b 100644 --- a/crates/socket-patch-cli/tests/hosted_memory_common/mod.rs +++ b/crates/socket-patch-cli/tests/hosted_memory_common/mod.rs @@ -305,7 +305,7 @@ pub struct DiskRun { /// node / pipenv / gem subprocesses), `HOME` and the language caches at /// empty directories, no socket-cli config, no telemetry. pub fn run_disk(server: &MockServer, files: &BTreeMap>, dry_run: bool) -> DiskRun { - run_disk_in(server, files, "", dry_run) + run_disk_with(server, files, dry_run, &[]) } /// [`run_disk`] with `--cwd` at the repo-relative `cwd_rel` of a checkout @@ -317,6 +317,76 @@ pub fn run_disk_in( cwd_rel: &str, dry_run: bool, ) -> DiskRun { + run_disk_in_with(server, files, cwd_rel, dry_run, &[]) +} + +/// A human (non-`--json`) disk scan with `args`: exit code, stdout, and the +/// files it changed. +pub fn run_disk_args( + server: &MockServer, + files: &BTreeMap>, + args: &[&str], +) -> (i32, String, BTreeMap>) { + let (checkout, _home, mut cmd) = disk_command(server, files, ""); + cmd.args(args); + let output = cmd.output().expect("spawn socket-patch"); + let stdout = String::from_utf8_lossy(&output.stdout).to_string(); + let after = read_tree(checkout.path()); + let changed = after + .into_iter() + .filter(|(rel, bytes)| files.get(rel) != Some(bytes)) + .collect(); + (output.status.code().unwrap_or(-1), stdout, changed) +} + +/// [`run_disk`] with extra `scan --json` arguments. +pub fn run_disk_with( + server: &MockServer, + files: &BTreeMap>, + dry_run: bool, + extra: &[&str], +) -> DiskRun { + run_disk_in_with(server, files, "", dry_run, extra) +} + +fn run_disk_in_with( + server: &MockServer, + files: &BTreeMap>, + cwd_rel: &str, + dry_run: bool, + extra: &[&str], +) -> DiskRun { + let (checkout, _home, mut cmd) = disk_command(server, files, cwd_rel); + cmd.arg("--json"); + if dry_run { + cmd.arg("--dry-run"); + } + cmd.args(extra); + let output = cmd.output().expect("spawn socket-patch"); + let stdout = String::from_utf8_lossy(&output.stdout).to_string(); + let stderr = String::from_utf8_lossy(&output.stderr).to_string(); + let envelope: Value = serde_json::from_str(&stdout) + .unwrap_or_else(|e| panic!("disk --json output is not JSON ({e}):\n{stdout}\n{stderr}")); + let after = read_tree(checkout.path()); + let changed = after + .into_iter() + .filter(|(rel, bytes)| files.get(rel) != Some(bytes)) + .collect(); + DiskRun { + envelope, + changed, + stderr, + } +} + +/// The scrubbed `scan --mode hosted` command over a checkout of `files`, +/// run at `cwd_rel` (the returned tempdirs, the checkout and `HOME`, must +/// outlive the run). +fn disk_command( + server: &MockServer, + files: &BTreeMap>, + cwd_rel: &str, +) -> (tempfile::TempDir, tempfile::TempDir, std::process::Command) { let checkout = tempfile::tempdir().unwrap(); let home = tempfile::tempdir().unwrap(); for (rel, bytes) in files { @@ -361,7 +431,6 @@ pub fn run_disk_in( "scan", "--mode", "hosted", - "--json", "--yes", "--cwd", cwd.to_str().unwrap(), @@ -372,24 +441,7 @@ pub fn run_disk_in( "--api-url", &server.uri(), ]); - if dry_run { - cmd.arg("--dry-run"); - } - let output = cmd.output().expect("spawn socket-patch"); - let stdout = String::from_utf8_lossy(&output.stdout).to_string(); - let stderr = String::from_utf8_lossy(&output.stderr).to_string(); - let envelope: Value = serde_json::from_str(&stdout) - .unwrap_or_else(|e| panic!("disk --json output is not JSON ({e}):\n{stdout}\n{stderr}")); - let after = read_tree(checkout.path()); - let changed = after - .into_iter() - .filter(|(rel, bytes)| files.get(rel) != Some(bytes)) - .collect(); - DiskRun { - envelope, - changed, - stderr, - } + (checkout, home, cmd) } /// The engine's changed files (text and binary) as bytes. diff --git a/crates/socket-patch-cli/tests/hosted_memory_rollout.rs b/crates/socket-patch-cli/tests/hosted_memory_rollout.rs new file mode 100644 index 00000000..ccaf92cc --- /dev/null +++ b/crates/socket-patch-cli/tests/hosted_memory_rollout.rs @@ -0,0 +1,639 @@ +//! The staged rollout (`maxNewPatches`) in the in-memory hosted engine, +//! held to the disk `scan --mode hosted --json --max-new-patches` run: one +//! root admits and defers the same rows run after run until converged; two +//! roots show memory's run-wide budget next to disk's per-directory one +//! (they differ by design, §5.2); a committed manifest, vendor ledger or +//! lockfile pin counts as recorded, never NEW. + +use std::collections::BTreeMap; + +use serde_json::{json, Value}; +use socket_patch_cli::hosted_memory::{HostedScanOptions, HostedScanOutput, MaxNewPatchesOption}; +use wiremock::matchers::{method, path, path_regex}; +use wiremock::{Mock, MockServer, Request, Respond, ResponseTemplate}; + +#[path = "hosted_memory_common/mod.rs"] +mod common; + +use common::{build_input, run_disk_args, run_disk_with, run_engine, ORG}; + +const TOKEN: &str = "22222222-2222-4222-8222-222222222222"; + +/// `(name, severities)`: one advisory per severity. +const PACKAGES: [(&str, &[&str]); 5] = [ + ("mem-a", &["low"]), + ("mem-b", &["critical"]), + ("mem-c", &["high"]), + ("mem-d", &["medium"]), + ("mem-e", &["critical", "high"]), +]; + +/// Most severe first: severity, then advisory count, then name. +const ORDER: [&str; 5] = ["mem-e", "mem-b", "mem-c", "mem-d", "mem-a"]; + +fn uuid(name: &str) -> String { + let n = PACKAGES.iter().position(|(p, _)| *p == name).unwrap() + 1; + format!("{n:08x}-3333-4333-8333-{n:012x}") +} + +fn purl(name: &str) -> String { + format!("pkg:npm/{name}@1.0.0") +} + +fn url(name: &str) -> String { + format!( + "https://patch.socket.dev/patch/npm/{name}/1.0.0/{TOKEN}/{}/{name}-1.0.0.tgz", + uuid(name) + ) +} + +fn name_of(purl: &str) -> Option<&'static str> { + PACKAGES + .iter() + .map(|(n, _)| *n) + .find(|n| purl.contains(&format!("{n}@")) || purl.contains(&format!("{n}%40"))) +} + +fn vulns(name: &str) -> Value { + let sevs = PACKAGES.iter().find(|(n, _)| *n == name).unwrap().1; + let mut map = serde_json::Map::new(); + for (i, sev) in sevs.iter().enumerate() { + map.insert( + format!("GHSA-{name}-{i}"), + json!({ "cves": [], "summary": "s", "severity": sev, "description": "d" }), + ); + } + Value::Object(map) +} + +struct Batch; +impl Respond for Batch { + fn respond(&self, request: &Request) -> ResponseTemplate { + let body: Value = serde_json::from_slice(&request.body).unwrap_or(Value::Null); + let packages: Vec = body["components"] + .as_array() + .into_iter() + .flatten() + .filter_map(|c| { + let p = c["purl"].as_str()?; + let name = name_of(p)?; + Some(json!({ "purl": p, "patches": [{ + "uuid": uuid(name), "purl": p, "tier": "free", "cveIds": [], + "ghsaIds": [format!("GHSA-{name}-0")], "severity": "high", "title": name + }]})) + }) + .collect(); + ResponseTemplate::new(200) + .set_body_json(json!({ "packages": packages, "canAccessPaidPatches": false })) + } +} + +struct ByPackage; +impl Respond for ByPackage { + fn respond(&self, request: &Request) -> ResponseTemplate { + let patches: Vec = name_of(request.url.path()) + .map(|name| { + vec![json!({ + "uuid": uuid(name), "purl": purl(name), + "publishedAt": "2024-01-01T00:00:00Z", + "description": name, "license": "MIT", "tier": "free", + "vulnerabilities": vulns(name), + })] + }) + .unwrap_or_default(); + ResponseTemplate::new(200) + .set_body_json(json!({ "patches": patches, "canAccessPaidPatches": false })) + } +} + +struct References; +impl Respond for References { + fn respond(&self, request: &Request) -> ResponseTemplate { + let body: Value = serde_json::from_slice(&request.body).unwrap_or(Value::Null); + let mut results = serde_json::Map::new(); + for requested in body["uuids"].as_array().into_iter().flatten() { + let Some(requested) = requested.as_str() else { + continue; + }; + if let Some((name, _)) = PACKAGES.iter().find(|(n, _)| uuid(n) == requested) { + results.insert( + requested.to_string(), + json!({ + "status": "granted", "url": url(name), "purl": null, + "artifacts": [{ "kind": "tarball", "url": url(name), + "integrity": { "sha512": format!("sha512-PATCHED{name}==") } }], + "registryOverride": null + }), + ); + } + } + ResponseTemplate::new(200).set_body_json(json!({ "results": results })) + } +} + +struct View; +impl Respond for View { + fn respond(&self, request: &Request) -> ResponseTemplate { + let requested = request.url.path().rsplit('/').next().unwrap_or(""); + match PACKAGES.iter().find(|(n, _)| uuid(n) == requested) { + Some((name, _)) => ResponseTemplate::new(200).set_body_json(json!({ + "uuid": uuid(name), "purl": purl(name), "publishedAt": "2024-01-01T00:00:00Z", + "files": { "package/index.js": { "beforeHash": "a".repeat(64), "afterHash": "b".repeat(64) } }, + "vulnerabilities": vulns(name), + "description": name, "license": "MIT", "tier": "free" + })), + None => ResponseTemplate::new(404), + } + } +} + +async fn mount(server: &MockServer) { + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{ORG}/patches/batch"))) + .respond_with(Batch) + .mount(server) + .await; + Mock::given(method("GET")) + .and(path_regex(format!( + "^/v0/orgs/{ORG}/patches/by-package/.+$" + ))) + .respond_with(ByPackage) + .mount(server) + .await; + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{ORG}/patches/package"))) + .respond_with(References) + .mount(server) + .await; + Mock::given(method("GET")) + .and(path_regex(format!("^/v0/orgs/{ORG}/patches/view/.+$"))) + .respond_with(View) + .mount(server) + .await; +} + +/// A v3 package-lock at `dir` locking `names` (no install needed: both +/// engines read the lock). +fn lock(files: &mut BTreeMap>, dir: &str, names: &[&str]) { + let prefix = if dir.is_empty() { + String::new() + } else { + format!("{dir}/") + }; + let deps: serde_json::Map = names + .iter() + .map(|n| (n.to_string(), json!("1.0.0"))) + .collect(); + let mut packages = serde_json::Map::new(); + packages.insert( + String::new(), + json!({ "name": "c", "version": "0.0.0", "dependencies": deps }), + ); + for n in names { + packages.insert( + format!("node_modules/{n}"), + json!({ + "version": "1.0.0", + "resolved": format!("https://registry.npmjs.org/{n}/-/{n}-1.0.0.tgz"), + "integrity": "sha512-UPSTREAM==" + }), + ); + } + let mut text = serde_json::to_vec_pretty(&json!({ + "name": "c", "version": "0.0.0", "lockfileVersion": 3, "requires": true, + "packages": packages + })) + .unwrap(); + text.push(b'\n'); + files.insert(format!("{prefix}package-lock.json"), text); + files.insert( + format!("{prefix}package.json"), + serde_json::to_vec(&json!({ "name": "c", "version": "0.0.0", "dependencies": deps })) + .unwrap(), + ); +} + +fn options(cap: Option) -> HostedScanOptions { + HostedScanOptions { + org_slug: ORG.to_string(), + max_new_patches: cap.map(|n| MaxNewPatchesOption(Some(n))), + ..HostedScanOptions::default() + } +} + +async fn memory( + server: &MockServer, + files: &BTreeMap>, + o: HostedScanOptions, +) -> HostedScanOutput { + run_engine(server, build_input(files, &[], o)).await +} + +/// [`memory`] through two-phase path selection, as a host with a root +/// socket.yml runs it: select with the policy text, fetch what selection +/// asks for, then pass its `policyPaths` / `policySha256` to the session. +async fn memory_selected( + server: &MockServer, + files: &BTreeMap>, + mut o: HostedScanOptions, +) -> HostedScanOutput { + use socket_patch_cli::hosted_memory::{select_paths, PolicyFileInput, SelectOptions, TreeEntryInput}; + let entries: Vec = files + .iter() + .map(|(p, bytes)| TreeEntryInput { + path: p.clone(), + mode: "100644".into(), + kind: "blob".into(), + size: Some(bytes.len() as u64), + }) + .collect(); + let policy_files: Vec = ["socket.yml", "socket.yaml"] + .iter() + .filter_map(|name| { + files.get(*name).map(|bytes| PolicyFileInput { + path: name.to_string(), + text: Some(String::from_utf8(bytes.clone()).unwrap()), + missing: None, + }) + }) + .collect(); + let selection = select_paths( + &entries, + &SelectOptions { + policy_files: Some(policy_files), + no_socket_yml: o.no_socket_yml, + ..SelectOptions::default() + }, + ); + assert!(selection.policy_error.is_none(), "{:?}", selection.policy_error); + let fetched: BTreeMap> = selection + .fetch_text + .iter() + .chain(selection.fetch_binary.iter()) + .map(|p| (p.clone(), files[p].clone())) + .collect(); + let present: Vec<&str> = selection.present_only.iter().map(String::as_str).collect(); + o.policy_paths = Some(selection.policy_paths.clone()); + o.policy_sha256 = selection.policy_sha256.clone(); + run_engine(server, build_input(&fetched, &present, o)).await +} + +/// `files` with the engine's changed files applied: the next run's input. +fn apply(files: &BTreeMap>, out: &HostedScanOutput) -> BTreeMap> { + let mut next = files.clone(); + for f in &out.changed_files { + next.insert(f.path.clone(), f.content.clone().into_bytes()); + } + next +} + +fn pinned(files: &BTreeMap>, lock_path: &str) -> Vec<&'static str> { + let text = String::from_utf8_lossy(&files[lock_path]).into_owned(); + ORDER + .iter() + .copied() + .filter(|n| text.contains(&uuid(n))) + .collect() +} + +#[tokio::test] +async fn one_root_disk_and_memory_admit_and_defer_the_same_rows_until_converged() { + let server = MockServer::start().await; + mount(&server).await; + let mut files = BTreeMap::new(); + let names: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + lock(&mut files, "", &names); + + for (run, expected) in [&ORDER[..2], &ORDER[..4], &ORDER[..5], &ORDER[..5]] + .iter() + .enumerate() + { + let disk = run_disk_with(&server, &files, false, &["--max-new-patches", "2"]); + let mem = memory(&server, &files, options(Some(2))).await; + let project = &mem.projects[0]; + assert!(project.error.is_none(), "{:?}", project.error); + assert_eq!( + mem.rollout, + disk.envelope["rollout"], + "run {}: the rollout blocks agree\nstderr: {}", + run + 1, + disk.stderr + ); + assert_eq!( + project.redirect, + disk.envelope["redirect"], + "run {}: the redirect blocks agree", + run + 1 + ); + let next = apply(&files, &mem); + let memory_changed: Vec<&String> = mem.changed_files.iter().map(|f| &f.path).collect(); + assert_eq!( + memory_changed, + disk.changed.keys().collect::>(), + "run {}: the same files change", + run + 1 + ); + for (rel, bytes) in &disk.changed { + assert_eq!( + String::from_utf8_lossy(&next[rel]), + String::from_utf8_lossy(bytes), + "run {}: {rel}", + run + 1 + ); + } + assert_eq!(pinned(&next, "package-lock.json"), expected.to_vec()); + let deferred: Vec<&str> = project.deferred.iter().map(|d| d.purl.as_str()).collect(); + let want: Vec = ORDER[expected.len()..].iter().map(|n| purl(n)).collect(); + assert_eq!( + deferred, + want.iter().map(String::as_str).collect::>() + ); + let skipped: Vec<&str> = project + .skipped + .iter() + .filter(|s| s.reason == "rollout_deferred") + .map(|s| s.purl.as_str()) + .collect(); + assert_eq!(skipped, deferred, "deferred rows are mirrored in skipped[]"); + files = next; + } + assert_eq!( + memory(&server, &files, options(Some(2))).await.rollout["counts"], + json!({ "new": 0, "deferred": 0, "upgrade": 0, "already": 5 }) + ); +} + +#[tokio::test] +async fn two_roots_spend_one_budget_in_memory_and_one_per_directory_on_disk() { + let server = MockServer::start().await; + mount(&server).await; + let mut files = BTreeMap::new(); + lock(&mut files, "a", &["mem-a", "mem-b"]); + lock(&mut files, "b", &["mem-c", "mem-d", "mem-e"]); + + // Memory: one run-wide queue, e (b/), b (a/) first. + let mem = memory(&server, &files, options(Some(2))).await; + assert_eq!( + mem.rollout["counts"], + json!({ "new": 2, "deferred": 3, "upgrade": 0, "already": 0 }) + ); + let next = apply(&files, &mem); + assert_eq!(pinned(&next, "a/package-lock.json"), ["mem-b"]); + assert_eq!(pinned(&next, "b/package-lock.json"), ["mem-e"]); + let mut ranks: Vec<(String, u32)> = mem + .projects + .iter() + .flat_map(|p| p.deferred.iter().map(|d| (d.purl.clone(), d.rank))) + .collect(); + ranks.sort(); + assert_eq!( + ranks, + [(purl("mem-a"), 5), (purl("mem-c"), 3), (purl("mem-d"), 4)], + "ranks are run-wide" + ); + + // Disk: the directories spend the budget in sorted order, so `a/` + // takes both slots before `b/` is visited. + let (code, stdout, changed) = + run_disk_args(&server, &files, &["--max-new-patches", "2", "a", "b"]); + assert_eq!(code, 0, "{stdout}"); + let mut disk_files = files.clone(); + disk_files.extend(changed); + assert_eq!( + pinned(&disk_files, "a/package-lock.json"), + ["mem-b", "mem-a"] + ); + assert!( + pinned(&disk_files, "b/package-lock.json").is_empty(), + "{stdout}" + ); +} + +/// The socket.yml policy and the cap together (§9.3): `includePaths` +/// keeps `legacy/` out, `minSeverity: high` leaves only e, b and c +/// eligible, and `maxNewPatches: 2` paces them; both engines converge in +/// two runs. `--no-socket-yml` then drops the file's floor, paths and cap +/// but keeps a flag cap, and each engine spends it in its own scope. +#[tokio::test] +async fn socket_yml_policy_and_cap_converge_on_disk_and_in_memory() { + let server = MockServer::start().await; + mount(&server).await; + let mut files = BTreeMap::new(); + files.insert( + "socket.yml".to_string(), + b"version: 2\npatches:\n includePaths: [\"/apps/\"]\n minSeverity: high\n maxNewPatches: 2\n" + .to_vec(), + ); + lock(&mut files, "apps/one", &["mem-a", "mem-b", "mem-c", "mem-d", "mem-e"]); + lock(&mut files, "apps/two", &["mem-b", "mem-c", "mem-d"]); + lock(&mut files, "legacy", &["mem-b", "mem-e"]); + let dirs = ["apps/one", "apps/two", "legacy"]; + let pins = |f: &BTreeMap>| -> Vec> { + dirs.iter() + .map(|d| pinned(f, &format!("{d}/package-lock.json"))) + .collect() + }; + let expected: [Vec>; 3] = [ + vec![vec!["mem-e", "mem-b"], vec!["mem-b"], vec![]], + vec![vec!["mem-e", "mem-b", "mem-c"], vec!["mem-b", "mem-c"], vec![]], + vec![vec!["mem-e", "mem-b", "mem-c"], vec!["mem-b", "mem-c"], vec![]], + ]; + + let mut mem_files = files.clone(); + let mut disk_files = files.clone(); + for (run, want) in expected.iter().enumerate() { + let mem = memory_selected(&server, &mem_files, options(None)).await; + assert_eq!( + mem.rollout["maxNewPatches"], + json!({ "value": 2, "source": "file" }), + "run {}", + run + 1 + ); + assert_eq!(mem.policy.as_ref().map(|p| p["source"].clone()), Some(json!("file"))); + mem_files = apply(&mem_files, &mem); + assert_eq!(&pins(&mem_files), want, "memory run {}", run + 1); + + let (code, stdout, changed) = run_disk_args(&server, &disk_files, &dirs); + assert_eq!(code, 0, "{stdout}"); + assert!( + stdout.contains("maxNewPatches=2 from socket.yml"), + "disk run {}: {stdout}", + run + 1 + ); + disk_files.extend(changed); + assert_eq!(&pins(&disk_files), want, "disk run {}: {stdout}", run + 1); + } + + // Disk visits the directories in sorted order: `apps/one` spends the + // one slot on d, `apps/two` gets d free, and `legacy` defers e and b. + let (code, stdout, changed) = run_disk_args( + &server, + &disk_files, + &["--no-socket-yml", "--max-new-patches", "1", "apps/one", "apps/two", "legacy"], + ); + assert_eq!(code, 0, "{stdout}"); + disk_files.extend(changed); + assert_eq!( + pins(&disk_files), + [ + vec!["mem-e", "mem-b", "mem-c", "mem-d"], + vec!["mem-b", "mem-c", "mem-d"], + vec![], + ], + "{stdout}" + ); + + // Memory ranks every root in one queue: legacy's critical e goes first. + let mut o = options(Some(1)); + o.no_socket_yml = Some(true); + let mem = memory_selected(&server, &mem_files, o).await; + assert_eq!( + mem.rollout["maxNewPatches"], + json!({ "value": 1, "source": "flag" }) + ); + mem_files = apply(&mem_files, &mem); + assert_eq!( + pins(&mem_files), + [ + vec!["mem-e", "mem-b", "mem-c"], + vec!["mem-b", "mem-c"], + vec!["mem-e"], + ] + ); +} + +#[tokio::test] +async fn memory_counts_a_committed_manifest_vendor_entry_or_pin_as_recorded() { + let server = MockServer::start().await; + mount(&server).await; + let mut files = BTreeMap::new(); + let names: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + lock(&mut files, "", &names); + // mem-e is recorded by the agent manifest, mem-b by the vendor ledger. + files.insert( + ".socket/manifest.json".into(), + serde_json::to_vec(&json!({ "patches": { purl("mem-e"): { + "uuid": uuid("mem-e"), "exportedAt": "", "files": {}, "vulnerabilities": {}, + "description": "", "license": "", "tier": "free" + }}})) + .unwrap(), + ); + files.insert( + ".socket/vendor/state.json".into(), + serde_json::to_vec(&json!({ + "version": 1, + "entries": { purl("mem-b"): { + "ecosystem": "npm", "uuid": uuid("mem-b"), "basePurl": purl("mem-b"), + "artifact": { "path": format!(".socket/vendor/npm/{}/mem-b-1.0.0.tgz", uuid("mem-b")) }, + "wiring": [] + }} + })) + .unwrap(), + ); + let mem = memory(&server, &files, options(Some(1))).await; + assert_eq!( + mem.rollout["counts"], + json!({ "new": 1, "deferred": 2, "upgrade": 0, "already": 2 }), + "{:#}", + mem.rollout + ); + let deferred: Vec<&str> = mem.projects[0] + .deferred + .iter() + .map(|d| d.purl.as_str()) + .collect(); + assert_eq!(deferred, [purl("mem-d"), purl("mem-a")]); + // mem-e (manifest) re-confirms its pin; mem-b (vendored) is refused as a + // takeover; mem-c is the one NEW patch admitted. + let next = apply(&files, &mem); + assert_eq!(pinned(&next, "package-lock.json"), ["mem-e", "mem-c"]); + // The rerun lands the next one. + let again = memory(&server, &next, options(Some(1))).await; + assert_eq!( + again.rollout["counts"], + json!({ "new": 1, "deferred": 1, "upgrade": 0, "already": 3 }) + ); + assert_eq!( + pinned(&apply(&next, &again), "package-lock.json"), + ["mem-e", "mem-c", "mem-d"] + ); +} + +#[tokio::test] +async fn memory_defers_new_rows_when_the_reference_lookup_fails_under_a_cap() { + let server = MockServer::start().await; + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{ORG}/patches/package"))) + .respond_with(ResponseTemplate::new(500)) + .with_priority(1) + .mount(&server) + .await; + mount(&server).await; + let mut files = BTreeMap::new(); + let names: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + lock(&mut files, "", &names); + let mem = memory(&server, &files, options(Some(2))).await; + let project = &mem.projects[0]; + assert!(project.error.is_none(), "{:?}", project.error); + assert_eq!(project.deferred.len(), 5); + assert!(mem.changed_files.is_empty()); + let codes: Vec<&str> = mem.warnings.iter().map(|w| w.code.as_str()).collect(); + assert!(codes.contains(&"rollout_reference_failed"), "{codes:?}"); + // Uncapped, the failure is the root's. + let mem = memory(&server, &files, options(None)).await; + assert_eq!( + mem.projects[0].error.as_ref().map(|e| e.code.as_str()), + Some("reference_lookup_failed") + ); +} + +#[tokio::test] +async fn memory_in_flight_patches_go_first_and_the_server_cap_tightens() { + let server = MockServer::start().await; + mount(&server).await; + let mut files = BTreeMap::new(); + let names: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + lock(&mut files, "", &names); + let mem = memory( + &server, + &files, + HostedScanOptions { + max_new_patches: Some(MaxNewPatchesOption(None)), + max_new_patches_cap: Some(1), + in_flight_patches: Some(vec![purl("mem-a")]), + ..options(None) + }, + ) + .await; + assert_eq!( + mem.rollout["maxNewPatches"], + json!({ "value": 1, "source": "cap" }) + ); + assert_eq!( + pinned(&apply(&files, &mem), "package-lock.json"), + ["mem-a"], + "the in-flight patch keeps its slot" + ); +} + +#[test] +fn the_max_new_patches_option_takes_a_count_or_none() { + let parse = |v: Value| { + serde_json::from_value::(json!({ "orgSlug": "o", "maxNewPatches": v })) + }; + assert_eq!( + parse(json!(3)).unwrap().max_new_patches, + Some(MaxNewPatchesOption(Some(3))) + ); + assert_eq!( + parse(json!("none")).unwrap().max_new_patches, + Some(MaxNewPatchesOption(None)) + ); + for bad in [json!(-1), json!(1.5), json!("all"), json!(4294967296u64)] { + assert!(parse(bad.clone()).is_err(), "{bad}"); + } + assert_eq!( + serde_json::to_value(MaxNewPatchesOption(None)).unwrap(), + json!("none") + ); +} diff --git a/crates/socket-patch-cli/tests/in_process_cargo_apply.rs b/crates/socket-patch-cli/tests/in_process_cargo_apply.rs index 8c7dc960..fa9021fc 100644 --- a/crates/socket-patch-cli/tests/in_process_cargo_apply.rs +++ b/crates/socket-patch-cli/tests/in_process_cargo_apply.rs @@ -246,6 +246,7 @@ async fn cargo_fetch_scan_sync_patches_real_file() { mode: None, all_releases: false, vex: Default::default(), + rollout: Default::default(), }; // CARGO_HOME must be set in this process's env so the cargo crawler // probes the isolated location (not the developer's real ~/.cargo). @@ -365,6 +366,7 @@ async fn cargo_apply_refuses_on_before_hash_mismatch() { mode: None, all_releases: false, vex: Default::default(), + rollout: Default::default(), }; std::env::set_var("CARGO_HOME", &cargo_home); @@ -463,6 +465,7 @@ async fn cargo_crawler_finds_real_fetched_crate() { mode: None, all_releases: false, vex: Default::default(), + rollout: Default::default(), }; assert_eq!(scan_run(args).await, 0); diff --git a/crates/socket-patch-cli/tests/in_process_gem_apply.rs b/crates/socket-patch-cli/tests/in_process_gem_apply.rs index 45cbaedd..8efb7392 100644 --- a/crates/socket-patch-cli/tests/in_process_gem_apply.rs +++ b/crates/socket-patch-cli/tests/in_process_gem_apply.rs @@ -224,6 +224,7 @@ async fn gem_install_scan_sync_patches_real_file() { mode: None, all_releases: false, vex: Default::default(), + rollout: Default::default(), }; let code = scan_run(args).await; assert_eq!( @@ -336,6 +337,7 @@ async fn gem_crawler_finds_real_installed_gem() { mode: None, all_releases: false, vex: Default::default(), + rollout: Default::default(), }; assert_eq!(scan_run(args).await, 0); diff --git a/crates/socket-patch-cli/tests/in_process_gem_multi_platform.rs b/crates/socket-patch-cli/tests/in_process_gem_multi_platform.rs index 1b0f8e35..be197eec 100644 --- a/crates/socket-patch-cli/tests/in_process_gem_multi_platform.rs +++ b/crates/socket-patch-cli/tests/in_process_gem_multi_platform.rs @@ -244,6 +244,7 @@ fn scan_args(cwd: &Path, api_url: String, all_releases: bool) -> ScanArgs { mode: None, all_releases, vex: Default::default(), + rollout: Default::default(), } } diff --git a/crates/socket-patch-cli/tests/in_process_pypi_apply.rs b/crates/socket-patch-cli/tests/in_process_pypi_apply.rs index e169dffe..4cbac457 100644 --- a/crates/socket-patch-cli/tests/in_process_pypi_apply.rs +++ b/crates/socket-patch-cli/tests/in_process_pypi_apply.rs @@ -273,6 +273,7 @@ async fn pypi_install_scan_sync_patches_real_file() { mode: None, all_releases: false, vex: Default::default(), + rollout: Default::default(), }; // Avoid borrow problem with into_iter let _ = &mut args; @@ -349,6 +350,7 @@ async fn pypi_scan_then_apply_force_patches_real_file() { mode: None, all_releases: false, vex: Default::default(), + rollout: Default::default(), }; let scan_code = scan_run(scan_args).await; assert_eq!(scan_code, 0, "scan --sync should succeed (exit 0)"); @@ -458,6 +460,7 @@ async fn pypi_apply_dry_run_does_not_modify_file() { mode: None, all_releases: false, vex: Default::default(), + rollout: Default::default(), }; // Require success: otherwise an early crash (before the apply path // is ever reached) would leave the file untouched and let this test @@ -587,6 +590,7 @@ async fn pypi_crawler_finds_real_installed_six() { mode: None, all_releases: false, vex: Default::default(), + rollout: Default::default(), }; assert_eq!(scan_run(args).await, 0); diff --git a/crates/socket-patch-cli/tests/in_process_pypi_multi_release.rs b/crates/socket-patch-cli/tests/in_process_pypi_multi_release.rs index 28fa350a..bdc63edb 100644 --- a/crates/socket-patch-cli/tests/in_process_pypi_multi_release.rs +++ b/crates/socket-patch-cli/tests/in_process_pypi_multi_release.rs @@ -322,6 +322,7 @@ fn scan_args(tmp: &Path, api_url: String, all_releases: bool) -> ScanArgs { mode: None, all_releases, vex: Default::default(), + rollout: Default::default(), } } diff --git a/crates/socket-patch-cli/tests/in_process_python_envs.rs b/crates/socket-patch-cli/tests/in_process_python_envs.rs index 855b04fd..97a730bc 100644 --- a/crates/socket-patch-cli/tests/in_process_python_envs.rs +++ b/crates/socket-patch-cli/tests/in_process_python_envs.rs @@ -139,6 +139,7 @@ fn default_args(cwd: &Path, api_url: String) -> ScanArgs { mode: None, all_releases: false, vex: Default::default(), + rollout: Default::default(), } } diff --git a/crates/socket-patch-cli/tests/in_process_redirect.rs b/crates/socket-patch-cli/tests/in_process_redirect.rs index 25273828..3c132eec 100644 --- a/crates/socket-patch-cli/tests/in_process_redirect.rs +++ b/crates/socket-patch-cli/tests/in_process_redirect.rs @@ -65,6 +65,7 @@ fn redirect_args(cwd: &Path, api_url: String) -> ScanArgs { mode: Some(socket_patch_cli::commands::scan::ScanMode::Hosted), all_releases: false, vex: Default::default(), + rollout: Default::default(), } } diff --git a/crates/socket-patch-cli/tests/in_process_redirect_pdm.rs b/crates/socket-patch-cli/tests/in_process_redirect_pdm.rs index a72ca5c3..97f4a171 100644 --- a/crates/socket-patch-cli/tests/in_process_redirect_pdm.rs +++ b/crates/socket-patch-cli/tests/in_process_redirect_pdm.rs @@ -166,6 +166,7 @@ fn hosted_args(cwd: &Path, api_url: String, vex: Option<&Path>) -> ScanArgs { vex: vex.map(Path::to_path_buf), ..Default::default() }, + rollout: Default::default(), } } diff --git a/crates/socket-patch-cli/tests/in_process_redirect_pipenv.rs b/crates/socket-patch-cli/tests/in_process_redirect_pipenv.rs index 738d2c91..bf2e00fd 100644 --- a/crates/socket-patch-cli/tests/in_process_redirect_pipenv.rs +++ b/crates/socket-patch-cli/tests/in_process_redirect_pipenv.rs @@ -99,6 +99,7 @@ fn hosted_args(cwd: &Path, api_url: String, vex: Option<&Path>) -> ScanArgs { vex_product: vex.map(|_| "pkg:pypi/pipenv-fixture@0.1.0".to_string()), ..Default::default() }, + rollout: Default::default(), } } diff --git a/crates/socket-patch-cli/tests/in_process_redirect_pnpm.rs b/crates/socket-patch-cli/tests/in_process_redirect_pnpm.rs index 4ad77067..592d72c1 100644 --- a/crates/socket-patch-cli/tests/in_process_redirect_pnpm.rs +++ b/crates/socket-patch-cli/tests/in_process_redirect_pnpm.rs @@ -97,6 +97,7 @@ fn hosted_args(cwd: &Path, api_url: String) -> ScanArgs { mode: Some(ScanMode::Hosted), all_releases: false, vex: Default::default(), + rollout: Default::default(), } } diff --git a/crates/socket-patch-cli/tests/in_process_redirect_poetry.rs b/crates/socket-patch-cli/tests/in_process_redirect_poetry.rs index d90c2039..ec348334 100644 --- a/crates/socket-patch-cli/tests/in_process_redirect_poetry.rs +++ b/crates/socket-patch-cli/tests/in_process_redirect_poetry.rs @@ -135,6 +135,7 @@ fn hosted_args(cwd: &Path, api_url: String, vex: Option<&Path>) -> ScanArgs { vex: vex.map(Path::to_path_buf), ..Default::default() }, + rollout: Default::default(), } } diff --git a/crates/socket-patch-cli/tests/in_process_remote_ecosystems_apply.rs b/crates/socket-patch-cli/tests/in_process_remote_ecosystems_apply.rs index 7a50373a..3b000c46 100644 --- a/crates/socket-patch-cli/tests/in_process_remote_ecosystems_apply.rs +++ b/crates/socket-patch-cli/tests/in_process_remote_ecosystems_apply.rs @@ -97,6 +97,7 @@ fn default_scan_args(cwd: &Path, eco: &str, api_url: String) -> ScanArgs { mode: None, all_releases: false, vex: Default::default(), + rollout: Default::default(), } } diff --git a/crates/socket-patch-cli/tests/in_process_rollback_hosted.rs b/crates/socket-patch-cli/tests/in_process_rollback_hosted.rs index d79e89f8..b093b492 100644 --- a/crates/socket-patch-cli/tests/in_process_rollback_hosted.rs +++ b/crates/socket-patch-cli/tests/in_process_rollback_hosted.rs @@ -90,6 +90,7 @@ fn hosted_scan_args(cwd: &Path, api_url: String) -> ScanArgs { mode: Some(ScanMode::Hosted), all_releases: false, vex: Default::default(), + rollout: Default::default(), } } diff --git a/crates/socket-patch-cli/tests/in_process_scan.rs b/crates/socket-patch-cli/tests/in_process_scan.rs index 5521dded..5ca5608a 100644 --- a/crates/socket-patch-cli/tests/in_process_scan.rs +++ b/crates/socket-patch-cli/tests/in_process_scan.rs @@ -43,6 +43,7 @@ fn default_args(cwd: &Path) -> ScanArgs { mode: None, all_releases: false, vex: Default::default(), + rollout: Default::default(), } } diff --git a/crates/socket-patch-cli/tests/in_process_vendor.rs b/crates/socket-patch-cli/tests/in_process_vendor.rs index 20bcfa5f..492538f1 100644 --- a/crates/socket-patch-cli/tests/in_process_vendor.rs +++ b/crates/socket-patch-cli/tests/in_process_vendor.rs @@ -3266,6 +3266,7 @@ snapshots: mode: Some(ScanMode::Hosted), all_releases: false, vex: Default::default(), + rollout: Default::default(), } } diff --git a/crates/socket-patch-cli/tests/scan_rollout_e2e.rs b/crates/socket-patch-cli/tests/scan_rollout_e2e.rs new file mode 100644 index 00000000..2e0fba55 --- /dev/null +++ b/crates/socket-patch-cli/tests/scan_rollout_e2e.rs @@ -0,0 +1,1038 @@ +//! `scan --max-new-patches` end to end (staged rollout, work item B): a +//! project with nine patchable packages and a cap of three rolls forward +//! three packages per run, most severe first, in hosted, agent and vendored +//! mode; a fourth run changes nothing. Mock API, the built binary. + +use std::path::{Path, PathBuf}; +use std::process::Command; + +use serde_json::{json, Value}; +use sha2::{Digest, Sha256}; +use wiremock::matchers::{method, path, path_regex}; +use wiremock::{Mock, MockServer, ResponseTemplate}; + +const ORG: &str = "test-org"; +const TOKEN: &str = "22222222-2222-4222-8222-222222222222"; +const HOST: &str = "http://patch.test"; + +/// `(name, severities)`: one advisory per severity; none = unknown. +const PACKAGES: [(&str, &[&str]); 9] = [ + ("roll-a", &["low"]), + ("roll-b", &["critical"]), + ("roll-c", &["high"]), + ("roll-d", &["medium"]), + ("roll-e", &["critical", "high"]), + ("roll-f", &[]), + ("roll-g", &["high", "low", "low"]), + ("roll-h", &["medium"]), + ("roll-i", &["low"]), +]; + +/// The rollout order: severity, then advisory count, then name. +const ORDER: [&str; 9] = [ + "roll-e", "roll-b", "roll-g", "roll-c", "roll-d", "roll-h", "roll-a", "roll-i", "roll-f", +]; + +fn binary() -> PathBuf { + env!("CARGO_BIN_EXE_socket-patch").into() +} + +fn uuid(name: &str) -> String { + let n = PACKAGES.iter().position(|(p, _)| *p == name).unwrap() + 1; + format!("{n:08x}-1111-4111-8111-{n:012x}") +} + +fn purl(name: &str) -> String { + format!("pkg:npm/{name}@1.0.0") +} + +fn hosted_url(name: &str) -> String { + format!( + "{HOST}/patch/npm/{name}/1.0.0/{TOKEN}/{}/{name}-1.0.0.tgz", + uuid(name) + ) +} + +fn before(name: &str) -> Vec { + format!("module.exports = '{name} before';\n").into_bytes() +} + +fn after(name: &str) -> Vec { + format!("module.exports = '{name} after';\n").into_bytes() +} + +fn git_sha256(content: &[u8]) -> String { + let mut hasher = Sha256::new(); + hasher.update(format!("blob {}\0", content.len()).as_bytes()); + hasher.update(content); + hex::encode(hasher.finalize()) +} + +fn b64(bytes: &[u8]) -> String { + use base64::Engine; + base64::engine::general_purpose::STANDARD.encode(bytes) +} + +fn vulns(name: &str, severities: &[&str]) -> Value { + let mut map = serde_json::Map::new(); + for (i, sev) in severities.iter().enumerate() { + map.insert( + format!("GHSA-{name}-{i}"), + json!({ "cves": [], "summary": "s", "severity": sev, "description": "d" }), + ); + } + Value::Object(map) +} + +/// A v3 npm project in `dir` with `names` installed and locked. +fn write_project(dir: &Path, names: &[&str]) { + std::fs::create_dir_all(dir).unwrap(); + let deps: serde_json::Map = names + .iter() + .map(|n| (n.to_string(), json!("1.0.0"))) + .collect(); + std::fs::write( + dir.join("package.json"), + serde_json::to_vec_pretty(&json!({ + "name": "rollout-consumer", "version": "0.0.0", "dependencies": deps + })) + .unwrap(), + ) + .unwrap(); + let mut packages = serde_json::Map::new(); + packages.insert( + String::new(), + json!({ "name": "rollout-consumer", "version": "0.0.0", "dependencies": deps }), + ); + for name in names { + packages.insert( + format!("node_modules/{name}"), + json!({ + "version": "1.0.0", + "resolved": format!("https://registry.npmjs.org/{name}/-/{name}-1.0.0.tgz"), + "integrity": "sha512-UPSTREAMupstream==", + "license": "MIT" + }), + ); + let pkg = dir.join("node_modules").join(name); + std::fs::create_dir_all(&pkg).unwrap(); + std::fs::write( + pkg.join("package.json"), + format!(r#"{{"name":"{name}","version":"1.0.0"}}"#), + ) + .unwrap(); + std::fs::write(pkg.join("index.js"), before(name)).unwrap(); + } + let mut lock = serde_json::to_vec_pretty(&json!({ + "name": "rollout-consumer", + "version": "0.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": packages, + })) + .unwrap(); + lock.push(b'\n'); + std::fs::write(dir.join("package-lock.json"), lock).unwrap(); +} + +/// How the mock answers one package's reference grant. +#[derive(Clone, Copy, PartialEq)] +enum Grant { + Granted, + Withdrawn, + BadPurl, +} + +/// Batch, by-package, view and reference mocks for every package. +async fn mount_api(mock: &MockServer, grant: impl Fn(&str) -> Grant) { + let batch: Vec = PACKAGES + .iter() + .map(|(name, sevs)| { + json!({ + "purl": purl(name), + "patches": [{ + "uuid": uuid(name), "purl": purl(name), "tier": "free", + "cveIds": [], + "ghsaIds": (0..sevs.len()).map(|i| format!("GHSA-{name}-{i}")).collect::>(), + "severity": sevs.first().copied(), + "title": name, + }] + }) + }) + .collect(); + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{ORG}/patches/batch"))) + .respond_with( + ResponseTemplate::new(200) + .set_body_json(json!({ "packages": batch, "canAccessPaidPatches": false })), + ) + .mount(mock) + .await; + let mut results = serde_json::Map::new(); + for (name, sevs) in PACKAGES { + Mock::given(method("GET")) + .and(path_regex(format!( + "^/v0/orgs/{ORG}/patches/by-package/.*{name}(%40|@)1\\.0\\.0$" + ))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "patches": [{ + "uuid": uuid(name), "purl": purl(name), + "publishedAt": "2026-01-01T00:00:00Z", + "description": name, "license": "MIT", "tier": "free", + "vulnerabilities": vulns(name, sevs), + }], + "canAccessPaidPatches": false, + }))) + .mount(mock) + .await; + Mock::given(method("GET")) + .and(path(format!("/v0/orgs/{ORG}/patches/view/{}", uuid(name)))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "uuid": uuid(name), "purl": purl(name), + "publishedAt": "2026-01-01T00:00:00Z", + "files": { "package/index.js": { + "beforeHash": git_sha256(&before(name)), + "afterHash": git_sha256(&after(name)), + "blobContent": b64(&after(name)), + }}, + "vulnerabilities": vulns(name, sevs), + "description": name, "license": "MIT", "tier": "free", + }))) + .mount(mock) + .await; + let entry = match grant(name) { + Grant::Granted => json!({ + "status": "granted", + "url": hosted_url(name), + "purl": purl(name), + "artifacts": [{ + "kind": "tarball", "url": hosted_url(name), + "integrity": { "sha512": format!("sha512-PATCHED{name}==") } + }], + "registryOverride": null + }), + Grant::Withdrawn => json!({ "status": "withdrawn" }), + Grant::BadPurl => json!({ + "status": "granted", "url": hosted_url(name), "purl": "garbage", + "artifacts": [], "registryOverride": null + }), + }; + results.insert(uuid(name), entry); + } + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{ORG}/patches/package"))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ "results": results }))) + .mount(mock) + .await; +} + +fn run(root: &Path, mock: &MockServer, args: &[&str]) -> (i32, String, String) { + let uri = mock.uri(); + let mut argv = vec![ + "scan", + "--yes", + "--api-url", + uri.as_str(), + "--api-token", + "fake-token", + "--org", + ORG, + ]; + argv.extend_from_slice(args); + let mut cmd = Command::new(binary()); + cmd.args(&argv).current_dir(root); + for (key, _) in std::env::vars_os() { + if key.to_string_lossy().starts_with("SOCKET_") { + cmd.env_remove(&key); + } + } + cmd.env("SOCKET_TELEMETRY_DISABLED", "1"); + let out = cmd.output().expect("run socket-patch"); + ( + out.status.code().unwrap_or(-1), + String::from_utf8_lossy(&out.stdout).into_owned(), + String::from_utf8_lossy(&out.stderr).into_owned(), + ) +} + +fn run_json(root: &Path, mock: &MockServer, args: &[&str]) -> Value { + let mut all = vec!["--json"]; + all.extend_from_slice(args); + let (code, stdout, stderr) = run(root, mock, &all); + assert_eq!(code, 0, "stdout={stdout}\nstderr={stderr}"); + serde_json::from_str(stdout.trim()).unwrap_or_else(|e| panic!("{e}: {stdout}")) +} + +/// How many times the mock served `view/{uuid}`. +async fn view_fetches(mock: &MockServer, uuid: &str) -> usize { + mock.received_requests() + .await + .unwrap_or_default() + .iter() + .filter(|r| r.url.path().ends_with(&format!("/patches/view/{uuid}"))) + .count() +} + +/// The packages whose hosted artifact the lockfile pins. +fn pinned(root: &Path) -> Vec { + let lock = std::fs::read_to_string(root.join("package-lock.json")).unwrap(); + ORDER + .iter() + .filter(|n| lock.contains(&hosted_url(n))) + .map(|n| n.to_string()) + .collect() +} + +fn names(list: &[&str]) -> Vec { + list.iter().map(|s| s.to_string()).collect() +} + +fn deferred_names(v: &Value) -> Vec { + v["rollout"]["deferred"] + .as_array() + .unwrap() + .iter() + .map(|d| { + d["purl"] + .as_str() + .unwrap() + .trim_start_matches("pkg:npm/") + .trim_end_matches("@1.0.0") + .to_string() + }) + .collect() +} + +fn counts(v: &Value) -> (u64, u64, u64, u64) { + let c = &v["rollout"]["counts"]; + ( + c["new"].as_u64().unwrap(), + c["deferred"].as_u64().unwrap(), + c["upgrade"].as_u64().unwrap(), + c["already"].as_u64().unwrap(), + ) +} + +#[tokio::test] +async fn hosted_cap_rolls_nine_packages_forward_three_per_run() { + let mock = MockServer::start().await; + mount_api(&mock, |_| Grant::Granted).await; + let tmp = tempfile::tempdir().unwrap(); + let names9: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + write_project(tmp.path(), &names9); + let args = [ + "--mode", + "hosted", + "--max-new-patches", + "3", + "--patch-server-url", + HOST, + ]; + + // The dry run predicts exactly what the wet run does. + let lock_before = std::fs::read(tmp.path().join("package-lock.json")).unwrap(); + let mut dry_args = args.to_vec(); + dry_args.push("--dry-run"); + let dry = run_json(tmp.path(), &mock, &dry_args); + assert_eq!( + std::fs::read(tmp.path().join("package-lock.json")).unwrap(), + lock_before, + "a dry run writes nothing" + ); + + let mut previous_lock = lock_before; + let mut dry = dry; + for run_no in 0..3 { + let v = run_json(tmp.path(), &mock, &args); + // The dry run before each wet run predicts it exactly. + // (Warning details switch tense: "would be written" / "was written".) + let decisions = |block: &Value| -> Value { + let mut block = block.clone(); + let obj = block.as_object_mut().unwrap(); + obj.remove("dryRun"); + let codes: Vec = obj["warnings"] + .as_array() + .unwrap() + .iter() + .map(|w| w["code"].clone()) + .collect(); + obj.insert("warnings".into(), Value::Array(codes)); + block + }; + let (predicted, actual) = (decisions(&dry["redirect"]), decisions(&v["redirect"])); + assert_eq!( + dry["rollout"], + v["rollout"], + "run {}: dry run == wet run", + run_no + 1 + ); + assert_eq!(predicted, actual, "run {}: dry run == wet run", run_no + 1); + if run_no == 0 { + assert_eq!( + v["rollout"]["deferred"][0], + json!({ + "purl": "pkg:npm/roll-c@1.0.0", + "uuids": [uuid("roll-c")], + "severity": "high", + "advisoryCount": 1, + "projects": [""], + "rank": 4, + }) + ); + } + assert_eq!( + v["rollout"]["maxNewPatches"], + json!({ "value": 3, "source": "flag" }) + ); + let done = 3 * run_no as u64; + assert_eq!( + counts(&v), + (3, 6 - done, 0, done), + "run {}: {v}", + run_no + 1 + ); + assert_eq!( + pinned(tmp.path()), + names(&ORDER[..3 * (run_no + 1)]), + "run {} pins the next three, most severe first", + run_no + 1 + ); + assert_eq!(deferred_names(&v), names(&ORDER[3 * (run_no + 1)..])); + let ranks: Vec = v["rollout"]["deferred"] + .as_array() + .unwrap() + .iter() + .map(|d| d["rank"].as_u64().unwrap()) + .collect(); + assert_eq!(ranks, (4..4 + ranks.len() as u64).collect::>()); + let skipped: Vec<&str> = v["redirect"]["skipped"] + .as_array() + .unwrap() + .iter() + .map(|s| s["reason"].as_str().unwrap()) + .collect(); + assert!(skipped.iter().all(|r| *r == "rollout_deferred")); + assert_eq!(skipped.len() as u64, 6 - done); + assert_eq!(v["redirect"]["redirected"], 3 * (run_no as u64 + 1)); + previous_lock = std::fs::read(tmp.path().join("package-lock.json")).unwrap(); + let mut next_dry = args.to_vec(); + next_dry.push("--dry-run"); + dry = run_json(tmp.path(), &mock, &next_dry); + } + + let v = run_json(tmp.path(), &mock, &args); + assert_eq!(counts(&v), (0, 0, 0, 9), "a fourth run adds nothing: {v}"); + assert_eq!( + std::fs::read(tmp.path().join("package-lock.json")).unwrap(), + previous_lock, + "a converged run is byte-stable" + ); +} + +#[tokio::test] +async fn hosted_converges_even_when_the_patch_server_is_not_configured() { + // Without --patch-server-url discovery does not recognize the test + // host's pins; the rollout still treats a lockfile that names the + // selected patch as recorded, so it never re-spends a slot on it. + let mock = MockServer::start().await; + mount_api(&mock, |_| Grant::Granted).await; + let tmp = tempfile::tempdir().unwrap(); + let names9: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + write_project(tmp.path(), &names9); + for run_no in 1..=3 { + run_json( + tmp.path(), + &mock, + &["--mode", "hosted", "--max-new-patches", "3"], + ); + assert_eq!(pinned(tmp.path()), names(&ORDER[..3 * run_no])); + } + let v = run_json( + tmp.path(), + &mock, + &["--mode", "hosted", "--max-new-patches", "3"], + ); + assert_eq!(counts(&v), (0, 0, 0, 9)); +} + +#[tokio::test] +async fn hosted_ineligible_top_ranked_patches_hold_no_slot() { + let mock = MockServer::start().await; + mount_api(&mock, |name| match name { + "roll-e" => Grant::Withdrawn, + "roll-b" => Grant::BadPurl, + _ => Grant::Granted, + }) + .await; + let tmp = tempfile::tempdir().unwrap(); + let names9: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + write_project(tmp.path(), &names9); + let v = run_json( + tmp.path(), + &mock, + &[ + "--mode", + "hosted", + "--max-new-patches", + "2", + "--patch-server-url", + HOST, + ], + ); + assert_eq!(pinned(tmp.path()), names(&["roll-g", "roll-c"]), "{v}"); + assert_eq!(counts(&v), (2, 5, 0, 0)); + assert_eq!( + deferred_names(&v), + names(&["roll-d", "roll-h", "roll-a", "roll-i", "roll-f"]) + ); + assert_eq!( + v["rollout"]["deferred"][0]["rank"], 3, + "ranks count eligible rows only" + ); + let reasons: Vec<(&str, &str)> = v["redirect"]["skipped"] + .as_array() + .unwrap() + .iter() + .map(|s| (s["purl"].as_str().unwrap(), s["reason"].as_str().unwrap())) + .collect(); + assert!( + reasons.contains(&("pkg:npm/roll-e@1.0.0", "withdrawn")), + "{reasons:?}" + ); + assert!(reasons.contains(&("garbage", "bad_purl")), "{reasons:?}"); +} + +#[tokio::test] +async fn a_failed_detail_lookup_admits_nothing_new_under_a_cap() { + let mock = MockServer::start().await; + // Mounted first, so it wins over the per-package mock below. + Mock::given(method("GET")) + .and(path_regex(format!( + "^/v0/orgs/{ORG}/patches/by-package/.*roll-i(%40|@)1\\.0\\.0$" + ))) + .respond_with(ResponseTemplate::new(500)) + .mount(&mock) + .await; + mount_api(&mock, |_| Grant::Granted).await; + let tmp = tempfile::tempdir().unwrap(); + let names9: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + write_project(tmp.path(), &names9); + let lock_before = std::fs::read(tmp.path().join("package-lock.json")).unwrap(); + let v = run_json( + tmp.path(), + &mock, + &[ + "--mode", + "hosted", + "--max-new-patches", + "3", + "--patch-server-url", + HOST, + ], + ); + assert_eq!(counts(&v), (0, 8, 0, 0), "{v}"); + assert_eq!( + std::fs::read(tmp.path().join("package-lock.json")).unwrap(), + lock_before + ); + let codes: Vec<&str> = v["warnings"] + .as_array() + .unwrap() + .iter() + .map(|w| w["code"].as_str().unwrap()) + .collect(); + assert!(codes.contains(&"rollout_incomplete_lookup"), "{codes:?}"); + assert!(codes.contains(&"patch_details_failed"), "{codes:?}"); +} + +#[tokio::test] +async fn agent_cap_rolls_forward_and_upgrades_ignore_the_cap() { + let mock = MockServer::start().await; + mount_api(&mock, |_| Grant::Granted).await; + let tmp = tempfile::tempdir().unwrap(); + let names9: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + write_project(tmp.path(), &names9); + let recorded = |root: &Path| -> Vec { + let m: Value = serde_json::from_slice( + &std::fs::read(root.join(".socket/manifest.json")).unwrap_or_else(|_| b"{}".to_vec()), + ) + .unwrap(); + ORDER + .iter() + .filter(|n| m["patches"].get(purl(n)).is_some()) + .map(|n| n.to_string()) + .collect() + }; + let args = ["--mode", "agent", "--max-new-patches", "3"]; + let dry = { + let mut a = args.to_vec(); + a.push("--dry-run"); + run_json(tmp.path(), &mock, &a) + }; + for run_no in 1..=3 { + let v = run_json(tmp.path(), &mock, &args); + if run_no == 1 { + assert_eq!(dry["rollout"], v["rollout"], "dry run == wet run"); + let added: Vec<&str> = dry["apply"]["patches"] + .as_array() + .unwrap() + .iter() + .map(|p| p["purl"].as_str().unwrap()) + .collect(); + assert_eq!(added.len(), 3, "{dry}"); + } + assert_eq!(recorded(tmp.path()), names(&ORDER[..3 * run_no]), "{v}"); + for name in &ORDER[..3 * run_no] { + assert_eq!( + std::fs::read(tmp.path().join("node_modules").join(name).join("index.js")).unwrap(), + after(name) + ); + } + for name in &ORDER[3 * run_no..] { + assert_eq!( + std::fs::read(tmp.path().join("node_modules").join(name).join("index.js")).unwrap(), + before(name), + "a deferred package is not touched" + ); + assert_eq!( + view_fetches(&mock, &uuid(name)).await, + 0, + "a deferred patch is never downloaded: {name}" + ); + } + } + let manifest = std::fs::read(tmp.path().join(".socket/manifest.json")).unwrap(); + let v = run_json(tmp.path(), &mock, &args); + assert_eq!(counts(&v), (0, 0, 0, 9)); + assert_eq!( + std::fs::read(tmp.path().join(".socket/manifest.json")).unwrap(), + manifest, + "a converged run is byte-stable" + ); + + // A newer patch for an applied package is an UPGRADE: it lands even + // with `--max-new-patches 0`, and is reported in `updates[]`. + mock.reset().await; + mount_api(&mock, |_| Grant::Granted).await; + let newer = "0000000a-1111-4111-8111-00000000000a"; + Mock::given(method("GET")) + .and(path_regex(format!( + "^/v0/orgs/{ORG}/patches/by-package/.*roll-a(%40|@)1\\.0\\.0$" + ))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "patches": [ + { "uuid": newer, "purl": purl("roll-a"), "publishedAt": "2026-06-01T00:00:00Z", + "description": "newer", "license": "MIT", "tier": "free", + "vulnerabilities": vulns("roll-a", &["low"]) }, + { "uuid": uuid("roll-a"), "purl": purl("roll-a"), "publishedAt": "2026-01-01T00:00:00Z", + "description": "roll-a", "license": "MIT", "tier": "free", + "vulnerabilities": vulns("roll-a", &["low"]) } + ], + "canAccessPaidPatches": false, + }))) + .with_priority(1) + .mount(&mock) + .await; + Mock::given(method("GET")) + .and(path(format!("/v0/orgs/{ORG}/patches/view/{newer}"))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "uuid": newer, "purl": purl("roll-a"), "publishedAt": "2026-06-01T00:00:00Z", + "files": { "package/index.js": { + "beforeHash": git_sha256(&before("roll-a")), + "afterHash": git_sha256(&after("roll-a")), + "blobContent": b64(&after("roll-a")), + }}, + "vulnerabilities": vulns("roll-a", &["low"]), + "description": "newer", "license": "MIT", "tier": "free", + }))) + .mount(&mock) + .await; + let v = run_json( + tmp.path(), + &mock, + &["--mode", "agent", "--max-new-patches", "0"], + ); + assert_eq!(counts(&v), (0, 0, 1, 8), "{v}"); + assert_eq!( + v["updates"], + json!([{ "purl": purl("roll-a"), "oldUuid": uuid("roll-a"), "newUuid": newer }]) + ); + let m: Value = + serde_json::from_slice(&std::fs::read(tmp.path().join(".socket/manifest.json")).unwrap()) + .unwrap(); + assert_eq!(m["patches"][purl("roll-a")]["uuid"], newer); +} + +#[tokio::test] +async fn vendored_cap_rolls_forward_three_per_run() { + let mock = MockServer::start().await; + mount_api(&mock, |_| Grant::Granted).await; + let tmp = tempfile::tempdir().unwrap(); + let names9: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + write_project(tmp.path(), &names9); + let vendored = |root: &Path| -> Vec { + let state: Value = std::fs::read(root.join(".socket/vendor/state.json")) + .ok() + .map(|b| serde_json::from_slice(&b).unwrap()) + .unwrap_or(json!({})); + ORDER + .iter() + .filter(|n| state["entries"].get(purl(n)).is_some()) + .map(|n| n.to_string()) + .collect() + }; + let args = ["--mode", "vendored", "--max-new-patches", "3"]; + let dry = { + let mut a = args.to_vec(); + a.push("--dry-run"); + run_json(tmp.path(), &mock, &a) + }; + assert!( + !tmp.path().join(".socket").exists(), + "a dry run writes nothing" + ); + for run_no in 1..=3 { + let v = run_json(tmp.path(), &mock, &args); + if run_no == 1 { + assert_eq!(dry["rollout"], v["rollout"], "dry run == wet run"); + assert_eq!(dry["vendor"]["patches"].as_array().unwrap().len(), 3); + } + assert_eq!(vendored(tmp.path()), names(&ORDER[..3 * run_no]), "{v}"); + } + let lock = std::fs::read(tmp.path().join("package-lock.json")).unwrap(); + let v = run_json(tmp.path(), &mock, &args); + assert_eq!(counts(&v), (0, 0, 0, 9)); + assert_eq!( + std::fs::read(tmp.path().join("package-lock.json")).unwrap(), + lock + ); +} + +#[tokio::test] +async fn project_directories_share_one_budget_in_sorted_order() { + let mock = MockServer::start().await; + mount_api(&mock, |_| Grant::Granted).await; + let tmp = tempfile::tempdir().unwrap(); + // `a/` sorts first and spends two slots; `b/` gets the one left, and + // its copy of roll-b rides free (already admitted in `a/`). + write_project(&tmp.path().join("a"), &["roll-a", "roll-b"]); + write_project( + &tmp.path().join("b"), + &["roll-b", "roll-c", "roll-e", "roll-h"], + ); + let (code, stdout, stderr) = run( + tmp.path(), + &mock, + &[ + "--mode", + "hosted", + "--max-new-patches", + "3", + "--patch-server-url", + HOST, + // Given out of order: the directories are still visited sorted. + "b", + "a", + ], + ); + assert_eq!(code, 0, "stdout={stdout}\nstderr={stderr}"); + assert_eq!(pinned(&tmp.path().join("a")), names(&["roll-b", "roll-a"])); + assert_eq!(pinned(&tmp.path().join("b")), names(&["roll-e", "roll-b"])); + assert!( + stdout.contains( + "Rollout: 2 of 2 new patches applied (maxNewPatches=3 from --max-new-patches, \ + shared by this run's directories, 1 left)" + ), + "{stdout}" + ); + assert!( + stdout.contains( + "Rollout: 2 of 4 new patches applied (maxNewPatches=3 from --max-new-patches, \ + shared by this run's directories, 0 left)" + ), + "b/ admits roll-b free and roll-e with the last slot: {stdout}" + ); + assert!( + stdout.contains("Next up: roll-c@1.0.0 (high), roll-h@1.0.0 (medium)"), + "{stdout}" + ); +} + +#[tokio::test] +async fn a_malformed_env_cap_is_a_usage_error_unless_the_flag_overrides_it() { + let mock = MockServer::start().await; + mount_api(&mock, |_| Grant::Granted).await; + let tmp = tempfile::tempdir().unwrap(); + write_project(tmp.path(), &["roll-a", "roll-b"]); + let mut cmd = Command::new(binary()); + cmd.args(["scan", "--json", "--api-url"]) + .arg(mock.uri()) + .args(["--api-token", "t", "--org", ORG]) + .current_dir(tmp.path()); + for (key, _) in std::env::vars_os() { + if key.to_string_lossy().starts_with("SOCKET_") { + cmd.env_remove(&key); + } + } + cmd.env("SOCKET_TELEMETRY_DISABLED", "1") + .env("SOCKET_MAX_NEW_PATCHES", "lots"); + let out = cmd.output().unwrap(); + assert_eq!(out.status.code(), Some(2)); + assert!(String::from_utf8_lossy(&out.stderr).contains("SOCKET_MAX_NEW_PATCHES")); + + // A flag overrides the env value without reading it. + let mut with_flag = Command::new(binary()); + with_flag + .args([ + "scan", + "--json", + "--dry-run", + "--max-new-patches", + "1", + "--api-url", + ]) + .arg(mock.uri()) + .args(["--api-token", "t", "--org", ORG]) + .current_dir(tmp.path()); + for (key, _) in std::env::vars_os() { + if key.to_string_lossy().starts_with("SOCKET_") { + with_flag.env_remove(&key); + } + } + with_flag + .env("SOCKET_TELEMETRY_DISABLED", "1") + .env("SOCKET_MAX_NEW_PATCHES", "lots"); + let out = with_flag.output().unwrap(); + assert_eq!( + out.status.code(), + Some(0), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + let v: Value = serde_json::from_slice(&out.stdout).unwrap(); + assert_eq!( + v["rollout"]["maxNewPatches"], + json!({ "value": 1, "source": "flag" }) + ); + + cmd.env("SOCKET_MAX_NEW_PATCHES", "1"); + let out = cmd.output().unwrap(); + let v: Value = serde_json::from_slice(&out.stdout).unwrap(); + assert_eq!( + v["rollout"]["maxNewPatches"], + json!({ "value": 1, "source": "env" }) + ); + assert_eq!(v["rollout"]["counts"]["new"], 1); + + cmd.args(["--max-new-patches", "none"]); + let out = cmd.output().unwrap(); + let v: Value = serde_json::from_slice(&out.stdout).unwrap(); + assert_eq!( + v["rollout"]["maxNewPatches"], + json!({ "value": null, "source": "flag" }) + ); + assert_eq!( + v["rollout"]["counts"]["new"], 1, + "roll-b landed on the capped run; roll-a now" + ); +} + +/// Mount, ahead of `mount_api`'s answers, a newer superseding patch for +/// `name`: offered first by-package, granted, and viewable. +async fn mount_newer(mock: &MockServer, name: &str, newer: &str) { + let sevs = PACKAGES.iter().find(|(n, _)| *n == name).unwrap().1; + Mock::given(method("GET")) + .and(path_regex(format!( + "^/v0/orgs/{ORG}/patches/by-package/.*{name}(%40|@)1\\.0\\.0$" + ))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "patches": [ + { "uuid": newer, "purl": purl(name), "publishedAt": "2026-06-01T00:00:00Z", + "description": "newer", "license": "MIT", "tier": "free", + "vulnerabilities": vulns(name, sevs) }, + { "uuid": uuid(name), "purl": purl(name), "publishedAt": "2026-01-01T00:00:00Z", + "description": name, "license": "MIT", "tier": "free", + "vulnerabilities": vulns(name, sevs) } + ], + "canAccessPaidPatches": false, + }))) + .with_priority(1) + .mount(mock) + .await; + let mut results = serde_json::Map::new(); + for (n, _) in PACKAGES { + results.insert( + uuid(n), + json!({ "status": "granted", "url": hosted_url(n), "purl": purl(n), + "artifacts": [{ "kind": "tarball", "url": hosted_url(n), + "integrity": { "sha512": format!("sha512-PATCHED{n}==") } }], + "registryOverride": null }), + ); + } + let newer_url = hosted_url(name).replace(&uuid(name), newer); + results.insert( + newer.to_string(), + json!({ "status": "granted", "url": newer_url, "purl": purl(name), + "artifacts": [{ "kind": "tarball", "url": newer_url, + "integrity": { "sha512": "sha512-NEWER==" } }], + "registryOverride": null }), + ); + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{ORG}/patches/package"))) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ "results": results }))) + .with_priority(1) + .mount(mock) + .await; +} + +#[tokio::test] +async fn hosted_upgrades_land_with_a_cap_of_zero() { + let mock = MockServer::start().await; + mount_api(&mock, |_| Grant::Granted).await; + let tmp = tempfile::tempdir().unwrap(); + let names9: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + write_project(tmp.path(), &names9); + let args = [ + "--mode", + "hosted", + "--max-new-patches", + "3", + "--patch-server-url", + HOST, + ]; + run_json(tmp.path(), &mock, &args); + assert_eq!(pinned(tmp.path()), names(&ORDER[..3])); + + // roll-e (applied) gains a newer patch; six packages still wait. + let newer = "0000000e-1111-4111-8111-00000000000e"; + mount_newer(&mock, "roll-e", newer).await; + let v = run_json( + tmp.path(), + &mock, + &[ + "--mode", + "hosted", + "--max-new-patches", + "0", + "--patch-server-url", + HOST, + ], + ); + assert_eq!(counts(&v), (0, 6, 1, 2), "{v}"); + assert_eq!( + v["updates"], + json!([{ "purl": purl("roll-e"), "oldUuid": uuid("roll-e"), "newUuid": newer }]) + ); + let lock = std::fs::read_to_string(tmp.path().join("package-lock.json")).unwrap(); + assert!(lock.contains(newer), "the upgrade landed: {lock}"); + assert_eq!( + pinned(tmp.path()), + names(&["roll-b", "roll-g"]), + "nothing new was added" + ); +} + +#[tokio::test] +async fn a_failed_reference_lookup_defers_new_rows_instead_of_failing() { + let tmp = tempfile::tempdir().unwrap(); + let names9: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + write_project(tmp.path(), &names9); + let lock_before = std::fs::read(tmp.path().join("package-lock.json")).unwrap(); + let args = [ + "--mode", + "hosted", + "--max-new-patches", + "3", + "--patch-server-url", + HOST, + ]; + + let failing = MockServer::start().await; + Mock::given(method("POST")) + .and(path(format!("/v0/orgs/{ORG}/patches/package"))) + .respond_with(ResponseTemplate::new(500)) + .with_priority(1) + .mount(&failing) + .await; + mount_api(&failing, |_| Grant::Granted).await; + + // Every row is NEW: the failure only affects rows the incomplete lookup + // defers anyway, so the capped run succeeds and writes nothing. + let v = run_json(tmp.path(), &failing, &args); + assert_eq!(counts(&v), (0, 9, 0, 0), "{v}"); + let codes: Vec<&str> = v["warnings"] + .as_array() + .unwrap() + .iter() + .map(|w| w["code"].as_str().unwrap()) + .collect(); + assert!(codes.contains(&"rollout_reference_failed"), "{codes:?}"); + assert!(codes.contains(&"rollout_incomplete_lookup"), "{codes:?}"); + assert_eq!( + std::fs::read(tmp.path().join("package-lock.json")).unwrap(), + lock_before + ); + assert!(!tmp.path().join(".socket").exists(), "no lock, no state"); + + // Without a cap the failure is the run's, as before. + let (code, stdout, _) = run(tmp.path(), &failing, &["--json", "--mode", "hosted"]); + assert_eq!(code, 1, "{stdout}"); + let v: Value = serde_json::from_str(stdout.trim()).unwrap(); + assert_eq!(v["status"], "error"); + assert!( + v.get("rollout").is_none(), + "no rollout block on an error envelope" + ); + + // With applied rows the failure affects them too: the run fails. + let ok = MockServer::start().await; + mount_api(&ok, |_| Grant::Granted).await; + run_json(tmp.path(), &ok, &args); + let mut failing_args = vec!["--json"]; + failing_args.extend_from_slice(&args); + let (code, stdout, _) = run(tmp.path(), &failing, &failing_args); + assert_eq!(code, 1, "{stdout}"); +} + +#[tokio::test] +async fn a_failed_detail_lookup_for_an_applied_package_does_not_freeze_new_rows() { + let mock = MockServer::start().await; + mount_api(&mock, |_| Grant::Granted).await; + let tmp = tempfile::tempdir().unwrap(); + let names9: Vec<&str> = PACKAGES.iter().map(|(n, _)| *n).collect(); + write_project(tmp.path(), &names9); + let args = [ + "--mode", + "hosted", + "--max-new-patches", + "3", + "--patch-server-url", + HOST, + ]; + run_json(tmp.path(), &mock, &args); + Mock::given(method("GET")) + .and(path_regex(format!( + "^/v0/orgs/{ORG}/patches/by-package/.*roll-e(%40|@)1\\.0\\.0$" + ))) + .respond_with(ResponseTemplate::new(500)) + .with_priority(1) + .mount(&mock) + .await; + let v = run_json(tmp.path(), &mock, &args); + assert_eq!( + counts(&v).0, + 3, + "roll-e is recorded, so the next three land: {v}" + ); + let codes: Vec<&str> = v["warnings"] + .as_array() + .unwrap() + .iter() + .map(|w| w["code"].as_str().unwrap()) + .collect(); + assert!(!codes.contains(&"rollout_incomplete_lookup"), "{codes:?}"); + assert!( + pinned(tmp.path()).contains(&"roll-e".to_string()), + "the pin stays" + ); +} diff --git a/crates/socket-patch-core/src/api/ranking.rs b/crates/socket-patch-core/src/api/ranking.rs index 395f790d..4aca7fac 100644 --- a/crates/socket-patch-core/src/api/ranking.rs +++ b/crates/socket-patch-core/src/api/ranking.rs @@ -25,6 +25,11 @@ //! //! `tier` is an access filter, not a ranking signal: callers drop the paid //! patches a free user cannot download before ranking. +//! +//! This order picks one patch per package. Which *packages* a capped scan +//! patches first is a different order, [`crate::rollout::rollout_cmp`]: +//! it reads the same severity ladder and advisory count, but never the +//! publish date, so a missing date cannot reshuffle the rollout queue. use std::cmp::{Ordering, Reverse}; @@ -159,7 +164,18 @@ pub fn cmp_search_results(a: &PatchSearchResult, b: &PatchSearchResult) -> Order /// never count, and neither does a missing date (the batch endpoint omits /// `publishedAt`), so an equal sibling is never reported as an update. pub fn batch_supersedes(candidate: &BatchPatchInfo, applied: &BatchPatchInfo) -> bool { - let (c, a) = (rank_batch_info(candidate), rank_batch_info(applied)); + key_supersedes(&rank_batch_info(candidate), &rank_batch_info(applied)) +} + +/// [`batch_supersedes`] over the by-package shape: the rule scan uses to +/// classify a recorded patch (ALREADY vs UPGRADE) and to report +/// `updates[]`, on the same records that pick the patch, so selection, +/// classification and reporting cannot disagree. +pub fn search_result_supersedes(candidate: &PatchSearchResult, recorded: &PatchSearchResult) -> bool { + key_supersedes(&rank_search_result(candidate), &rank_search_result(recorded)) +} + +fn key_supersedes(c: &RankKey<'_>, a: &RankKey<'_>) -> bool { if c.not_merged != a.not_merged { return !c.not_merged; } @@ -807,6 +823,45 @@ mod tests { ); } + // ── search_result_supersedes ───────────────────────────────────── + + #[test] + fn search_supersedes_on_merged_state_first() { + let merged = search_multi("z", "free", "2020-01-01T00:00:00Z", &["low", "low"]); + let single = search("a", "free", "2026-01-01T00:00:00Z", "critical"); + assert!(search_result_supersedes(&merged, &single)); + assert!(!search_result_supersedes(&single, &merged)); + } + + #[test] + fn search_supersedes_on_severity_between_unmerged() { + let crit = search("z", "free", "2020-01-01T00:00:00Z", "critical"); + let high = search("a", "free", "2026-01-01T00:00:00Z", "high"); + assert!(search_result_supersedes(&crit, &high)); + assert!(!search_result_supersedes(&high, &crit)); + } + + #[test] + fn search_supersedes_on_a_real_later_date_only() { + let newer = search("z", "free", "2026-01-01T00:00:00Z", "high"); + let older = search("a", "free", "2024-01-01T00:00:00Z", "high"); + assert!(search_result_supersedes(&newer, &older)); + assert!(!search_result_supersedes(&older, &newer)); + let undated = search("z", "free", "", "high"); + assert!(!search_result_supersedes(&undated, &older)); + assert!(!search_result_supersedes(&newer, &undated)); + } + + #[test] + fn search_supersedes_ignores_tier_and_uuid_tiebreaks() { + let paid = search("a", "paid", "2026-01-01T00:00:00Z", "high"); + let free = search("z", "free", "2026-01-01T00:00:00Z", "high"); + assert_eq!(cmp_search_results(&paid, &free), Ordering::Less); + assert!(!search_result_supersedes(&paid, &free)); + assert!(!search_result_supersedes(&free, &paid)); + assert!(!search_result_supersedes(&paid, &paid)); + } + #[test] fn batch_without_published_at_still_ranks_by_severity() { // The batch endpoint historically omits `publishedAt`; losing the diff --git a/crates/socket-patch-core/src/hosted/engine.rs b/crates/socket-patch-core/src/hosted/engine.rs index bc0afe8a..2d9173b9 100644 --- a/crates/socket-patch-core/src/hosted/engine.rs +++ b/crates/socket-patch-core/src/hosted/engine.rs @@ -287,7 +287,7 @@ pub fn withhold_everywhere( } /// The project's candidate files as the rewriters read them. -#[derive(Debug, Default)] +#[derive(Debug, Clone, Default)] pub struct CandidateFiles { /// Readable candidate texts, keyed by project-relative path. pub files: BTreeMap, diff --git a/crates/socket-patch-core/src/hosted/memory/discover.rs b/crates/socket-patch-core/src/hosted/memory/discover.rs index 1a7758c8..6e5b36e0 100644 --- a/crates/socket-patch-core/src/hosted/memory/discover.rs +++ b/crates/socket-patch-core/src/hosted/memory/discover.rs @@ -15,7 +15,7 @@ use std::time::Duration; use crate::api::client::{ApiError, ApiFuture, PatchApi}; use crate::api::ranking::cmp_search_results; use crate::api::types::{ - BatchPackagePatches, PackageVendorResult, PatchResponse, PatchSearchResult, SearchResponse, + BatchPackagePatches, PackageVendorResult, PatchResponse, SearchResponse, }; use crate::utils::purl::{normalize_purl, strip_purl_qualifiers}; @@ -280,29 +280,6 @@ pub(crate) async fn fetch_details( .collect() } -/// The disk `--json` selection over one root's merged detail results: -/// accessible patches only, then the top-ranked patch per purl, sorted by -/// purl, as `(purl, uuid)`. -pub(crate) fn select_top_ranked( - results: &[PatchSearchResult], - can_access_paid: bool, -) -> Vec<(String, String)> { - let mut by_purl: BTreeMap<&str, Vec<&PatchSearchResult>> = BTreeMap::new(); - for patch in results - .iter() - .filter(|p| can_access_paid || p.tier == "free") - { - by_purl.entry(patch.purl.as_str()).or_default().push(patch); - } - by_purl - .into_iter() - .filter_map(|(purl, mut group)| { - group.sort_by(|a, b| cmp_search_results(a, b)); - group.first().map(|p| (purl.to_string(), p.uuid.clone())) - }) - .collect() -} - /// Reference grants for every distinct uuid (`MAX_REFERENCE_BATCH` per /// request): the merged results, plus the uuids whose request failed. pub(crate) async fn fetch_references( @@ -389,6 +366,7 @@ pub(crate) async fn fetch_records( #[cfg(test)] mod tests { use super::*; + use crate::api::types::PatchSearchResult; use std::sync::atomic::{AtomicUsize, Ordering}; #[tokio::test] @@ -440,13 +418,21 @@ mod tests { result("pkg:npm/a@1", "a-paid", "paid", "critical"), result("pkg:npm/a@1", "a-free", "free", "low"), ]; + // The engine selects through the disk flow's own seam. + let pairs = |paid: bool| -> Vec<(String, String)> { + crate::rollout::stage::offers_from_results(&results, paid) + .selected + .into_iter() + .map(|(purl, p)| (purl, p.uuid)) + .collect() + }; assert_eq!( - select_top_ranked(&results, false), + pairs(false), vec![ ("pkg:npm/a@1".to_string(), "a-free".to_string()), ("pkg:npm/b@1".to_string(), "b-crit".to_string()) ] ); - assert_eq!(select_top_ranked(&results, true)[0].1, "a-paid"); + assert_eq!(pairs(true)[0].1, "a-paid"); } } diff --git a/crates/socket-patch-core/src/hosted/memory/limits.rs b/crates/socket-patch-core/src/hosted/memory/limits.rs index af11729c..9f895d6c 100644 --- a/crates/socket-patch-core/src/hosted/memory/limits.rs +++ b/crates/socket-patch-core/src/hosted/memory/limits.rs @@ -27,11 +27,30 @@ pub(crate) struct ResolvedOptions { pub(crate) provider_concurrency: usize, pub(crate) request_timeout: std::time::Duration, pub(crate) limits: ResolvedLimits, + /// `maxNewPatches` (`Some(None)` is `"none"`); see [`Self::max_new`]. + max_new_patches: Option>, + max_new_patches_cap: Option, + /// `inFlightPatches` as canonical base purls. + pub(crate) in_flight: std::collections::BTreeSet, pub(crate) policy_overrides: crate::policy::PolicyOverrides, pub(crate) policy_paths: Vec, pub(crate) policy_sha256: Option, } +impl ResolvedOptions { + /// The run-wide cap on NEW patches: the `maxNewPatches` option (reported + /// as `flag`), then the socket.yml `patches.maxNewPatches`, then + /// unlimited; `maxNewPatchesCap` only tightens it. + pub(crate) fn max_new(&self, file: Option) -> crate::rollout::MaxNew { + crate::rollout::resolve_max_new( + self.max_new_patches, + None, + file, + self.max_new_patches_cap, + ) + } +} + pub(crate) fn resolve_options(options: &HostedScanOptions) -> Result { if options.org_slug.trim().is_empty() { return Err(EngineError::invalid( @@ -128,6 +147,14 @@ pub(crate) fn resolve_options(options: &HostedScanOptions) -> Result, summary: ProjectSummary, packages: Vec, + /// Every accessible offer per purl, and the winner per purl. + offers: Offers, + /// Purls whose detail query failed. + failed_details: Vec, + /// The classified rows (§5.1) the run-wide rollout plan spends on. + rows: Vec, selected: Vec<(String, String)>, skipped: Vec, + deferred: Vec, /// Candidates the socket.yml policy withheld (`policy_*` reasons). policy_skipped: Vec, error: Option, @@ -365,6 +377,56 @@ fn unrooted_unsupported_warnings<'a>( } } +/// One root's recorded view (§5.1) in memory: its `.socket/manifest.json`, +/// the hosted pins its lockfiles name, and its vendor ledger — the disk +/// merge's precedence. A pin is a mention of an offered uuid for the purl +/// in one of the root's own files (a nested root's files are its own); a +/// pin to a patch the API no longer offers reads as NEW, which costs one +/// slot once instead of stalling. +fn memory_recorded( + project: &MemoryProject, + root: &str, + roots: &[String], + offers: &Offers, +) -> RecordedIndex { + let manifest = project + .text(select::MANIFEST_REL) + .and_then(|text| serde_json::from_str(text).ok()); + let vendor = stages::vendored_entries(project); + let nested: Vec = roots + .iter() + .filter(|other| other.as_str() != root) + .filter_map(|other| roots::strip_root(root, other).map(|rel| format!("{rel}/"))) + .filter(|rel| rel != "/") + .collect(); + let mut mentioned = std::collections::HashSet::new(); + for (path, entry) in project.entries() { + if path.starts_with(".socket/") || nested.iter().any(|n| path.starts_with(n.as_str())) { + continue; + } + if let MemoryEntry::Text(text) = entry { + mentioned_uuids(text, &mut mentioned); + } + } + let pins: Vec<(String, String)> = offers + .selected + .iter() + .filter_map(|(purl, selected)| { + let offered = offers.unfiltered.get(purl)?; + std::iter::once(selected) + .chain(offered.iter()) + .find(|p| mentioned.contains(&p.uuid.to_ascii_lowercase())) + .map(|p| (purl.clone(), p.uuid.clone())) + }) + .collect(); + let merged = crate::ledgers::merge_ledger_records_for_updates( + manifest.as_ref(), + vendor.as_ref(), + &pins, + ); + RecordedIndex::new(merged.as_deref(), &pins) +} + async fn engine( input: HostedScanInput, api: Arc, @@ -487,8 +549,12 @@ async fn engine( purls: Vec::new(), summary: ProjectSummary::default(), packages: Vec::new(), + offers: Offers::default(), + failed_details: Vec::new(), + rows: Vec::new(), selected: Vec::new(), skipped: Vec::new(), + deferred: Vec::new(), policy_skipped: Vec::new(), error: None, }) @@ -555,6 +621,8 @@ async fn engine( .collect(); let batch = discover::batch_search(&provider, &root_purls, options.batch_size).await; let can_access_paid = batch.can_access_paid_patches; + // A package a failed batch hid could have been NEW (§5.2). + let mut batch_failed = false; for state in states.iter_mut().filter(|s| s.error.is_none()) { state.summary.can_access_paid_patches = can_access_paid; let Some(outcome) = batch.roots.get(&state.root) else { @@ -569,6 +637,7 @@ async fn engine( continue; } if outcome.failed_purls > 0 { + batch_failed = true; warnings.push(EngineWarning::new( "batch_failed", format!( @@ -609,8 +678,14 @@ async fn engine( for pkg in &state.packages { match details.get(&pkg.purl) { Some(Ok(response)) => results.extend(response.patches.iter().cloned()), - Some(Err(error)) => failures.push(error.clone()), - None => failures.push("patch details were not fetched".to_string()), + Some(Err(error)) => { + failures.push(error.clone()); + state.failed_details.push(pkg.purl.clone()); + } + None => { + failures.push("patch details were not fetched".to_string()); + state.failed_details.push(pkg.purl.clone()); + } } } if !failures.is_empty() && failures.len() == state.packages.len() { @@ -632,7 +707,7 @@ async fn engine( Some(&state.root), )); } - state.selected = select_with_policy( + state.offers = select_with_policy( &policy, results, can_access_paid, @@ -643,6 +718,39 @@ async fn engine( } phases.mark("details"); + // Classify every root's selection against its recorded view (§5.1): + // the tree's manifest and vendor ledger, and the hosted pins its + // lockfiles name. ALREADY rows carry the recorded uuid, so a re-scan + // re-confirms a pin instead of swapping it. + let mut stage = Stage::new(options.max_new(policy.max_new_patches()), None, std::path::Path::new("")); + // A root whose every lookup failed hides packages that could have been + // NEW: a capped run then admits none anywhere (§5.2). + stage.incomplete |= states + .iter() + .any(|s| s.error.as_ref().is_some_and(|e| e.code == "patch_lookup_failed")); + let roots_by_path: Vec = states.iter().map(|s| s.root.clone()).collect(); + for state in states.iter_mut().filter(|s| s.error.is_none()) { + let Some(project) = state.project.as_ref() else { + continue; + }; + let recorded = memory_recorded(project, &state.root, &roots_by_path, &state.offers); + stage.incomplete |= lookup_incomplete( + &recorded, + &state.failed_details, + batch_failed, + ); + let mut rows = classify(&state.offers, &recorded, &state.root); + for row in &mut rows { + row.candidate.in_flight = options.in_flight.contains(&row.candidate.base_purl); + } + state.selected = rows + .iter() + .map(|r| (r.writer.purl.clone(), r.writer.uuid.clone())) + .collect(); + state.rows = rows; + } + phases.mark("classify"); + let uuids: BTreeSet = states .iter() .filter(|s| s.error.is_none()) @@ -653,12 +761,24 @@ async fn engine( } else { discover::fetch_references(&provider, &uuids).await }; + // Roots whose rows' eligibility is unknown: a reference failure that + // hit only NEW rows of a capped run defers them instead of failing the + // root (§5.2). + let mut unknown_roots: BTreeSet = BTreeSet::new(); for state in states.iter_mut().filter(|s| s.error.is_none()) { if let Some(error) = state .selected .iter() .find_map(|(_, uuid)| failed_refs.get(uuid)) { + if stage.capped() && state.rows.iter().all(|r| r.candidate.recorded.is_new()) { + stage.incomplete = true; + stage.reference_failed = Some(error.clone()); + unknown_roots.insert(state.root.clone()); + state.selected.clear(); + continue; + } + stage.incomplete = true; state.fail( "reference_lookup_failed", format!("failed to resolve patch references: {error}"), @@ -693,16 +813,20 @@ async fn engine( }; phases.mark("plan"); - let stage = StageOptions { + let stage_options = StageOptions { dry_run: options.dry_run, pipenv_major: options.pipenv_major, trust_lockfile_config: options.trust_lockfile_config, npm_allow_remote_config: options.npm_allow_remote_config, }; let mut rewritten: Vec<(usize, Rewritten)> = Vec::new(); + let mut first_plans: BTreeMap = BTreeMap::new(); for (index, plan) in planned { checkpoint(&cancel).await?; - match stages::rewrite(plan, &wheel_metadata, stage).await { + if stage.capped() { + first_plans.insert(index, plan.clone()); + } + match stages::rewrite(plan, &wheel_metadata, stage_options).await { Ok(done) => rewritten.push((index, done)), Err(RewriteRefused { refusal, skipped }) => { states[index].skipped = skipped; @@ -712,6 +836,103 @@ async fn engine( } phases.mark("rewrite"); + // The run-wide rollout plan (§5.2): one budget across every root, spent + // after each root's write-free checks (grants, takeover refusals, vlt, + // wheel metadata, the rewrite's confirmation probe). A root with + // deferred rows is rewritten again without them. + let confirmed: BTreeSet<(String, String)> = rewritten + .iter() + .flat_map(|(index, done)| { + let root = states[*index].root.clone(); + done.done + .confirmed + .iter() + .map(move |(_, uuid)| (root.clone(), uuid.clone())) + }) + .collect(); + let all_rows: Vec = states + .iter() + .filter(|s| s.error.is_none()) + .flat_map(|s| s.rows.iter().cloned()) + .collect(); + stage.plan(&all_rows, |row| { + unknown_roots.contains(&row.candidate.project) + || confirmed.contains(&(row.candidate.project.clone(), row.writer.uuid.clone())) + }); + let deferred_rows: Vec<(crate::rollout::Candidate, u32)> = + stage.plan.as_ref().map(|p| p.deferred.clone()).unwrap_or_default(); + if !deferred_rows.is_empty() { + let root_index: BTreeMap = states + .iter() + .enumerate() + .map(|(i, s)| (s.root.clone(), i)) + .collect(); + for (row, rank) in &deferred_rows { + let Some(&i) = root_index.get(&row.project) else { + continue; + }; + states[i].deferred.push(DeferredPatch { + purl: row.purl.clone(), + uuid: row.uuid.clone(), + severity: crate::rollout::severity_label(row.severity_order).into(), + rank: *rank, + }); + } + let deferred_skip = |d: &DeferredPatch| SkippedPatch { + purl: d.purl.clone(), + uuid: d.uuid.clone(), + reason: ROLLOUT_DEFERRED.to_string(), + detail: Some(format!( + "rank {} in the rollout queue; a later scan adds it", + d.rank + )), + }; + let mut again: Vec<(usize, Rewritten)> = Vec::with_capacity(rewritten.len()); + for (index, done) in rewritten { + let root_deferred: BTreeSet = states[index] + .deferred + .iter() + .map(|d| d.uuid.clone()) + .collect(); + if root_deferred.is_empty() { + again.push((index, done)); + continue; + } + checkpoint(&cancel).await?; + // The first pass's skips stay (wheel metadata the rewrite could + // not fetch): its candidates are already gone, so the second + // pass cannot report them again. + let Some(mut plan) = first_plans.remove(&index) else { + again.push((index, done)); + continue; + }; + plan.candidates + .retain(|c| !root_deferred.contains(&c.dep.patch_uuid)); + plan.skipped + .extend(states[index].deferred.iter().map(deferred_skip)); + match stages::rewrite(plan, &wheel_metadata, stage_options).await { + Ok(done) => again.push((index, done)), + Err(RewriteRefused { refusal, skipped }) => { + states[index].skipped = skipped; + states[index].error = Some(ProjectError::from(refusal)); + } + } + } + rewritten = again; + // Roots that never reached the rewrite (unknown eligibility) list + // their deferred rows as skipped too. + for state in states.iter_mut() { + if unknown_roots.contains(&state.root) && state.error.is_none() { + let extra: Vec = state.deferred.iter().map(deferred_skip).collect(); + state.skipped.extend(extra); + } + } + } + for (code, detail) in stage.warnings() { + warnings.push(EngineWarning::new(code, detail, None)); + } + phases.mark("rollout"); + let record_uuids: BTreeSet = if options.dry_run { BTreeSet::new() } else { @@ -768,6 +989,7 @@ async fn engine( summary: state.summary.clone(), redirected: Vec::new(), skipped, + deferred: state.deferred.clone(), error: state.error.clone(), }); } @@ -802,6 +1024,7 @@ async fn engine( changed_binary_files, deleted_files: Vec::new(), warnings, + rollout: stage.json(), stats, engine_version: engine_version(), policy: Some(policy_block(&policy, &policy_filtered, &[])), @@ -842,6 +1065,7 @@ fn policy_error_output( changed_binary_files: Vec::new(), deleted_files: Vec::new(), warnings, + rollout: serde_json::Value::Null, stats: EngineStats { files_input, bytes_input, @@ -867,7 +1091,7 @@ fn select_with_policy( root: &str, filtered: &mut Vec, skipped: &mut Vec, -) -> Vec<(String, String)> { +) -> Offers { let accessible: Vec = results .into_iter() .filter(|p| can_access_paid || p.tier == "free") @@ -883,8 +1107,8 @@ fn select_with_policy( .collect(), ) }; - let selected = discover::select_top_ranked(&admitted, true); - let chosen: BTreeSet<&str> = selected.iter().map(|(purl, _)| purl.as_str()).collect(); + let offers = offers_from_results(&admitted, true); + let chosen: BTreeSet<&str> = offers.selected.keys().map(String::as_str).collect(); let mut by_purl: BTreeMap> = BTreeMap::new(); for (patch, reason) in dropped { if !chosen.contains(patch.purl.as_str()) { @@ -908,7 +1132,7 @@ fn select_with_policy( reason, }); } - selected + offers } /// Records → the project's result and changed files. @@ -1000,6 +1224,7 @@ fn finish_root( summary: state.summary.clone(), redirected: Vec::new(), skipped, + deferred: state.deferred.clone(), error: Some(ProjectError { code: "conflicting_write".into(), message, @@ -1045,6 +1270,7 @@ fn finish_root( .map(|(purl, uuid)| RedirectedPatch { purl, uuid }) .collect(), skipped, + deferred: state.deferred.clone(), error: None, } } @@ -1062,8 +1288,12 @@ mod tests { purls: Vec::new(), summary: ProjectSummary::default(), packages: Vec::new(), + offers: Offers::default(), + failed_details: Vec::new(), + rows: Vec::new(), selected: Vec::new(), skipped: Vec::new(), + deferred: Vec::new(), policy_skipped: Vec::new(), error: None, } diff --git a/crates/socket-patch-core/src/hosted/memory/select.rs b/crates/socket-patch-core/src/hosted/memory/select.rs index be43ec8e..fb4dfc83 100644 --- a/crates/socket-patch-core/src/hosted/memory/select.rs +++ b/crates/socket-patch-core/src/hosted/memory/select.rs @@ -41,8 +41,12 @@ const MAX_PATH_DEPTH: usize = 64; /// The vendored-mode ledger (its presence refuses a vendored takeover). pub(crate) const VENDOR_STATE_REL: &str = ".socket/vendor/state.json"; +/// The agent-mode manifest: part of the recorded view the rollout cap +/// classifies against (a package it records is not NEW). +pub(crate) const MANIFEST_REL: &str = ".socket/manifest.json"; + /// Root-relative text files read beyond `REDIRECT_CANDIDATE_FILES`. -const EXTRA_TEXT_FILES: [&str; 3] = [PNPM_WORKSPACE_REL, NPMRC_REL, VENDOR_STATE_REL]; +const EXTRA_TEXT_FILES: [&str; 4] = [PNPM_WORKSPACE_REL, NPMRC_REL, VENDOR_STATE_REL, MANIFEST_REL]; /// The one directory name the disk Cargo member walk never enters (it /// follows `members`, `exclude`, path dependencies and `[patch]` paths diff --git a/crates/socket-patch-core/src/hosted/memory/stages.rs b/crates/socket-patch-core/src/hosted/memory/stages.rs index a2b8d403..61944e59 100644 --- a/crates/socket-patch-core/src/hosted/memory/stages.rs +++ b/crates/socket-patch-core/src/hosted/memory/stages.rs @@ -38,7 +38,7 @@ pub(crate) struct StageOptions { /// The vendored ledger's entries (the disk `vendor::load_state` parse, /// including its legacy `{mode}`-only shape); `None` when absent or /// unreadable. -fn vendored_entries(project: &MemoryProject) -> Option { +pub(crate) fn vendored_entries(project: &MemoryProject) -> Option { let bytes: Vec = match project.get(VENDOR_STATE_REL)? { MemoryEntry::Text(text) => text.as_bytes().to_vec(), MemoryEntry::Binary(bytes) => bytes.to_vec(), @@ -142,7 +142,7 @@ fn refuse_takeovers( /// A project's state between the reference grants and the wheel-metadata /// fetch. -#[derive(Debug)] +#[derive(Debug, Clone)] pub(crate) struct Planned { pub(crate) project: MemoryProject, pub(crate) candidates: Vec, diff --git a/crates/socket-patch-core/src/hosted/memory/types.rs b/crates/socket-patch-core/src/hosted/memory/types.rs index e982c14e..2a11daed 100644 --- a/crates/socket-patch-core/src/hosted/memory/types.rs +++ b/crates/socket-patch-core/src/hosted/memory/types.rs @@ -115,6 +115,68 @@ pub struct HostedScanOptions { /// `policyError` when the policy it reads differs. #[serde(default, skip_serializing_if = "Option::is_none")] pub policy_sha256: Option, + /// The run-wide cap on NEW patches (`scan --max-new-patches`); absent + /// or `"none"` is unlimited, 0 admits upgrades only. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_new_patches: Option, + /// A server ceiling applied on top of `maxNewPatches`, `"none"` + /// included: it can tighten the cap, never loosen it. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_new_patches_cap: Option, + /// Base purls already proposed in an open rollout PR: ranked first, so + /// a newly published patch never displaces one under review. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub in_flight_patches: Option>, +} + +/// `maxNewPatches`: a count, or `"none"` (`None`). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct MaxNewPatchesOption(pub Option); + +impl Serialize for MaxNewPatchesOption { + fn serialize(&self, s: S) -> Result { + match self.0 { + Some(n) => s.serialize_u32(n), + None => s.serialize_str("none"), + } + } +} + +impl<'de> Deserialize<'de> for MaxNewPatchesOption { + fn deserialize>(d: D) -> Result { + struct Visitor; + impl serde::de::Visitor<'_> for Visitor { + type Value = MaxNewPatchesOption; + fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result { + f.write_str("a patch count (0 to 4294967295) or \"none\"") + } + fn visit_u64(self, v: u64) -> Result { + u32::try_from(v) + .map(|n| MaxNewPatchesOption(Some(n))) + .map_err(|_| E::custom("maxNewPatches exceeds 4294967295")) + } + fn visit_i64(self, v: i64) -> Result { + u64::try_from(v) + .map_err(|_| E::custom("maxNewPatches must not be negative")) + .and_then(|v| self.visit_u64(v)) + } + fn visit_f64(self, v: f64) -> Result { + if v.fract() == 0.0 && (0.0..=f64::from(u32::MAX)).contains(&v) { + Ok(MaxNewPatchesOption(Some(v as u32))) + } else { + Err(E::custom("maxNewPatches must be a whole number of patches")) + } + } + fn visit_str(self, v: &str) -> Result { + if v == "none" { + Ok(MaxNewPatchesOption(None)) + } else { + Err(E::custom(format!("maxNewPatches must be a number or \"none\", not `{v}`"))) + } + } + } + d.deserialize_any(Visitor) + } } pub const DEFAULT_BATCH_SIZE: u32 = 100; @@ -244,10 +306,26 @@ pub struct ProjectResult { pub summary: ProjectSummary, pub redirected: Vec, pub skipped: Vec, + /// NEW patches over the run-wide `maxNewPatches` budget, in rank order + /// (also in `skipped[]` as `rollout_deferred`). + #[serde(default)] + pub deferred: Vec, #[serde(default, skip_serializing_if = "Option::is_none")] pub error: Option, } +/// One deferred NEW patch. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct DeferredPatch { + pub purl: String, + pub uuid: String, + /// `critical` … `unknown`. + pub severity: String, + /// 1-based rank among the run's eligible NEW base purls. + pub rank: u32, +} + #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] pub struct ChangedFile { @@ -292,6 +370,10 @@ pub struct HostedScanOutput { pub changed_binary_files: Vec, pub deleted_files: Vec, pub warnings: Vec, + /// The session-level `rollout` block, the CLI `--json` shape; absent + /// when the session failed on its socket.yml (`policyError`). + #[serde(default, skip_serializing_if = "serde_json::Value::is_null")] + pub rollout: serde_json::Value, pub stats: EngineStats, pub engine_version: String, /// The session-level `policy` block (the CLI's `policy` JSON shape); diff --git a/crates/socket-patch-core/src/hosted/vlt.rs b/crates/socket-patch-core/src/hosted/vlt.rs index e2e83aaf..85f02cf9 100644 --- a/crates/socket-patch-core/src/hosted/vlt.rs +++ b/crates/socket-patch-core/src/hosted/vlt.rs @@ -61,7 +61,7 @@ pub(crate) fn bun_lockb_present(view: &ProjectView<'_>) -> bool { } /// What the artifact preflight decided for this run's npm candidates. -#[derive(Debug, Default)] +#[derive(Debug, Clone, Default)] pub struct Preflight { /// Failed while vlt drives, or for a vlt-vendored takeover: withheld /// from every rewriter. diff --git a/crates/socket-patch-core/src/ledgers.rs b/crates/socket-patch-core/src/ledgers.rs index 5ad5e73d..9bcf5662 100644 --- a/crates/socket-patch-core/src/ledgers.rs +++ b/crates/socket-patch-core/src/ledgers.rs @@ -22,6 +22,7 @@ //! lockfile evidence (`vex`). The hosted records only ever join as the //! last store. +use std::borrow::Cow; use std::collections::{HashMap, HashSet}; use std::path::Path; @@ -370,6 +371,59 @@ pub fn uuid_only_record(uuid: &str) -> PatchRecord { } } + +/// Fold the hosted pins and the vendor ledger's patch records into the +/// manifest view update detection consults. Hosted mode records purl→uuid +/// ONLY in the lockfiles (`hosted_pins`, uuid only; v5 keeps no hosted +/// ledger) and vendored mode ONLY in `.socket/vendor/state.json`, so without +/// this fold a pure hosted or vendored project's `updates[]` would always +/// be empty. Precedence on a collision: manifest > hosted pins > vendor +/// ledger (the live lock over a possibly superseded vendored entry); the +/// vendor entries fold under the shared owner rule +/// ([`Ledgers::owned`]), so one the manifest +/// claims (its key or base purl) stays behind the manifest's record. +/// Vendor entries are keyed by their ledger key (`detect_updates` bridges +/// the spellings); a legacy entry without an embedded record contributes +/// its uuid alone. Borrows the manifest untouched when nothing else +/// contributes. +pub fn merge_ledger_records_for_updates<'a>( + manifest: Option<&'a PatchManifest>, + vendor: Option<&VendorState>, + hosted_pins: &[(String, String)], +) -> Option> { + let vendor = vendor.filter(|s| !s.entries.is_empty()); + if vendor.is_none() && hosted_pins.is_empty() { + return manifest.map(Cow::Borrowed); + } + let mut merged = manifest.cloned().unwrap_or_default(); + for (purl, uuid) in hosted_pins { + merged + .patches + .entry(purl.clone()) + .or_insert_with(|| uuid_only_record(uuid)); + } + let ledgers = Ledgers { + manifest, + vendor, + redirect: None, + }; + for owned in ledgers.owned() { + if owned.store == Store::Manifest { + continue; + } + merged + .patches + .entry(owned.key.to_string()) + .or_insert_with(|| { + owned + .record + .cloned() + .unwrap_or_else(|| uuid_only_record(owned.uuid)) + }); + } + Some(Cow::Owned(merged)) +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/socket-patch-core/src/lib.rs b/crates/socket-patch-core/src/lib.rs index c16adb55..f257d0be 100644 --- a/crates/socket-patch-core/src/lib.rs +++ b/crates/socket-patch-core/src/lib.rs @@ -8,6 +8,7 @@ pub mod ledgers; pub mod manifest; pub mod patch; pub mod policy; +pub mod rollout; pub mod telemetry; pub mod update; pub mod utils; diff --git a/crates/socket-patch-core/src/rollout.rs b/crates/socket-patch-core/src/rollout.rs new file mode 100644 index 00000000..ac955e4c --- /dev/null +++ b/crates/socket-patch-core/src/rollout.rs @@ -0,0 +1,696 @@ +//! Staged rollout: the per-run cap on NEW patches (`scan +//! --max-new-patches`, `patches.maxNewPatches` in socket.yml). +//! +//! Pure planning only. Callers classify each selected `(project, purl)` row +//! against the recorded state ([`Recorded`]), decide eligibility with their +//! planning pass, and hand the rows to [`plan_rollout`], which admits the +//! most critical NEW packages up to the budget and defers the rest. Rows +//! that already carry a patch (ALREADY, UPGRADE) never count toward the cap. +//! +//! The order ([`rollout_cmp`]) is total and time-independent, so the same +//! inputs always give the same plan and repeated runs converge: run k lands +//! the top N, run k+1 sees them as recorded and lands the next N. + +pub mod stage; + +use std::cmp::{Ordering, Reverse}; +use std::collections::{BTreeMap, BTreeSet}; + +/// What the recorded state (manifest > hosted pins > vendor ledger) says +/// about one selected row. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Recorded { + /// No patch recorded for this base purl in this project: NEW. + None, + /// The recorded uuid is the selected one: ALREADY. + Same, + /// A different uuid is recorded and the selection does not supersede + /// it: ALREADY, and the writer keeps `uuid`. + Kept { uuid: String }, + /// The selection supersedes the recorded uuid, or the recorded uuid is + /// no longer offered: UPGRADE. + Superseded { old_uuid: String }, +} + +impl Recorded { + pub fn is_new(&self) -> bool { + matches!(self, Recorded::None) + } +} + +/// One selected `(project, purl)` row. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Candidate { + /// Repo-relative project root; `""` is the repo root. + pub project: String, + pub purl: String, + /// [`canonical_base_purl`] of `purl`: the budget unit. + pub base_purl: String, + /// The selected uuid. + pub uuid: String, + /// `Ecosystem::cli_name`. + pub ecosystem: &'static str, + /// `ranking::max_severity_order` of the selected patch (0 = critical). + pub severity_order: u8, + pub advisory_count: usize, + pub recorded: Recorded, + /// Every check the planning pass can decide without writing passed. + /// Only read for NEW rows. + pub eligible: bool, + /// Already proposed in an open rollout PR (in-memory engine option). + pub in_flight: bool, +} + +/// Where the effective cap came from. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum MaxNewSource { + Flag, + Env, + File, + Default, + /// A server ceiling (`maxNewPatchesCap`) tightened the value. + Cap, +} + +impl MaxNewSource { + pub fn as_str(self) -> &'static str { + match self { + MaxNewSource::Flag => "flag", + MaxNewSource::Env => "env", + MaxNewSource::File => "file", + MaxNewSource::Default => "default", + MaxNewSource::Cap => "cap", + } + } +} + +/// The effective cap. `value: None` is unlimited. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct MaxNew { + pub value: Option, + pub source: MaxNewSource, +} + +impl MaxNew { + pub const UNLIMITED: MaxNew = MaxNew { + value: None, + source: MaxNewSource::Default, + }; +} + +/// Resolve the cap: flag > env > file > unlimited, then a server `cap` +/// tightens it (never loosens; it applies to `none` too). For `flag` and +/// `env`, `Some(None)` is an explicit `none`. +pub fn resolve_max_new( + flag: Option>, + env: Option>, + file: Option, + cap: Option, +) -> MaxNew { + let chosen = if let Some(value) = flag { + MaxNew { + value, + source: MaxNewSource::Flag, + } + } else if let Some(value) = env { + MaxNew { + value, + source: MaxNewSource::Env, + } + } else if let Some(value) = file { + MaxNew { + value: Some(value), + source: MaxNewSource::File, + } + } else { + MaxNew::UNLIMITED + }; + match cap { + Some(cap) if chosen.value.is_none_or(|v| v > cap) => MaxNew { + value: Some(cap), + source: MaxNewSource::Cap, + }, + _ => chosen, + } +} + +/// The budget unit: ecosystem + name + version, qualifiers stripped, +/// percent-decoded and case-folded where the ecosystem is case-insensitive +/// (discovery's [`crate::vex::discover::canonical_base_purl`], the key +/// hosted pins carry), so qualifier twins (a wheel and its sdist, gem +/// platforms), the API's encoded spelling and a lockfile's `Newtonsoft.Json` +/// vs a pin's `newtonsoft.json` are one package. +pub fn canonical_base_purl(purl: &str) -> String { + crate::vex::discover::canonical_base_purl(purl) +} + +/// Rollout order, most urgent first: in-flight, severity, advisory count +/// (descending), ecosystem, base purl, uuid. Total, and free of +/// time-dependent keys. +pub fn rollout_cmp(a: &Candidate, b: &Candidate) -> Ordering { + rollout_key(a).cmp(&rollout_key(b)) +} + +type RolloutKey<'a> = (bool, u8, Reverse, &'a str, &'a str, &'a str); + +fn rollout_key(c: &Candidate) -> RolloutKey<'_> { + ( + !c.in_flight, + c.severity_order, + Reverse(c.advisory_count), + c.ecosystem, + c.base_purl.as_str(), + c.uuid.as_str(), + ) +} + +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct RolloutCounts { + /// Distinct base purls admitted as NEW. + pub new: u32, + /// Distinct base purls deferred. + pub deferred: u32, + /// UPGRADE rows. + pub upgrade: u32, + /// ALREADY rows (same or kept). + pub already: u32, +} + +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct RolloutPlan { + /// Rows the writers receive: admitted NEW rows plus every ALREADY and + /// UPGRADE row, in input order. Ineligible NEW rows are in neither + /// list; they keep their own skip reasons. + pub admitted: Vec, + /// Eligible NEW rows over the budget, in rank order, each with the + /// 1-based rank of its base purl among eligible NEW base purls. + pub deferred: Vec<(Candidate, u32)>, + pub counts: RolloutCounts, + /// Budget left for the next project directory; `None` is unlimited. + pub remaining: Option, + /// Base purls admitted so far in this invocation (input set included). + pub admitted_base_purls: BTreeSet, +} + +/// Admit eligible NEW base purls in [`rollout_cmp`] order until the budget +/// is spent. A base purl already in `already_admitted` (an earlier +/// directory of the same invocation) is admitted without spending budget. +/// With a finite cap and `incomplete` set (a lookup failed for a package +/// that could have been NEW), no NEW row is admitted: a missing package +/// must not let lower-ranked ones take its slot. +pub fn plan_rollout( + candidates: Vec, + max_new: &MaxNew, + incomplete: bool, + already_admitted: &BTreeSet, +) -> RolloutPlan { + let mut counts = RolloutCounts::default(); + let mut remaining = max_new.value; + let mut admitted_base_purls = already_admitted.clone(); + + let mut new_rows: Vec<&Candidate> = candidates + .iter() + .filter(|c| c.recorded.is_new() && c.eligible) + .collect(); + new_rows.sort_by(|a, b| rollout_cmp(a, b)); + // Base purl → (rank, admitted), in rank order. + let mut decisions: BTreeMap<&str, (u32, bool)> = BTreeMap::new(); + let mut rank = 0u32; + for row in &new_rows { + if decisions.contains_key(row.base_purl.as_str()) { + continue; + } + rank += 1; + let admit = if incomplete && max_new.value.is_some() { + false + } else if already_admitted.contains(&row.base_purl) { + true + } else { + match remaining.as_mut() { + None => true, + Some(0) => false, + Some(left) => { + *left -= 1; + true + } + } + }; + if admit { + counts.new += 1; + admitted_base_purls.insert(row.base_purl.clone()); + } else { + counts.deferred += 1; + } + decisions.insert(row.base_purl.as_str(), (rank, admit)); + } + + let deferred: Vec<(Candidate, u32)> = new_rows + .iter() + .filter_map(|row| match decisions[row.base_purl.as_str()] { + (rank, false) => Some(((*row).clone(), rank)), + _ => None, + }) + .collect(); + let mut admitted = Vec::new(); + for row in &candidates { + match &row.recorded { + Recorded::None => { + if row.eligible && decisions[row.base_purl.as_str()].1 { + admitted.push(row.clone()); + } + } + Recorded::Superseded { .. } => { + counts.upgrade += 1; + admitted.push(row.clone()); + } + Recorded::Same | Recorded::Kept { .. } => { + counts.already += 1; + admitted.push(row.clone()); + } + } + } + RolloutPlan { + admitted, + deferred, + counts, + remaining, + admitted_base_purls, + } +} + +/// The severity label for a `ranking::severity_order` value. +pub fn severity_label(order: u8) -> &'static str { + match order { + 0 => "critical", + 1 => "high", + 2 => "medium", + 3 => "low", + _ => "unknown", + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn row(project: &str, purl: &str, uuid: &str, severity: u8, advisories: usize) -> Candidate { + Candidate { + project: project.to_string(), + purl: purl.to_string(), + base_purl: canonical_base_purl(purl), + uuid: uuid.to_string(), + ecosystem: purl + .strip_prefix("pkg:") + .and_then(|r| r.split('/').next()) + .map(|e| match e { + "npm" => "npm", + "pypi" => "pypi", + "cargo" => "cargo", + "gem" => "gem", + _ => "golang", + }) + .unwrap_or("npm"), + severity_order: severity, + advisory_count: advisories, + recorded: Recorded::None, + eligible: true, + in_flight: false, + } + } + + fn cap(n: u32) -> MaxNew { + MaxNew { + value: Some(n), + source: MaxNewSource::Flag, + } + } + + fn admitted_purls(plan: &RolloutPlan) -> Vec<&str> { + plan.admitted.iter().map(|c| c.purl.as_str()).collect() + } + + fn deferred_purls(plan: &RolloutPlan) -> Vec<(&str, u32)> { + plan.deferred + .iter() + .map(|(c, r)| (c.purl.as_str(), *r)) + .collect() + } + + fn nine() -> Vec { + vec![ + row("", "pkg:npm/a@1", "u1", 3, 1), + row("", "pkg:npm/b@1", "u2", 0, 1), + row("", "pkg:npm/c@1", "u3", 1, 1), + row("", "pkg:npm/d@1", "u4", 2, 1), + row("", "pkg:npm/e@1", "u5", 0, 2), + row("", "pkg:npm/f@1", "u6", 4, 1), + row("", "pkg:npm/g@1", "u7", 1, 3), + row("", "pkg:npm/h@1", "u8", 2, 1), + row("", "pkg:npm/i@1", "u9", 3, 1), + ] + } + + fn permutations(items: &[T]) -> Vec> { + if items.len() <= 1 { + return vec![items.to_vec()]; + } + let mut out = Vec::new(); + for i in 0..items.len() { + let mut rest = items.to_vec(); + let head = rest.remove(i); + for mut tail in permutations(&rest) { + tail.insert(0, head.clone()); + out.push(tail); + } + } + out + } + + #[test] + fn rollout_cmp_is_a_total_order_every_permutation_sorts_the_same() { + let mut in_flight = row("", "pkg:pypi/z@1", "u9", 4, 0); + in_flight.in_flight = true; + let items = vec![ + row("", "pkg:npm/b@1", "u2", 1, 1), + row("", "pkg:npm/a@1", "u3", 1, 1), + row("", "pkg:cargo/a@1", "u1", 1, 1), + row("", "pkg:npm/m@1", "u0", 1, 2), + row("", "pkg:npm/a@1", "u1", 1, 1), + in_flight, + ]; + let expected: Vec<(String, String)> = vec![ + ("pkg:pypi/z@1".into(), "u9".into()), + ("pkg:npm/m@1".into(), "u0".into()), + ("pkg:cargo/a@1".into(), "u1".into()), + ("pkg:npm/a@1".into(), "u1".into()), + ("pkg:npm/a@1".into(), "u3".into()), + ("pkg:npm/b@1".into(), "u2".into()), + ]; + for mut perm in permutations(&items) { + perm.sort_by(rollout_cmp); + let got: Vec<(String, String)> = perm + .iter() + .map(|c| (c.purl.clone(), c.uuid.clone())) + .collect(); + assert_eq!(got, expected); + } + } + + #[test] + fn severity_then_advisory_count_decide_before_names() { + let mut rows = [ + row("", "pkg:npm/a@1", "u", 2, 5), + row("", "pkg:npm/b@1", "u", 0, 1), + row("", "pkg:npm/c@1", "u", 0, 3), + row("", "pkg:npm/d@1", "u", 4, 9), + ]; + rows.sort_by(rollout_cmp); + let order: Vec<&str> = rows.iter().map(|c| c.purl.as_str()).collect(); + assert_eq!( + order, + ["pkg:npm/c@1", "pkg:npm/b@1", "pkg:npm/a@1", "pkg:npm/d@1"] + ); + } + + #[test] + fn a_cap_admits_the_most_critical_and_defers_the_rest_with_ranks() { + let plan = plan_rollout(nine(), &cap(3), false, &BTreeSet::new()); + assert_eq!( + admitted_purls(&plan), + ["pkg:npm/b@1", "pkg:npm/e@1", "pkg:npm/g@1"] + ); + assert_eq!( + deferred_purls(&plan), + [ + ("pkg:npm/c@1", 4), + ("pkg:npm/d@1", 5), + ("pkg:npm/h@1", 6), + ("pkg:npm/a@1", 7), + ("pkg:npm/i@1", 8), + ("pkg:npm/f@1", 9), + ] + ); + assert_eq!( + plan.counts, + RolloutCounts { + new: 3, + deferred: 6, + upgrade: 0, + already: 0 + } + ); + assert_eq!(plan.remaining, Some(0)); + } + + #[test] + fn three_runs_roll_nine_packages_forward_and_a_fourth_changes_nothing() { + let mut state = nine(); + let mut landed: Vec> = Vec::new(); + for _ in 0..4 { + let plan = plan_rollout(state.clone(), &cap(3), false, &BTreeSet::new()); + let new: Vec = plan + .admitted + .iter() + .filter(|c| c.recorded.is_new()) + .map(|c| c.purl.clone()) + .collect(); + for c in &mut state { + if new.contains(&c.purl) { + c.recorded = Recorded::Same; + } + } + landed.push(new); + } + assert_eq!(landed[0], ["pkg:npm/b@1", "pkg:npm/e@1", "pkg:npm/g@1"]); + assert_eq!(landed[1], ["pkg:npm/c@1", "pkg:npm/d@1", "pkg:npm/h@1"]); + assert_eq!(landed[2], ["pkg:npm/a@1", "pkg:npm/f@1", "pkg:npm/i@1"]); + assert!(landed[3].is_empty()); + } + + #[test] + fn zero_admits_no_new_rows_but_keeps_upgrades_and_already() { + let mut rows = nine(); + rows[0].recorded = Recorded::Superseded { + old_uuid: "old".into(), + }; + rows[1].recorded = Recorded::Same; + rows[2].recorded = Recorded::Kept { uuid: "k".into() }; + let plan = plan_rollout(rows, &cap(0), false, &BTreeSet::new()); + assert_eq!( + admitted_purls(&plan), + ["pkg:npm/a@1", "pkg:npm/b@1", "pkg:npm/c@1"] + ); + assert_eq!(plan.counts.new, 0); + assert_eq!(plan.counts.deferred, 6); + assert_eq!(plan.counts.upgrade, 1); + assert_eq!(plan.counts.already, 2); + assert_eq!(plan.deferred[0].1, 1, "ranks start at 1 among NEW rows"); + } + + #[test] + fn a_cap_of_one_admits_exactly_the_top_package() { + let plan = plan_rollout(nine(), &cap(1), false, &BTreeSet::new()); + assert_eq!(admitted_purls(&plan), ["pkg:npm/e@1"]); + assert_eq!(plan.counts.deferred, 8); + } + + #[test] + fn unlimited_and_a_cap_above_the_supply_admit_everything() { + for max in [MaxNew::UNLIMITED, cap(50)] { + let plan = plan_rollout(nine(), &max, false, &BTreeSet::new()); + assert_eq!(plan.admitted.len(), 9); + assert!(plan.deferred.is_empty()); + assert_eq!(plan.counts.new, 9); + } + let plan = plan_rollout(nine(), &cap(50), false, &BTreeSet::new()); + assert_eq!(plan.remaining, Some(41)); + let plan = plan_rollout(nine(), &MaxNew::UNLIMITED, false, &BTreeSet::new()); + assert_eq!(plan.remaining, None); + } + + #[test] + fn ineligible_rows_hold_no_slot_and_are_not_reported_as_deferred() { + let mut rows = nine(); + // The top two by rank cannot land. + rows[4].eligible = false; // e + rows[1].eligible = false; // b + let plan = plan_rollout(rows, &cap(2), false, &BTreeSet::new()); + assert_eq!(admitted_purls(&plan), ["pkg:npm/c@1", "pkg:npm/g@1"]); + assert!(plan + .deferred + .iter() + .all(|(c, _)| c.purl != "pkg:npm/b@1" && c.purl != "pkg:npm/e@1")); + assert_eq!(plan.counts.deferred, 5); + assert_eq!(plan.deferred[0].1, 3); + } + + #[test] + fn incomplete_lookups_admit_nothing_new_under_a_cap() { + let mut rows = nine(); + rows[0].recorded = Recorded::Superseded { + old_uuid: "old".into(), + }; + let plan = plan_rollout(rows.clone(), &cap(3), true, &BTreeSet::new()); + assert_eq!(admitted_purls(&plan), ["pkg:npm/a@1"]); + assert_eq!(plan.counts.deferred, 8); + assert_eq!(plan.remaining, Some(3), "nothing was spent"); + // Without a cap there is no slot to steal. + let plan = plan_rollout(rows, &MaxNew::UNLIMITED, true, &BTreeSet::new()); + assert_eq!(plan.admitted.len(), 9); + } + + #[test] + fn one_package_across_roots_costs_one_slot() { + let rows = vec![ + row("services/api", "pkg:npm/qs@6.5.2", "u1", 1, 1), + row("services/web", "pkg:npm/qs@6.5.2", "u1", 1, 1), + row("", "pkg:npm/minimist@1.2.5", "u2", 0, 1), + row("", "pkg:npm/zzz@1.0.0", "u3", 3, 1), + ]; + let plan = plan_rollout(rows, &cap(2), false, &BTreeSet::new()); + assert_eq!( + admitted_purls(&plan), + [ + "pkg:npm/qs@6.5.2", + "pkg:npm/qs@6.5.2", + "pkg:npm/minimist@1.2.5" + ] + ); + assert_eq!(plan.counts.new, 2); + assert_eq!(deferred_purls(&plan), [("pkg:npm/zzz@1.0.0", 3)]); + } + + #[test] + fn qualifier_twins_share_a_base_purl_and_a_rank() { + assert_eq!( + canonical_base_purl("pkg:pypi/foo@1.0?artifact_id=abc"), + canonical_base_purl("pkg:pypi/foo@1.0?artifact_id=def") + ); + assert_eq!( + canonical_base_purl("pkg:npm/%40scope/x@1.0.0"), + "pkg:npm/@scope/x@1.0.0" + ); + assert_eq!( + canonical_base_purl("pkg:nuget/Newtonsoft.Json@13.0.3"), + canonical_base_purl("pkg:nuget/newtonsoft.json@13.0.3") + ); + assert_eq!( + canonical_base_purl("pkg:pypi/Foo_Bar@1.0"), + canonical_base_purl("pkg:pypi/foo-bar@1.0") + ); + let rows = vec![ + row("", "pkg:pypi/foo@1.0?artifact_id=whl", "u2", 1, 1), + row("", "pkg:pypi/foo@1.0?artifact_id=sdist", "u1", 1, 1), + row("", "pkg:pypi/bar@1.0", "u3", 0, 1), + ]; + let plan = plan_rollout(rows, &cap(1), false, &BTreeSet::new()); + assert_eq!(admitted_purls(&plan), ["pkg:pypi/bar@1.0"]); + assert_eq!( + deferred_purls(&plan), + [ + ("pkg:pypi/foo@1.0?artifact_id=sdist", 2), + ("pkg:pypi/foo@1.0?artifact_id=whl", 2) + ] + ); + assert_eq!(plan.counts.deferred, 1); + } + + // The ecosystem key agrees with the purl type prefix of the base purl + // key below it; the test pins the combined order. + #[test] + fn ties_across_ecosystems_break_by_ecosystem_then_name() { + let rows = vec![ + row("", "pkg:npm/x@1", "u1", 1, 1), + row("", "pkg:cargo/x@1", "u2", 1, 1), + row("", "pkg:pypi/x@1", "u3", 1, 1), + row("", "pkg:gem/x@1", "u4", 1, 1), + ]; + let plan = plan_rollout(rows, &cap(2), false, &BTreeSet::new()); + assert_eq!(admitted_purls(&plan), ["pkg:cargo/x@1", "pkg:gem/x@1"]); + assert_eq!( + deferred_purls(&plan), + [("pkg:npm/x@1", 3), ("pkg:pypi/x@1", 4)] + ); + } + + #[test] + fn in_flight_rows_go_first_whatever_their_severity() { + let mut rows = nine(); + rows[5].in_flight = true; // f, unknown severity + let plan = plan_rollout(rows, &cap(1), false, &BTreeSet::new()); + assert_eq!(admitted_purls(&plan), ["pkg:npm/f@1"]); + } + + #[test] + fn remaining_budget_carries_across_directories() { + let first = vec![ + row("a", "pkg:npm/x@1", "u1", 0, 1), + row("a", "pkg:npm/y@1", "u2", 1, 1), + ]; + let plan_a = plan_rollout(first, &cap(3), false, &BTreeSet::new()); + assert_eq!(plan_a.remaining, Some(1)); + let carried = MaxNew { + value: plan_a.remaining, + source: MaxNewSource::Flag, + }; + let second = vec![ + row("b", "pkg:npm/x@1", "u1", 0, 1), + row("b", "pkg:npm/z@1", "u3", 2, 1), + row("b", "pkg:npm/w@1", "u4", 3, 1), + ]; + let plan_b = plan_rollout(second, &carried, false, &plan_a.admitted_base_purls); + assert_eq!(admitted_purls(&plan_b), ["pkg:npm/x@1", "pkg:npm/z@1"]); + assert_eq!(plan_b.remaining, Some(0)); + assert_eq!(deferred_purls(&plan_b), [("pkg:npm/w@1", 3)]); + assert_eq!(plan_b.admitted_base_purls.len(), 3); + } + + #[test] + fn an_empty_plan_is_empty() { + let plan = plan_rollout(Vec::new(), &cap(3), true, &BTreeSet::new()); + assert!(plan.admitted.is_empty() && plan.deferred.is_empty()); + assert_eq!(plan.counts, RolloutCounts::default()); + assert_eq!(plan.remaining, Some(3)); + } + + #[test] + fn resolve_max_new_precedence_table() { + use MaxNewSource::*; + type Case = ( + Option>, + Option>, + Option, + Option, + Option, + MaxNewSource, + ); + let cases: [Case; 10] = [ + (None, None, None, None, None, Default), + (None, None, Some(5), None, Some(5), File), + (None, Some(Some(2)), Some(5), None, Some(2), Env), + (Some(Some(1)), Some(Some(2)), Some(5), None, Some(1), Flag), + (Some(None), None, Some(5), None, None, Flag), + (None, Some(None), Some(5), None, None, Env), + (Some(None), None, None, Some(4), Some(4), Cap), + (None, None, None, Some(4), Some(4), Cap), + (Some(Some(9)), None, None, Some(4), Some(4), Cap), + (Some(Some(2)), None, None, Some(4), Some(2), Flag), + ]; + for (flag, env, file, cap, value, source) in cases { + assert_eq!( + resolve_max_new(flag, env, file, cap), + MaxNew { value, source }, + "flag={flag:?} env={env:?} file={file:?} cap={cap:?}" + ); + } + assert_eq!(resolve_max_new(None, None, Some(4), Some(4)).source, File); + } + + #[test] + fn severity_labels_follow_the_ladder() { + assert_eq!( + (0..=5).map(severity_label).collect::>(), + ["critical", "high", "medium", "low", "unknown", "unknown"] + ); + } +} diff --git a/crates/socket-patch-core/src/rollout/stage.rs b/crates/socket-patch-core/src/rollout/stage.rs new file mode 100644 index 00000000..eb6c08df --- /dev/null +++ b/crates/socket-patch-core/src/rollout/stage.rs @@ -0,0 +1,505 @@ +//! The per-run rollout stage (`docs/design/staged-rollout.md` §5, §9.2) +//! the disk scan and the in-memory engine share: classify the selected +//! offers against the recorded state, spend the budget on NEW packages +//! most critical first once each mode's eligibility checks ran, and +//! render the `rollout` block. + +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; +use std::path::{Path, PathBuf}; +use std::sync::{Arc, Mutex}; + +use crate::api::ranking::{cmp_search_results, max_severity_order, search_result_supersedes}; +use crate::api::types::PatchSearchResult; +use crate::crawlers::Ecosystem; +use crate::manifest::schema::PatchManifest; +pub use crate::policy::Offers; + +use super::{ + canonical_base_purl, plan_rollout, severity_label, Candidate, MaxNew, MaxNewSource, Recorded, + RolloutPlan, +}; + +/// The env binding of `scan --max-new-patches`. +pub const MAX_NEW_PATCHES_ENV: &str = "SOCKET_MAX_NEW_PATCHES"; + +/// What one invocation's project directories share: the cap as configured, +/// the budget left, and the base purls already admitted (admitted free in +/// later directories). +#[derive(Debug)] +pub struct Carry { + pub configured: MaxNew, + pub remaining: Option, + pub admitted: BTreeSet, + /// The directory the project paths are relative to. + pub root: PathBuf, +} + +#[derive(Debug, Clone)] +pub struct RolloutCarry(pub Arc>); + +impl RolloutCarry { + pub fn new(configured: MaxNew, root: PathBuf) -> Self { + RolloutCarry(Arc::new(Mutex::new(Carry { + configured, + remaining: configured.value, + admitted: BTreeSet::new(), + root, + }))) + } + + pub fn lock(&self) -> std::sync::MutexGuard<'_, Carry> { + self.0.lock().unwrap_or_else(|e| e.into_inner()) + } +} + +/// Warning: a lookup failed for a package that could have been NEW, so a +/// capped run admitted no NEW patch. +pub const ROLLOUT_INCOMPLETE_LOOKUP: &str = "rollout_incomplete_lookup"; +/// Warning: the reference lookup failed, but only for rows that were +/// deferred anyway, so the run went on. +pub const ROLLOUT_REFERENCE_FAILED: &str = "rollout_reference_failed"; +/// `skipped[].reason` of a deferred row. +pub const ROLLOUT_DEFERRED: &str = "rollout_deferred"; + +/// Group the by-package records per purl into the step 5 → 7 seam (work +/// item A's [`Offers`]), dropping paid patches the org cannot download, and +/// take the top-ranked offer per purl. +pub fn offers_from_results(results: &[PatchSearchResult], can_access_paid: bool) -> Offers { + let mut unfiltered: BTreeMap> = BTreeMap::new(); + for p in results { + if can_access_paid || p.tier == "free" { + unfiltered + .entry(p.purl.clone()) + .or_default() + .push(p.clone()); + } + } + for group in unfiltered.values_mut() { + group.sort_by(cmp_search_results); + group.dedup_by(|a, b| a.uuid == b.uuid); + } + let selected = unfiltered + .iter() + .filter_map(|(purl, group)| group.first().map(|p| (purl.clone(), p.clone()))) + .collect(); + Offers { + unfiltered, + selected, + } +} + +/// One classified row plus the offer its writer receives: the selection, +/// or the recorded patch when the selection does not supersede it. +#[derive(Debug, Clone)] +pub struct Row { + pub candidate: Candidate, + pub writer: PatchSearchResult, +} + +/// The recorded view (§5.1), indexed once so every row is an O(1) lookup: +/// the merged manifest (manifest > hosted pins > vendor ledger) plus every +/// hosted pin (the merge keeps one per key; a project can pin each +/// qualifier twin of a package to its own patch). +#[derive(Debug, Default)] +pub struct RecordedIndex { + exact: HashMap>, + /// Discovery's folded base purl plus the raw qualifier suffix. + qualified: HashMap>, + by_base: HashMap>, +} + +/// The recorded view one project root classifies against: the merged +/// manifest (`updates[]`'s batch fallback reads it) and its index. +pub struct RecordedState<'a> { + pub manifest: Option<&'a PatchManifest>, + pub index: RecordedIndex, +} + +/// `purl`'s folded base plus its qualifiers: equal for two spellings of the +/// same qualified purl (percent-encoding, case where it does not matter). +pub fn qualified_key(purl: &str) -> String { + let suffix = purl.find(['?', '#']).map_or("", |i| &purl[i..]); + format!("{}{suffix}", canonical_base_purl(purl)) +} + +impl RecordedIndex { + pub fn new(manifest: Option<&PatchManifest>, pins: &[(String, String)]) -> Self { + let mut index = RecordedIndex::default(); + let entries = manifest + .into_iter() + .flat_map(|m| m.patches.iter().map(|(k, r)| (k.as_str(), r.uuid.as_str()))) + .chain(pins.iter().map(|(p, u)| (p.as_str(), u.as_str()))); + for (key, uuid) in entries { + index + .exact + .entry(key.to_string()) + .or_default() + .push(uuid.to_string()); + index + .qualified + .entry(qualified_key(key)) + .or_default() + .push(uuid.to_string()); + index + .by_base + .entry(canonical_base_purl(key)) + .or_default() + .push(uuid.to_string()); + } + for list in index + .exact + .values_mut() + .chain(index.qualified.values_mut()) + .chain(index.by_base.values_mut()) + { + list.sort(); + list.dedup(); + } + index + } + + /// The uuids recorded for `purl`, sorted: the exact key, else the same + /// purl in another spelling, else any qualifier twin. + pub fn uuids(&self, purl: &str) -> &[String] { + self.exact + .get(purl) + .or_else(|| self.qualified.get(&qualified_key(purl))) + .or_else(|| self.by_base.get(&canonical_base_purl(purl))) + .map_or(&[], Vec::as_slice) + } + + /// Whether any patch is recorded for `purl`'s base purl. + pub fn records_package(&self, purl: &str) -> bool { + self.by_base.contains_key(&canonical_base_purl(purl)) + } +} + +/// Classify every selected purl of one project root (§5.1). `recorded` is +/// the merged view (manifest > hosted pins > vendor ledger). +pub fn classify(offers: &Offers, recorded: &RecordedIndex, project: &str) -> Vec { + offers + .selected + .iter() + .map(|(purl, selected)| { + let uuids = recorded.uuids(purl); + let offered = offers + .unfiltered + .get(purl) + .map(Vec::as_slice) + .unwrap_or(&[]); + let (class, writer) = if uuids.is_empty() { + (Recorded::None, selected.clone()) + } else if uuids.contains(&selected.uuid) { + (Recorded::Same, selected.clone()) + } else { + let old = uuids[0].clone(); + match offered.iter().find(|p| p.uuid == old) { + Some(prior) if !search_result_supersedes(selected, prior) => { + (Recorded::Kept { uuid: old }, prior.clone()) + } + _ => (Recorded::Superseded { old_uuid: old }, selected.clone()), + } + }; + Row { + candidate: Candidate { + project: project.to_string(), + purl: purl.clone(), + base_purl: canonical_base_purl(purl), + uuid: selected.uuid.clone(), + ecosystem: Ecosystem::from_purl(purl).map_or("", |e| e.cli_name()), + severity_order: max_severity_order( + selected + .vulnerabilities + .values() + .map(|v| v.severity.as_str()), + ), + advisory_count: selected.vulnerabilities.len(), + recorded: class, + eligible: true, + in_flight: false, + }, + writer, + } + }) + .collect() +} + +/// Whether a failed detail lookup hit a package that could have been NEW +/// (nothing recorded for it), or a whole batch failed (its packages are +/// unknown). +pub fn lookup_incomplete( + recorded: &RecordedIndex, + failed_details: &[String], + batch_failed: bool, +) -> bool { + batch_failed + || failed_details + .iter() + .any(|purl| !recorded.records_package(purl)) +} + +/// Every canonical-shaped uuid (`8-4-4-4-12` hex) `text` mentions, +/// lowercased, in one linear pass. +pub fn mentioned_uuids(text: &str, out: &mut HashSet) { + let bytes = text.as_bytes(); + if bytes.len() < 36 { + return; + } + let mut i = 0; + while i + 36 <= bytes.len() { + let window = &bytes[i..i + 36]; + let shaped = window.iter().enumerate().all(|(k, b)| match k { + 8 | 13 | 18 | 23 => *b == b'-', + _ => b.is_ascii_hexdigit(), + }); + if shaped { + out.insert(String::from_utf8_lossy(window).to_ascii_lowercase()); + i += 36; + } else { + i += 1; + } + } +} + +/// Mark NEW rows whose selected uuid the project's lockfile texts already +/// mention as ALREADY. A hosted pin on a patch server discovery does not +/// recognize (an origin missing from `--patch-server-url`) would otherwise +/// read as NEW on every run and hold its slot forever; patch uuids are +/// unique, so a mention is a pin. +pub fn mark_pinned(rows: &mut [Row], texts: &[&str]) { + if !rows.iter().any(|r| r.candidate.recorded.is_new()) { + return; + } + let mut mentioned = HashSet::new(); + for text in texts { + mentioned_uuids(text, &mut mentioned); + } + for row in rows.iter_mut().filter(|r| r.candidate.recorded.is_new()) { + if mentioned.contains(&row.candidate.uuid.to_ascii_lowercase()) { + row.candidate.recorded = Recorded::Same; + } + } +} + +/// One directory's budget and the outcome of its plan. +#[derive(Debug, Clone)] +pub struct Stage { + /// The cap as configured (reported). + pub configured: MaxNew, + /// The budget this directory may spend (the carried remainder). + pub budget: MaxNew, + pub already_admitted: BTreeSet, + /// A lookup failed for a package that could have been NEW. + pub incomplete: bool, + pub project: String, + pub carry: Option, + /// Set by [`Self::plan`]. + pub plan: Option, + /// The reference lookup failed and only deferred rows were affected. + pub reference_failed: Option, +} + +impl Stage { + /// The stage for one scan of `cwd`: a fresh budget, or the invocation's + /// shared one. + pub fn new(configured: MaxNew, carry: Option, cwd: &Path) -> Self { + let (budget, already_admitted, project) = match &carry { + Some(c) => { + let c = c.lock(); + ( + MaxNew { + value: c.remaining, + source: c.configured.source, + }, + c.admitted.clone(), + crate::policy::repo_relative( + &c.root, + &std::fs::canonicalize(cwd).unwrap_or_else(|_| cwd.to_path_buf()), + ), + ) + } + None => (configured, BTreeSet::new(), String::new()), + }; + Stage { + configured, + budget, + already_admitted, + incomplete: false, + project, + carry, + plan: None, + reference_failed: None, + } + } + + pub fn capped(&self) -> bool { + self.configured.value.is_some() + } + + /// Whether any NEW row can be admitted at all. + pub fn may_admit_new(&self) -> bool { + !(self.capped() && self.incomplete) && self.budget.value != Some(0) + } + + /// Plan `rows` with `eligible` deciding each NEW row, record the plan + /// and hand the remaining budget to the next directory. + pub fn plan(&mut self, rows: &[Row], eligible: impl Fn(&Row) -> bool) -> &RolloutPlan { + let candidates: Vec = rows + .iter() + .map(|row| Candidate { + eligible: !row.candidate.recorded.is_new() || eligible(row), + ..row.candidate.clone() + }) + .collect(); + let plan = plan_rollout( + candidates, + &self.budget, + self.incomplete, + &self.already_admitted, + ); + if let Some(carry) = &self.carry { + let mut c = carry.lock(); + c.remaining = plan.remaining; + c.admitted = plan.admitted_base_purls.clone(); + } + self.plan.insert(plan) + } + + /// `(purl, uuid)` of every deferred row. + pub fn deferred_keys(&self) -> HashSet<(String, String)> { + self.plan + .iter() + .flat_map(|p| &p.deferred) + .map(|(c, _)| (c.purl.clone(), c.uuid.clone())) + .collect() + } + + /// Run-level warnings the plan earned. + pub fn warnings(&self) -> Vec<(&'static str, String)> { + let mut out = Vec::new(); + if let Some(detail) = &self.reference_failed { + out.push(( + ROLLOUT_REFERENCE_FAILED, + format!( + "the hosted reference lookup failed ({detail}); only new patches were \ + affected and they were deferred to the next scan" + ), + )); + } + let deferred = self.plan.as_ref().map_or(0, |p| p.counts.deferred); + if self.capped() && self.incomplete && deferred > 0 { + out.push(( + ROLLOUT_INCOMPLETE_LOOKUP, + format!( + "a patch lookup failed for a package that could get its first patch, so \ + no new patches were added this run ({} deferred) and none can take the \ + missing package's place; re-run once the API answers", + if deferred == 1 { "1 package".to_string() } else { format!("{deferred} packages") } + ), + )); + } + out + } + + /// The top-level `rollout` block (§5.5). + pub fn json(&self) -> serde_json::Value { + rollout_json(&self.configured, self.plan.as_ref()) + } + + /// `redirect.skipped[]` entries mirroring the deferred rows. + pub fn deferred_skips(&self) -> Vec { + self.plan + .iter() + .flat_map(|p| &p.deferred) + .map(|(c, rank)| crate::hosted::engine::SkippedPatch { + purl: c.purl.clone(), + uuid: c.uuid.clone(), + reason: ROLLOUT_DEFERRED.to_string(), + detail: Some(format!("rank {rank} in the rollout queue; a later scan adds it")), + }) + .collect() + } +} + +pub fn source_label(max: &MaxNew) -> &'static str { + match max.source { + MaxNewSource::Flag => "--max-new-patches", + MaxNewSource::Env => MAX_NEW_PATCHES_ENV, + MaxNewSource::File => "socket.yml", + MaxNewSource::Cap => "the server cap", + MaxNewSource::Default => "default", + } +} + +/// One deferred package, grouped over its rows. +pub struct DeferredGroup { + pub base_purl: String, + pub uuids: BTreeSet, + pub severity_order: u8, + pub advisory_count: usize, + pub projects: BTreeSet, + pub rank: u32, +} + +pub fn deferred_groups(plan: &RolloutPlan) -> Vec { + let mut groups: Vec = Vec::new(); + let mut at: HashMap<&str, usize> = HashMap::new(); + for (c, rank) in &plan.deferred { + match at.get(c.base_purl.as_str()).map(|&i| &mut groups[i]) { + Some(g) => { + g.uuids.insert(c.uuid.clone()); + g.projects.insert(c.project.clone()); + g.severity_order = g.severity_order.min(c.severity_order); + g.advisory_count = g.advisory_count.max(c.advisory_count); + } + None => { + at.insert(c.base_purl.as_str(), groups.len()); + groups.push(DeferredGroup { + base_purl: c.base_purl.clone(), + uuids: BTreeSet::from([c.uuid.clone()]), + severity_order: c.severity_order, + advisory_count: c.advisory_count, + projects: BTreeSet::from([c.project.clone()]), + rank: *rank, + }) + } + } + } + groups.sort_by_key(|g| g.rank); + groups +} + +/// The `rollout` block. `plan: None` (a run that planned nothing) reports +/// zero counts. +pub fn rollout_json(configured: &MaxNew, plan: Option<&RolloutPlan>) -> serde_json::Value { + let counts = plan.map(|p| p.counts).unwrap_or_default(); + let deferred: Vec = plan + .map(deferred_groups) + .unwrap_or_default() + .into_iter() + .map(|g| { + serde_json::json!({ + "purl": g.base_purl, + "uuids": g.uuids, + "severity": severity_label(g.severity_order), + "advisoryCount": g.advisory_count, + "projects": g.projects, + "rank": g.rank, + }) + }) + .collect(); + serde_json::json!({ + "maxNewPatches": { + "value": configured.value, + "source": configured.source.as_str(), + }, + "counts": { + "new": counts.new, + "deferred": counts.deferred, + "upgrade": counts.upgrade, + "already": counts.already, + }, + "deferred": deferred, + }) +} + diff --git a/crates/socket-patch-core/src/vendor/lock_inventory/view.rs b/crates/socket-patch-core/src/vendor/lock_inventory/view.rs index 02e55b99..acd48d72 100644 --- a/crates/socket-patch-core/src/vendor/lock_inventory/view.rs +++ b/crates/socket-patch-core/src/vendor/lock_inventory/view.rs @@ -84,7 +84,6 @@ impl MemoryProject { matches!(self.entries.get(rel), Some(MemoryEntry::Symlink)) } - #[cfg(test)] pub(crate) fn entries(&self) -> impl Iterator { self.entries.iter().map(|(k, v)| (k.as_str(), v)) } diff --git a/crates/socket-patch-node/npm/index.d.ts b/crates/socket-patch-node/npm/index.d.ts index c70f28fa..a6cadfba 100644 --- a/crates/socket-patch-node/npm/index.d.ts +++ b/crates/socket-patch-node/npm/index.d.ts @@ -54,6 +54,9 @@ export interface HostedScanSessionOptions { providerConcurrency?: number // default 8 requestTimeoutMs?: number // per provider call, default 60000 limits?: HostedScanLimits + maxNewPatches?: number | 'none' // run-wide cap on NEW patches, most severe first; 0 = upgrades only; absent/'none' = unlimited + maxNewPatchesCap?: number // server ceiling: tightens maxNewPatches (including 'none'), never loosens it + inFlightPatches?: string[] // base purls already in the open rollout PR: ranked first noSocketYml?: boolean // ignore the repo's socket.yml (built-in test/fixture ignores still apply); default false minSeverity?: 'critical' | 'high' | 'medium' | 'moderate' | 'low' | 'none' // beats socket.yml patches.minSeverity policyPaths?: string[] // selectHostedScanPaths' policyPaths; each must be streamed with content or the session returns policyError @@ -73,15 +76,23 @@ export interface ProjectResult { redirect: Record // same shape as CLI `--json` `redirect` block summary: { scannedPackages: number; packagesWithPatches: number; totalPatches: number; freePatches: number; paidPatches: number; canAccessPaidPatches: boolean } redirected: { purl: string; uuid: string }[] - skipped: { purl: string; uuid: string; reason: string; detail?: string }[] // reasons include the policy_* codes + skipped: { purl: string; uuid: string; reason: string; detail?: string }[] // reasons include the policy_* codes; deferred rows appear as `rollout_deferred` + deferred: DeferredPatch[] // NEW patches over the maxNewPatches budget, rank order error?: { code: string; message: string } // project-level failure (e.g. corrupt_ledger, patch_lookup_failed) } +export interface DeferredPatch { purl: string; uuid: string; severity: 'critical' | 'high' | 'medium' | 'low' | 'unknown'; rank: number } +export interface RolloutBlock { // same shape as CLI `scan --json` `rollout` + maxNewPatches: { value: number | null; source: 'flag' | 'env' | 'file' | 'default' | 'cap' } + counts: { new: number; deferred: number; upgrade: number; already: number } + deferred: { purl: string; uuids: string[]; severity: string; advisoryCount: number; projects: string[]; rank: number }[] +} export interface HostedScanResult { projects: ProjectResult[] changedFiles: { path: string; content: string }[] // repo-relative, sorted, only byte-changed, includes ledgers (wet runs only) changedBinaryFiles: { path: string; content: Buffer }[] deletedFiles: string[] warnings: EngineWarning[] + rollout?: RolloutBlock // session-level: one budget across every project; absent only with policyError stats: { projects: number; filesInput: number; bytesInput: number; packagesScanned: number; packagesWithPatches: number; patchesSelected: number; patchesRedirected: number; filesChanged: number; providerCalls: Record; phaseMs: Record } engineVersion: string policy?: PolicyBlock // absent only with policyError diff --git a/crates/socket-patch-node/npm/test/smoke.mjs b/crates/socket-patch-node/npm/test/smoke.mjs index 826932ca..0009f955 100644 --- a/crates/socket-patch-node/npm/test/smoke.mjs +++ b/crates/socket-patch-node/npm/test/smoke.mjs @@ -234,6 +234,24 @@ test('dry run previews the lockfile without patch fetches', async () => { assert.equal(calls.fetchPatch, 0) }) +test('maxNewPatches defers new patches and reports the rollout block', async () => { + const { provider } = fakeProvider() + const selection = addon.selectHostedScanPaths(tree) + const session = new addon.HostedScanSession({ orgSlug: 'test-org', maxNewPatches: 0 }, provider) + streamSelection(session, selection) + const result = await session.finish() + const [project] = result.projects + assert.deepEqual(project.redirected, []) + assert.deepEqual( + project.deferred.map((d) => [d.purl, d.rank]), + [['pkg:npm/left-pad@1.3.0', 1]], + ) + assert.ok(project.skipped.some((s) => s.reason === 'rollout_deferred')) + assert.deepEqual(result.rollout.maxNewPatches, { value: 0, source: 'flag' }) + assert.deepEqual(result.rollout.counts, { new: 0, deferred: 1, upgrade: 0, already: 0 }) + assert.deepEqual(result.changedFiles, []) +}) + test('provider failures become project errors, never rejections', async () => { for (const searchPatchesBatch of [ async () => ({ ok: false, error: { kind: 'unauthorized', message: 'token revoked' } }), diff --git a/docs/design/staged-rollout.md b/docs/design/staged-rollout.md index 5a9c32ca..09a0f5a9 100644 --- a/docs/design/staged-rollout.md +++ b/docs/design/staged-rollout.md @@ -1108,3 +1108,73 @@ Gaps and contradictions A resolved with the smallest reasonable decision the docs, to keep validators from drifting. - Recognizing a dependency version bump of an already-patched package as exempt from the cap (needs state hosted mode does not keep). + +## 11. Implementation notes (work item B) + +Where the plan left a gap, work item B made the smallest decision that +keeps its rules intact: + +- **`updates[]` without by-package data.** A `--json` report-only run + (`--prune` / `--global` with no mode) fetches no by-package records, so + its `updates[]` stays on the batch rungs (`batch_supersedes`). In every + other run `updates[]` is the UPGRADE rows, plus the batch-derived entry + for a package the by-package lookup returned no offer for (nothing was + selected there to disagree with). +- **Report-only and empty scans** carry the `rollout` block with zero + counts: they plan nothing. +- **Unrecognized hosted pins.** Discovery only treats URLs on + `patch.socket.dev` or a `--patch-server-url` origin as hosted pins. A + scan against another server without that flag would read every pin as + NEW and re-spend the same N slots forever. The hosted gate therefore + also counts a NEW row as ALREADY when a lockfile names its selected + uuid (uuids are unique; one linear scan per file). +- **In-memory recorded view.** The engine has no disk discovery, so a + root's hosted pins are the offered uuids of each purl that its own + files mention (nested roots' files excluded), merged with the tree's + `.socket/manifest.json` (now selected by `selectHostedScanPaths`) and + `.socket/vendor/state.json` at the disk precedence. A pin to a patch + the API no longer offers reads as NEW: it costs one slot once, and the + next run sees the new pin. +- **Option source.** The in-memory `maxNewPatches` option reports + `source: "flag"` (it is the caller's explicit layer); `maxNewPatchesCap` + reports `cap` when it tightens the value. +- **Key 6 across a base purl's rows.** Rows are sorted with + `rollout_cmp` and a base purl takes the rank of its first row, which is + the "minimum key over its rows" of 5.2; the uuid key only orders rows of + one base purl, which are admitted or deferred together. +- **Reference failures.** On disk the reference lookup is one call; when + it fails in a capped run whose rows are all NEW, the rows are deferred + with `rollout_reference_failed` and `rollout_incomplete_lookup` instead + of failing the run. In memory the same rule applies per root. +- **Human output.** The `Rollout:` line prints only when a cap is set. + Hosted mode appends the deferred lines to its existing next steps; + agent and vendored mode print them under a `Next steps:` heading. +- **Lock.** A wet hosted run whose only candidates are NEW rows it cannot + admit (budget 0, or incomplete data) takes no apply lock and writes + nothing, `.socket/` included. +- **Folding.** `canonical_base_purl` is discovery's, so a nuget or + composer pin (lowercased) and a pypi pin (PEP 503 name) match the API's + spelling, and a package in two spellings is one budget unit. Every hosted + pin joins the recorded index, not just the first per key, so both pinned + qualifier twins read ALREADY. +- **Flag over env.** An explicit `--max-new-patches` wins without parsing + `SOCKET_MAX_NEW_PATCHES`; whitespace-only env values are unset. +- **Known limits.** (1) In memory, the symlink / unreadable-file refusals + run inside the first rewrite, before the plan, so a candidate file that + only a deferred NEW row would rewrite still refuses its root (disk runs + its symlink guard after the gate). (2) On disk, a project directory that + fails outright spends nothing and does not freeze later directories. (3) + A NEW row that `mark_pinned` turns ALREADY in a run that could admit no + NEW row takes the apply lock only after its files were read. (4) Memory + pin evidence is any fetched text file of the root; disk's is its + discovery plus the candidate files. (5) No e2e case covers a + vlt-withheld top-ranked row; it takes the same ineligible path as the + withdrawn and `bad_purl` rows the e2e tests cover. +- **socket.yml layer.** B resolves the cap as flag > env > file > + unlimited through `resolve_max_new`; the file value is A's + `SelectionPolicy::max_new_patches()` (none when bypassed). The rollout + classifies what A's selection keeps: a retained package is neither NEW + nor an UPGRADE, and a severity floor removes a NEW row before it can + take a slot (9.3). The in-memory engine resolves the cap after the + session's socket.yml loads; a session that fails on it has no + `rollout` block.