From eadc047a8e8b2194f42190de23e83435dabf3ebe Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 4 Oct 2026 16:32:00 +0000 Subject: [PATCH 1/3] Start fix for #790 Assisted-by: Claude Code:claude-opus-5-5 From 77071a5fa461da4b611793b9c6a708686ea947f8 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 4 Oct 2026 16:37:03 +0000 Subject: [PATCH 2/3] Name the lock category in the Pipenv remedy When a warm Pipenv virtualenv still holds the unpatched release, the hosted and vendored warnings told users to run `pipenv run pip uninstall -y && pipenv sync` or `pipenv --rm && pipenv sync`. Plain `pipenv sync` installs only the default category, so for a package pinned in [dev-packages] or a named category, following the advice removed the package instead of reinstalling it patched, and the --rm form dropped every dev and category package. Both warnings now build the remedy from the Pipfile.lock: the targeted form re-syncs the categories that pin the package (`--dev`, or `--categories ""`), and the --rm form re-syncs every non-empty category. Fixes #790 Assisted-by: Claude Code:claude-opus-5-5 --- crates/socket-patch-cli/CLI_CONTRACT.md | 4 +- .../src/commands/scan/hosted.rs | 5 + .../src/commands/scan/hosted/python.rs | 81 +++++++- crates/socket-patch-core/src/vendor/pypi.rs | 51 +++++- .../src/vendor/pypi_pipenv.rs | 173 ++++++++++++++++++ docs/testing/pipenv-compatibility.md | 7 +- 6 files changed, 307 insertions(+), 14 deletions(-) diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 69fb09df9..dce546433 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -167,7 +167,7 @@ The rewriter reads a fixed set of candidate files from the project root: the npm **Gem stale-install guard (additive warning — the canonical narrative; other mentions point here)**: the gem hosted rewrite is pure Gemfile/lock text, so a gem ALREADY materialized under the project's bundle paths keeps its upstream bytes — the next `bundle install` prints `Using ` and never refetches, on **every** bundler major (live-verified 2026-08-19 on 1.17.3 / 2.7.2 / 4.0.18: bundler 4's CHECKSUMS verify at download time only, and nothing is downloaded; `bundle install --force`/`--redownload` re-install from the stale cached `.gem` instead of re-fetching — bundler 1 silently, bundler 4 with an exit-37 checksum refusal that still leaves the upstream bytes installed; the **verified** remedy is removing the installed dir + cache `.gem` + `specifications` entry, then `bundle install`). After the rewrite, a hosted run therefore probes the installed-gem discovery paths (the same ruby-crawler discovery `apply` uses, honoring `--global`/`--global-prefix` like scan's own discovery) for each confirmed gem redirect and judges the materialization against the patch record's `afterHash` file map. Judgment rules: records are found **by uuid** among this run's fetched records (v5.0: hosted mode persists no records, so a purl whose `/patches/view` fetch failed this run is not judged; the warning re-fires on every re-scan whose fetch succeeds, until the stale materialization is gone); a materialization with every file at `afterHash` is already patched and never warns (an agent→hosted migration stays quiet by construction), and when several confirmed variant purls resolve to one installed dir, ANY of them judging it patched keeps it quiet; staleness needs **positive evidence** — at least one record file whose bytes were actually read and hash to neither state's expectation — so missing or unreadable files never produce a warning. Warnings emit `redirect_gem_stale_install` (JSON `redirect.warnings[]` + a code-tagged stderr line) in three flavors: a PROJECT-LOCAL dir gets the verified delete-list remedy (installed dir, cache `.gem`, `specifications` entry — plus the project's committed `/.gem` when present and not proven to be the patched artifact, since bundler installs from its cache dir in preference to fetching); a SHARED gem-env home gets a caveat that the home is shared machine-wide and prefers migrating the project to a local bundle path over deleting shared files; and a committed cache-dir archive whose sha256 differs from the patched artifact's warns standalone even with no installed dir at all (a fresh checkout with a committed stale cache re-materializes the upstream bytes forever). A stale-flagged purl is additionally **excluded from the same run's `--vex` `assume_applied` set** — the envelope must never attest a CVE its own warning says is live; the purl falls back to normal installed-tree verification (a patched install still attests, a stale one is omitted). The cache dir is bundler's `cache_path` setting (`Bundler.app_cache`), resolved in `Bundler::Settings` priority: `BUNDLE_CACHE_PATH:` in the bundler app config (`$BUNDLE_APP_CONFIG/config`, else `.bundle/config`) first, then the `BUNDLE_CACHE_PATH` environment variable, else `vendor/cache`; a relative value is read against the project root. With `BUNDLE_IGNORE_CONFIG` set (any value) bundler reads no config file, so the app config is skipped here too and only the environment and the default count — the same holds for the `BUNDLE_GEMFILE:` app-config setting. The probe is read-only (nothing is deleted) and skipped on `--dry-run` — deliberately explicit, since nothing was rewritten. Exit code and `status` are unchanged (warning-only, the hosted-refusal posture); a same-run `--vex` may still fail on "nothing to attest" per the embedded-VEX contract. -**Pipenv hosted redirect (`Pipfile.lock`, pipfile-spec 6)**: every category other than `_meta` (`default`, `develop`, and Pipenv 2022+ named categories) that pins the package at the patched version is rewritten to the hosted reference — `{"file" | "path": "#sha256=", "hashes": ["sha256:"]}` with `markers`/`extras`/`index` kept exactly as Pipenv wrote them (present or absent: whether Pipenv records `index` depends on its release, the Pipfile spelling and the locking environment, so only the entry itself knows) and `version` dropped; `_meta` (the Pipfile content hash) and the Pipfile itself are never touched, so `pipenv install --deploy`/`sync`/`verify` keep passing. The reference KEY depends on the installing Pipenv: releases 7–11 only install `path` references, 2018 and later `file` ones (0–6 write pipfile-spec < 6 and are refused). The release is probed once per command with `pipenv --version`, resolved on ABSOLUTE `PATH` entries only (a relative entry would run a `pipenv` planted in the scanned repository; `.bat`/`.cmd` shims are found through `PATHEXT` on Windows), only when a pypi patch actually targets an entry of the lock, and `SOCKET_PIPENV_MAJOR=` pins the answer without spawning anything. An unknown installer selects `file` and warns `redirect_pipenv_installer_unknown` only when the lock was rewritten. **Refusal scope**: a pin/source CONFLICT (another version pinned, a foreign `file`/`path` source, a VCS/editable dependency) refuses the whole dependency atomically across categories as `redirect_pipenv_refused` AND vetoes the sibling Python rewriters (requirements.txt / uv.lock / pyproject) for that patch — the project's Pipenv install could not pick the patch up, so a half-redirected checkout is refused; anything else (no entry for the package, an old pipfile-spec, an unparseable lock, a digest-less patch) is `redirect_pipenv_skipped` and leaves the siblings alone (a stale Pipfile.lock in a uv/Poetry/requirements project must not block them). The veto applies to a LIVE lock only: a `Pipfile.lock` with no `Pipfile` beside it is abandoned, so its conflict refuses that file but never the siblings. Hash enforcement at install time is split by era — the `#sha256=` URL fragment is what Pipenv 2023+ verifies, the `hashes` list what 2018–2022 verify, Pipenv 11 either — so both are load-bearing. **Pipenv stale-install guard**: Pipenv never reinstalls a release that is already present (`pipenv install`, `install --deploy` and `sync` all exit 0 and keep the installed bytes — measured on 11.10.4, 2018.11.26 and 2026.8.0, hosted and vendored), so after the rewrite the run probes the Python crawler's site-packages (VIRTUAL_ENV, `./.venv`, `./venv`, Pipenv's out-of-tree `WORKON_HOME` venv; `--global`/`--global-prefix` honoured) for each confirmed Pipfile.lock redirect with the same rules as the gem guard (records by uuid from this run's fetch, PATCHED = `verify_patch_record` Ok, STALE needs positive evidence, read-only, skipped on `--dry-run`, stale purls excluded from the same-run `--vex` `assume_applied` set) and the Python stale-install guard (`redirect_pypi_stale_install`, see above) names the site-packages dir and the Pipenv-specific verified remedy: `pipenv run pip uninstall -y && pipenv sync` (or `pipenv --rm && pipenv sync`) — NOT `pipenv uninstall`, which rewrites the Pipfile and re-locks the patch away. The vendored backend emits the twin `pypi_pipenv_stale_install` (`skipped` warning event). **Rollback** (v5.0, upstream restore): each hosted entry gets its registry shape back — `"version": "=="`, the entry's own `index` carried back unchanged (refused unless it — and the Pipfile's explicit `index`, if any — names a PyPI source in `_meta.sources`), and every release file's sha256 from PyPI's JSON API (`SOCKET_PYPI_JSON_API`), sorted by filename as Pipenv records them; an entry that pins another version beside the hosted reference is refused with the `git checkout` remedy (see "Hosted unwind coverage"). A Pipfile names no project, so a same-run `--vex` on a Pipenv project needs `--vex-product` (or a git remote) to detect a product purl. **Discovery**: `Pipfile.lock` is part of the lockfile inventory (every category's `==` pins, with the lock's digest set as `Sha256AnyOf` integrity so a lock-only checkout can be vendored by fetching the pure wheel through PyPI's JSON API — only when `_meta.sources` name the public index; a private-index lock stays discovery-only and never reaches pypi.org), and Socket's own hosted / vendored references stay discoverable as the package they replace, so a re-scan of an already-redirected or already-vendored lock-only checkout re-confirms it (`--vex` attests, vendored reports `already_vendored`) instead of finding nothing. +**Pipenv hosted redirect (`Pipfile.lock`, pipfile-spec 6)**: every category other than `_meta` (`default`, `develop`, and Pipenv 2022+ named categories) that pins the package at the patched version is rewritten to the hosted reference — `{"file" | "path": "#sha256=", "hashes": ["sha256:"]}` with `markers`/`extras`/`index` kept exactly as Pipenv wrote them (present or absent: whether Pipenv records `index` depends on its release, the Pipfile spelling and the locking environment, so only the entry itself knows) and `version` dropped; `_meta` (the Pipfile content hash) and the Pipfile itself are never touched, so `pipenv install --deploy`/`sync`/`verify` keep passing. The reference KEY depends on the installing Pipenv: releases 7–11 only install `path` references, 2018 and later `file` ones (0–6 write pipfile-spec < 6 and are refused). The release is probed once per command with `pipenv --version`, resolved on ABSOLUTE `PATH` entries only (a relative entry would run a `pipenv` planted in the scanned repository; `.bat`/`.cmd` shims are found through `PATHEXT` on Windows), only when a pypi patch actually targets an entry of the lock, and `SOCKET_PIPENV_MAJOR=` pins the answer without spawning anything. An unknown installer selects `file` and warns `redirect_pipenv_installer_unknown` only when the lock was rewritten. **Refusal scope**: a pin/source CONFLICT (another version pinned, a foreign `file`/`path` source, a VCS/editable dependency) refuses the whole dependency atomically across categories as `redirect_pipenv_refused` AND vetoes the sibling Python rewriters (requirements.txt / uv.lock / pyproject) for that patch — the project's Pipenv install could not pick the patch up, so a half-redirected checkout is refused; anything else (no entry for the package, an old pipfile-spec, an unparseable lock, a digest-less patch) is `redirect_pipenv_skipped` and leaves the siblings alone (a stale Pipfile.lock in a uv/Poetry/requirements project must not block them). The veto applies to a LIVE lock only: a `Pipfile.lock` with no `Pipfile` beside it is abandoned, so its conflict refuses that file but never the siblings. Hash enforcement at install time is split by era — the `#sha256=` URL fragment is what Pipenv 2023+ verifies, the `hashes` list what 2018–2022 verify, Pipenv 11 either — so both are load-bearing. **Pipenv stale-install guard**: Pipenv never reinstalls a release that is already present (`pipenv install`, `install --deploy` and `sync` all exit 0 and keep the installed bytes — measured on 11.10.4, 2018.11.26 and 2026.8.0, hosted and vendored), so after the rewrite the run probes the Python crawler's site-packages (VIRTUAL_ENV, `./.venv`, `./venv`, Pipenv's out-of-tree `WORKON_HOME` venv; `--global`/`--global-prefix` honoured) for each confirmed Pipfile.lock redirect with the same rules as the gem guard (records by uuid from this run's fetch, PATCHED = `verify_patch_record` Ok, STALE needs positive evidence, read-only, skipped on `--dry-run`, stale purls excluded from the same-run `--vex` `assume_applied` set) and the Python stale-install guard (`redirect_pypi_stale_install`, see above) names the site-packages dir and the Pipenv-specific verified remedy: `pipenv run pip uninstall -y && pipenv sync` (or `pipenv --rm && pipenv sync`), with the `sync` arguments following the lock, since plain `pipenv sync` installs only `default`: the targeted form re-syncs the categories that pin the package (`--dev` for `develop`, `--categories ""` for a named category) and the `--rm` form re-syncs every non-empty category — NOT `pipenv uninstall`, which rewrites the Pipfile and re-locks the patch away. The vendored backend emits the twin `pypi_pipenv_stale_install` (`skipped` warning event). **Rollback** (v5.0, upstream restore): each hosted entry gets its registry shape back — `"version": "=="`, the entry's own `index` carried back unchanged (refused unless it — and the Pipfile's explicit `index`, if any — names a PyPI source in `_meta.sources`), and every release file's sha256 from PyPI's JSON API (`SOCKET_PYPI_JSON_API`), sorted by filename as Pipenv records them; an entry that pins another version beside the hosted reference is refused with the `git checkout` remedy (see "Hosted unwind coverage"). A Pipfile names no project, so a same-run `--vex` on a Pipenv project needs `--vex-product` (or a git remote) to detect a product purl. **Discovery**: `Pipfile.lock` is part of the lockfile inventory (every category's `==` pins, with the lock's digest set as `Sha256AnyOf` integrity so a lock-only checkout can be vendored by fetching the pure wheel through PyPI's JSON API — only when `_meta.sources` name the public index; a private-index lock stays discovery-only and never reaches pypi.org), and Socket's own hosted / vendored references stay discoverable as the package they replace, so a re-scan of an already-redirected or already-vendored lock-only checkout re-confirms it (`--vex` attests, vendored reports `already_vendored`) instead of finding nothing. **Mode ledgers (contract surfaces).** Vendored mode persists its state at a stable repo-relative path; external tools (and the depscan backend's GitHub-app PR flows) read and write it, so path + schema are part of the contract. Hosted mode (v5.0) persists nothing but its lockfile / config edits: @@ -1233,7 +1233,7 @@ Every `--json` invocation emits a single JSON object that follows the **unified | `pypi_pipenv_version_mismatch` | `failed` | vendor (pipenv): a category pins a different version than the patch — refused before any write. (`pypi_pipenv_invalid_wheel` retired in v5.0: the backend takes the orchestrator's resolved version instead of parsing the wheel filename.) | | `pypi_poetry_symlink_unsupported` / `pypi_pipenv_symlink_unsupported` / `pypi_requirements_symlink_unsupported` | `failed` | vendor (pypi, v5.0): a target file (`pyproject.toml` / `poetry.lock`, `Pipfile` / `Pipfile.lock`, or any planned `requirements*.txt`) is a symlink — refused before any write on wire AND on revert (the revert keeps the artifact, `kept_artifact`); the twins of the existing pdm/uv symlink refusals. | | `pypi_poetry_changed` / `pypi_pdm_changed` / `pypi_pipenv_changed` / `pypi_uv_changed` | `failed` | vendor (pypi, v5.0): the lock / project file changed between the read that planned the edit and the first write — refused before any write (worded like `pypi_lock_changed`: " changed during vendoring; re-run"). | -| `pypi_pipenv_stale_install` | `skipped` (warning) | vendor (pipenv): the vendored twin of `redirect_pypi_stale_install` — the project's venv still holds the upstream release Pipenv will not reinstall over; the detail names the `pipenv run pip uninstall -y && pipenv sync` remedy. | +| `pypi_pipenv_stale_install` | `skipped` (warning) | vendor (pipenv): the vendored twin of `redirect_pypi_stale_install` — the project's venv still holds the upstream release Pipenv will not reinstall over; the detail names the `pipenv run pip uninstall -y && pipenv sync` remedy, with the same lock-category `sync` arguments as the hosted warning. | | `pypi_pipenv_installer_unknown` | `skipped` (warning) | vendor (pipenv): no `pipenv` answered on PATH; the vendored references assume Pipenv 2018 or later (7–11 cannot consume them — use hosted mode there); `SOCKET_PIPENV_MAJOR` pins the release. | | `vendor_lock_entry_relocked` | revert `warnings[]` | vendor `--revert` / rollback (pipenv): a relock regenerated the wired entry to a registry reference, or removed it; the record is retired (artifact removed, ledger entry dropped) instead of drift-kept. | | `pypi_{poetry,pdm,pipenv}_no_lockfile` | `failed` | vendor (pypi): a lock-less tool marker with no `requirements.txt` fallback — run ` lock`. | diff --git a/crates/socket-patch-cli/src/commands/scan/hosted.rs b/crates/socket-patch-cli/src/commands/scan/hosted.rs index 97e6866ce..18f707cfc 100644 --- a/crates/socket-patch-cli/src/commands/scan/hosted.rs +++ b/crates/socket-patch-cli/src/commands/scan/hosted.rs @@ -1152,6 +1152,11 @@ pub(crate) async fn run_redirect_selected( common, &confirmed, &rewrite.confirmed_pipenv_uuids, + rewrite + .files + .get("Pipfile.lock") + .or_else(|| done.files.get("Pipfile.lock")) + .map(String::as_str), &records, ) .await diff --git a/crates/socket-patch-cli/src/commands/scan/hosted/python.rs b/crates/socket-patch-cli/src/commands/scan/hosted/python.rs index 8a5445673..b6fb5a2a8 100644 --- a/crates/socket-patch-cli/src/commands/scan/hosted/python.rs +++ b/crates/socket-patch-cli/src/commands/scan/hosted/python.rs @@ -18,11 +18,16 @@ pub(super) async fn stale_install_warnings( common: &crate::args::GlobalArgs, confirmed: &[(String, String)], pipenv_uuids: &BTreeSet, + // The run's final Pipfile.lock text: the remedy's `pipenv sync` + // arguments follow the categories that pin each package. + pipenv_lock: Option<&str>, // This run's fetched records MERGED with the ledger's persisted ones // (the caller hands the post-merge ledger map), looked up by uuid. records: &BTreeMap, ) -> StaleInstallOutcome { let mut out = StaleInstallOutcome::default(); + let pipenv_lock: Option = + pipenv_lock.and_then(|text| serde_json::from_str(text.trim_start_matches('\u{feff}')).ok()); let candidates: Vec<_> = confirmed .iter() .filter(|(purl, _)| purl.starts_with("pkg:pypi/")) @@ -111,15 +116,15 @@ pub(super) async fn stale_install_warnings( .and_then(|rest| rest.split('@').next()) .unwrap_or("") .to_string(); + let remedy = socket_patch_core::vendor::pypi_pipenv::stale_install_remedy( + pipenv_lock.as_ref(), + &name, + ); format!( "Pipenv does not reinstall a release that is already present (`pipenv \ install`, `pipenv install --deploy` and `pipenv sync` all keep those \ bytes), so the rewritten Pipfile.lock only protects fresh installs. \ - Reinstall it from the lock without touching the Pipfile: `pipenv run pip \ - uninstall -y {name} && pipenv sync` (`pipenv install --deploy` before \ - Pipenv 2018), or `pipenv --rm && pipenv sync` for a clean virtualenv — \ - NOT `pipenv uninstall`, which rewrites the Pipfile and re-locks the \ - patch away; then `socket-patch vex --product ` re-verifies the \ + {remedy}; then `socket-patch vex --product ` re-verifies the \ installed files." ) } else { @@ -194,7 +199,8 @@ mod tests { ("one".into(), record("first-uuid", "first.py", b"patched")), ("two".into(), record("second-uuid", "second.py", b"patched")), ]); - let out = stale_install_warnings(&common, &confirmed, &BTreeSet::new(), &ledger).await; + let out = + stale_install_warnings(&common, &confirmed, &BTreeSet::new(), None, &ledger).await; assert_eq!(out.stale_purls, BTreeSet::from([first.to_string()])); assert_eq!(out.warnings.len(), 1); assert!(out.warnings[0]["detail"] @@ -209,7 +215,8 @@ mod tests { "variant".into(), )); ledger.insert("three".into(), record("variant", "first.py", b"upstream")); - let out = stale_install_warnings(&common, &confirmed, &BTreeSet::new(), &ledger).await; + let out = + stale_install_warnings(&common, &confirmed, &BTreeSet::new(), None, &ledger).await; assert!(out.stale_purls.is_empty()); assert!(out.warnings.is_empty()); } @@ -233,8 +240,66 @@ mod tests { let purl = "pkg:pypi/six@1.16.0"; let confirmed = vec![(purl.to_string(), "six-uuid".to_string())]; let ledger = BTreeMap::from([("k".into(), record("six-uuid", "six.py", b"patched"))]); - let out = stale_install_warnings(&common, &confirmed, &BTreeSet::new(), &ledger).await; + let out = + stale_install_warnings(&common, &confirmed, &BTreeSet::new(), None, &ledger).await; assert_eq!(out.stale_purls, BTreeSet::from([purl.to_string()])); assert_eq!(out.warnings[0]["code"], "redirect_pypi_stale_install"); } + + /// #790: the Pipenv remedy names the lock category that pins the + /// package. Plain `pipenv sync` installs only `default`, so for a + /// `[dev-packages]` entry it uninstalled the package and left it + /// uninstalled. + #[tokio::test] + async fn pipenv_remedy_resyncs_the_category_that_pins_the_package() { + let tmp = tempfile::tempdir().unwrap(); + let site = tmp.path().join("site-packages"); + std::fs::create_dir_all(site.join("six-1.16.0.dist-info")).unwrap(); + std::fs::write(site.join("six.py"), b"upstream").unwrap(); + let common = crate::args::GlobalArgs { + cwd: tmp.path().to_path_buf(), + global_prefix: Some(site.clone()), + ..Default::default() + }; + let purl = "pkg:pypi/six@1.16.0"; + let confirmed = vec![(purl.to_string(), "six-uuid".to_string())]; + let ledger = BTreeMap::from([("k".into(), record("six-uuid", "six.py", b"patched"))]); + let pipenv = BTreeSet::from(["six-uuid".to_string()]); + let detail = |lock: &str| { + let lock = lock.to_string(); + let (common, confirmed, ledger, pipenv) = (&common, &confirmed, &ledger, &pipenv); + async move { + let out = + stale_install_warnings(common, confirmed, pipenv, Some(&lock), ledger).await; + assert_eq!(out.warnings.len(), 1); + out.warnings[0]["detail"].as_str().unwrap().to_string() + } + }; + + let develop = detail( + r#"{"_meta": {"pipfile-spec": 6}, "default": {}, "develop": {"six": {"file": "x"}}}"#, + ) + .await; + assert!( + develop.contains("`pipenv run pip uninstall -y six && pipenv sync --dev`"), + "{develop}" + ); + assert!( + develop.contains("`pipenv --rm && pipenv sync --dev`"), + "{develop}" + ); + + let docs = detail( + r#"{"_meta": {"pipfile-spec": 6}, "default": {"requests": {}}, "docs": {"six": {}}}"#, + ) + .await; + assert!( + docs.contains("-y six && pipenv sync --categories \"docs\"`"), + "{docs}" + ); + assert!( + docs.contains("`pipenv --rm && pipenv sync --categories \"packages docs\"`"), + "{docs}" + ); + } } diff --git a/crates/socket-patch-core/src/vendor/pypi.rs b/crates/socket-patch-core/src/vendor/pypi.rs index 74883c23e..4c2a41b3e 100644 --- a/crates/socket-patch-core/src/vendor/pypi.rs +++ b/crates/socket-patch-core/src/vendor/pypi.rs @@ -608,6 +608,7 @@ async fn pipenv_stale_install_warning( purl: &str, record: &PatchRecord, listings: &InstalledSiteListings, + lock: &serde_json::Value, ) -> Option { use crate::crawlers::python_crawler::{find_local_venv_site_packages, PythonCrawler}; use crate::patch::apply::{verify_file_patch, VerifyStatus}; @@ -657,10 +658,11 @@ async fn pipenv_stale_install_warning( .map(|d| d.display().to_string()) .collect::>() .join(", "); + let remedy = super::pypi_pipenv::stale_install_remedy(Some(lock), &name); Some(VendorWarning::new( "pypi_pipenv_stale_install", format!( - "{purl}: the UNPATCHED upstream release is still installed in {listed}. Pipenv does not reinstall a release that is already present (`pipenv install`, `pipenv install --deploy` and `pipenv sync` all keep those bytes), so the wired Pipfile.lock only protects fresh installs. Reinstall it from the lock without touching the Pipfile: `pipenv run pip uninstall -y {name} && pipenv sync` (`pipenv install --deploy` before Pipenv 2018), or `pipenv --rm && pipenv sync` for a clean virtualenv — NOT `pipenv uninstall`, which rewrites the Pipfile and re-locks the patch away; then `socket-patch vex` re-verifies the installed files." + "{purl}: the UNPATCHED upstream release is still installed in {listed}. Pipenv does not reinstall a release that is already present (`pipenv install`, `pipenv install --deploy` and `pipenv sync` all keep those bytes), so the wired Pipfile.lock only protects fresh installs. {remedy}; then `socket-patch vex` re-verifies the installed files." ), )) } @@ -878,8 +880,14 @@ async fn pypi_prelude<'p>( } // Both a fresh vendor and a re-run over an already-wired lock // keep warning while the venv still holds the upstream release. - if let Some(stale) = - pipenv_stale_install_warning(project_root, purl, record, installed_sites).await + if let Some(stale) = pipenv_stale_install_warning( + project_root, + purl, + record, + installed_sites, + &project.lock, + ) + .await { warnings.push(stale); } @@ -2275,6 +2283,43 @@ mod tests { } } + /// #790: the vendored stale-install remedy re-syncs the lock category + /// that pins the package. Plain `pipenv sync` installs only `default`, + /// so for a `[dev-packages]` entry it uninstalled the package and left + /// it uninstalled. + #[tokio::test] + async fn pipenv_stale_install_remedy_names_the_develop_category() { + let fx = e2e_fixture().await; + let lock: serde_json::Value = serde_json::from_str( + r#"{"_meta": {"pipfile-spec": 6}, "default": {}, "develop": {"six": {"version": "==1.16.0"}}}"#, + ) + .unwrap(); + let warning = pipenv_stale_install_warning( + &fx.root, + "pkg:pypi/six@1.16.0", + &fx.record, + &InstalledSiteListings::default(), + &lock, + ) + .await + .expect("the upstream six.py is installed"); + assert_eq!(warning.code, "pypi_pipenv_stale_install"); + assert!( + warning + .detail + .contains("`pipenv run pip uninstall -y six && pipenv sync --dev`"), + "{}", + warning.detail + ); + assert!( + warning + .detail + .contains("`pipenv --rm && pipenv sync --dev` for a clean virtualenv"), + "{}", + warning.detail + ); + } + #[tokio::test] async fn end_to_end_requirements_vendor_and_revert() { let fx = e2e_fixture().await; diff --git a/crates/socket-patch-core/src/vendor/pypi_pipenv.rs b/crates/socket-patch-core/src/vendor/pypi_pipenv.rs index 6665ebb8c..ee4ff198a 100644 --- a/crates/socket-patch-core/src/vendor/pypi_pipenv.rs +++ b/crates/socket-patch-core/src/vendor/pypi_pipenv.rs @@ -616,6 +616,93 @@ fn find_entries<'a>(lock: &'a Value, canon_name: &str) -> Vec<(&'a str, String, out } +/// The `pipenv sync` arguments that install every lock section in +/// `sections`. Plain `pipenv sync` installs only `default`; `--dev` adds +/// `develop` (every release); a Pipenv 2022+ named category needs +/// `--categories`, which takes Pipfile names (`packages`, `dev-packages`, +/// ``) and installs exactly the categories listed. +fn sync_args(sections: &[&str]) -> String { + if sections.iter().all(|s| *s == "default") { + String::new() + } else if sections.iter().all(|s| *s == "default" || *s == "develop") { + " --dev".to_string() + } else { + let names: Vec<&str> = sections + .iter() + .map(|s| match *s { + "default" => "packages", + "develop" => "dev-packages", + other => other, + }) + .collect(); + format!(" --categories \"{}\"", names.join(" ")) + } +} + +/// `default`, then `develop`, then named categories in lock order — so the +/// rendered `--categories` list is stable. +fn ordered_sections(mut sections: Vec<&str>) -> Vec<&str> { + let rank = |s: &str| match s { + "default" => 0, + "develop" => 1, + _ => 2, + }; + sections.sort_by_key(|s| rank(s)); + let mut seen = std::collections::HashSet::new(); + sections.retain(|s| seen.insert(*s)); + sections +} + +/// The verified way to reinstall `name` from `lock` over a warm virtualenv +/// that still holds the upstream release, shared by the hosted +/// (`redirect_pypi_stale_install`) and vendored (`pypi_pipenv_stale_install`) +/// warnings. The `sync` arguments follow the lock: the uninstall-and-sync +/// remedy re-syncs every category that pins the package (plain `pipenv sync` +/// would uninstall a `[dev-packages]` or named-category package and leave +/// it uninstalled), and the clean-virtualenv remedy re-syncs every non-empty +/// category, so `--rm` drops nothing the project had installed. With no +/// readable lock, falls back to the `default`-only commands. +pub fn stale_install_remedy(lock: Option<&Value>, name: &str) -> String { + let canon = canonicalize_pypi_name(name); + let (package, all): (Vec<&str>, Vec<&str>) = match lock { + Some(lock) => ( + ordered_sections( + find_entries(lock, &canon) + .into_iter() + .map(|(section, _, _)| section) + .collect(), + ), + ordered_sections( + lock.as_object() + .into_iter() + .flat_map(|map| map.iter()) + .filter(|(section, value)| { + section.as_str() != "_meta" + && value.as_object().is_some_and(|m| !m.is_empty()) + }) + .map(|(section, _)| section.as_str()) + .collect(), + ), + ), + None => (Vec::new(), Vec::new()), + }; + let reinstall = sync_args(&package); + let clean = sync_args(&all); + // Named categories arrived in Pipenv 2022, so the pre-2018 spelling only + // ever needs `--dev`. + let legacy = if package.contains(&"develop") { + " --dev" + } else { + "" + }; + format!( + "Reinstall it from the lock without touching the Pipfile: `pipenv run pip uninstall \ + -y {name} && pipenv sync{reinstall}` (`pipenv install --deploy{legacy}` before \ + Pipenv 2018), or `pipenv --rm && pipenv sync{clean}` for a clean virtualenv — NOT \ + `pipenv uninstall`, which rewrites the Pipfile and re-locks the patch away" + ) +} + /// Pipenv preserves a lock's CRLF line endings; so do we, on both writes. fn with_line_ending(text: String, crlf: bool) -> String { if crlf { @@ -1895,4 +1982,90 @@ mod tests { "the live lock is left alone" ); } + + /// #790: the stale-install remedy re-syncs the category that pins the + /// package — plain `pipenv sync` installs only `default`, so for a + /// `[dev-packages]` or named-category entry it uninstalled the package + /// and left it uninstalled — and `--rm` re-syncs every category. + #[test] + fn stale_install_remedy_follows_the_lock_categories() { + let lock = |body: &str| -> Value { + serde_json::from_str(&format!(r#"{{"_meta": {{"pipfile-spec": 6}}, {body}}}"#)).unwrap() + }; + let six = r#"{"version": "==1.16.0"}"#; + + // default only: unchanged commands. + let only_default = lock(&format!(r#""default": {{"six": {six}}}, "develop": {{}}"#)); + let remedy = stale_install_remedy(Some(&only_default), "six"); + assert!( + remedy.contains("`pipenv run pip uninstall -y six && pipenv sync`"), + "{remedy}" + ); + assert!( + remedy.contains("(`pipenv install --deploy` before Pipenv 2018)"), + "{remedy}" + ); + assert!( + remedy.contains("`pipenv --rm && pipenv sync` for a clean"), + "{remedy}" + ); + + // develop ([dev-packages]): `--dev` on every spelling. + let develop = lock(&format!( + r#""default": {{"requests": {six}}}, "develop": {{"six": {six}}}"# + )); + let remedy = stale_install_remedy(Some(&develop), "Six"); + assert!( + remedy.contains("`pipenv run pip uninstall -y Six && pipenv sync --dev`"), + "{remedy}" + ); + assert!( + remedy.contains("(`pipenv install --deploy --dev` before Pipenv 2018)"), + "{remedy}" + ); + assert!( + remedy.contains("`pipenv --rm && pipenv sync --dev` for a clean"), + "{remedy}" + ); + + // A default package in a project that also has dev-packages: the + // targeted re-sync stays default-only, `--rm` must restore develop. + let remedy = stale_install_remedy(Some(&develop), "requests"); + assert!(remedy.contains("-y requests && pipenv sync` ("), "{remedy}"); + assert!( + remedy.contains("`pipenv --rm && pipenv sync --dev` for a clean"), + "{remedy}" + ); + + // A Pipenv 2022+ named category: `--categories` with Pipfile names. + let named = lock(&format!( + r#""default": {{"requests": {six}}}, "develop": {{"pytest": {six}}}, "docs": {{"six": {six}}}"# + )); + let remedy = stale_install_remedy(Some(&named), "six"); + assert!( + remedy.contains("-y six && pipenv sync --categories \"docs\"`"), + "{remedy}" + ); + assert!( + remedy.contains( + "`pipenv --rm && pipenv sync --categories \"packages dev-packages docs\"` for a clean" + ), + "{remedy}" + ); + + // Pinned in several categories: every one of them is re-synced. + let both = lock(&format!( + r#""default": {{"six": {six}}}, "docs": {{"six": {six}}}"# + )); + let remedy = stale_install_remedy(Some(&both), "six"); + assert!( + remedy.contains("pipenv sync --categories \"packages docs\"`"), + "{remedy}" + ); + + // No readable lock: the default-only spelling. + let remedy = stale_install_remedy(None, "six"); + assert!(remedy.contains("-y six && pipenv sync` ("), "{remedy}"); + assert!(remedy.contains("NOT `pipenv uninstall`"), "{remedy}"); + } } diff --git a/docs/testing/pipenv-compatibility.md b/docs/testing/pipenv-compatibility.md index da3499c08..3919d6d00 100644 --- a/docs/testing/pipenv-compatibility.md +++ b/docs/testing/pipenv-compatibility.md @@ -45,7 +45,12 @@ import on modern Pythons); 2018–2022 on Python 3.8; 2023+ on Python 3.12. (`redirect_pypi_stale_install` / `pypi_pipenv_stale_install`) while a venv still holds the upstream release. Verified remedies (Pipfile byte-untouched): `pipenv run pip uninstall -y && pipenv sync`, or - `pipenv --rm && pipenv sync`. `pipenv uninstall ` is **not** a remedy: + `pipenv --rm && pipenv sync`, for a `default` package. Plain `pipenv sync` + installs only `default`, so the printed `sync` arguments follow the lock: + `--dev` when the package is pinned in `develop`, and + `--categories ""` (Pipfile names: `packages`, `dev-packages`, + ``) when it is pinned in a named category. The `--rm` form + re-syncs every non-empty category (#790). `pipenv uninstall ` is **not** a remedy: it rewrites the Pipfile and re-locks the patch away. `PIP_FORCE_REINSTALL=1 pipenv sync` works on 2018 but is ignored by 2026. - **Relocking drops the reference.** `pipenv lock` (and `update`, and From fbdfa6fa73e40e590a2f29042126e11fc06dc2fe Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 4 Oct 2026 17:10:43 +0000 Subject: [PATCH 3/3] Put the Pipenv remedy test's venv where Windows probes The develop-category regression test built only the POSIX lib/python3.12/site-packages layout, which the Windows venv probe never reads, so no stale install was found and the warning never fired. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01K9MfJ84pKgWnWZUoTYq8Zg --- crates/socket-patch-core/src/vendor/pypi.rs | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/crates/socket-patch-core/src/vendor/pypi.rs b/crates/socket-patch-core/src/vendor/pypi.rs index 4c2a41b3e..90be6c4ca 100644 --- a/crates/socket-patch-core/src/vendor/pypi.rs +++ b/crates/socket-patch-core/src/vendor/pypi.rs @@ -2290,6 +2290,21 @@ mod tests { #[tokio::test] async fn pipenv_stale_install_remedy_names_the_develop_category() { let fx = e2e_fixture().await; + // The venv probe reads `.venv\Lib\site-packages` on Windows, so mirror + // the fixture's POSIX-layout install there. + if cfg!(windows) { + let sp = fx.root.join(".venv").join("Lib").join("site-packages"); + let di = sp.join("six-1.16.0.dist-info"); + std::fs::create_dir_all(&di).unwrap(); + std::fs::copy(fx.site_packages.join("six.py"), sp.join("six.py")).unwrap(); + for leaf in ["METADATA", "WHEEL", "RECORD"] { + std::fs::copy( + fx.site_packages.join("six-1.16.0.dist-info").join(leaf), + di.join(leaf), + ) + .unwrap(); + } + } let lock: serde_json::Value = serde_json::from_str( r#"{"_meta": {"pipfile-spec": 6}, "default": {}, "develop": {"six": {"version": "==1.16.0"}}}"#, )