Fix gem pair model ignoring custom lockfile and Bundler 1 twins (#749, #751)

[Mikola Lysenko (mikolalysenko)]

Fixes #749
Fixes #751

Summary

Hosted gem mode wired the wrong files in two Bundler layouts. In both, the scan reported success (and --vex attested the patch) while Bundler installed the unpatched gem or every frozen install failed. Both now wire the pair Bundler actually loads, or refuse before writing anything.

Root cause

formats::gem::manifest::LoadedManifest is the single model of "which manifest/lock pair does Bundler load", and hosted mode (keep_bundler_loaded_gem_files) and vendored mode (gem_manifest_refusal) both rely on it. It modelled BUNDLE_GEMFILE, but:

• Hosted gem redirect ignores Bundler 4's custom lockfile (lockfile setting / BUNDLE_LOCKFILE), so it never pins the lock Bundler uses and frozen installs fail with no warning #749: it derived the lock only from the manifest name, so it never saw Bundler 4's custom lockfile (BUNDLE_LOCKFILE env, or lockfile in .bundle/config). The lock Bundler reads was never pinned. A leftover Gemfile.lock was rewritten instead, even though Bundler ignores it.
• Hosted gem redirect wires gems.rb in a Gemfile/gems.rb twin locked by Bundler 1.17, which loads Gemfile, so the install stays unpatched while the in-run VEX attests it #751: default discovery for a Gemfile + gems.rb twin always followed Bundler ≥ 2 (gems.rb first). Bundler 1.x loads Gemfile first.

Fix

• LoadedManifest::with_lockfile resolves the configured lockfile the way Bundler::CLI does (env first, then app config, then the global ~/.bundle/config that Gem settings resolution skips Bundler's global config (~/.bundle/config / BUNDLE_USER_CONFIG), so a global cache_path or gemfile gets no warning or refusal and VEX attests an unpatched install #577 / Fix Bundler global config being ignored (#577) #621 added; BUNDLE_IGNORE_CONFIG honoured; relative to the project root). If it names the loaded pair's own lock, nothing changes. Anything else is the new UnsupportedLockfile:
  • hosted mode refuses with redirect_gem_bundle_lockfile_unsupported (nothing written, nothing attested);
  • vendored mode refuses with gemfile_not_loaded.
• default_twin_manifest reads both twin locks' BUNDLED WITH (new shared formats::gem::bundled_with_major):
  • every recorded major is 1.x → wire Gemfile + Gemfile.lock;
  • none is 1.x → gems.rb (unchanged);
  • the locks disagree → refuse with redirect_gem_twin_bundler_versions_diverge.
• Docs: CLI_CONTRACT.md (new additive warning codes), docs/ecosystems.md (RubyGems row). The CHANGELOG entry was dropped in 7c84ba8.

Tests (red → green)

Issue	Test	Without fix	With fix
#749	hosted_memory_engine::a_bundler4_custom_lockfile_is_refused	FAILED (rewrote the leftover lock)	ok
#749	e2e_redirect_gem_build::gem_hosted_bundler4_custom_lockfile_redirects_nothing (real Bundler 4.0.17)	FAILED: redirected: 1, rewrittenFiles: [Gemfile, Gemfile.lock], vex statements: 1	ok; frozen bundle install of the untouched project succeeds
#749	ruby_crawler::loaded_manifest_reads_the_lockfile_setting (disk, no leftover lock; env vs config priority; BUNDLE_IGNORE_CONFIG)	new API	ok
#749 × #577	manifest::with_lockfile_accepts_only_the_pairs_own_lock (global tier, shadowed by the app config) and ruby_crawler::loaded_manifest_reads_the_global_config_below_local_and_env (bundle config set --global lockfile)	new (merge)	ok
#749	vendor::gem::a_bundler4_custom_lockfile_is_refused	new	ok
#749	hosted_memory_engine::a_lockfile_setting_naming_the_default_lock_is_wired (control)	ok	ok
#751	hosted_memory_engine::a_bundler1_twin_wires_the_gemfile_pair	FAILED (wired gems.rb)	ok
#751	hosted_memory_engine::a_twin_with_diverging_bundler_majors_is_refused	FAILED	ok
#751	hosted_memory_engine::a_bundler2_twin_still_wires_gems_rb (control)	ok	ok
#751	e2e_redirect_gem_build::gem_hosted_bundler1_twin_wires_the_gemfile_and_installs	runs on the CI bundler 1.17.3 leg; skips on ≥ 2	skipped locally (Bundler 4)
both	manifest.rs unit tests (with_lockfile_*, default_twin_manifest_*, config_lockfile_*), bundled_with_major_reads_the_version_line	new	ok

Local results

• cargo clippy --workspace --all-features -- -D warnings: clean.
• cargo fmt --all -- --check is not clean on main itself (≈500 diffs, and CI has no fmt gate), so I formatted only the hunks I touched.
• cargo test --workspace --all-features --lib --bins: the core lib has 4848 passed and 4 failed. All four (copy_tree::relax_loop_must_not_traverse_symlinked_root, vlt_heal::an_unremovable_hidden_lock_keeps_every_store_entry, pypi_poetry::wire_write_failure_*, pypi_requirements::wire_failure_rolls_back_*) are chmod-based write-failure tests that can't fail writes when run as root (uid 0) in this sandbox. They don't touch gem code.
• Integration tests: hosted_memory_engine 33/33, hosted_memory_parity, in_process_vendor and e2e_redirect_gem_stale_install all ok. in_process_redirect has 104 passed and 3 failed, again only the chmod-based write-failure tests (root sandbox).
• e2e_redirect_gem_build --ignored (real Bundler 4.0.17): new custom-lockfile, dual-boot and gems.rb arms pass.
• I couldn't run the full cargo test --workspace locally because the sandbox's disk allowance runs out building every integration-test binary. CI runs the full suite.

Merge with main (d502a07)

main gained #577 (Bundler's global config, #621) while this PR was open, and the two overlapped in LoadedManifest. The conflicts are resolved by keeping both. Because Bundler 4 reads lockfile through Bundler.settings, which includes the global file, with_lockfile now takes the global tier too. Without that, bundle config set --global lockfile custom.lock would have slipped past the #749 guard. BundlerEnv carries the global config path, and #621's call sites use it. After the merge: cargo clippy --workspace --all-features -- -D warnings is clean; the gem/ruby core lib tests (253), crawler_ruby_e2e (26), hosted_memory_engine (34), hosted_memory_parity, in_process_vendor (106), e2e_redirect_gem_stale_install, and real-Bundler e2e_redirect_gem_build --ignored (17) all pass. The full core lib has 5010 passing and the same 4 chmod-based failures from running as root here.

Bugbot follow-up (9e7af6e)

• Lock-only scans are no longer silent. Since Lock inventory reads only Gemfile.lock, so a gems.rb project's gems.locked is invisible and a stale Gemfile.lock is read instead #736, a gem lock bundler loads but socket-patch can't read (a custom BUNDLE_LOCKFILE, an unsupported BUNDLE_GEMFILE, or a diverging twin) left lock inventory empty with no warning. Inventory now adds a gem_lock_unsupported diagnosis, which scan and the in-memory engine report as a run-level warnings[] entry (documented in CLI_CONTRACT.md). Test: lock_inventory::tests::gem_inventory_diagnoses_a_lock_it_cannot_read (red→green). The hosted_memory_engine refusal tests assert the warning again.
• An empty BUNDLE_LOCKFILE shadows the tiers below it, as Settings#[] does, so an empty env or app-config value now clears a global custom lock.

Notes / follow-ups

• PR Fix gem lock readers ignoring gems.locked (#736) #750 (Lock inventory reads only Gemfile.lock, so a gems.rb project's gems.locked is invisible and a stale Gemfile.lock is read instead #736) routes the lock readers (VEX discovery, lock inventory) through LoadedManifest::pair. Once both land, a Bundler 1.x twin's post-install VEX should use default_twin_manifest too. This PR doesn't touch those readers, to avoid conflicting with Fix gem lock readers ignoring gems.locked (#736) #750.
• The Gemfile lockfile "custom.lock" DSL (issue matrix row 4) is Ruby code the model can't read. Bundler 4.0.17 itself fails frozen installs on that shape before any scan.
• A BUNDLE_LOCKFILE setting is refused even on Bundler < 4 (which ignores it). That's a conservative, fail-closed choice.

🤖 Generated with Claude Code

https://claude.ai/code/session_018ULgrJMQMWEBAsiuprY449

---

Note

Medium Risk
Changes which lockfiles get rewritten in hosted/vendored gem mode and when scans attest patches; wrong behavior previously caused false success and broken frozen installs, but lockfile edits remain high-impact for Ruby projects.

Overview
Hosted and vendored gem flows now follow the manifest/lock pair Bundler actually loads, instead of rewriting ignored files while reporting success.

LoadedManifest gains with_lockfile (Bundler 4 BUNDLE_LOCKFILE from env → app → global config) and default_twin_manifest (uses both locks’ BUNDLED WITH majors to choose Gemfile vs gems.rb under default discovery). Unsupported custom lockfiles refuse with redirect_gem_bundle_lockfile_unsupported / vendored gemfile_not_loaded; diverging twin majors refuse with redirect_gem_twin_bundler_versions_diverge.

Lockfile-only scan no longer treats unreadable gem layouts as “no gems”: inventory emits gem_lock_unsupported when gem files exist but no lock socket-patch can read.

Docs (CLI_CONTRACT.md, ecosystems.md), hosted engine candidate filtering, VEX discovery messaging, and e2e/memory tests cover the new behaviors. Minor unrelated refactor: shared sha1_hex_of / sha256_hex_of helpers in a few JVM/Gradle paths.

^{Reviewed by Cursor Bugbot for commit cf746ec. Configure here.}

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

Ready for review at 731f489.

• CI: 402/402 non-skipped checks green on the current head (9/9 workflows, path-filtered).
• Bugbot: reviewed 731f489 and found no issues. There are no unresolved review threads.
• Mergeable: yes, no conflicts with main. The diff is 10 files, all in the gem lane, plus docs and changelog.
• For the reviewer: look at the LoadedManifest pair model in formats/gem/manifest.rs (the configured BUNDLE_LOCKFILE and the BUNDLED WITH major of a Gemfile/gems.rb twin) and the matching ruby_crawler.rs changes.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

coverage (the Linux test gate) failed on the merge commit d502a07. Its one failing target is -p socket-patch-cli --lib. coverage was green on 731f489, before the merge with main.

I can't see which test failed yet. The job log's download URL (productionresultssa5.blob.core.windows.net) is blocked by my sandbox's network policy, and the MCP log tool returns only the last ~5000 of the log's 12022 lines. The CLI lib test section is earlier than that, so the panic isn't in what I can read. The check annotations only carry the exit code.

What I checked locally on d502a07:

• cargo test -p socket-patch-cli --lib with default features and with --all-features: 835/835 pass.
• With CI's SOCKET_PATCH_GO_E2E_REQUIRED=1 / SOCKET_PATCH_GO_E2E_VERSION=1.24 (Go 1.24.7): 835/835 pass.
• The same test binary run as a non-root user: 835/835 pass.
• This PR doesn't touch crates/socket-patch-cli/src. That code is identical to main's.

Next: the macOS and Windows test legs run the same CLI lib tests on this commit, and I'll use them to tell a real failure from an instrumentation or timing one. If anyone can open the job log (coverage → "Run tests with coverage", search for FAILED), the failing test name and panic would let me fix it directly.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Found it, and this PR doesn't cause it: main is red at 4646693. coverage and test-release test the PR merged into the current main. Both fail on the same two CLI lib tests, and they fail identically on main alone:

• commands::vex_consumed::tests::hosted_expands_alias_only_copies (vex_consumed.rs:771, assert!(installed.is_empty()))
• commands::vex_consumed::tests::hosted_reuses_expanded_npm_copies_and_merges_alias_variants (vex_consumed.rs:718, assert_eq!(installed_again, installed))

Cause: the two tests came in with #738 (#356) and assume the name-keyed resolver find_manifest_package_copies_reusing can't see aliased copies. #605 (#601/#603, merged as 4646693) made that resolver probe store entries' bundled trees, so it now returns the aliases, the nested alias and every store peer variant itself. I ran both tests on main with the asserts relaxed:

• calls == []: Fix agent mode skipping npm-aliased copies (#356) #738's alias expansion is no longer needed.
• The final paths set is the complete, correct one: {host/…/lp, left-pad(peer@1.0.0), left-pad(peer@1.0.1)} in the first test, and every root, alias, nested-alias and peer copy in the second.

So the behaviour is right and only the tests' premise is stale.

Proposed patch (for main, not this PR):

• In hosted_expands_alias_only_copies, replace assert!(installed.is_empty()) and assert_eq!(calls, vec[vec[alias.clone()]]) with:
  • installed[&purl] (sorted) equals peers + [alias] (sorted);
  • calls.is_empty();
  • paths == installed[&purl].
• In hosted_reuses_expanded_npm_copies_and_merges_alias_variants:
  • Expect installed_again[&purl] to already contain nm/lp, host/node_modules/lp and the nested peers.
  • Replace calls.len() == 1 / the alias-inputs check with calls.is_empty().
  • Keep the final sorted-set and canonical-uniqueness assertions.
  • The earlier nm/lp step (calls == [[alias]]) needs the same update if the resolver also finds that alias. On main it currently doesn't fail first, so check it when applying.

I'm not adding this to #768. It's in code unrelated to the gem change, and no fix for it is open yet. Once main is green, I'll merge it into this branch and re-run CI.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Burn-down agent: pushed 8c2ba59.

• test (macos-latest), test-release and coverage were red on d502a07. The failures were vex_consumed::tests::hosted_expands_alias_only_copies and hosted_reuses_expanded_npm_copies_and_merges_alias_variants, which come from main and have been red there since Fix npm store copies missed by agent apply and vex (#601, #603) #605. I merged main and cherry-picked Fix vex alias tests broken by store-copy merge #851, a test-only change that becomes a no-op once Fix vex alias tests broken by store-copy merge #851 lands.
• Locally: clippy (CI command) is clean, and socket-patch-core --lib gem (280), socket-patch-cli --lib vex_consumed (10) and --test hosted_memory_engine (34) all pass.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

Burn-down agent: labeled Ready for review.

• Head: 8c2ba59421a2e49711197f1a85f8b7ad7ea879a6
• CI: 458/458 check runs green (success/skipped) on this head; mergeable clean
• Bugbot: reviewed 8c2ba59, no new issues; no open threads
• Reviewer focus: gem pair model honoring a custom lockfile and Bundler 1 twins (Hosted gem redirect ignores Bundler 4's custom lockfile (lockfile setting / BUNDLE_LOCKFILE), so it never pins the lock Bundler uses and frozen installs fail with no warning #749, Hosted gem redirect wires gems.rb in a Gemfile/gems.rb twin locked by Bundler 1.17, which loads Gemfile, so the install stays unpatched while the in-run VEX attests it #751)

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Unsupported gem layouts silent on lock-only scans
  • Modified the condition in hosted/engine.rs to also check for gem manifest files presence, ensuring keep_bundler_loaded_gem_files runs even when there are no gem candidates, allowing warnings to be surfaced for unsupported layouts on lock-only scans.

Or push these changes by commenting:

@cursor push 992984be56

Preview (992984be56)

diff --git a/crates/socket-patch-core/src/hosted/engine.rs b/crates/socket-patch-core/src/hosted/engine.rs
--- a/crates/socket-patch-core/src/hosted/engine.rs
+++ b/crates/socket-patch-core/src/hosted/engine.rs
@@ -545,7 +545,11 @@
             }
         }
     }
-    if candidates.iter().any(|c| c.dep.ecosystem == "gem") {
+    if candidates.iter().any(|c| c.dep.ecosystem == "gem")
+        || GEM_MANIFEST_FILES
+            .iter()
+            .any(|&f| out.files.contains_key(f))
+    {
         keep_bundler_loaded_gem_files(view, &mut out).await;
     }
     // A Gradle build: every script, catalog and lock file its script graph

_{You can send follow-ups to the cloud agent here.}

[Mikola Lysenko (mikolalysenko)]

test (windows-latest) failed on 9e7af6e, but not because of this PR. The failure is utils::digest::tests::production_digests_go_through_the_helpers, which also fails on main (9c43dfc) on Linux. crawlers/gradle_cache.rs, patch/jvm_jar.rs and patch/sidecars/maven.rs compute digests inline and aren't on the pending list. #878 fixes it and is still open, so I've ported its change in cf746ec, cherry-picked from #885's 4d8cad2. It becomes a no-op once #878 lands. With it, the digest check and the Gradle/JAR/sidecar tests pass locally, and clippy is clean.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit cf746ec. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Burn-down agent: labeled Ready for review at cf746ec (cf746ec8b25b87cb6f7fb0a2ae8e70416c116f37).

• CI: 506/506 workflow checks green on the head commit (6 skipped by matrix rule). The only non-green entries are the 4 CodeQL default-setup Analyze jobs (run "PR Fix gem pair model ignoring custom lockfile and Bundler 1 twins (#749, #751) #768"). GitHub cancelled them during today's Actions runner outage, and GitHub doesn't allow re-running them ("This workflow run cannot be retried"). They'll run again on the next push.
• Bugbot: reviewed cf746ec with no new issues. Both earlier findings are resolved, and no review threads are open.
• Mergeable against main, with no conflicts.
• Reviewer note: this changes which Gemfile/lockfile pair hosted and vendored gem modes rewrite (BUNDLE_LOCKFILE, Bundler 1 gems.rb twins). Please look most closely at the new refusal codes.

---

Generated by Claude Code