v5 WS4/WS6: one hosted engine for disk + memory; unified Ledgers view - #282
Conversation
Extract the plan -> rewrite -> edits stages of `run_redirect_selected` into `socket_patch_core::hosted::engine`, a set of pure functions over a `ProjectView` (build_candidates, bun_lockb_symlinked, withhold_everywhere, read_candidate_files, wheel_targets, rewrite, guard). The disk flow (`scan`/`get --mode hosted`) keeps only the apply lock, the host probes (pipenv version, gem/python/vlt stale installs), the vendored takeover, the symlink refusal and the commit of the rewritten files. The in-memory engine moves to `socket_patch_core::hosted::memory` and runs the same stages over `ProjectView::Memory`; its duplicated redirect.rs / ledger.rs orchestration is deleted. The pnpm trust / npm allow-remote planners move to `hosted::guidance`, the vlt preflight to `hosted::vlt`, and the redirect-ledger delta to `hosted::ledger`, which the engine never calls, so removing the hosted ledger only touches the two callers. `socket-patch-node` now depends on socket-patch-core only; the CLI re-exports `hosted_memory` for `hosted-bundle` and the tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
The previous commit ran `cargo fmt --all` over a tree that is not rustfmt-clean, reformatting ~30 files it does not otherwise touch. Restore those files; no code changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
`socket_patch_core::ledgers` holds the one owner-precedence rule for the manifest, the vendor ledger and the hosted redirect ledger (manifest > vendored > hosted by ledger key; a manifest key claims every vendor entry filed under it or naming it as base purl) and the views every reader derives from it: `owned` (one group per owner key, losers as alternates), `listed` (every copy worth showing), `matching` (remove/rollback identifiers) and `hosted_vendored_overlap`. It replaces list's combined_entries, fold_vendor_records / vendor_record_is_unowned, scan's merge_ledger_records_for_updates, vex_sources' build_candidates and overlap_from_states, and rollback/remove's per-store matching loops. `LoadedLedgers::load` loads the three stores once, each with its own outcome so every caller keeps its error posture. `commands::context::ProjectContext` lazily loads the stores, the lockfile inventory and the wiring discovery at most once per run; scan's discovery phase, list and get read through it. `get`'s installed-version narrowing now reuses scan's lockfile and vendored-ledger supplements instead of its own inventory and ledger reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
|
This is not caused by this PR. The same test fails the same way on Root cause: v5 No fix exists yet. Proposed patch, a test-only change I'm not adding here to keep this PR's scope: --- a/crates/socket-patch-cli/tests/e2e_vlt.rs
+++ b/crates/socket-patch-cli/tests/e2e_vlt.rs
@@ async fn vlt_pinned_matrix_agent_get_and_remove() {
- let out = socket_api(&fx.proj, &fx.svc, &["get", UUID], &[]);
+ let out = socket_api(&fx.proj, &fx.svc, &["get", UUID, "--mode", "agent"], &[]);Generated by Claude Code |
|
This isn't from this PR. Likely cause, not yet verified: step (2) runs the "read-only" embedded Generated by Claude Code |
|
Generated by Claude Code |
Mikola Lysenko (mikolalysenko)
left a comment
There was a problem hiding this comment.
v5 review at 5dcf9882 — moving the hosted engine into core and removing the Node addon's CLI dependency is the right boundary. Keep it. I did not confirm a new functional regression, but there is avoidable work and unfinished integration.
- Index ledger ownership once.
Ledgers::owned()sorts and scans all vendor keys inside each manifest-record iteration (ledgers.rs:173–192). For M manifest and V vendor entries this adds O(M·V log V) work, even for direct key matches. Build a deterministic owner-to-entry index once and derive the different views from it. This is a source-level complexity finding, not a measured wall-clock regression. - Finish the ledger-free integration before calling this complete. Disk and memory still serialize the hosted ledger, and the new abstraction models three persistent stores. With #280, retain only legacy read/migration support for hosted records; remove the hosted writer/merge path from both engines. The #280/#282 heads conflict across these paths, so parity must be checked again on the combined tree.
- Cache one project snapshot, not two interpretations of the files.
ProjectContext::locks()anddiscovery()independently read/parse the project, and the context has no shared crawl snapshot. Back both with the shared format models and a consistent project root; rebuild/invalidate the snapshot after writes before embedded VEX. Thread it through vendor/VEX as well as scan/get/list. - Keep rendering out of orchestration. Return a typed plan/result from the shared engine, then let disk, memory, JSON and human adapters consume it. This makes the reduction durable rather than relocating the old CLI state machine into core.
Validation: source/diff review; existing parity/golden coverage inspected, not rerun here.
`Ledgers::owned` re-sorted and scanned every vendor key for each manifest key. Group the vendor entries under their claiming manifest key in one pass over the sorted ledger (`claims`), so each view is O(V log V + M) instead of O(M * V log V). Same ordering and alternates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
|
Thanks, replies by item:
I'll push 1 once the suite is clean, then do 3 and 4 as separate commits on this PR. Item 2 waits on the #280 ordering decision. Generated by Claude Code |
The engine's own warnings (rush repo-state, pnpm trustLockfile, npm allow-remote, vlt artifact-unverifiable, record_fetch_failed, the in-memory takeover refusal) were built as serde_json values inside orchestration. They are now RewriteWarning values; the new hosted::render module holds the only JSON spelling (warnings, skipped entries, the nested redirect block), consumed by the disk adapter and the in-memory engine. No output change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
ProjectView gains a Snapshot variant: the disk under a read-through cache (DiskSnapshot), so each lock or config file is read at most once per run and every reader sees the same bytes; probes that are not content reads still go to the disk. Lockfile discovery's guarded reads now go through a ProjectView (discover_patched_refs_in), and ProjectContext backs both locks() and discovery() with one snapshot of --cwd. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
|
Status on the review as of
CI on Generated by Claude Code |
|
Restack request from the v5 coordinator (please act now). #280 (ledger-free hosted) is now merged into
|
|
Owner decision (via coordinator): on ordering, #280 has landed first ( |
…sted-engine Resolved toward #280's model: v5 hosted mode keeps no ledger. - hosted::ledger (the redirect-ledger merge, in-memory load and serializer) is deleted; neither the disk flow nor the in-memory engine reads or writes .socket/vendor/redirect-state.json. - Ledgers' hosted store is now the hosted records: the lockfiles' hosted pins (or a run's fetched records), plus a pre-v5 ledger read only for migration. hosted_vendored_overlap drops the edits-only fallback. - list, scan's updates[] and rollback run the shared owner rule over the manifest and the vendor ledger and take the hosted pins from the lockfiles; updates[] keeps #280's precedence (manifest > pins > vendor ledger). scan reads the pins through ProjectContext's one discovery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
|
I merged Resolved toward #280's ledger-free model:
Local validation on the merge:
I'll mark the PR ready once CI on Follow-up list (non-blocking, not in this PR):
Generated by Claude Code |
|
This PR is ready for review at head CI on
Local run on the merge:
I'll merge the base again after each of #283, #281 and #279 lands. Generated by Claude Code |
|
#283 landed on release/v5-prerelease as 06437d2; please merge origin/release/v5-prerelease again, resolve conflicts, get green, and keep it ready. Generated by Claude Code |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
|
This PR is at head CI on
Local checks on the merge:
Generated by Claude Code |
…ed-engine Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
|
#281 landed on release/v5-prerelease as 73c0c4f; please merge origin/release/v5-prerelease again, resolve conflicts, get green, and keep it ready. Generated by Claude Code |
…into v5/one-hosted-engine #281's format-registry changes to the hosted flow land where this branch moved that code: core hosted::engine derives REDIRECT_CANDIDATE_FILES and file_ecosystem from formats::registry, hosted::guidance re-exports the pnpm lock-version sniffs from formats::pnpm, and the in-memory root detection reads registry::root_marker. The CLI hosted_memory redirect.rs #281 edited is already gone on this branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
|
CI on 94dc5d2 (the merge of #281 base 73c0c4f). Two reds so far:
The rest of CI is still running. I'll post the head SHA and CI summary when it finishes. Generated by Claude Code |
|
#279 landed on release/v5-prerelease as f6bdad5; please merge origin/release/v5-prerelease again, resolve conflicts, get green, and keep it ready. Generated by Claude Code |
…to v5/one-hosted-engine setup.rs goes, with the setup-only fold_vendor_records helper this branch had moved onto the Ledgers view; vendor_record_is_unowned has no callers left here (list, vex and scan read Ledgers), so it goes too. #279's hosted wording lands where this branch moved the code: the pnpm trust guidance in core hosted::guidance, the record_fetch_failed and rush repo-state texts in core hosted::engine (shared by the memory engine), and the human warning/summary text in the CLI scan/hosted.rs, which also gets the one-line npm allow-remote note and its test. The CHANGELOG owner-rule entry drops its `setup --check` clauses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
|
v5 coordinator: #282 is next to land and is up to date with the base (
Generated by Claude Code |
|
Head Local validation before the push:
CI on this head
These four are exactly the failed jobs of #279's final vlt run, 36466226104 (head Every other workflow on this head is green: pnpm, Poetry, Go, Pipenv, npm, PDM and Bun. Generated by Claude Code |
14a9cb0
into
release/v5-prerelease
Brings in #282, which moves the in-memory engine into socket-patch-core (`hosted::memory`). The two-phase policy selection moves with it. Conflicts: - `canon`, `FilteredEntry`, `RetainedEntry` and `policy_block` move from the CLI into `socket_patch_core::policy`, so the core engine and disk scans share one `policy` block. - `roots` is now public in core, for disk scans' marker lookup. - scan reads the ledgers and wiring through the base's `ProjectContext`, and keeps loading the recorded view before filtering. - CLI_CONTRACT's hosted-bundle row takes the base's path, plus the policy fields. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BKsyzefGhAnPkYmXCwq3H3
Brings in #282 through A: the in-memory engine now runs the disk engine's stages in socket_patch_core::hosted::memory. The rollout moves with it. The stage, the recorded view and the classification live in socket_patch_core::rollout::stage, and the ledger fold for updates lives in socket_patch_core::ledgers, so disk and memory share them. scan keeps updates[], the hosted gate and the human lines. On disk the gate plans after the engine's first rewrite and rewrites again without the deferred rows; in memory each root keeps its plan for that second pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Implements WS4 and a first cut of WS6 from
docs/design/v5-plan.md, on top of #280 (ledger-free hosted,686e5fb).WS4: one hosted engine (disk + memory)
run_redirect_selectednow live insocket_patch_core::hosted::engine, as pure functions overProjectView:build_candidates,bun_lockb_symlinked,withhold_everywhere,read_candidate_files,wheel_targets,rewriteandguard.scan/get --mode hosted) keeps only the host side:vendored_takeover)socket_patch_core::hosted::memoryand runs the same stages overProjectView::Memory. Its duplicated redirect orchestration is deleted.trustLockfile/ npmallow-remoteplanners →hosted::guidancehosted::vlthosted::render; the engine emits typedRewriteWarnings, notserde_json::Value..socket/vendor/redirect-state.json. This branch's oldhosted::ledgermodule (the redirect-ledger merge, in-memory load and serializer) is deleted, not moved.socket-patch-nodenow depends onsocket-patch-coreonly. The CLI re-exportshosted_memoryforhosted-bundleand the tests.hosted_memory_parity,redirect_goldenand v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280'supstream_restore_goldenpass on the merged tree.WS6: one
Ledgersview +ProjectContextsocket_patch_core::ledgersholds the one owner-precedence rule:The hosted records are what v5 has instead of a ledger: the lockfiles' hosted pins, or a run's fetched records, plus a pre-v5 ledger that is read, never written, for migration.
The rule exposes these views:
owned: one group per owner key, with the losing copies as alternates;listed: every copy worth showing;matching: store entries a remove/rollback identifier matches;hosted_vendored_overlap.These views replace:
list's manifest and vendor folding (hosted pins are listed one per pin, as in v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280);fold_vendor_records/vendor_record_is_unowned;merge_ledger_records_for_updates(v5 WS1+WS2: ledger-free hosted mode, upstream-restore rollback, vendor ejects hosted projects #280's precedence is kept: manifest > hosted pins > vendor ledger);overlap_from_states;vex_sources::build_candidates;rollbackandremove.LoadedLedgers::loadloads the manifest, the vendor ledger and any pre-v5 redirect ledger once, and each store keeps its own load outcome.commands::context::ProjectContextloads the stores, the lockfile inventory and the wiring discovery lazily, at most once per run:locks()anddiscovery()read--cwdthrough one read-throughDiskSnapshot(ProjectView::Snapshot), so each lock or config file is read at most once and both see the same bytes.listandgetgo through it.Behavior differences, listed in the CHANGELOG:
scan'supdates[]no longer folds a vendor entry the manifest claims by base purl.vextreats every vendor entry a manifest key claims as a fallback copy of that key's record.setup --checkno longer folds a detached vendor entry whose base purl is in the manifest.getfalls back to the committed artifacts on a corrupt vendor ledger, asscandoes.Follow-ups (not in this PR)
Listed in a PR comment: the rest of review item 3 (share the parsed models; thread the context into
vendor/vex) and item 4 (a typedHostedOutcomewith JSON, human and memory adapters), plus F17.Validation (on
4cabaf1, the merge)cargo clippy --workspace --all-targets --all-features -- -D warningsis clean.cargo test --workspace --all-features --no-fail-fast -j4:release/v5-prerelease@686e5fb, same sandbox: 10211 passed, 18 failed.🤖 Generated with Claude Code
https://claude.ai/code/session_01KHZ8uzdXfkG2zH8ZYDG8ju
Note
Medium Risk
Broad refactor of how patch stores are merged and how hosted rewrites run, with intentional behavior changes across scan, vex, get, and setup rather than cosmetic CLI moves.
Overview
This PR centralizes hosted-mode lockfile rewriting in
socket_patch_core::hostedso the CLI disk path and the in-memory engine (Node addon /hosted-bundle) share the same plan → rewrite stages overProjectView. The CLI keeps host-only concerns (apply lock, vendored takeover, symlink guards, commits); pnpm/npm install-policy text, vlt preflight, and JSON rendering move into core.socket-patch-nodeno longer depends on the CLI—it uses core only, withhosted_memoryre-exported for compatibility.It also introduces
socket_patch_core::ledgerswith a single precedence rule (manifest → vendor ledger → hosted records, including manifest claims by ledger key or base purl) andProjectContextto load stores, lock inventory, and wiring discovery once per run (sharedDiskSnapshotfor lock/config reads).Commands
list,scan,vex,get,setup --check,rollback, andremovenow read patch stores through that view instead of ad hoc merging. User-visible fixes include:scanupdates[]no longer double-counts vendor entries the manifest owns by base purl;vextreats claimed vendor rows as fallback copies of the manifest record;setup --checkstops folding detached vendor entries when the manifest already records the base purl;getmatchesscanwhen the vendor ledger is corrupt (fallback to committed artifacts). Manifest record building from API patches moves fromgetinto core for reuse.Reviewed by Cursor Bugbot for commit 4cabaf1. Configure here.