Skip to content

pool: a stale VM's attached volumes stay held until it is removed - #349

Merged
CMGS merged 5 commits into
mainfrom
fix/348-restart-volume-holds
Oct 17, 2026
Merged

CMGS merged 5 commits into
mainfrom
fix/348-restart-volume-holds

Conversation

@CMGS

@CMGS CMGS commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Closes #348.

Problem

A teardown keeps a claim's volume holds until cocoon confirms the VM is gone. The pending-removal queue that carries them lives only in memory.

After a sandboxd restart with a removal still pending, the startup sweep retried removing the unowned VM but held none of its volumes. A conflicting claim (a writer, or a reader of a volume the old VM held writable) was admitted. It then failed at cocoon vm disk attach on the hypervisor's image lock: a 500, and a warm VM destroyed, instead of a 409.

Fix

When the startup sweep cannot remove a running unowned sbx- VM, it reads the VM's hot-attached disks from cocoon vm inspect (attached_devices.disks, available since cocoon v0.4.9). It holds those volumes, rw as a writer and ro as a reader, and queues the removal with the holds attached. The existing pending-removal path releases them once the VM is confirmed gone.

  • No journal change. Rolling sandboxd back is unaffected.
  • A removal that succeeds needs no inspect, because Reconcile finishes before the server starts.
  • removeOrRetry loses its tap parameter, which no caller sets any more.
  • docs/deploy.md gains one sentence about the startup sweep.

Cost

Zero on the claim path. At startup there is one extra cocoon vm inspect per stale VM whose removal fails.

Verification

  • New pool test, with two cases (old claim rw, old claim ro):

    • after the restart, the registry holds a writer or a reader respectively;
    • after the stale VM is removed, it is empty.
    • Without the fix, flipping the rw/ro mapping, or inspecting on a successful removal, the tests fail.
  • The engine test parses cocoon's real inspect JSON shape.

  • make go-lint: 10 × 0 issues. (5 modules × linux/darwin). asl is clean on both GOOS, and make go-test passes.

  • Hardware, a bare-metal testbed with cocoon v0.7.2 and Cloud Hypervisor. The procedure: claim a writable volume, make vm rm fail for that VM, release, then restart sandboxd.

    step this branch main e4e8acb
    new rw claim of the volume, old claim rw 409 "volume is held by another claim", VM list unchanged 500 "Cannot lock images", one warm VM destroyed
    new rw claim, old claim ro 409 —
    after vm rm stops failing old VM removed by the reap tick (~3.7 s), claim 200 200

    Rerun on the final revision (0d1bae4):

    check this branch main e4e8acb
    old claim rw, then a new rw claim 409, VM list unchanged; 200 once the old VM is removed (3.8 s) 500 "Cannot lock images", one warm VM destroyed
    old claim ro, then a new rw claim 409; 200 after removal —
    restart with one failing stale VM plus 2 stale warm VMs vm rm ×3 at +120 ms; one extra vm inspect, only for the VM whose removal failed vm rm ×3 at +122 ms
    clean restart, 2 stale warm VMs 0 vm inspect 0 vm inspect

    Start to healthz is 445–448 ms on both builds.

CMGS added 5 commits October 17, 2026 12:02
Startup sweeps unowned sbx- VMs but reserved nothing for the volumes they still had attached, so after a restart with a pending removal a conflicting claim reached cocoon vm disk attach and failed on the hypervisor's image lock with a 500. When a running stale VM's removal fails, the sweep now reads its hot-attached disks from cocoon vm inspect and queues the removal with holds on them, so they drop only once the VM is confirmed gone. Reconcile ends before the server starts, so a removal that succeeds needs no inspect. removeOrRetry loses its tap parameter, which no caller sets any more. Closes #348.
@CMGS
CMGS merged commit 1ca70d8 into main Oct 17, 2026
1 check passed
@CMGS
CMGS deleted the fix/348-restart-volume-holds branch October 17, 2026 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pool: volume holds of a VM whose removal is pending are lost across a sandboxd restart

1 participant