Repository navigation
pool: a stale VM's attached volumes stay held until it is removed - #349
Merged
Merged
Conversation
Startup sweeps unowned sbx- VMs but reserved nothing for the volumes they still had attached, so after a restart with a pending removal a conflicting claim reached cocoon vm disk attach and failed on the hypervisor's image lock with a 500. When a running stale VM's removal fails, the sweep now reads its hot-attached disks from cocoon vm inspect and queues the removal with holds on them, so they drop only once the VM is confirmed gone. Reconcile ends before the server starts, so a removal that succeeds needs no inspect. removeOrRetry loses its tap parameter, which no caller sets any more. Closes #348.
…r a fixed message
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #348.
Problem
A teardown keeps a claim's volume holds until cocoon confirms the VM is gone. The pending-removal queue that carries them lives only in memory.
After a sandboxd restart with a removal still pending, the startup sweep retried removing the unowned VM but held none of its volumes. A conflicting claim (a writer, or a reader of a volume the old VM held writable) was admitted. It then failed at
cocoon vm disk attachon the hypervisor's image lock: a 500, and a warm VM destroyed, instead of a 409.Fix
When the startup sweep cannot remove a running unowned
sbx-VM, it reads the VM's hot-attached disks fromcocoon vm inspect(attached_devices.disks, available since cocoon v0.4.9). It holds those volumes, rw as a writer and ro as a reader, and queues the removal with the holds attached. The existing pending-removal path releases them once the VM is confirmed gone.removeOrRetryloses itstapparameter, which no caller sets any more.docs/deploy.mdgains one sentence about the startup sweep.Cost
Zero on the claim path. At startup there is one extra
cocoon vm inspectper stale VM whose removal fails.Verification
New pool test, with two cases (old claim rw, old claim ro):
The engine test parses cocoon's real inspect JSON shape.
make go-lint: 10 ×0 issues.(5 modules × linux/darwin). asl is clean on both GOOS, andmake go-testpasses.Hardware, a bare-metal testbed with cocoon v0.7.2 and Cloud Hypervisor. The procedure: claim a writable volume, make
vm rmfail for that VM, release, then restart sandboxd.vm rmstops failingRerun on the final revision (0d1bae4):
vm rm×3 at +120 ms; one extravm inspect, only for the VM whose removal failedvm rm×3 at +122 msvm inspectvm inspectStart to healthz is 445–448 ms on both builds.