Skip to content

feat(disk): vm disk attach --from-snapshot/--from-dir attaches a snapshot's COW copy - #285

Merged
CMGS merged 9 commits into
masterfrom
feat/284-disk-attach-from-snapshot
Oct 18, 2026
Merged

CMGS merged 9 commits into
masterfrom
feat/284-disk-attach-from-snapshot

Conversation

@CMGS

@CMGS CMGS commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Closes #284.

What

vm disk attach VM --from-snapshot SNAPSHOT and --from-dir DIR hot-attach a private read-write copy of a snapshot's COW disk to a running Cloud Hypervisor VM. The reader VM runs the same image as the sleeping VM, so it already has the image's read-only disks (erofs layers for OCI, the qcow2 base for cloudimg); cocoon hands over only the COW copy. The memory image is never read.

  • Identity: the snapshot's layers in order and its image_blob_ids must equal the target VM's, read from the record reloaded under the ops lock. Digests only, so another node's export works when the image is local. A mismatch is refused with both sides in the error.
  • The role=cow entry of cocoon.json names the file (cow.raw / overlay.qcow2). It is reflinked, without fsync, into the target's run dir as snapdisk-<name>.<raw|qcow2> and attached through the existing add-disk path. A qcow2 copy is always rebased (qemu-img rebase -u) onto the local base blob, since an imported snapshot keeps the source node's backing path.
  • Pre-check, copy and attach run under the ops lock: a retry finds the attached disk and returns it. A lost vm.add-disk response leaves the copy for the retry; a lost vm.remove-device response still reclaims it on the next detach.
  • The copy is removed by detach after the eject, by vm rm with the run dir, and swept wherever runtime files are cleaned after the VMM is dead (stop, hibernate, restore kill, start prepare, failed launch): CleanupRuntimeFiles now expands glob patterns.
  • snapshot.json gains the additive disk_attach_files (["cocoon.json", <cow file>]) in both the tar and --to-dir exports, so an archive consumer fetches only those.
  • Output is the existing attach result plus format (raw | qcow2). The copy is always named snap-<first 8 chars of the snapshot id>, so a repeat of the same snapshot is a no-op and two snapshots never share a disk; --name, --readonly and --directio are refused with the snapshot sources. Firecracker is unsupported (its drives carry no serial).

Nothing is pinned and GC is unchanged: the target VM's own record pins the image blobs. No change on the create, clone or claim paths; each lifecycle cleanup adds one ReadDir of the run dir.

Verification

  • go build ./..., go vet ./..., asl -forwarder=false ./...: clean.
  • make lint: 0 issues. for GOOS=linux and GOOS=darwin.
  • gofumpt -extra -l ., goimports -l: no output.
  • go test ./... on darwin and, for the touched packages, in docker run --platform linux/arm64 golang:1.27.1: all ok.
  • Unit tests cover the identity check (same image under another root, layers in another order, another blob set, cloudimg base), copy naming and format, detach removal, the runtime-file sweep, the envelope field in both export paths, and the CLI flag groups.

Hardware acceptance on a bare-metal node is in the first comment: real attach and guest mount with journal replay on OCI and cloudimg, the source snapshot byte-identical afterwards, the memory image never opened, qcow2 rebase from another root, repeat attach, detach, stop and vm rm cleanup, --from-dir from a three-file export, image mismatch and Firecracker refusals, and the copy time on XFS (0.05 s) versus ext4 (1.1 s for 1.6 GiB allocated).

CMGS added 9 commits October 18, 2026 11:41
…p take glob patterns

The snapshot disk attach copies only the sidecar's cow entry, by basename, so another node's sidecar works. Stop needs to sweep per-name copies, which fixed runtime names cannot express; non-pattern names keep the plain remove so dangling symlinks still go.
Identity check, pre-check, copy and vm.add-disk all run under one ops lock, so a retry after a lost result returns the attached disk instead of truncating its file. Detach removes the copy after the eject; stop, restore and start sweep snapdisk-* with the runtime files.
…lope

Both export paths fill disk_attach_files from the sidecar, so an archive consumer fetches only those. A snapshot without a sidecar exports as before, without the field.
…apshot/--from-dir

--from-snapshot holds the snapshot's read lease across the attach; --from-dir reads the envelope. The cmd layer resolves the local cloudimg base so the hypervisor stays free of image-store knowledge. --name defaults to snap-<id prefix>; --readonly and --directio do not apply to the copy.
The sidecar is cocoon-written and a degenerate cow path already fails
the copy loudly, so the basename guard defended nothing reachable.
… and remove responses

A lost vm.add-disk response may have attached the copy, so the error
path no longer unlinks it; a retry, detach or stop reclaims it. A
detach whose vm.remove-device response was lost finds no device, so
the copy cleanup now also runs on that path.
With a caller-chosen name two snapshots of one image could share a
copy path, and the idempotent pre-check then returned the first
snapshot's disk for the second. Deriving the name from the snapshot id
makes a repeat of the same snapshot a no-op and gives different
snapshots different disks.
@CMGS

CMGS commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Hardware acceptance

Bare-metal node (Linux 6.12 amd64, XFS data volume), Cloud Hypervisor v54, isolated lane with its own root, run dir and CNI bridge. Images: ghcr.io/cocoonstack/cocoon/ubuntu:24.04 and ubuntu:22.04 (OCI), jammy-server-cloudimg-amd64.img (cloudimg). Guest checks through vm exec (OCI) and SSH (cloudimg). Steps 1–8 ran on a4855b8, steps 9–13 on 099b468; the two differ only in the --name refusal, the attach path is the same. Evidence logs are kept on the node under the lane's evidence/.

# Check Result
1 OCI --from-snapshot on a hibernated 4 GiB VM {"name":"snap-otuy647v","id":"cocoon-disk-snap-otuy647v","format":"raw"}; copy shares 73 extents with the source (reflink); guest mount -o ro logs EXT4-fs (vdd): recovery complete, the known file and a 1.5 GiB file's sha256 match what the source wrote
1 Source untouched snapshot cow.raw sha256 3266554141fc…754e67 before and after the guest mount; the copy's sha256 changed (replay landed on the copy)
2 Memory image never read attach wall time 0.051 s (0.0515–0.0572 s over 5 repeats) with a 4 GiB memory-ranges file; strace -f -e trace=openat: 18 opens, 0 of memory-ranges, under the snapshot dir only cocoon.json and cow.raw once each
3 Repeat attach same id, exit 0, copy inode/mtime/size unchanged
4 Detach, vm rm detach (0.24 s) removes the copy; vm rm --force removes the run dir; inspect shows no disks
5 Stop sweep after vm stop the run dir holds only api.sock.lock and cow.raw; vm start and vm exec work afterwards
6 Image mismatch ubuntu:22.04 target: exit 1, no copy left: snapshot image (layers [66e9b213… a5afae19…], blobs [66e9b213… a5afae19…]) does not match vm AUOMNUDO… (layers [88d1ff49… 2d09a75d…], blobs [2d09a75d… 88d1ff49…]); an OCI snapshot against a cloudimg VM is refused the same way with layers [] on the VM side
7 --from-dir snapshot export --to-dir (45 ms) writes "disk_attach_files": ["cocoon.json","cow.raw"]; a dir holding only snapshot.json, cocoon.json, cow.raw attaches and the guest reads the known file; the same with every cocoon.json path rewritten to /nonexistent/node-b/root/… also attaches; vm clone --from-dir on that dir exits 1 with parse source config: … config.json: no such file or directory, no VM left
8 ext4 reader run dir on an ext4 loop mount: ReflinkCopy logs reflink unsupported on the filesystem … (ficlone: operation not supported), the copy is a real copy (du 1.6G = source's allocated 1.6G, 0 shared extents, 10G apparent), attach 1.092 s (ext4→ext4 --from-dir 1.091 s) versus 0.051 s on XFS; guest read and sha256 match; detach 0.42 s removes it
9 cloudimg snapshot holds overlay.qcow2 (376 MB, backing = local base blob); attach {"format":"qcow2"} in 0.071 s, copy backing = local base, 228 shared extents; guest sees a whole GPT disk (/dev/disk/by-id/virtio-snap-j2obsbl5, -part1/14/15), mount -o ro /dev/vdc1 logs recovery complete, known file and a 256 MiB file match; snapshot overlay.qcow2 sha256 unchanged; export writes ["cocoon.json","overlay.qcow2"]; a partial dir whose overlay was rebased to /nonexistent/node-b/… attaches in 0.064 s with the copy's backing rewritten to the local blob while the dir's file stays as it was; snapshot rm of the source after attach leaves the guest reading
10 Firecracker vm disk attach fcvm --from-snapshot S: snapshot disk attach: backend firecracker: backend does not support disk attach, no file created
11 Flag groups, --path unchanged --readonly/--directio/--path/--name with a snapshot source each exit 1 with cobra's group error; --path attach/detach output and behaviour unchanged
12 Cleanup VMs, snapshots, bridge, loop mount and scratch removed; pgrep -fa "^cloud-hypervisor" 5 before and 5 after, same PIDs, none from the lane

Not run on hardware: same layers in a different order (no such image pair exists; unit test only).

Timings (XFS unless noted):

case attach copy
OCI, 1.6G allocated / 10G virtual, 4 GiB memory 0.051 s reflink
cloudimg, 355 MiB / 10G, 2 GiB memory 0.071 s reflink
OCI, snapshot on XFS, run dir on ext4 1.092 s full sparse copy
OCI, --from-dir, ext4 to ext4 1.091 s full sparse copy

@CMGS
CMGS merged commit 2375369 into master Oct 18, 2026
4 checks passed
@CMGS
CMGS deleted the feat/284-disk-attach-from-snapshot branch October 18, 2026 04:37
CMGS added a commit that referenced this pull request Oct 18, 2026
…deps, docs) (#286)

Orphan sweep matches the VMM cmdline instead of kill(0) so an exited, unreaped scan match is not reported killed; Firecracker validates the disk detach name before locking; modern-go and reuse cleanup; dependency and lint tool bumps; docs synced with the code and the v0.7.2 release.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vm disk attach: attach a snapshot's COW copy to a running VM (--from-snapshot / --from-dir)

1 participant