Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
- name: golangci-lint
uses: golangci/golangci-lint-action@v9.3.0
with:
version: v2.13.2
version: v2.14.0

- name: Check formatting (gofumpt + goimports)
run: golangci-lint fmt --diff
6 changes: 3 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,15 @@ $(LOCALBIN):
mkdir -p $(LOCALBIN)

## Tool versions
GOLANGCILINT_VERSION ?= v2.13.2
GOLANGCILINT_VERSION ?= v2.14.0
GOLANGCILINT_ROOT := $(LOCALBIN)/golangci-lint-$(GOLANGCILINT_VERSION)
GOLANGCILINT := $(GOLANGCILINT_ROOT)/golangci-lint

GOFUMPT_VERSION ?= v0.11.0
GOFUMPT_VERSION ?= v0.12.0
GOFUMPT_ROOT := $(LOCALBIN)/gofumpt-$(GOFUMPT_VERSION)
GOFMT := $(GOFUMPT_ROOT)/gofumpt

GOIMPORTS_VERSION ?= v0.49.0
GOIMPORTS_VERSION ?= v0.51.0
GOIMPORTS_ROOT := $(LOCALBIN)/goimports-$(GOIMPORTS_VERSION)
GOIMPORTS := $(GOIMPORTS_ROOT)/goimports

Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Lightweight MicroVM engine with dual hypervisor backends: [Cloud Hypervisor](htt
- **Docker-like CLI** — `create`, `run`, `start`, `stop`, `list`, `inspect`, `console`, `exec`, `rm`, `clone`, `status`
- **Snapshot & clone** — capture a running VM (memory, disks, config) and clone it into new VMs with fresh network identity; clones restore memory copy-on-write by default (page cache shared across sibling clones) and hold shared source leases so parallel clone bursts survive concurrent source removal; restore and atomic hibernate in place; portable export/import between hosts
- **CNI networking** — multi-queue virtio-net TAPs wired via TC redirect (no bridge in the data path), multi-NIC, bridge mode, NIC hot-resize
- **Data disks & runtime attach** — extra virtio-blk disks at create/clone time; hot-plug vhost-user-fs shares and VFIO PCI devices (CH); external raw disks and NIC resize on CH or Firecracker `--pci` VMs
- **Data disks & runtime attach** — extra virtio-blk disks at create/clone time; hot-plug vhost-user-fs shares and VFIO PCI devices (CH); external raw disks and NIC resize on CH or Firecracker `--pci` VMs; a private copy of a snapshot's COW disk on CH
- **Windows guests** — UEFI + Hyper-V enlightenments via the cocoonstack CH/firmware forks
- **Firecracker backend** — `--fc` for ~125ms boots and <5 MiB per-VM overhead (OCI images only)
- **Daemon-optional architecture** — one hypervisor process per VM, every command standalone; `cocoon daemon` optionally adopts running VMs to converge crashes as they happen; modular lock-safe GC with snapshot LRU eviction
Expand All @@ -19,12 +19,12 @@ Lightweight MicroVM engine with dual hypervisor backends: [Cloud Hypervisor](htt
## Quick Start

```bash
# Install the binary (Linux amd64; arm64: cocoon_0.7.1_Linux_arm64.tar.gz)
curl -fsSL -o cocoon.tar.gz https://github.com/cocoonstack/cocoon/releases/download/v0.7.1/cocoon_0.7.1_Linux_x86_64.tar.gz
# Install the binary (Linux amd64; arm64: cocoon_0.7.2_Linux_arm64.tar.gz)
curl -fsSL -o cocoon.tar.gz https://github.com/cocoonstack/cocoon/releases/download/v0.7.2/cocoon_0.7.2_Linux_x86_64.tar.gz
tar -xzf cocoon.tar.gz && install -m 0755 cocoon /usr/local/bin/

# One-time environment setup (installs CH, firmware, CNI plugins)
curl -fsSL -o cocoon-check https://raw.githubusercontent.com/cocoonstack/cocoon/refs/tags/v0.7.1/doctor/check.sh
curl -fsSL -o cocoon-check https://raw.githubusercontent.com/cocoonstack/cocoon/refs/tags/v0.7.2/doctor/check.sh
install -m 0755 cocoon-check /usr/local/bin/ && sudo cocoon-check --upgrade

# Pull an image and run a VM
Expand Down
1 change: 0 additions & 1 deletion cmd/core/vmconfig_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,6 @@ func TestCloneVMConfigKnobFlagsOverrideSnapshot(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
cmd := &cobra.Command{}
cmd.Flags().String("name", "c", "")
cmd.Flags().Int("nics", 0, "")
cmd.Flags().Int("queue-size", 0, "")
cmd.Flags().Int("disk-queue-size", 0, "")
cmd.Flags().Int("cpu-weight", 0, "")
Expand Down
18 changes: 6 additions & 12 deletions cmd/others/handler_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (
)

func TestParseSnapshotPolicy_Defaults(t *testing.T) {
cmd := buildGCCmd()
cmd := gcCommand()
if err := cmd.ParseFlags(nil); err != nil {
t.Fatal(err)
}
Expand All @@ -33,7 +33,7 @@ func TestParseSnapshotPolicy_NegativeRejected(t *testing.T) {
}
for _, tt := range cases {
t.Run(tt.name, func(t *testing.T) {
cmd := buildGCCmd()
cmd := gcCommand()
if err := cmd.ParseFlags(tt.args); err != nil {
t.Fatal(err)
}
Expand All @@ -57,7 +57,7 @@ func TestParseSnapshotPolicy_SubFlagRequiresSnapshot(t *testing.T) {
}
for _, args := range cases {
t.Run(args[0], func(t *testing.T) {
cmd := buildGCCmd()
cmd := gcCommand()
if err := cmd.ParseFlags(args); err != nil {
t.Fatal(err)
}
Expand All @@ -70,7 +70,7 @@ func TestParseSnapshotPolicy_SubFlagRequiresSnapshot(t *testing.T) {
}

func TestParseSnapshotPolicy_HappyPath(t *testing.T) {
cmd := buildGCCmd()
cmd := gcCommand()
if err := cmd.ParseFlags([]string{
"--snapshot",
"--snapshot-keep=10",
Expand All @@ -89,12 +89,6 @@ func TestParseSnapshotPolicy_HappyPath(t *testing.T) {
}
}

func buildGCCmd() *cobra.Command {
cmd := &cobra.Command{Use: "gc"}
cmd.Flags().Bool("snapshot", false, "")
cmd.Flags().Int("snapshot-keep", 0, "")
cmd.Flags().Duration("snapshot-age", 0, "")
cmd.Flags().String("snapshot-size", "", "")
cmd.Flags().Bool("snapshot-dry-run", false, "")
return cmd
func gcCommand() *cobra.Command {
return Commands(Handler{})[0]
}
6 changes: 3 additions & 3 deletions cmd/storebench/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"os/exec"
"path/filepath"
"runtime/pprof"
"slices"
"strconv"
"time"

Expand Down Expand Up @@ -182,10 +183,9 @@ func runMicro(ctx context.Context, engine, op string, n, ops int, dir string) er
for i := range n {
seed = append(seed, fmt.Sprintf("S%023d", i))
}
for base := 0; base < n; base += 1000 {
hi := min(base+1000, n)
for chunk := range slices.Chunk(seed, 1000) {
if err := store.Update(ctx, meta.Scope{Write: ns}, meta.CommitDurable, func(w meta.Writer) error {
for _, id := range seed[base:hi] {
for _, id := range chunk {
if err := w.PutRaw(ctx, ns, hypervisor.TableRecords, id, benchPayload, false); err != nil {
return err
}
Expand Down
3 changes: 1 addition & 2 deletions daemon/reconcile_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -409,8 +409,7 @@ func newFake() *fakeSupervisor {

func newTestDaemon(t *testing.T, f *fakeSupervisor) *Daemon {
t.Helper()
d := New(Config{RootDir: t.TempDir()}, nil, []Supervisor{f})
return d
return New(Config{RootDir: t.TempDir()}, nil, []Supervisor{f})
}

func runningRec(id string, gen uint64) *hypervisor.VMRecord {
Expand Down
9 changes: 5 additions & 4 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ cocoon
│ │ ├── attach [flags] VM Attach a VFIO PCI device (CH only)
│ │ └── detach [flags] VM Detach a VFIO PCI device by --id
│ ├── disk
│ │ ├── attach [flags] VM Hot-attach an existing raw disk file (CH, or FC with --pci)
│ │ └── detach [flags] VM Detach a hot-attached disk by --name (keeps the file)
│ │ ├── attach [flags] VM Hot-attach a raw disk file (CH, or FC with --pci) or a snapshot's COW disk copy (CH)
│ │ └── detach [flags] VM Detach a hot-attached disk by --name (keeps a --path file, removes a snapshot copy)
│ ├── net [flags] VM Resize NIC count on a running VM (CH, or FC with --pci)
│ └── debug [flags] IMAGE Generate hypervisor launch command (dry run)
├── snapshot
Expand Down Expand Up @@ -294,8 +294,9 @@ Applies to `cocoon vm debug`:

`vm debug` prints the VMM launch command only. It does not prepare a netns or a
TAP, so it emits no `--net` and no `ip=`, and `--nics`, `--queue-size`,
`--network` and `--bridge` are ignored — setting any of them warns on stderr,
like `--data-disk`. `--max-cpu`, `--balloon`, `--cow` and `--ch` shape the Cloud
`--network`, `--bridge` and the cgroup CPU knobs (`--cpu-weight`,
`--cpu-quota-us`, `--cpu-period-us`, `--cpu-burst-us`, `--cpuset-cpus`) are
ignored — setting any of them warns on stderr, like `--data-disk`. `--max-cpu`, `--balloon`, `--cow` and `--ch` shape the Cloud
Hypervisor command only; `vm debug --fc` prints the Firecracker REST sequence
and ignores all four. Queue placement happens at launch against the store, so
the printed command carries no `queue_affinity` whatever `--cpuset-cpus` says.
Expand Down
4 changes: 2 additions & 2 deletions docs/devices.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# Runtime Device Attach

Hot-plug vhost-user-fs shares, VFIO PCI devices, and external raw disks on a running VM (Cloud Hypervisor; raw disks also on Firecracker VMs created with `--pci`).
Hot-plug vhost-user-fs shares, VFIO PCI devices, external raw disks, and copies of a snapshot's COW disk on a running VM (Cloud Hypervisor; raw disks also on Firecracker VMs created with `--pci`).

## Overview

Cocoon can hot-plug three classes of external resources onto a running VM:

- **Vhost-user-fs** — a file share served by an external `virtiofsd` (or compatible backend) over a Unix socket. Attach surfaces a virtio-fs device in the guest, accessible via `mount -t virtiofs <tag> /mnt/...`.
- **VFIO PCI passthrough** — a host PCI device bound to `vfio-pci` (GPU, NIC, NVMe). Attach hands the device to the guest with IOMMU isolation.
- **Data disks** — an existing raw disk file, surfaced as `/dev/disk/by-id/virtio-<name>`. Cocoon never creates or deletes the backing file: it can outlive any VM and be re-attached elsewhere (a persistent volume).
- **Data disks** — an existing raw disk file, surfaced as `/dev/disk/by-id/virtio-<name>`. Cocoon never creates or deletes the backing file: it can outlive any VM and be re-attached elsewhere (a persistent volume). On Cloud Hypervisor the disk can instead be a private read-write copy of a snapshot's COW disk (`--from-snapshot` / `--from-dir`), which cocoon creates in the VM's run dir and removes on detach, stop, or `vm rm`.

All three attaches are **runtime-only**: the device lives only for the current VM process and is gone after stop/restart — re-attach after the next start. cocoon refuses `snapshot save` and `vm hibernate` while any of the three is attached — umount an external disk in-guest, detach, capture, then re-attach after the wake/restore; on both backends the refusal arrives before the guest is paused. Cloud Hypervisor does not enforce this itself: it captures a snapshot that names the vhost-user socket, and restoring that snapshot either fails once the backend is gone or hangs against a fresh one. External volumes are host state cocoon does not own: it never creates, copies, or deletes the backing file, and snapshot, restore, and clone carry no trace of them. All mutating verbs on one VM (attach/detach, net resize, snapshot, hibernate, restore, stop) serialize on a per-VM lock, so concurrent invocations cannot interleave with a capture window. While the VM runs, `vm inspect` reports the live attach set under `.attached_devices` (`fs`, `devices`, `disks`).

Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ cocoon CLI ──► images: OCI (EROFS layers, direct boot) | cloudimg (qcow2,
- **TC redirect I/O path** — veth ↔ TAP wired via ingress qdisc + mirred redirect (no bridge in the data path)
- **DNS configuration** — custom DNS servers injected into VMs via kernel cmdline (OCI) or cloud-init network-config (cloudimg)
- **Cloud-init metadata** — automatic NoCloud cidata FAT12 disk for cloudimg VMs (hostname, configurable user/password via `--user`/`--password`, multi-NIC Netplan v2 network-config); cidata is automatically skipped on subsequent boots
- **User data disks** — `--data-disk` attaches additional virtio-blk disks per VM, with optional ext4 mkfs at create time, cloud-init `mounts:` auto-mount on cloudimg+CH (via `/dev/disk/by-id/virtio-<name>`), per-disk DirectIO override, and 1:1 inheritance through snapshot/clone/restore; `vm clone --data-disk` adds fresh disks to a clone and `vm disk attach/detach` hot-plugs existing raw files on a running VM (CH, or Firecracker with `--pci`)
- **User data disks** — `--data-disk` attaches additional virtio-blk disks per VM, with optional ext4 mkfs at create time, cloud-init `mounts:` auto-mount on cloudimg+CH (via `/dev/disk/by-id/virtio-<name>`), per-disk DirectIO override, and 1:1 inheritance through snapshot/clone/restore; `vm clone --data-disk` adds fresh disks to a clone and `vm disk attach/detach` hot-plugs existing raw files on a running VM (CH, or Firecracker with `--pci`) or a private copy of a snapshot's COW disk (CH)
- **Copy-on-write clone restore** — Cloud Hypervisor clones of plain private-anon snapshots default to `mmap` memory restore: no eager copy, page cache shared across sibling clones; hugepages/shared snapshots fall back to eager copy with a warning
- **Hugepages** — opt-in via `vm create --hugepages` (Cloud Hypervisor only); backs VM memory with hugetlbfs at the cost of the mmap restore fast path for that VM's snapshots (never supported on Firecracker, whose snapshots cannot restore from hugetlbfs)
- **Mergeable memory** — opt-in `--mergeable` (Cloud Hypervisor only) madvises guest memory `MADV_MERGEABLE` so host KSM can dedup identical pages across VMs; excludes `--hugepages`/`--shared-memory`
Expand Down
6 changes: 3 additions & 3 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,10 @@ Download pre-built binaries from [GitHub Releases](https://github.com/cocoonstac

```bash
# Linux amd64
curl -fsSL -o cocoon.tar.gz https://github.com/cocoonstack/cocoon/releases/download/v0.7.1/cocoon_0.7.1_Linux_x86_64.tar.gz
curl -fsSL -o cocoon.tar.gz https://github.com/cocoonstack/cocoon/releases/download/v0.7.2/cocoon_0.7.2_Linux_x86_64.tar.gz

# Linux arm64
curl -fsSL -o cocoon.tar.gz https://github.com/cocoonstack/cocoon/releases/download/v0.7.1/cocoon_0.7.1_Linux_arm64.tar.gz
curl -fsSL -o cocoon.tar.gz https://github.com/cocoonstack/cocoon/releases/download/v0.7.2/cocoon_0.7.2_Linux_arm64.tar.gz

tar -xzf cocoon.tar.gz
install -m 0755 cocoon /usr/local/bin/
Expand All @@ -54,7 +54,7 @@ Cocoon ships a diagnostic script that checks your environment and can auto-insta

```bash
# Get script
curl -fsSL -o cocoon-check https://raw.githubusercontent.com/cocoonstack/cocoon/refs/tags/v0.7.1/doctor/check.sh
curl -fsSL -o cocoon-check https://raw.githubusercontent.com/cocoonstack/cocoon/refs/tags/v0.7.2/doctor/check.sh
install -m 0755 cocoon-check /usr/local/bin/

# Check only — reports PASS/FAIL for each requirement
Expand Down
6 changes: 3 additions & 3 deletions docs/known-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,9 +246,9 @@ On CNI plugins with strict per-veth MAC enforcement (Cilium eBPF, Calico eBPF),

`cocoon vm fs attach` only works on CH VMs that were created with `--shared-memory`. CH's `memory shared=on` is fixed at VM creation: backend processes (e.g. virtiofsd) need to mmap guest memory via the negotiated memfd, and the memory model cannot be flipped on a running VM. If `--shared-memory` was omitted at create time, the only path is to recreate the VM. Cocoon's preflight reads `vm.info` and surfaces a clear error rather than letting CH return a vague rejection.

## Snapshotting a VM with attached vhost-user-fs / VFIO is refused
## Snapshotting a VM with attached vhost-user-fs / VFIO / hot-attached disks is refused

Cloud Hypervisor captures a snapshot of a VM that holds a vhost-user-fs share or a VFIO PCI passthrough device, but the result cannot be restored: it names a backend socket that is gone (restore fails after 60s) or hangs against a fresh one. Cocoon refuses the capture itself — the pre-flight reads the live `vm.info` before pausing the guest and fails with `hot-attached vhost-user-fs "<tag>": detach before snapshot or hibernate` (or `hot-attached device "<path>": ...`) before anything is captured or recorded. `cocoon vm fs detach` / `cocoon vm device detach` first to clear runtime devices, then snapshot.
Cloud Hypervisor captures a snapshot of a VM that holds a vhost-user-fs share or a VFIO PCI passthrough device, but the result cannot be restored: it names a backend socket that is gone (restore fails after 60s) or hangs against a fresh one. Cocoon refuses the capture itself — the pre-flight reads the live `vm.info` before pausing the guest and fails with `hot-attached vhost-user-fs "<tag>": detach before snapshot or hibernate` (or `hot-attached device "<path>": ...`, `hot-attached disk "<name>": ...`) before anything is captured or recorded. `cocoon vm fs detach` / `cocoon vm device detach` / `cocoon vm disk detach` first to clear runtime devices, then snapshot.

## Snapshot lease files are never reclaimed (fixed in v0.6.8)

Expand All @@ -262,7 +262,7 @@ On v0.6.7 and older every `snapshot save` leaves that zero-byte file behind and

## Runtime attached devices do not survive VM stop / clone / restore

Attaches via `cocoon vm fs attach` and `cocoon vm device attach` are runtime-only — they live in the CH process state and are never written into the VM record, sidecar, or snapshot. After `vm stop`, `vm clone`, or `vm restore`, the user must re-run the attach commands. `vm inspect` reflects the live CH `vm.info` for running VMs and omits `attached_devices` for stopped VMs. This is by design: cocoon does not own the backend (virtiofsd / vfio-pci binding) and cannot guarantee the resource still exists across host events.
Attaches via `cocoon vm fs attach`, `cocoon vm device attach` and `cocoon vm disk attach` are runtime-only — they live in the VMM process state and are never written into the VM record, sidecar, or snapshot. After `vm stop`, `vm clone`, or `vm restore`, the user must re-run the attach commands. `vm inspect` reflects the live VMM state for running VMs and omits `attached_devices` for stopped VMs. This is by design: cocoon does not own the backend (virtiofsd, the vfio-pci binding, an external disk file) and cannot guarantee the resource still exists across host events.

## virtiofsd is a single-shot daemon

Expand Down
2 changes: 1 addition & 1 deletion docs/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ cocoon vm clone imported-snap
cocoon snapshot export my-snap -o - | ssh host2 cocoon snapshot import --name my-snap
```

The archive contains the snapshot config, VM config, COW disk, memory ranges, and device state — sparse files carry cocoon's own pax records so their holes survive the round-trip — everything needed to reconstruct the snapshot on a different machine. The envelope also lists the files the archive carries, and `snapshot import` refuses an archive that ends before every listed file has landed — a plain tar cut at an entry boundary otherwise reads as complete. `snapshot export --to-dir` writes the same files as a directory that `vm clone --from-dir` and `vm restore --from-dir` consume without a tar round-trip. Do not repack an export with a third-party tar: dropping cocoon's sparse records can rebuild a short, shifted disk. `--from-dir` and `snapshot import` reject a `cow.raw` whose size differs from the envelope's recorded storage size.
The archive contains the snapshot config, VM config, COW disk, memory ranges, and device state — sparse files carry cocoon's own pax records so their holes survive the round-trip — everything needed to reconstruct the snapshot on a different machine. The envelope also lists the files the archive carries, and `snapshot import` refuses an archive that ends before every listed file has landed — a plain tar cut at an entry boundary otherwise reads as complete. `snapshot export --to-dir` writes the same files as a directory that `vm clone --from-dir`, `vm restore --from-dir` and `vm disk attach --from-dir` (a copy of the COW disk on a running VM of the same image; see [Devices](devices.md)) consume without a tar round-trip. Do not repack an export with a third-party tar: dropping cocoon's sparse records can rebuild a short, shifted disk. `--from-dir` and `snapshot import` reject a `cow.raw` whose size differs from the envelope's recorded storage size.

#### Cross-Node Clone

Expand Down
Loading
Loading