Read-only file access to hibernated or archived sandboxes without waking them

README.md

@@ -36,7 +36,9 @@ performance) — source in
   snapshot exports (online-retunable), claim/release/renew/hibernate/fork/
   promote/checkpoint HTTP API, operator-catalog dataset volumes (read-only or
   writable; read-only ones change on a running claim), signed preview URLs,
-  the HTTP-upgrade byte relays to silkd and to any guest port, claim env with
+  the HTTP-upgrade byte relays to silkd and to any guest port, read-only file
+  access to hibernated or archived claims from a warm VM of their pool
+  (opt-in per pool `reader`), claim env with
   host-side credential injection and per-claim upstream proxies, runtime
   tenants, config reload (SIGHUP), usage + audit journals, /metrics, reap +
   restart reconcile with VMM cold restart, memberlist mesh with redirect

docs/cluster.md

@@ -334,7 +334,9 @@ checkpoint store and keeps each claim's registry row on the latest snapshot:
 - **Self-fencing.** A node whose renewal is overdue by 12 seconds (the lease
   length minus a 3 second margin, counted from the start of the last renewal
   that answered) stops serving every replicated claim it holds: relay, exec,
-  ports and preview answer `503 lease lost; retry`, and `/owner` answers 404 so
+  ports and preview answer `503 lease lost; retry` (a read through a
+  [reader](deploy.md#reading-sleeping-claims-without-waking-them) after the `101`
+  gets an `error` frame instead), and `/owner` answers 404 so
   a `Lookup` moves on to the other nodes. A claim is replicated when its policy
   pool has `replicate_seconds`, from the moment it is claimed, adopted or
   loaded, whether or not a round has run, so the fence covers it before its

docs/deploy.md

@@ -14,7 +14,7 @@ the cocoon CLI and needs a template image with silkd baked in.
   empty `snapshot list --format json`, which sandboxd's snapshot reconcile
   reads. sandboxd logs a warning at startup when the detected cocoon is below
   v0.6.9 (a dev/`master-<sha>` build is assumed current); changing volumes on
-  a running claim needs v0.7.2
+  a running claim needs v0.7.2, and a pool's `reader` needs v0.7.3
 - The sandbox boot artifact installed where cocoon finds it
   (`/boot/vmlinuz-sandbox`, `/boot/initrd.img-sandbox` — from
   `ghcr.io/cocoonstack/sandbox/boot:<kernel-ver>`)
@@ -141,6 +141,7 @@ gets the intersection of the two.
 | `warm_max` (pool entry) | 0 (static) | turns on the demand-adaptive watermark for that pool: the warm target rises from `warm` toward `warm_max` while claims arrive faster than the measured provision lead covers, and decays back over ~a minute of silence. The pool follows the target down: warm VMs above it are destroyed once the count has stayed above it for a full decay period, so a burst does not leave the pool parked at its high-water mark |
 | `warmup` (pool entry) | unset | argv run in the golden VM after readiness and before its snapshot, so the files it touches are page-cache-resident in every clone, and again in every clone before it joins the warm pool, so those pages are already faulted into the restored VM when the first command runs — e.g. `["node", "-e", "0"]` on a Node flavor. It runs under the engine's 2-minute command timeout in silkd's base environment (`PATH`, `TERM`, and the guest image's proxy variables wherever nothing routes directly — the none lane and the locked bridge egress lane — with the proxy not yet serving, since a door pre-bound at refill only starts serving at claim); a non-zero exit or a timeout fails the golden build, so the pool stays unfilled until the config is fixed. Config-owned like `egress`: `PUT /v1/pools` rejects it, and a golden built with a different warmup is rebuilt |
 | `capture_trim` (pool entry) | `false` | Trim the guest's copy-on-write disk before a promote or checkpoint of this pool's sandboxes (and of the clones of a template promoted from it), so the record carries only live data: blocks a build deleted — a `COPY` step's archive, a `RUN` step's caches — are no longer captured. sandboxd mounts the disk init resolved from `cocoon.cow` a second time at `/run/sandboxd-trim` through silkd as root, runs `fstrim` and unmounts, under the capture's transition lock (about 15 ms on a warm clone). A failed trim is logged and the capture proceeds, larger. A hibernated or archived sandbox is captured untrimmed. Config-owned like `egress`: `PUT /v1/pools` rejects it |
+| `reader` (pool entry) | `false` | Serve read-only file calls on this pool's hibernated or archived claims from a warm VM of the same pool instead of waking them; needs `warm` of at least 1 and a cocoon whose `vm disk attach` takes `--from-snapshot` and `--from-dir`. See [Reading sleeping claims](#reading-sleeping-claims-without-waking-them). Config-owned like `capture_trim`: `PUT /v1/pools` rejects it |
 | `egress_classes` | unset | named tenant egress layers: `[{"name": "desk", "egress": {…}, "egress_upstream_env": "DESK_UPSTREAM"}]`. A tenant names one with `egress_class` through the [tenant API](sandboxd-api.md#tenants-v1tenants) and its claims take that policy intersected with the pool's. `egress` is required; `intercept` stays a pool-rule flag. Reloadable: a class's policy reaches live claims on their next request. A class removed while tenants still name it leaves them with no egress and is logged at load and at reload |
 | `egress_upstream_env` (pool or egress class entry) | unset | names a node environment variable holding the default upstream proxy URL for the pool's or class's claims (class before pool, a claim's own entry before both); needs `egress_upstream`, and the URL must be on its `allow`. Read and checked at startup. Config-owned like `egress`: `PUT /v1/pools` rejects it |
 | `replicate_seconds` (pool entry) | 0 (off) | snapshot every running claim of this pool to the checkpoint store this often and point the claim's registry row at the snapshot, so another node restores the claim, memory and processes included, when this node is lost (see [running claims across node loss](cluster.md#running-claims-across-node-loss)). Each round pauses the guest for the length of a memory and disk capture. A restore resumes from the last committed replica and loses everything after it: up to the longer of the interval and one replication pass, plus the 5-second tick, plus one more pass, since a claim can run first in one pass and last in the next. A pass covers all the node's due claims of the replicating pools with bounded concurrency, so it grows with claim count and guest memory; the first round of a claim after a wake or adoption uploads the whole claim, so that pass is the long pass. The claim's row carries no retention deadline while it runs. Requires `archive_failover` and a `none`-lane pool, both checked at config load; a claim with volumes is skipped. Config-owned like `egress`: `PUT /v1/pools` rejects it. Restart only: a claim is replicated by its pool's setting from the moment it is claimed, so a reload that changes it is refused (a pool restarted with 0 drops its claims' rows and replicas on the first pass) |
@@ -166,6 +167,84 @@ fragment the warm pools):
 | `xlarge` | 4 | 8G |
 | `2xlarge` | 8 | 16G |
 
+### Reading sleeping claims without waking them
+
+A client that only lists or reads files (a UI showing an idle agent's
+workspace, a tool collecting results) would otherwise wake the claim and
+hold its memory until the next idle hibernate, and for an archived claim
+fetch the whole checkpoint first. With `reader` set on the claim's pool,
+`fs_stat`, `fs_list`, `fs_read` and `fs_pull` on a hibernated or archived
+claim are served from a reader instead; anything else on the connection wakes
+the claim as before ([API](sandboxd-api.md#get-v1sandboxesidagent)).
+
+```json
+{
+  "pools": [
+    {"template": "rt:24.04", "warm": 4, "reader": true, "idle_hibernate_seconds": 600}
+  ]
+}
+```
+
+It needs Cloud Hypervisor and a cocoon whose `vm disk attach` takes
+`--from-snapshot` and `--from-dir` (v0.7.3 or newer). Leave `reader` off
+on a Firecracker node or with an older cocoon: there every read spends a warm VM
+on an attach that fails, then wakes the claim. An archive exported by an older
+cocoon, whose `snapshot.json` has no `disk_attach_files`, wakes before any VM is
+taken. Unpooled claims (template and checkpoint claims) always wake.
+`reader` is reloadable like `capture_trim`; turning it off leaves open reader
+connections to finish, and the next connection wakes the claim.
+
+A reader is a warm VM taken from the claim's own pool, so it boots the claim's
+image with the claim's layers, and it has that pool's lane: no NIC on the none
+lane, the bridge NIC the claim had on the bridge lane (egress-lane claims never
+hibernate). Taking one removes a VM from the pool until the refill replaces it,
+and an empty pool never cold-boots one: the call wakes the claim. A reader is never shared: it
+serves one claim, and only the claim's own token reaches it. The guest's own
+kernel does the parsing, as it does for the claim:
+
+- a hibernated claim's copy-on-write disk is copied by cocoon (`vm disk attach
+  --from-snapshot`) into the reader VM and hot-attached read-write; cocoon uses
+  a reflink where the filesystem has them, copies only that disk, never the
+  memory image, and removes the copy with the VM. The claim's transition lock is
+  held for the copy so a wake cannot consume the snapshot meanwhile;
+- an archived claim's record is fetched only for `snapshot.json` and the files
+  its `disk_attach_files` lists, into the reader's own directory under
+  `data_dir` (the dir backend copies them out of the record, the s3 backend
+  downloads only the chunks those files name, holes stay holes), then attached with `--from-dir`. The fetched files
+  are deleted once cocoon has its copy, and nothing lands in the store's shared
+  fetch cache;
+- the reader mounts the attached disk writable, so the guest replays the ext4
+  journal onto the copy and never onto the original, and mounts read-only the
+  layers named by its own `cocoon.layers=` kernel parameter. It assembles a
+  read-only overlay at `/view` and starts the claim's own `silkd`
+  (`/usr/local/bin/silkd`) chrooted into it on guest vsock port 2051. Paths and
+  symlinks resolve inside the claim's filesystem, and a write fails at the
+  read-only overlay.
+
+The pool's image must offer `mount`, `chroot`, `env`, `mkdir` and `sync` and
+boot a kernel that mounts erofs, ext4 and overlayfs, which the shipped images
+do. The claim's image must ship `silkd` at `/usr/local/bin/silkd`, as the
+shipped Linux images do; the android image does not, so its claims always wake.
+A missing silkd is detected right after the view is mounted, so such a claim
+pays one reader VM, the copy and the mounts before its call wakes it, never a
+probe timeout. A reader is kept for 30 s after its last connection, then its VM
+is destroyed. A wake waits for a reader only while cocoon copies the disk:
+milliseconds with a reflink, the time to copy the whole disk without one. A
+reader serves only the sleep it was built from: a later hibernate or archive
+gets a fresh one. The claim's own code runs in its reader with root, as it did
+in the claim, and the reader serves one claim and is destroyed afterwards, so
+the claim's bytes never reach a shared VM or the host kernel.
+
+Whenever a reader cannot be built (no warm VM in the claim's pool, the disk
+reserve is low, a failed attach, an archive without
+`disk_attach_files`, an image without `silkd` at that path), the call wakes the
+claim as it does without the feature. With the feature on for the pool, every
+hibernate of its claims runs a guest `sync` right before the pause, and a reader
+shows the disk as of that `sync`: a write the guest makes in the instant between
+the `sync` and the pause is visible only after a wake. A hibernate whose `sync`
+fails or times out is marked not reader-safe, as is any sleep that began before
+the pool had `reader`, and its reads wake the claim as without the feature.
+
 ### Dataset volumes
 
 Running claims can change mounted read-only catalog entries through the
@@ -711,7 +790,7 @@ validates it with the same code as boot, and applies the reloadable settings
 all at once or not at all. Every field falls into one of three classes:
 
 - **Reloaded**:
-  - per pool: `egress`, `warmup`, `capture_trim`, `storage`, `egress_upstream_env`, for existing keys and for keys new in the file;
+  - per pool: `egress`, `warmup`, `capture_trim`, `storage`, `reader`, `egress_upstream_env`, for existing keys and for keys new in the file;
   - `egress_classes`: each class's `egress` and `egress_upstream_env`;
   - node-wide: `egress_internal_allow`, `secrets`, `egress_upstream`, `egress_usage_bytes`, `egress_conn_rate` and `disk_reserve`.
 - **Left to their API**: a pool's targets (`warm`, `warm_max`, idle and archive

docs/sandboxd-api.md

@@ -393,7 +393,8 @@ Auth: the sandbox's own token, or the root token by id (operator, like
 release). Atomically snapshots the VM and stops it,
 freeing its memory; the next agent access restores it transparently
 (sessions, processes, and memory state intact — cocoon's hibernate keeps the
-snapshot point and the stop coincident). Idempotent on an already-hibernated
+snapshot point and the stop coincident), except a read-only file call that a
+[reader](#get-v1sandboxesidagent) serves without waking it. Idempotent on an already-hibernated
 sandbox. The TTL keeps running: a hibernated sandbox is still reaped (VM and
 snapshot) at its deadline, unless it claimed `on_expire: archive` or its pool
 archives. When to hibernate is the caller's policy — the
@@ -578,7 +579,7 @@ is deleted; a pool kept with `warm: 0` keeps its golden.
 Answers the fresh `GET /v1/info` payload. 400 bad key, negative warm/idle,
 `warm_max` below `warm`, `idle_hibernate_seconds` on an egress pool, a
 negative archive duration or an `archive_after_seconds` not above the pool's
-`idle_hibernate_seconds`, duplicate pool, or a config-owned `egress`/`warmup`/`capture_trim`/`storage`/`replicate_seconds`/`egress_upstream_env`
+`idle_hibernate_seconds`, duplicate pool, or a config-owned `egress`/`warmup`/`capture_trim`/`storage`/`replicate_seconds`/`reader`/`egress_upstream_env`
 field; 401 bad api
 token; 409 egress pool on a node without an egress attachment; 503 the
 `meta_store` cannot take a cell's write.
@@ -964,6 +965,33 @@ while its node's lease is overdue (see
 claim of a replicating pool that is already open when the lease turns overdue
 is closed, whether or not it has a replica yet.
 
+A hibernated or archived sandbox wakes on the first frame that opens an RPC,
+unless the sandbox's pool sets [`reader`](deploy.md#reading-sleeping-claims-without-waking-them).
+With it, the node answers the `101` before it touches the sandbox, reads the
+first frame, and serves `fs_stat`, `fs_list`, `fs_read` and `fs_pull` from a
+reader VM, a warm VM of the sandbox's pool that mounts a copy of the sandbox's disk read-only: the sandbox stays
+hibernated or archived, no `wake` usage event is recorded, and the read stamps
+neither the idle nor the archive clock. An `info` frame pipelined ahead of the
+call (the SDKs' first frames) rides along to the reader; one alone, with no call
+behind it for 100 ms, wakes the sandbox. The first frame on the connection that
+is anything else, or any later one, wakes the sandbox and hands the same
+connection to its silkd with the unread bytes replayed, so a client sees one
+silkd and no close. When the sandbox sleeps again meanwhile (a new hibernate or
+an archive), the connection's next read moves to a fresh reader of the new sleep;
+when it wakes, the connection moves to the woken sandbox, so reads never come
+from a stale sleep. Files
+and symlinks resolve inside the sandbox's own filesystem as of the guest `sync`
+that runs right before the last hibernate's pause; a write the guest made in
+the instant between that `sync` and the pause is visible only after a wake. A
+hibernate whose `sync` failed, or one made before the pool had `reader`,
+is not served by a reader: its reads wake the sandbox. When no reader can be built (no warm VM in the pool, a failed attach, an archive from an older cocoon) the call wakes the sandbox as if the
+feature were off. Because the `101` has gone out by then, a failure on this path
+(unknown sandbox, a refused wake, an unreachable guest, an overdue node lease)
+arrives as a silkd `error` frame (`not_found` for an unknown sandbox, else
+`internal`) rather than an HTTP status, and the connection ends. A connection
+served by a reader, or handed over to the woken sandbox, of a claim of a
+replicating pool is closed when the lease turns overdue, as an open relay is.
+
 ## GET /v1/sandboxes/{id}/ports/{port}
 
 Auth: the sandbox's own token, or the root `api_token` for a control-plane

docs/sdk.md

@@ -376,6 +376,11 @@ automatically with the same transparent wake. A claim with a connection live
 when the sweep checks it (a relay stream, a buffered exec, a preview dial, an
 egress request) is not swept; the idle clock restarts when that connection ends.
 
+When the sandbox's pool sets [`reader`](deploy.md#reading-sleeping-claims-without-waking-them),
+`Stat`, `ReadFile`, `ReadFileTo`, `ListDir` and `Pull` on a hibernated or archived handle are
+served without waking it, as of the guest `sync` just before the last hibernate, and the handle's next call
+that is anything else wakes the sandbox on the same connection.
+
 `WithArchiveOnExpire()` makes the lease's end archive the sandbox instead of
 destroying it, whatever the deployment's archive settings: the node hibernates
 it if it is running and moves it to the checkpoint store, and the next call

docs/silkd.md

@@ -6,7 +6,10 @@ the port) for the host only — a peer with any other vsock CID is refused —
 and started at sysinit in parallel with boot. It is what actually
 "runs something" inside a sandbox; sandboxd relays SDK connections to it
 byte-for-byte. Reaching silkd is the
-claim-readiness signal — a claim returns only once silkd answers.
+claim-readiness signal — a claim returns only once silkd answers. A
+[reader VM](deploy.md#reading-sleeping-claims-without-waking-them) runs a
+second copy, the claim's own, chrooted into a read-only view of the claim's
+filesystem on vsock port 2051.
 
 Most users never speak the protocol directly — the [Go SDK](sdk.md) and
 [Python SDK](sdk-python.md) cover the full surface. This page is the wire

e2e/fakeengine_test.go

@@ -28,6 +28,10 @@ type fakeEngine struct {
 	volumeVMs map[string]bool
 	guestEnvs map[string]string
 	seq       int
+
+	readerRoot     string
+	readerStarts   []string
+	readerAttaches []string
 }
 
 func newFakeEngine(dir string) *fakeEngine {
@@ -88,6 +92,33 @@ func (f *fakeEngine) SnapshotExport(_ context.Context, _, toDir string) error {
 	return os.MkdirAll(toDir, 0o750)
 }
 
+func (f *fakeEngine) DiskAttachSnapshot(_ context.Context, vmName, snapName string) (string, error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.readerAttaches = append(f.readerAttaches, vmName+":"+snapName)
+	return "snap-" + snapName, nil
+}
+
+func (f *fakeEngine) DiskAttachDir(context.Context, string, string) (string, error) {
+	return "", errors.New("no archive in this fake")
+}
+
+func (f *fakeEngine) StartReader(_ context.Context, vsockSocket, _ string) error {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.readerStarts = append(f.readerStarts, vsockSocket)
+	return nil
+}
+
+func (f *fakeEngine) DialReader(_ context.Context, _ string) (net.Conn, error) {
+	f.mu.Lock()
+	root := f.readerRoot
+	f.mu.Unlock()
+	client, server := net.Pipe()
+	go silkdtest.NewFake(root).ServeConn(server)
+	return client, nil
+}
+
 func (f *fakeEngine) SnapshotRemove(_ context.Context, _ string) error { return nil }
 
 func (f *fakeEngine) SnapshotList(_ context.Context) ([]string, error) { return nil, nil }