Skip to content

feat(sandbox): add opt-in nested virtualization - #1

Merged
martinothamar merged 1 commit into
main-digdirfrom
feat/nested-virt
Oct 23, 2026
Merged

martinothamar merged 1 commit into
main-digdirfrom
feat/nested-virt

Conversation

@martinothamar

@martinothamar martinothamar commented Oct 19, 2026 •

Copy link
Copy Markdown

Adds a per-sandbox opt-in to the Rust SDK that exposes hardware virtualization to the guest, so it has /dev/kvm and can run its own VMs. This is the microsandbox part of digdir/digdir-agents#173, which tracks nested Sandboxes for Agents. digdir-agents only uses the Rust SDK, so the CLI and the other language SDKs are left unchanged.

One commit on main-digdir at v0.7.8-digdir.1 (45315a8), to ship in 0.7.8-digdir.2. It lands after the version bump with a rebase merge.

agentctl will expose the setting in the agent manifest, so whoever writes an agent definition opts in. Until v0.7.8-digdir.1, every sandbox on Linux x86_64 got nesting implicitly; msb_krun 0.1.41 masks VMX/SVM unless the setting is on. agentctl needs modify() to keep nesting for its existing sandboxes, and the host check to validate a manifest that asks for it.

What changes

  • SandboxResources.nested_virt is stored with the sandbox. It is off by default and left out of serialized records and launch payloads while off, so existing records and launches don't change.
  • SandboxBuilder::nested_virt(bool) sets it at create.
  • SandboxModificationBuilder::nested_virt(bool) changes it for an existing sandbox. It is fixed at boot, so it is restart-backed on a running sandbox, like max_cpus. Enabling it checks the host and the runtime before anything is stopped or saved; disabling it always works, so a sandbox on a host that lost support can still be started without it.
  • microsandbox_runtime::nested_virt::host_support() probes the host. The SDK re-exports it as microsandbox::runtime::nested_virt_host_support(), so callers can check before asking:
    • macOS on Apple silicon: checks for EL2 support in Hypervisor.framework (M3 or newer, macOS 15 or newer). This works without the hypervisor entitlement.
    • Linux x86_64: checks that the kvm_intel/kvm_amd module has nested enabled, and says so separately when the module isn't loaded.
    • Every other host is reported as unsupported.
  • The runtime advertises nested_virt in msb __launch-protocol, next to the other launch capabilities.
  • The SDK checks the host and the runtime's capability in validate_runtime_config, before replace deletes an existing sandbox, in modify() when enabling it, and again at every start. Failures are MicrosandboxError::Unsupported, labelled with the refused operation: create, modify or start.
  • The runtime checks the host again before building the VM, then passes nested_virt to the VMM:
    • On macOS the guest boots at EL2.
    • On Linux x86_64, msb_krun 0.1.41 passes VMX or SVM through. It masks them otherwise, so without the setting guests have no /dev/kvm on either platform.
  • The msb binary changes in two lines: the capability, and the pass-through in machine_cmd.rs, its hidden VM-process entry point.
  • The docs cover the security trade-off: with the setting on, code in the sandbox can reach the host hypervisor's nested virtualization support.

Refusals

  • Older runtimes: a v0.7 runtime that doesn't advertise the capability is refused before replace or modify. If one gets an enabled launch from a current SDK anyway, it rejects it, because launch JSON uses deny_unknown_fields. The previous launch generation (v0.6 runtimes) and the cloud backend refuse it explicitly. An older runtime or CLI that builds its own launch from a stored record ignores the field and starts the sandbox without nesting; the field stays in the record. COMPATIBILITY.md records this.
  • Execution state: forks and full restores of a nested sandbox are refused in the SDK before replace. A full snapshot of a running nested guest is refused by its runtime, and comes back as MicrosandboxError::ControlClient, not Unsupported. The runtime goes by the setting the running VM booted with; the SDK goes by the stored setting. The VMM doesn't capture the state of the guest's own VMs, and the HVF backend also can't capture EL2 state on M-series chips correctly.
  • Disk-only snapshots: still work. The refusal comes first in capture_to, before the sandbox is paused. A sandbox restored from a disk-only snapshot starts without nesting, which the docs note.

Tests and CI

  • Unit tests cover:
    • record and launch encoding;
    • the builder setting, in its own test;
    • planning and applying modify().nested_virt(): next start when stopped or with next_start, restart-required when running, and a refused enable that leaves the record unchanged while disabling still works;
    • the capability probe, and the operation each refusal is labelled with;
    • the refusal on the previous launch generation;
    • the refusal before replace;
    • the runtime's restore refusal and the coordinator's capture refusal;
    • parsing of the KVM nested values.
  • The integration tests in sdk/rust/tests/nested_virt.rs run in the KVM lane:
    • a default sandbox has no /dev/kvm;
    • a nested one has it, and a full snapshot is refused while it keeps running;
    • modify().nested_virt(true).restart() gives an existing sandbox /dev/kvm, and false takes it away again;
    • where the host lacks nested KVM, they check only the refusals and say so.
  • The KVM integration lane prints the runner's KVM nested parameter. When it is Y or 1, the lane sets MSB_TEST_EXPECT_NESTED_VIRT=1, and the tests then fail instead of checking only the refusal, so a wrong host check can't leave CI green.

Testing

The platform sections below tested d8901a4, the first version of this commit. f6fe87d adds modify().nested_virt(), the re-exported host check, the operation labels on refusals, the CI expectation and the docs. The Linux section covers those on AMD; the TODO list after it covers what to rerun elsewhere.

Checks (done)

Locally on f6fe87d, with toolchain 1.99.0, the checks from check-digdir.yml pass:

  • fmt for the workspace and agentd, and clippy -D warnings;
  • api_compat and the feature-surface checks;
  • cargo doc -D warnings;
  • cargo test --workspace --exclude microsandbox-agentd: 4,557 passed.

Check Digdir run 39872735407 on f6fe87d:

  • Passed: all 18 checks, including Rust quality, the unit tests on every platform and the Windows jobs.
  • The integration lane passed on its second attempt, with all 84 tests and the fixtures. The first attempt failed at the "Enable KVM" step, because the hosted runner it got had no /dev/kvm.
  • The runner had kvm_amd with nested: 1, so the lane set MSB_TEST_EXPECT_NESTED_VIRT=1, and both nested tests took the full path, including modify().

Check Digdir run 39856855099 on d8901a4:

  • Passed:
    • Rust quality;
    • the unit tests on linux-x86_64, linux-aarch64 and darwin-aarch64;
    • the Windows jobs;
    • the integration lane, with all 83 tests and the fixtures.
  • The integration runner had kvm_amd with nested: 1, so nested_virt_exposes_kvm_only_when_enabled took the full path: a default sandbox has no /dev/kvm, a nested one has it, and a full snapshot is refused while it keeps running.
  • The CLI smoke job passed on its third attempt. The first two failed at the "Enable KVM" step, because the hosted runners they got had no /dev/kvm.

macOS (done)

On an Apple M6 Mac mini, macOS 27.0.1, with a release build of this branch and the v0.7.8-digdir.1 agentd and libkrunfw, through the Rust SDK:

  • Without the setting: no /dev/kvm.
  • With .nested_virt(true):
    • the guest's KVM starts in nested mode, and the guest has /dev/kvm;
    • the v0.7.8-digdir.1 Linux aarch64 msb inside it boots an inner sandbox;
    • a full snapshot is refused with execution-state capture is not supported with nested virtualization; use a disk-only snapshot, and the sandbox keeps running;
    • a fork is refused before the source is touched;
    • after a stop and start the setting is still on.
  • With the released v0.7.8-digdir.1 msb, which doesn't advertise the capability: a replace() create is refused, and the sandbox it would have replaced keeps running.
  • The new integration test passes.
  • Rough overhead at the inner level, measured earlier on 0.7.4 with the same HVF code: boot ~1.4 s vs ~0.28 s, CPU loop ~1.1×, process spawn ~4×, fsync'd 256 MB write ~13×.
  • Not covered: the refusal on an unsupported Mac (M1/M2, or macOS older than 15), which only the unit test covers.

Linux (done on AMD; Intel and aarch64 not covered)

On an AMD Ryzen 9 9950X3D host (kvm_amd with nested: 1), CachyOS with kernel 7.2.10, with a ci-profile build of d8901a4 (embed-binaries,net,ssh) and the v0.7.8-digdir.1 agentd and libkrunfw 5.6.1, through the Rust SDK, in an isolated MSB_HOME:

  • With nested KVM enabled:
    • a default sandbox has no /dev/kvm;
    • a .nested_virt(true) sandbox (debian:bookworm-slim, 2 vCPUs, 2 GiB) has /dev/kvm. The guest sees svm, and its kvm_amd reports nested: 1. The v0.7.8-digdir.1 Linux x86_64 msb, bind-mounted into it, boots an inner alpine sandbox in ~3 s. The inner sandbox has no /dev/kvm, as expected without the setting.
    • full snapshot and fork are refused, and the sandbox keeps running. The fork leaves no child. The deprecated branch() is refused the same way. The messages are:
      • full snapshot: control client error: control operation rejected by peer: execution-state capture is not supported with nested virtualization; use a disk-only snapshot
      • fork: Sandbox::create is not supported by this backend: full snapshot restores and forks are not supported with nested virtualization
    • the setting survives a stop and start, through both the SDK and this branch's msb start.
    • A live disk-only snapshot works, and a sandbox restored from it has no /dev/kvm, as the docs say.
  • With nested KVM disabled, or the module unloaded: .nested_virt(true).replace() is refused with the matching reason, and the existing sandbox keeps running. Starting the stopped nested sandbox is refused the same way through the SDK and through msb start. The messages are:
    • disabled: Sandbox::start is not supported by this backend: nested virtualization is disabled in the host's kvm_intel or kvm_amd module
    • not loaded: Sandbox::start is not supported by this backend: nested virtualization needs the kvm_intel or kvm_amd module, which is not loaded
    • These were simulated, not set up for real. The host had other VMs running, so instead of reloading kvm_amd the SDK ran in an unprivileged user and mount namespace, with 0 bind-mounted over /sys/module/kvm_amd/parameters/nested, or a tmpfs over /sys/module/kvm_amd.
  • cargo nextest run -p microsandbox --test nested_virt --run-ignored=only passes, and takes the full path. MSB_HOME has to be short, because a long scratch path fails on the 108-byte Unix socket limit.
  • An aarch64 Linux host refuses with "not supported on this host". This wasn't run because no aarch64 Linux host was available. The CI unit tests on linux-aarch64 cover the probe.
  • With the released v0.7.8-digdir.1 msb as the runtime, .nested_virt(true).replace() is refused with selected msb runtime does not support resources.nested_virt; upgrade msb in the configured home or select a newer runtime explicitly, and the existing sandbox keeps running.
  • With the released v0.7.8-digdir.1 msb CLI and runtime, msb start on a sandbox created with .nested_virt(true) boots it without nesting and without an error. The record still has the setting, and this branch's SDK brings /dev/kvm back on the next start. The deny_unknown_fields refusal only applies to launches that a newer SDK builds.
  • On f6fe87d, with a ci-profile build of it:
    • both integration tests pass and take the full path: modify().nested_virt(true).restart() gives an existing sandbox /dev/kvm, and false takes it away again;
    • with nested faked off as above, both take their refusal paths: modify().nested_virt(true) is refused with Sandbox::modify is not supported by this backend: nested virtualization is disabled in the host's kvm_intel or kvm_amd module, and the sandbox keeps running;
    • with nested faked off and MSB_TEST_EXPECT_NESTED_VIRT=1, both fail, as the CI expectation intends;
    • the modify() unit tests pass with nesting available and faked off, covering a refused enable that leaves the record unchanged and a disable that still applies.
  • Not covered:
    • Intel (kvm_intel): no Intel host was available, so, like CI, this ran on AMD only.
    • The runtime's own host check before it builds the VM: with a wrapper msb that sees nested: 0, the SDK refuses the runtime as no tested sandbox launch contract, so that check couldn't be isolated from the SDK's.

Windows (done on Intel; Arm not covered)

On an Intel Core i7-11800H laptop, Windows 11 Pro 10.0.26300 with Windows Hypervisor Platform, with a ci-profile build of d8901a4 (embed-binaries,net,ssh, x86_64-pc-windows-msvc, toolchain 1.99.0) and the v0.7.8-digdir.1 agentd and Windows libkrunfw, through the Rust SDK, in an isolated MSB_HOME:

  • .nested_virt(true).replace() over a running sandbox is refused with Sandbox::start is not supported by this backend: nested virtualization is not supported on this host. The existing sandbox is still running, and a file written in it before the attempt is still there. With the released v0.7.8-digdir.1 msb as the runtime, the host check comes first, so the message is the same.
  • Sandboxes without the setting are unaffected: an alpine sandbox boots through both the SDK and msb run, and has no /dev/kvm and no vmx flag in /proc/cpuinfo.
  • The integration test passes through its refusal path, and prints the reason.
  • The nested unit tests that build on Windows pass: 4 in microsandbox-runtime, 2 in microsandbox and 1 in microsandbox-types. nested_virt_requires_an_explicit_runtime_capability is cfg(unix) and doesn't run there.
  • The refusal is right for this VMM. msb_krun 0.1.41's WHP backend masks VMX and SVM in CPUID while the setting is off, but it never enables nested virtualization on the WHP partition, so a guest could not use VMX even with the setting on.
  • Not covered: Windows on Arm, and AMD hosts.

Windows WSL distro (done on Intel)

On the same laptop, in an Arch Linux WSL 2 distro (kernel 6.18.33.2-microsoft-standard-WSL2, no .wslconfig), with a ci-profile build of d8901a4 (embed-binaries,net,ssh) and the v0.7.8-digdir.1 agentd and libkrunfw 5.6.1, through the Rust SDK, in a short isolated MSB_HOME. This is also the Intel (kvm_intel) coverage that the Linux section lacks.

  • WSL exposes nested virtualization by default: there is no nestedVirtualization override, the distro has /dev/kvm, and kvm_intel reports nested: Y.
  • The Linux checks pass:
    • a default sandbox has no /dev/kvm and no vmx flag;
    • a .nested_virt(true) sandbox (debian:bookworm-slim, 2 vCPUs, 2 GiB) has /dev/kvm. The guest sees vmx, and its kvm_intel reports nested: Y. The v0.7.8-digdir.1 Linux x86_64 msb, bind-mounted into it, boots an inner alpine sandbox: ~9.9 s for the first msb run, including the image pull, and ~4.3 s for a second. That is the third level under Hyper-V. The inner sandbox has no /dev/kvm, as expected without the setting.
      • The guest needs ca-certificates installed before the inner msb can pull. Pulling from mirror.gcr.io inside the guest failed with a 404 on the blob redirect, but Docker Hub worked.
    • cargo nextest run -p microsandbox --test nested_virt --run-ignored=only passes and takes the full path, including the full snapshot refusal.
    • a fork is refused with full snapshot restores and forks are not supported with nested virtualization. It leaves no child, and the sandbox keeps running.
    • the setting survives a stop and start through the SDK.
  • Disabled and unloaded nesting were set up for real, by reloading kvm_intel in the distro, not simulated:
    • nested=0: .nested_virt(true).replace() is refused with nested virtualization is disabled in the host's kvm_intel or kvm_amd module, and the existing sandbox keeps running. Starting a stopped nested sandbox is refused with the same reason, through the SDK and through this branch's msb start. The sandbox stays stopped. msb start prints the error but exits 0; the released v0.7.8-digdir.1 msb also exits 0 on errors, so that isn't from this branch.
    • module unloaded: starting the stopped nested sandbox is refused with nested virtualization needs the kvm_intel or kvm_amd module, which is not loaded. The integration test fails here, because without KVM its default sandbox cannot boot.
  • With the released v0.7.8-digdir.1 msb as the runtime, .nested_virt(true).replace() is refused with selected msb runtime does not support resources.nested_virt; upgrade msb in the configured home or select a newer runtime explicitly. The existing sandbox's record and disk are kept. It was stopped after the run, because the scratch test panicked on that path, so this run doesn't show whether it kept running.
  • Not covered: disk-only snapshots and the deprecated branch(), which the AMD run covers.

Retest on f6fe87d (done except recording macOS)

Run cargo nextest run -p microsandbox --test nested_virt --run-ignored=only, which now has two tests.

  • macOS, M3 or newer: both tests take the full path; modify().nested_virt(true).restart() adds /dev/kvm and false removes it; the guest's dmesg shows kvm [1]: Hyp nVHE mode initialized successfully, which the docs now quote from the Linux 6.12 source.
    • On the Apple M6 Mac mini, macOS 27.0.1, with a release build of f6fe87d and the v0.7.8-digdir.1 agentd and libkrunfw. The tests ran with cargo test -p microsandbox --test nested_virt -- --ignored --test-threads=1 rather than nextest, with MSB_TEST_EXPECT_NESTED_VIRT=1, a scratch MSB_HOME and MSB_PATH set to the branch runtime. Isolated-home mode always takes ~/.microsandbox/bin/msb, which here is the installed release.
    • The same toggle through the Rust SDK: /dev/kvm is absent by default, present after enabling, and absent after disabling, and each plan reports applied. After enabling, the guest logged [ 0.028640] kvm [1]: Hyp nVHE mode initialized successfully.
  • WSL on Intel (kvm_intel), with ci-profile builds of f6fe87d in the setup above: both tests take the full path, with MSB_TEST_EXPECT_NESTED_VIRT=1 set. With kvm_intel reloaded with nested=0:
    • both tests take the refusal path, and with MSB_TEST_EXPECT_NESTED_VIRT=1 set both fail with the host has nested KVM, but the host check refused it, so the CI guard works;
    • modify().next_start().nested_virt(true) on a running sandbox is refused with Sandbox::modify is not supported by this backend: nested virtualization is disabled in the host's kvm_intel or kvm_amd module, and the sandbox keeps running;
    • a stopped sandbox with the setting can't start, modify().next_start().nested_virt(false) applies, and it then starts without /dev/kvm.
    • The 11 nested virtualization unit tests pass with nested=Y and with nested=0, so applying_nested_virt_checks_the_host_before_persisting ran both of its paths.
  • agentctl's upgrade path, in WSL with nested=Y: a sandbox created by the 0.7.4-digdir.3 msb that agentctl pins has /dev/kvm. Started with this branch's SDK and runtime in the same home, it has none. modify().next_start().nested_virt(true) plans one NextStart change and leaves the running sandbox alone. After a stop and start it has /dev/kvm.
  • Windows on Intel, with a ci-profile build of f6fe87d in the setup above: both tests take the refusal path with nested virtualization is not supported on this host. modify().nested_virt(true).restart() is refused as Unsupported with Operation::SandboxModify, and the sandbox keeps running without /dev/kvm. The 10 nested virtualization unit tests that build on Windows pass; nested_virt_requires_an_explicit_runtime_capability is cfg(unix).
  • Optional, unit-tested only: a refusal on an M1/M2 Mac or macOS older than 15, and on Linux aarch64.

🤖 Generated with Claude Code

@martinothamar

martinothamar commented Oct 22, 2026 •

Copy link
Copy Markdown
Author

Review: macOS (Apple M6, macOS 27.0.1)

An independent review of d8901a4, done on macOS. The macOS results are in the PR description. Linux, Windows and a Windows WSL distro will be reviewed and tested separately, using the TODO lists there.

The review found no blockers. The refusals run before replace and before any pause, and nothing captures or restores the execution state of a nested guest.

Context: this fork only serves agentctl

The findings are triaged for how agentctl uses the SDK:

  • it uses SingleTenant only;
  • it doesn't use snapshots, forks, pause or replace();
  • it changes resources of existing Sandboxes with modify().next_start();
  • the plan is to enable nested virtualization wherever the host supports it.

To fix in this PR

  1. The CI test can pass without testing nesting (sdk/rust/tests/nested_virt.rs, check-digdir.yml). The test takes its refusal path on any error from host_support(). If the check ever wrongly reported "unsupported" on a capable runner, nesting would quietly be off everywhere and CI would stay green. Fix: when the runner reports KVM nested as Y or 1, the CI step sets an environment variable, and the test then fails if it takes the refusal path.
  2. The Rust SDK reference has no entry for the new method (docs/sdk/rust/sandbox.mdx). Every builder method has one, including thp() and guest_clock(). Add nested_virt() beside them.
  3. Docs wording (docs/sandboxes/bootstrap.mdx):
    • Replace "The CLI and the TypeScript, Python, and Go SDKs do not expose the setting", which leaves out Ruby, with "Only the Rust SDK exposes it".
    • Make the nVHE sentence describe what the guest's log shows.
  4. Separate test for the builder (sdk/rust/lib/sandbox/builder.rs). The nested_virt assertion sits in upstream's shared builder test, which often conflicts when upstream adds fields. A small test of its own avoids that.
  5. PR body: it should say msb_krun 0.1.41, not libkrun 0.1.41.

For agentctl enabling it on every supported host

  1. The host check isn't reachable from the SDK. It lives in microsandbox-runtime, which digdir-agents doesn't depend on. Re-export it from the SDK so agentctl can call .nested_virt(supported), instead of trying true and falling back on an Unsupported error.
  2. The cost to the outer sandbox on macOS hasn't been measured. The M6 figures compare the outer sandbox with the inner one, both with nesting on. With nesting always on, the comparison that matters is the outer sandbox with it on versus off: boot, CPU, syscalls, process spawn, I/O. On macOS the guest runs at EL2 under Hypervisor.framework; on Linux x86_64, exposing VMX/SVM should cost nothing.
  3. Existing Sandboxes stay without nesting. modify() only changes CPUs and memory, so Sandboxes created before this change get nesting only when recreated. An open decision is whether modify().nested_virt(...) should switch it on at the next start.
  4. Security trade-off: every Agent sandbox gets the hypervisor's nested-virtualization emulation to attack, EL2 emulation in Hypervisor.framework on macOS and nested VMX/SVM on Linux. This is a deliberate choice.

Verified

  • Serialization: the setting is left out while off and defaults to off when missing, in stored records and in the launch payload. The runtime's capability flag defaults to false too.
  • No other way to set it: global and CLI config can't set it, modify() can't change it, and restore doesn't copy the source's resources.
  • Order before replace: the host check, the restore and fork refusal and the capability check all run before the replace target is touched, and again at every start.
  • Capture:
    • every full capture over the control socket goes through capture_to, which refuses before any freeze or pause;
    • the sandbox keeps running after a refusal;
    • disk-only capture is a separate path and is unaffected.
  • msb_krun 0.1.41:
    • Hypervisor.framework enables EL2 through hv_vm_config_set_el2_enabled, and multi-vCPU nested guests work;
    • Linux x86_64 masks VMX/SVM unless nesting is enabled;
    • Windows and Linux aarch64 have no nesting, so refusing there is correct.
  • macOS host check: the function signature and return type match the framework, null handles and symbols are checked, dlclose comes after the last call, and the framework path matches msb_krun_hvf. It only compiles on Apple silicon.
  • Older runtimes and cloud:
    • the v0.6 launch generation refuses the setting;
    • v0.7 runtimes without the capability are refused before replace;
    • the cloud backend refuses it on its only create path.
  • Builds: the code compiles on every target, and the SDK's local feature keeps building.
  • Guest kernel: both libkrunfw configs have CONFIG_KVM, so libkrunfw needs no change.
  • Fork rules: one commit with its docs, no upstream references (the grep is clean), and no AI trailers.

@martinothamar

martinothamar commented Oct 22, 2026 •

Copy link
Copy Markdown
Author

Review notes from Linux testing

The test results are in the Linux section of the description (AMD host, kvm_amd with nested: 1). Nested guests, the refusals and the integration test all work there. The notes below come from that testing and from reading the diff. They are ranked for agentctl, the fork's only consumer. agentctl pins its runtime and keeps sandboxes long-lived: it restarts them with start_detached() and changes them through modify().next_start(). It doesn't use snapshots, forks or restores.

  1. nested_virt can't be changed on an existing sandbox, and agentctl sandboxes have nested virtualization today. On Linux x86_64 nesting has been on for every sandbox until now, and agentctl's self-dev agents depend on it. I checked a default sandbox on this host: with the 0.7.4-digdir.3 runtime agentctl pins, it has /dev/kvm and sees svm. With the released 0.7.8-digdir.1 runtime it has neither, because msb_krun 0.1.41 masks VMX/SVM unless the setting is on. So the loss arrives with 0.7.8-digdir.1, and this PR is how agentctl gets nesting back. But only SandboxBuilder has the setting; SandboxModificationBuilder doesn't. agentctl changes existing sandboxes through modify(), so every sandbox created before the upgrade would come back without /dev/kvm and stay that way until it is recreated. An agent definition that later turns nesting on or off would hit the same wall. Suggest adding SandboxModificationBuilder::nested_virt(bool), applied at the next start, before agentctl moves past 0.7.4.
  2. There is no host probe in the microsandbox crate. host_support() lives only in microsandbox_runtime, which agentctl doesn't depend on. agentctl can only find out by trying a create and matching the error. Suggest re-exporting it from microsandbox. Then agentctl can reject an agent definition that asks for nesting on a host without it, with the host's reason, before creating anything.
  3. Security trade-off. With the setting on, the guest can reach the host kernel's nested SVM/VMX code, which has had guest-to-host vulnerabilities. Every agentctl sandbox on Linux x86_64 has that exposure today. So instead of keeping it on for every agent, we're considering exposing it in the agent manifest, so that the people writing an agent definition decide. Agents that need it, like self-dev, would then declare it. One sentence on the trade-off in docs/sandboxes/bootstrap.mdx and the builder doc would help them.

Minor:

  • The refusals read Sandbox::start is not supported by this backend: … and Sandbox::create is not supported by this backend: …, which suggests the local backend lacks the feature. The host refusal during create is labelled SandboxStart. The full-snapshot refusal comes back as control client error: control operation rejected by peer: … instead of Unsupported, unlike the others.
  • msb inspect and msb ps show THP but not this setting.
  • The SDK's restore check (launch_contract.rs:579) treats any checkpoint_restore as a full restore, so it also matches disk-only restores. It can't trigger today, because RestoreBuilder has no nested_virt().
  • The released v0.7.8-digdir.1 msb start boots a sandbox created with the setting without nesting and without an error. The description says older runtimes reject it, but that only holds for launches a newer SDK builds. agentctl isn't affected, because it pins its runtime.

Not verified on Linux: Intel hosts, aarch64 hosts, and the runtime's own host check on its own, separately from the SDK's.

Let a sandbox expose hardware virtualization to its guest, so the guest has
/dev/kvm and can run its own VMs, including microsandbox.

`SandboxBuilder::nested_virt(true)` persists the setting in the sandbox's
resources, and `SandboxModificationBuilder::nested_virt` changes it for an
existing sandbox at its next boot. It is off by default and omitted from
serialized records and launch payloads while off. With it, the VMM boots the
guest at EL2 on macOS and passes VMX or SVM through on Linux x86_64; without
it, guests have no /dev/kvm.

Creating, starting, or enabling it through modify fails before an existing
sandbox is replaced or changed when the host cannot honor the setting or the
runtime does not advertise it; disabling it always works. macOS needs EL2
support in Hypervisor.framework (an Apple M3 or newer chip and macOS 15 or
newer), and Linux x86_64 needs nested KVM. `runtime::nested_virt_host_support`
lets callers check the host first. The previous launch generation and the
cloud backend refuse it. Full snapshots, forks and full restores are refused
while it is enabled, because the VMM does not capture the state of the guest's
own VMs; disk-only snapshots still work, but restore without nesting.

The KVM integration lane reports whether the runner has nested KVM and, when
it does, makes the integration tests fail instead of checking only the refusal.
@martinothamar
martinothamar merged commit 5607d63 into main-digdir Oct 23, 2026
40 of 50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant