Repository navigation
feat(sandbox): add opt-in nested virtualization - #1
Conversation
2a998fd to
6330b54
Compare
37a477d to
45315a8
Compare
6330b54 to
d8901a4
Compare
Review: macOS (Apple M6, macOS 27.0.1)An independent review of d8901a4, done on macOS. The macOS results are in the PR description. Linux, Windows and a Windows WSL distro will be reviewed and tested separately, using the TODO lists there. The review found no blockers. The refusals run before Context: this fork only serves agentctlThe findings are triaged for how agentctl uses the SDK:
To fix in this PR
For agentctl enabling it on every supported host
Verified
|
Review notes from Linux testingThe test results are in the Linux section of the description (AMD host,
Minor:
Not verified on Linux: Intel hosts, aarch64 hosts, and the runtime's own host check on its own, separately from the SDK's. |
Let a sandbox expose hardware virtualization to its guest, so the guest has /dev/kvm and can run its own VMs, including microsandbox. `SandboxBuilder::nested_virt(true)` persists the setting in the sandbox's resources, and `SandboxModificationBuilder::nested_virt` changes it for an existing sandbox at its next boot. It is off by default and omitted from serialized records and launch payloads while off. With it, the VMM boots the guest at EL2 on macOS and passes VMX or SVM through on Linux x86_64; without it, guests have no /dev/kvm. Creating, starting, or enabling it through modify fails before an existing sandbox is replaced or changed when the host cannot honor the setting or the runtime does not advertise it; disabling it always works. macOS needs EL2 support in Hypervisor.framework (an Apple M3 or newer chip and macOS 15 or newer), and Linux x86_64 needs nested KVM. `runtime::nested_virt_host_support` lets callers check the host first. The previous launch generation and the cloud backend refuse it. Full snapshots, forks and full restores are refused while it is enabled, because the VMM does not capture the state of the guest's own VMs; disk-only snapshots still work, but restore without nesting. The KVM integration lane reports whether the runner has nested KVM and, when it does, makes the integration tests fail instead of checking only the refusal.
d8901a4 to
f6fe87d
Compare
Adds a per-sandbox opt-in to the Rust SDK that exposes hardware virtualization to the guest, so it has
/dev/kvmand can run its own VMs. This is the microsandbox part of digdir/digdir-agents#173, which tracks nested Sandboxes for Agents. digdir-agents only uses the Rust SDK, so the CLI and the other language SDKs are left unchanged.One commit on
main-digdirat v0.7.8-digdir.1 (45315a8), to ship in0.7.8-digdir.2. It lands after the version bump with a rebase merge.agentctl will expose the setting in the agent manifest, so whoever writes an agent definition opts in. Until v0.7.8-digdir.1, every sandbox on Linux x86_64 got nesting implicitly; msb_krun 0.1.41 masks VMX/SVM unless the setting is on. agentctl needs
modify()to keep nesting for its existing sandboxes, and the host check to validate a manifest that asks for it.What changes
SandboxResources.nested_virtis stored with the sandbox. It is off by default and left out of serialized records and launch payloads while off, so existing records and launches don't change.SandboxBuilder::nested_virt(bool)sets it at create.SandboxModificationBuilder::nested_virt(bool)changes it for an existing sandbox. It is fixed at boot, so it is restart-backed on a running sandbox, likemax_cpus. Enabling it checks the host and the runtime before anything is stopped or saved; disabling it always works, so a sandbox on a host that lost support can still be started without it.microsandbox_runtime::nested_virt::host_support()probes the host. The SDK re-exports it asmicrosandbox::runtime::nested_virt_host_support(), so callers can check before asking:kvm_intel/kvm_amdmodule hasnestedenabled, and says so separately when the module isn't loaded.nested_virtinmsb __launch-protocol, next to the other launch capabilities.validate_runtime_config, beforereplacedeletes an existing sandbox, inmodify()when enabling it, and again at every start. Failures areMicrosandboxError::Unsupported, labelled with the refused operation: create, modify or start.nested_virtto the VMM:/dev/kvmon either platform.msbbinary changes in two lines: the capability, and the pass-through inmachine_cmd.rs, its hidden VM-process entry point.Refusals
deny_unknown_fields. The previous launch generation (v0.6 runtimes) and the cloud backend refuse it explicitly. An older runtime or CLI that builds its own launch from a stored record ignores the field and starts the sandbox without nesting; the field stays in the record.COMPATIBILITY.mdrecords this.MicrosandboxError::ControlClient, notUnsupported. The runtime goes by the setting the running VM booted with; the SDK goes by the stored setting. The VMM doesn't capture the state of the guest's own VMs, and the HVF backend also can't capture EL2 state on M-series chips correctly.capture_to, before the sandbox is paused. A sandbox restored from a disk-only snapshot starts without nesting, which the docs note.Tests and CI
modify().nested_virt(): next start when stopped or withnext_start, restart-required when running, and a refused enable that leaves the record unchanged while disabling still works;nestedvalues.sdk/rust/tests/nested_virt.rsrun in the KVM lane:/dev/kvm;modify().nested_virt(true).restart()gives an existing sandbox/dev/kvm, andfalsetakes it away again;nestedparameter. When it isYor1, the lane setsMSB_TEST_EXPECT_NESTED_VIRT=1, and the tests then fail instead of checking only the refusal, so a wrong host check can't leave CI green.Testing
The platform sections below tested d8901a4, the first version of this commit. f6fe87d adds
modify().nested_virt(), the re-exported host check, the operation labels on refusals, the CI expectation and the docs. The Linux section covers those on AMD; the TODO list after it covers what to rerun elsewhere.Checks (done)
Locally on f6fe87d, with toolchain 1.99.0, the checks from
check-digdir.ymlpass:-D warnings;api_compatand the feature-surface checks;cargo doc -D warnings;cargo test --workspace --exclude microsandbox-agentd: 4,557 passed.Check Digdir run 39872735407 on f6fe87d:
/dev/kvm.kvm_amdwithnested: 1, so the lane setMSB_TEST_EXPECT_NESTED_VIRT=1, and both nested tests took the full path, includingmodify().Check Digdir run 39856855099 on d8901a4:
kvm_amdwithnested: 1, sonested_virt_exposes_kvm_only_when_enabledtook the full path: a default sandbox has no/dev/kvm, a nested one has it, and a full snapshot is refused while it keeps running./dev/kvm.macOS (done)
On an Apple M6 Mac mini, macOS 27.0.1, with a release build of this branch and the v0.7.8-digdir.1
agentdand libkrunfw, through the Rust SDK:/dev/kvm..nested_virt(true):/dev/kvm;msbinside it boots an inner sandbox;execution-state capture is not supported with nested virtualization; use a disk-only snapshot, and the sandbox keeps running;msb, which doesn't advertise the capability: areplace()create is refused, and the sandbox it would have replaced keeps running.Linux (done on AMD; Intel and aarch64 not covered)
On an AMD Ryzen 9 9950X3D host (
kvm_amdwithnested: 1), CachyOS with kernel 7.2.10, with aci-profile build of d8901a4 (embed-binaries,net,ssh) and the v0.7.8-digdir.1agentdand libkrunfw 5.6.1, through the Rust SDK, in an isolatedMSB_HOME:/dev/kvm;.nested_virt(true)sandbox (debian:bookworm-slim, 2 vCPUs, 2 GiB) has/dev/kvm. The guest seessvm, and itskvm_amdreportsnested: 1. The v0.7.8-digdir.1 Linux x86_64msb, bind-mounted into it, boots an inner alpine sandbox in ~3 s. The inner sandbox has no/dev/kvm, as expected without the setting.branch()is refused the same way. The messages are:control client error: control operation rejected by peer: execution-state capture is not supported with nested virtualization; use a disk-only snapshotSandbox::create is not supported by this backend: full snapshot restores and forks are not supported with nested virtualizationmsb start./dev/kvm, as the docs say..nested_virt(true).replace()is refused with the matching reason, and the existing sandbox keeps running. Starting the stopped nested sandbox is refused the same way through the SDK and throughmsb start. The messages are:Sandbox::start is not supported by this backend: nested virtualization is disabled in the host's kvm_intel or kvm_amd moduleSandbox::start is not supported by this backend: nested virtualization needs the kvm_intel or kvm_amd module, which is not loadedkvm_amdthe SDK ran in an unprivileged user and mount namespace, with0bind-mounted over/sys/module/kvm_amd/parameters/nested, or a tmpfs over/sys/module/kvm_amd.cargo nextest run -p microsandbox --test nested_virt --run-ignored=onlypasses, and takes the full path.MSB_HOMEhas to be short, because a long scratch path fails on the 108-byte Unix socket limit.msbas the runtime,.nested_virt(true).replace()is refused withselected msb runtime does not support resources.nested_virt; upgrade msb in the configured home or select a newer runtime explicitly, and the existing sandbox keeps running.msbCLI and runtime,msb starton a sandbox created with.nested_virt(true)boots it without nesting and without an error. The record still has the setting, and this branch's SDK brings/dev/kvmback on the next start. Thedeny_unknown_fieldsrefusal only applies to launches that a newer SDK builds.ci-profile build of it:modify().nested_virt(true).restart()gives an existing sandbox/dev/kvm, andfalsetakes it away again;nestedfaked off as above, both take their refusal paths:modify().nested_virt(true)is refused withSandbox::modify is not supported by this backend: nested virtualization is disabled in the host's kvm_intel or kvm_amd module, and the sandbox keeps running;nestedfaked off andMSB_TEST_EXPECT_NESTED_VIRT=1, both fail, as the CI expectation intends;modify()unit tests pass with nesting available and faked off, covering a refused enable that leaves the record unchanged and a disable that still applies.kvm_intel): no Intel host was available, so, like CI, this ran on AMD only.msbthat seesnested: 0, the SDK refuses the runtime asno tested sandbox launch contract, so that check couldn't be isolated from the SDK's.Windows (done on Intel; Arm not covered)
On an Intel Core i7-11800H laptop, Windows 11 Pro 10.0.26300 with Windows Hypervisor Platform, with a
ci-profile build of d8901a4 (embed-binaries,net,ssh,x86_64-pc-windows-msvc, toolchain 1.99.0) and the v0.7.8-digdir.1agentdand Windows libkrunfw, through the Rust SDK, in an isolatedMSB_HOME:.nested_virt(true).replace()over a running sandbox is refused withSandbox::start is not supported by this backend: nested virtualization is not supported on this host. The existing sandbox is still running, and a file written in it before the attempt is still there. With the released v0.7.8-digdir.1msbas the runtime, the host check comes first, so the message is the same.msb run, and has no/dev/kvmand novmxflag in/proc/cpuinfo.microsandbox-runtime, 2 inmicrosandboxand 1 inmicrosandbox-types.nested_virt_requires_an_explicit_runtime_capabilityiscfg(unix)and doesn't run there.Windows WSL distro (done on Intel)
On the same laptop, in an Arch Linux WSL 2 distro (kernel 6.18.33.2-microsoft-standard-WSL2, no
.wslconfig), with aci-profile build of d8901a4 (embed-binaries,net,ssh) and the v0.7.8-digdir.1agentdand libkrunfw 5.6.1, through the Rust SDK, in a short isolatedMSB_HOME. This is also the Intel (kvm_intel) coverage that the Linux section lacks.nestedVirtualizationoverride, the distro has/dev/kvm, andkvm_intelreportsnested: Y./dev/kvmand novmxflag;.nested_virt(true)sandbox (debian:bookworm-slim, 2 vCPUs, 2 GiB) has/dev/kvm. The guest seesvmx, and itskvm_intelreportsnested: Y. The v0.7.8-digdir.1 Linux x86_64msb, bind-mounted into it, boots an inner alpine sandbox: ~9.9 s for the firstmsb run, including the image pull, and ~4.3 s for a second. That is the third level under Hyper-V. The inner sandbox has no/dev/kvm, as expected without the setting.ca-certificatesinstalled before the innermsbcan pull. Pulling frommirror.gcr.ioinside the guest failed with a 404 on the blob redirect, but Docker Hub worked.cargo nextest run -p microsandbox --test nested_virt --run-ignored=onlypasses and takes the full path, including the full snapshot refusal.full snapshot restores and forks are not supported with nested virtualization. It leaves no child, and the sandbox keeps running.kvm_intelin the distro, not simulated:nested=0:.nested_virt(true).replace()is refused withnested virtualization is disabled in the host's kvm_intel or kvm_amd module, and the existing sandbox keeps running. Starting a stopped nested sandbox is refused with the same reason, through the SDK and through this branch'smsb start. The sandbox stays stopped.msb startprints the error but exits 0; the released v0.7.8-digdir.1msbalso exits 0 on errors, so that isn't from this branch.nested virtualization needs the kvm_intel or kvm_amd module, which is not loaded. The integration test fails here, because without KVM its default sandbox cannot boot.msbas the runtime,.nested_virt(true).replace()is refused withselected msb runtime does not support resources.nested_virt; upgrade msb in the configured home or select a newer runtime explicitly. The existing sandbox's record and disk are kept. It was stopped after the run, because the scratch test panicked on that path, so this run doesn't show whether it kept running.branch(), which the AMD run covers.Retest on f6fe87d (done except recording macOS)
Run
cargo nextest run -p microsandbox --test nested_virt --run-ignored=only, which now has two tests.modify().nested_virt(true).restart()adds/dev/kvmandfalseremoves it; the guest'sdmesgshowskvm [1]: Hyp nVHE mode initialized successfully, which the docs now quote from the Linux 6.12 source.agentdand libkrunfw. The tests ran withcargo test -p microsandbox --test nested_virt -- --ignored --test-threads=1rather than nextest, withMSB_TEST_EXPECT_NESTED_VIRT=1, a scratchMSB_HOMEandMSB_PATHset to the branch runtime. Isolated-home mode always takes~/.microsandbox/bin/msb, which here is the installed release./dev/kvmis absent by default, present after enabling, and absent after disabling, and each plan reportsapplied. After enabling, the guest logged[ 0.028640] kvm [1]: Hyp nVHE mode initialized successfully.kvm_intel), withci-profile builds of f6fe87d in the setup above: both tests take the full path, withMSB_TEST_EXPECT_NESTED_VIRT=1set. Withkvm_intelreloaded withnested=0:MSB_TEST_EXPECT_NESTED_VIRT=1set both fail withthe host has nested KVM, but the host check refused it, so the CI guard works;modify().next_start().nested_virt(true)on a running sandbox is refused withSandbox::modify is not supported by this backend: nested virtualization is disabled in the host's kvm_intel or kvm_amd module, and the sandbox keeps running;modify().next_start().nested_virt(false)applies, and it then starts without/dev/kvm.nested=Yand withnested=0, soapplying_nested_virt_checks_the_host_before_persistingran both of its paths.nested=Y: a sandbox created by the 0.7.4-digdir.3msbthat agentctl pins has/dev/kvm. Started with this branch's SDK and runtime in the same home, it has none.modify().next_start().nested_virt(true)plans oneNextStartchange and leaves the running sandbox alone. After a stop and start it has/dev/kvm.ci-profile build of f6fe87d in the setup above: both tests take the refusal path withnested virtualization is not supported on this host.modify().nested_virt(true).restart()is refused asUnsupportedwithOperation::SandboxModify, and the sandbox keeps running without/dev/kvm. The 10 nested virtualization unit tests that build on Windows pass;nested_virt_requires_an_explicit_runtime_capabilityiscfg(unix).🤖 Generated with Claude Code