diff --git a/.github/workflows/check-digdir.yml b/.github/workflows/check-digdir.yml index 26ca9b8d1..66fadb98c 100644 --- a/.github/workflows/check-digdir.yml +++ b/.github/workflows/check-digdir.yml @@ -1040,6 +1040,15 @@ jobs: Set-MsvcEnvironment -Architecture ${{ matrix.vs_arch }} -HostArchitecture ${{ matrix.vs_host_arch }} cargo test --locked --no-default-features --features local,net,runner,ssh -p microsandbox-utils -p microsandbox-runtime -p microsandbox -p microsandbox-cli --lib --target ${{ matrix.rust_target }} windows_drive + - name: Test nested virtualization host check + shell: pwsh + # Exercises the WHP probe on x86_64; hosted runners may or may not offer nesting. + run: | + $ErrorActionPreference = "Stop" + . "$env:GITHUB_WORKSPACE\vendor\libkrunfw\scripts\msvc-env.ps1" + Set-MsvcEnvironment -Architecture ${{ matrix.vs_arch }} -HostArchitecture ${{ matrix.vs_host_arch }} + cargo test --locked --no-default-features -p microsandbox-runtime --lib --target ${{ matrix.rust_target }} nested_virt + - name: Test bind rootfs backend shell: pwsh # Tests build without embed-binaries and take the guest agent from this override. diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index 1b905a0ce..3ddd70e59 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -160,7 +160,7 @@ HTTP CONNECT outbound proxies accept optional Basic authentication through an op That probe covers SDK launches only. v0.7.x `machine` runtimes and v0.7.x SDKs decode the proxy object leniently, so a saved configuration reopened by an older release drops `credentials` and would connect without authentication; a read-modify-write through an older SDK can then persist the stripped representation. No representability check in `crates/db/lib/compat/` covers this field today, and the downgrade preflight matches v0.6.x and v0.7.0-v0.7.6 rather than v0.7.7. Adding `credentials` to an existing variant also breaks Rust consumers that construct `OutboundProxy::HttpConnect` literally or match its fields exactly, and `OutboundProxyBuildError` is not `#[non_exhaustive]`. -Nested virtualization uses the optional `nested_virt` launch key and `resources.nested_virt` record field, both omitted while off, so sandboxes without it are byte-identical for every runtime and catalog reader. Enabling it, at create or through `modify`, needs the runtime `nested_virt` capability and host support; the SDK refuses before replacing, changing, or starting a sandbox, a v0.7 runtime without the capability rejects the unknown launch key, and the previous launch contract refuses it. A runtime or CLI that predates the field builds its own launch from the stored record and ignores the field, so it starts such a sandbox without nested virtualization instead of refusing. The field stays in the record, and applies again on the next start through a current SDK, unless an older peer rewrites the record: an older `modify` drops it silently. +Nested virtualization uses the optional `nested_virt` launch key and `resources.nested_virt` record field, both omitted while off, so sandboxes without it are byte-identical for every runtime and catalog reader. Enabling it, at create or through `modify`, needs the runtime `nested_virt` capability and host support; the SDK refuses before replacing, changing, or starting a sandbox, a v0.7 runtime without the capability rejects the unknown launch key, and the previous launch contract refuses it. A runtime or CLI that predates the field builds its own launch from the stored record and ignores the field, so it starts such a sandbox without nested virtualization instead of refusing. The field stays in the record, and applies again on the next start through a current SDK, unless an older peer rewrites the record: an older `modify` drops it silently. Windows x86_64 hosts support it from the release that bumps libkrun to 0.1.42, and both host checks must accept the host. An older SDK refuses before launch. Older Windows runtimes advertise `nested_virt` but refuse nesting on Windows when the VM starts, so a current SDK that launches one passes its own admission and the start fails after `replace` or a `modify` restart has already changed the sandbox. That needs an SDK that resolves an older runtime binary. Runtimes ship together with the SDK, so this is accepted rather than guarded by a separate capability. Starting a sandbox crosses a private process boundary. On Unix, launch JSON is passed through inherited descriptor 96, the parent watchdog uses descriptor 97, startup JSON uses descriptor 98, and the lifecycle lock uses descriptor 99. Windows uses a short-lived launch-config file and platform-specific startup plumbing. Detach acknowledgement bytes and graceful-shutdown signals are also part of this contract. @@ -180,6 +180,8 @@ Sources: [`crates/runtime/lib/client/launch.rs`](crates/runtime/lib/client/launc Windows disk attachment retains the existing exclusive sidecar filenames and sharing mode. The launcher duplicates non-inheritable sidecar handles into the exact spawned process and sends their values through a bounded startup-only stdin message, required by the hidden `msb machine --disk-locks-stdin` flag. The runtime adopts them before opening guest disks; successful handoff closes the launcher's copies without an unlocked interval. The SDK sends this flag only to modern runtimes; previous runtimes retain launcher-owned locks and never receive the unsupported flag. Failed or cancelled handoffs terminate the child before storage cleanup; confirmed exit also clears any remaining parent copies. Stop/start observes owned-disk sidecar release, independently of retained SDK objects. This changes the Windows private launch contract, not SDK APIs, agent framing, snapshot formats, or disk bytes. +On Windows, detached runtimes are started with `CREATE_BREAKAWAY_FROM_JOB` so that they outlive the launcher's job object. When that job does not allow breakaway, as under `cargo test` or in some sandboxed tool shells, `CreateProcess` refuses with `ERROR_ACCESS_DENIED`. The SDK then logs a warning and starts the runtime inside the job instead of failing. Such a runtime ends when the job is terminated or its last handle closes with kill-on-close, so a detached sandbox can stop when that test run or shell exits. Launchers in no job, or in a job that allows breakaway, are unchanged. If both attempts fail, the error names the process start. Launch arguments, JSON, handles and runtime state are unchanged. + Unix disk-lock descriptors stay close-on-exec in the SDK parent and become inheritable only in the intended child's pre-exec callback. Successful spawn closes the parent's copies without explicitly unlocking the child's shared open-file descriptions. The runtime receives the same locked descriptors; this Unix handoff preserves fixed launch descriptors, JSON and paths. Stop and terminal restart additionally observe this sandbox's owned-disk marker locks under transition/lifecycle ownership, since a zombie leader and free lifecycle lock do not prove deferred file teardown finished. Named/external disks retain ordinary conflict admission. An already-running older development SDK can still retain its own parent lock copy; the new observer waits or times out rather than forcibly unlocking that genuine holder. Drop that old handle or restart that SDK process before retesting with the corrected handoff. Launch JSON requires an explicit `execution` intent (`boot` or `restore`) and rejects unknown fields. Restores also pass the internal `msb machine --restore` argument: a runtime predating this contract rejects the unknown argument rather than ignoring a JSON restore source and cold-booting. The argument, intent, and complete strictly validated `checkpoint_restore` source must agree before VM construction. Unsupported restore behavior is an error, never a fresh-boot fallback. These #8 development contracts replace superseded unreleased forms without shims; they do not change portable snapshot bytes. @@ -388,6 +390,8 @@ Do not independently substitute or upgrade one component because its upstream AB Normal x86 boot now declares `krun.poweroff=i8042` in libkrun's default kernel command line. Matching libkrunfw registers a low-priority final poweroff handler that sends the existing i8042 exit byte after orderly Linux shutdown. Generic KVM identity is not enough to enable it. Without both the host declaration and the new kernel, Linux may halt without releasing VMM ownership; StopWithTimeout reports expiry without killing. Existing running guests and full snapshots retain their captured kernel and require a fresh boot/recapture to acquire the handler. ARM platform poweroff, explicit custom command-line replacement and SEV/TDX built-in overrides are unchanged; they are not implicitly qualified by the normal Linux x86 test. +`msb_krun` 0.1.42 changes Windows x86_64 full snapshots. WHP execution state now records the emulated PIT, so new captures use backend state ABI 2, where earlier runtimes wrote ABI 1 without it. Each side refuses the other's state before the guest runs: this runtime rejects an earlier Windows full snapshot, fork or branch with libkrun's missing-PIT error, and an earlier runtime rejects the new ABI as incompatible. Restart such a sandbox from the snapshot's disks and capture a new full snapshot. Disk-only snapshots, Linux and macOS execution state, and non-Windows hosts are unchanged. Windows guests also boot with `tsc_early_khz` from the WHP processor clock, prepended to the default kernel command line so explicit caller values still win; this changes no launch field. + ## 13. Networking, DNS, Published Ports, and Secret Substitution Strict hostname policy is enabled by default. The v0.6.0–v0.6.17 launch gate rejects strict policies containing outbound hostname allow rules, including domain suffixes and rules applying in both directions. Policies without those rules retain their existing behavior on older runtimes because strict enforcement is unused. Explicit `strict: false` remains supported. diff --git a/Cargo.lock b/Cargo.lock index d44117bf1..8741b21ed 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4305,9 +4305,9 @@ dependencies = [ [[package]] name = "msb_krun" -version = "0.1.41" +version = "0.1.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d96bbf2bc289bfe0c1365e65b24ecb0a49ad59f5ec3078d91a34aa739572cf75" +checksum = "9cc8c72e61d6342ed8f381f049e5659c3c3c7a2d589eb2407f4abecc5aa633f2" dependencies = [ "crossbeam-channel", "kvm-bindings", @@ -4325,9 +4325,9 @@ dependencies = [ [[package]] name = "msb_krun_arch" -version = "0.1.41" +version = "0.1.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3c9433d7cef70cb736af38753f8335454705b7f908b39d0fa4972a06b31ce254" +checksum = "ac38ac5290daf53011289f0b3420ce69e577db101b39fa4ac8c73869635dc9e1" dependencies = [ "kvm-bindings", "kvm-ioctls", @@ -4340,15 +4340,15 @@ dependencies = [ [[package]] name = "msb_krun_arch_gen" -version = "0.1.41" +version = "0.1.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8a05a3d17076af32243e26e309515e5bfd647d834b9e5e7cf0572b905bee1ee" +checksum = "372c943ecdd64cc741630ff76a4763aad41732f62dba368be0f530b83373ec85" [[package]] name = "msb_krun_cpuid" -version = "0.1.41" +version = "0.1.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1abc03040ba405b76254227ab2995c08cd2b9b4b87d49295466e856ba569ebfd" +checksum = "ef38e1dc441567a43111aceade3837b7111d29c1f8c1ca6dd5f4ff82e92b6824" dependencies = [ "kvm-bindings", "kvm-ioctls", @@ -4357,9 +4357,9 @@ dependencies = [ [[package]] name = "msb_krun_devices" -version = "0.1.41" +version = "0.1.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b039a4c3e2f3b81abc15f4c761d249628fe7717259f32785869f7a8ffce50fe7" +checksum = "a030aa4be892ea9b2215c2828d2ce02d77d4b02133f0af6dc842bae5a802ed87" dependencies = [ "bincode", "bitflags 1.3.2", @@ -4388,9 +4388,9 @@ dependencies = [ [[package]] name = "msb_krun_hvf" -version = "0.1.41" +version = "0.1.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b881c01f289b2151552c0b1bd5f4e992665c5c2fc0d88805bc0628dd91bb56b5" +checksum = "b7eafee262ce993bafc612abb3dd6582c102f0f462f15fb22159bf04184dc908" dependencies = [ "crossbeam-channel", "libloading", @@ -4401,9 +4401,9 @@ dependencies = [ [[package]] name = "msb_krun_kernel" -version = "0.1.41" +version = "0.1.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "49cbd9f0d74adebae3d4621658171e530c4d0ab9ba701bf3c4ead31252976017" +checksum = "51ad8e40e3fe51a18effce0245e099c318e8302e851e7917eef296d46bbe143d" dependencies = [ "msb-vm-memory", "msb_krun_utils", @@ -4411,9 +4411,9 @@ dependencies = [ [[package]] name = "msb_krun_polly" -version = "0.1.41" +version = "0.1.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6088d48466c67615e161eb9a565627952c12aa877de254adfaa3d06d7fb8fc3d" +checksum = "fad6d7aae45e030546e31349f1c8fe2d257223b95360f84ca80b6bc0ef232f22" dependencies = [ "libc", "msb_krun_utils", @@ -4421,18 +4421,18 @@ dependencies = [ [[package]] name = "msb_krun_smbios" -version = "0.1.41" +version = "0.1.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b24757a311e696ed0685d985adc4760562ffbffc00996619ea17a04c328aa9c6" +checksum = "fdd434257469b722d2920d179a5a927b09e795c95035eca01674c5033bf83caa" dependencies = [ "msb-vm-memory", ] [[package]] name = "msb_krun_utils" -version = "0.1.41" +version = "0.1.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3af020b5b380c50fbe1ac58af1e406b0ab5129387a53f510140543a9225f5658" +checksum = "4b4ecd659171dc62624ff73007f5abdf4fffaf48131ef00fed1ea3d164fb2e6f" dependencies = [ "bitflags 1.3.2", "crossbeam-channel", @@ -4446,9 +4446,9 @@ dependencies = [ [[package]] name = "msb_krun_vmm" -version = "0.1.41" +version = "0.1.42" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "400ebdfc24faad502a5f0ef7e1e91625873dfb54cf4ed7aa8b3ad6944f46c0bf" +checksum = "284e283d6f1b5e5496afe615c6a4912e41b8b0210d40b1e7d4e0647a59ed6175" dependencies = [ "bincode", "bzip2", diff --git a/Cargo.toml b/Cargo.toml index 7d7deebaf..21c0a16d6 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -93,8 +93,8 @@ microsandbox-runtime = { version = "=0.7.11-digdir.1", path = "crates/runtime", microsandbox-utils = { version = "=0.7.11-digdir.1", path = "crates/utils" } microsandbox-vsock = { version = "=0.7.11-digdir.1", path = "crates/vsock" } # Keep runtime APIs and shared metrics types on the same published release. -msb_krun = "=0.1.41" -msb_krun_utils = "=0.1.41" +msb_krun = "=0.1.42" +msb_krun_utils = "=0.1.42" test-macros = { path = "crates/testing/macros" } test-utils = { path = "crates/testing/utils" } diff --git a/crates/runtime/Cargo.toml b/crates/runtime/Cargo.toml index 049a0883b..06bf5bc97 100644 --- a/crates/runtime/Cargo.toml +++ b/crates/runtime/Cargo.toml @@ -72,7 +72,7 @@ tracing.workspace = true zeroize.workspace = true [target.'cfg(windows)'.dependencies] -windows-sys = { version = "0.61.2", features = ["Win32_Foundation", "Win32_Security", "Win32_Security_Authorization", "Win32_Storage_FileSystem", "Win32_System_Kernel", "Win32_System_SystemInformation", "Win32_System_Threading"] } +windows-sys = { version = "0.61.2", features = ["Win32_Foundation", "Win32_Security", "Win32_Security_Authorization", "Win32_Storage_FileSystem", "Win32_System_Hypervisor", "Win32_System_Kernel", "Win32_System_LibraryLoader", "Win32_System_SystemInformation", "Win32_System_Threading"] } [target.'cfg(unix)'.dependencies] microsandbox-agent-client = { workspace = true, features = ["uds"], optional = true } diff --git a/crates/runtime/lib/nested_virt.rs b/crates/runtime/lib/nested_virt.rs index 3b28a440b..a31203a46 100644 --- a/crates/runtime/lib/nested_virt.rs +++ b/crates/runtime/lib/nested_virt.rs @@ -6,9 +6,29 @@ //! - Linux x86_64: the `kvm_intel` or `kvm_amd` module has `nested` enabled. //! - macOS on Apple silicon: Hypervisor.framework supports EL2 guests (an M3 or newer chip and //! macOS 15 or newer). +//! - Windows x86_64: the Windows Hypervisor Platform sets up a partition with nested +//! virtualization enabled (Windows 11 or Windows Server 2022 or newer). //! //! Other hosts do not support it. +#[cfg(all(target_os = "windows", target_arch = "x86_64"))] +use std::ffi::c_void; + +#[cfg(all(target_os = "windows", target_arch = "x86_64"))] +use windows_sys::Win32::Foundation::FreeLibrary; +#[cfg(all(target_os = "windows", target_arch = "x86_64"))] +use windows_sys::Win32::System::Hypervisor::{ + WHV_PARTITION_HANDLE, WHV_PARTITION_PROPERTY_CODE, + WHvPartitionPropertyCodeLocalApicEmulationMode, WHvPartitionPropertyCodeNestedVirtualization, + WHvPartitionPropertyCodeProcessorCount, WHvX64LocalApicEmulationModeXApic, +}; +#[cfg(all(target_os = "windows", target_arch = "x86_64"))] +use windows_sys::Win32::System::LibraryLoader::{ + GetProcAddress, LOAD_LIBRARY_SEARCH_SYSTEM32, LoadLibraryExW, +}; +#[cfg(all(target_os = "windows", target_arch = "x86_64"))] +use windows_sys::core::HRESULT; + //-------------------------------------------------------------------------------------------------- // Constants //-------------------------------------------------------------------------------------------------- @@ -23,6 +43,37 @@ const KVM_NESTED_PARAMETERS: [&str; 2] = [ const HYPERVISOR_FRAMEWORK: &std::ffi::CStr = c"/System/Library/Frameworks/Hypervisor.framework/Versions/A/Hypervisor"; +//-------------------------------------------------------------------------------------------------- +// Types +//-------------------------------------------------------------------------------------------------- + +/// An export as `GetProcAddress` returns it, before it is cast to its declared signature. +#[cfg(all(target_os = "windows", target_arch = "x86_64"))] +type WhpExport = unsafe extern "system" fn() -> isize; + +#[cfg(all(target_os = "windows", target_arch = "x86_64"))] +type WhpCreatePartition = unsafe extern "system" fn(*mut WHV_PARTITION_HANDLE) -> HRESULT; + +#[cfg(all(target_os = "windows", target_arch = "x86_64"))] +type WhpSetPartitionProperty = unsafe extern "system" fn( + WHV_PARTITION_HANDLE, + WHV_PARTITION_PROPERTY_CODE, + *const c_void, + u32, +) -> HRESULT; + +#[cfg(all(target_os = "windows", target_arch = "x86_64"))] +type WhpPartitionCall = unsafe extern "system" fn(WHV_PARTITION_HANDLE) -> HRESULT; + +/// The `WinHvPlatform.dll` exports that the Windows probe calls. +#[cfg(all(target_os = "windows", target_arch = "x86_64"))] +struct WhpPartitionApi { + create: WhpCreatePartition, + set_property: WhpSetPartitionProperty, + setup: WhpPartitionCall, + delete: WhpPartitionCall, +} + //-------------------------------------------------------------------------------------------------- // Functions //-------------------------------------------------------------------------------------------------- @@ -93,14 +144,143 @@ fn host_support_impl() -> Result<(), String> { result } +#[cfg(all(target_os = "windows", target_arch = "x86_64"))] +fn host_support_impl() -> Result<(), String> { + // Load WHP at run time, as macOS loads Hypervisor.framework: importing it would keep every + // process that links this crate from starting on hosts without the optional feature. + // SAFETY: `w!` yields a NUL-terminated UTF-16 string. + let module = unsafe { + LoadLibraryExW( + windows_sys::w!("WinHvPlatform.dll"), + std::ptr::null_mut(), + LOAD_LIBRARY_SEARCH_SYSTEM32, + ) + }; + if module.is_null() { + return Err( + "nested virtualization needs the Windows Hypervisor Platform, which is not enabled" + .into(), + ); + } + + // SAFETY: `module` is a live handle and the symbol names are NUL-terminated. + let symbols = unsafe { + ( + GetProcAddress(module, c"WHvCreatePartition".as_ptr().cast()), + GetProcAddress(module, c"WHvSetPartitionProperty".as_ptr().cast()), + GetProcAddress(module, c"WHvSetupPartition".as_ptr().cast()), + GetProcAddress(module, c"WHvDeletePartition".as_ptr().cast()), + ) + }; + let result = match symbols { + (Some(create), Some(set_property), Some(setup), Some(delete)) => { + // SAFETY: each export has the signature that WinHvPlatform.h declares for it. + let api = unsafe { + WhpPartitionApi { + create: std::mem::transmute::(create), + set_property: std::mem::transmute::( + set_property, + ), + setup: std::mem::transmute::(setup), + delete: std::mem::transmute::(delete), + } + }; + probe_whp_nested_partition(&api) + } + _ => Err("nested virtualization needs a newer Windows Hypervisor Platform".into()), + }; + + // SAFETY: `module` came from LoadLibraryExW and its functions are not used after this. + unsafe { FreeLibrary(module) }; + result +} + #[cfg(not(any( all(target_os = "linux", target_arch = "x86_64"), - all(target_os = "macos", target_arch = "aarch64") + all(target_os = "macos", target_arch = "aarch64"), + all(target_os = "windows", target_arch = "x86_64") )))] fn host_support_impl() -> Result<(), String> { Err("nested virtualization is not supported on this host".into()) } +//-------------------------------------------------------------------------------------------------- +// Functions: Helpers +//-------------------------------------------------------------------------------------------------- + +/// Creates a one-vCPU partition, sets it up for nesting as the VMM does, and deletes it. +#[cfg(all(target_os = "windows", target_arch = "x86_64"))] +fn probe_whp_nested_partition(api: &WhpPartitionApi) -> Result<(), String> { + let mut partition: WHV_PARTITION_HANDLE = 0; + // SAFETY: `partition` outlives the call. + let status = unsafe { (api.create)(&mut partition) }; + if status < 0 { + return Err(format!( + "nested virtualization support check failed to create a partition with status {status:#x}" + )); + } + + let result = set_up_nested_partition(api, partition); + + // SAFETY: `partition` is live and not used after this. + unsafe { (api.delete)(partition) }; + result +} + +/// Configures a new partition for nesting and sets it up, which is where WHP refuses it. +#[cfg(all(target_os = "windows", target_arch = "x86_64"))] +fn set_up_nested_partition( + api: &WhpPartitionApi, + partition: WHV_PARTITION_HANDLE, +) -> Result<(), String> { + let set = |code: WHV_PARTITION_PROPERTY_CODE, value: u32| { + // SAFETY: `partition` is live and `value` outlives the call. + unsafe { + (api.set_property)( + partition, + code, + (&value as *const u32).cast(), + size_of::() as u32, + ) + } + }; + + let status = set(WHvPartitionPropertyCodeProcessorCount, 1); + if status < 0 { + return Err(format!( + "nested virtualization support check failed to configure a partition with status {status:#x}" + )); + } + + let status = set(WHvPartitionPropertyCodeNestedVirtualization, 1); + if status < 0 { + return Err(format!( + "nested virtualization is not offered by this Windows Hypervisor Platform (status {status:#x})" + )); + } + + // Setup refuses nesting unless the partition also emulates a local APIC. + let status = set( + WHvPartitionPropertyCodeLocalApicEmulationMode, + WHvX64LocalApicEmulationModeXApic as u32, + ); + if status < 0 { + return Err(format!( + "nested virtualization needs local APIC emulation, which the Windows Hypervisor Platform refused with status {status:#x}" + )); + } + + // SAFETY: `partition` is live and configured. + let status = unsafe { (api.setup)(partition) }; + if status < 0 { + return Err(format!( + "nested virtualization was refused by the Windows Hypervisor Platform with status {status:#x}" + )); + } + + Ok(()) +} + //-------------------------------------------------------------------------------------------------- // Tests //-------------------------------------------------------------------------------------------------- @@ -123,4 +303,25 @@ mod tests { assert!(reason.starts_with("nested virtualization"), "{reason}"); } } + + /// Set `MSB_TEST_EXPECT_NESTED_VIRT=1` where the host supports nesting, so a check that + /// wrongly refuses it fails instead of passing as an unsupported host. + #[test] + fn hosts_expected_to_nest_pass_the_check() { + if std::env::var_os("MSB_TEST_EXPECT_NESTED_VIRT").is_some() { + assert_eq!(super::host_support(), Ok(())); + } + } + + /// Every Windows check creates and deletes a partition, so repeated checks must agree + /// instead of exhausting partitions. + #[cfg(all(target_os = "windows", target_arch = "x86_64"))] + #[test] + fn windows_check_is_repeatable() { + let first = super::host_support(); + + for _ in 0..16 { + assert_eq!(super::host_support(), first); + } + } } diff --git a/docs/sandboxes/bootstrap.mdx b/docs/sandboxes/bootstrap.mdx index 7ee056a8c..2d27aa423 100644 --- a/docs/sandboxes/bootstrap.mdx +++ b/docs/sandboxes/bootstrap.mdx @@ -303,10 +303,11 @@ Without the setting, guests have no `/dev/kvm`. Creating, starting, or enabling - macOS with an Apple M3 or newer chip and macOS 15 or newer. The guest's kernel logs `Hyp nVHE mode initialized successfully`, and its own VMs cannot nest further. - Linux x86_64, when the `kvm_intel` or `kvm_amd` module has `nested` enabled. +- Windows x86_64 with Windows 11 or Windows Server 2022 or newer, when the Windows Hypervisor Platform accepts a partition with nested virtualization. Intel and AMD processors both work; Windows on Arm does not. Full snapshots, forks, and full restores capture execution state, which does not cover the VMs a guest runs, so they are refused while nested virtualization is enabled. Disk-only snapshots remain available, but a sandbox restored from one starts without nested virtualization. VMs inside the guest pay extra overhead on every exit to the hypervisor, so their disk and device I/O are noticeably slower than the outer sandbox's. Only the Rust SDK exposes the setting. -The guest can then reach the host hypervisor's nested virtualization support: KVM's nested VMX or SVM handling on Linux, and EL2 emulation in Hypervisor.framework on macOS. That adds to what code in the sandbox can attack on the host, so enable it only for sandboxes that need to run VMs. +The guest can then reach the host hypervisor's nested virtualization support: KVM's nested VMX or SVM handling on Linux, Hyper-V's on Windows, and EL2 emulation in Hypervisor.framework on macOS. That adds to what code in the sandbox can attack on the host, so enable it only for sandboxes that need to run VMs. ## Custom init system diff --git a/sdk/rust/lib/runtime/spawn.rs b/sdk/rust/lib/runtime/spawn.rs index f7b2124db..c4ee0df74 100644 --- a/sdk/rust/lib/runtime/spawn.rs +++ b/sdk/rust/lib/runtime/spawn.rs @@ -42,7 +42,8 @@ use tokio::{ }; #[cfg(windows)] use windows_sys::Win32::Foundation::{ - GetHandleInformation, HANDLE, HANDLE_FLAG_INHERIT, INVALID_HANDLE_VALUE, SetHandleInformation, + ERROR_ACCESS_DENIED, GetHandleInformation, HANDLE, HANDLE_FLAG_INHERIT, INVALID_HANDLE_VALUE, + SetHandleInformation, }; #[cfg(windows)] use windows_sys::Win32::System::Console::{ @@ -695,11 +696,6 @@ pub async fn spawn_sandbox( // Build the command. let mut cmd = Command::new(&msb_path); - #[cfg(windows)] - if matches!(mode, SpawnMode::Detached) { - let flags = DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP | CREATE_BREAKAWAY_FROM_JOB; - cmd.creation_flags(flags); - } cmd.args(visible); // Agentd selection is process-wide for the VMM. Forward the explicit @@ -1021,9 +1017,44 @@ fn spawn_runtime_command( } else { None }; + #[cfg(windows)] + if matches!(_mode, SpawnMode::Detached) { + return spawn_detached_windows_command(cmd); + } cmd.spawn() } +/// Start a detached runtime outside the caller's job object when the job allows it. +/// +/// `CreateProcess` refuses `CREATE_BREAKAWAY_FROM_JOB` with `ERROR_ACCESS_DENIED` when the caller +/// is in a job without `JOB_OBJECT_LIMIT_BREAKAWAY_OK`, such as `cargo test` or a sandboxed tool +/// shell. The runtime then starts inside that job instead, and ends when the job is terminated or +/// its last handle closes with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE`. +#[cfg(windows)] +fn spawn_detached_windows_command(cmd: &mut Command) -> std::io::Result { + const FLAGS: u32 = DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP; + cmd.creation_flags(FLAGS | CREATE_BREAKAWAY_FROM_JOB); + let breakaway_err = match cmd.spawn() { + Err(err) if err.raw_os_error() == Some(ERROR_ACCESS_DENIED as i32) => err, + result => return result, + }; + tracing::warn!( + error = %breakaway_err, + "the caller's job object does not allow breakaway; the detached sandbox process stays \ + in that job and ends if the job is terminated or closed with kill-on-close" + ); + cmd.creation_flags(FLAGS); + cmd.spawn().map_err(|err| { + std::io::Error::new( + err.kind(), + format!( + "failed to start sandbox process {}: {err} (with job breakaway: {breakaway_err})", + cmd.as_std().get_program().to_string_lossy() + ), + ) + }) +} + fn block_writeback_policy( config: &RuntimeConfig, ) -> MicrosandboxResult<(Option, Option)> { @@ -3634,6 +3665,141 @@ mod tests { assert!(child.wait().await.unwrap().success()); } + #[cfg(windows)] + #[test] + fn windows_detached_job_grandchild() { + if std::env::var_os("MSB_TEST_DETACHED_JOB_GRANDCHILD").is_some() { + // Outlive the assertions; closing the test job ends this process. + std::thread::sleep(std::time::Duration::from_secs(60)); + } + } + + #[cfg(windows)] + fn windows_detached_job_grandchild_command() -> std::process::Command { + let mut command = std::process::Command::new(std::env::current_exe().unwrap()); + command + .args([ + "--exact", + "runtime::spawn::tests::windows_detached_job_grandchild", + ]) + .env_remove("MSB_TEST_DETACHED_JOB_CHILD") + .stdout(std::process::Stdio::null()); + command + } + + #[cfg(windows)] + #[tokio::test] + async fn windows_detached_job_child() { + use std::io::BufRead; + use std::os::windows::process::CommandExt; + + if std::env::var_os("MSB_TEST_DETACHED_JOB_CHILD").is_none() { + return; + } + // The parent assigns this process to its job before releasing it. + std::io::stdin() + .lock() + .read_line(&mut String::new()) + .unwrap(); + + let breakaway = windows_detached_job_grandchild_command() + .creation_flags( + super::DETACHED_PROCESS + | super::CREATE_NEW_PROCESS_GROUP + | super::CREATE_BREAKAWAY_FROM_JOB, + ) + .spawn(); + assert_eq!( + breakaway.err().and_then(|err| err.raw_os_error()), + Some(super::ERROR_ACCESS_DENIED as i32), + "the test job must refuse breakaway" + ); + + let mut command = tokio::process::Command::from(windows_detached_job_grandchild_command()); + command.env("MSB_TEST_DETACHED_JOB_GRANDCHILD", "1"); + let child = + super::spawn_runtime_command(&mut command, super::SpawnMode::Detached, None).unwrap(); + println!("grandchild-pid={}", child.id().unwrap()); + } + + #[cfg(windows)] + #[test] + fn windows_detached_spawn_stays_in_job_without_breakaway() { + use std::io::{BufRead, Write}; + use std::os::windows::io::{AsRawHandle, FromRawHandle, OwnedHandle}; + use windows_sys::Win32::System::JobObjects::{ + AssignProcessToJobObject, CreateJobObjectW, IsProcessInJob, + JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, JOBOBJECT_EXTENDED_LIMIT_INFORMATION, + JobObjectExtendedLimitInformation, SetInformationJobObject, + }; + use windows_sys::Win32::System::Threading::{ + OpenProcess, PROCESS_QUERY_LIMITED_INFORMATION, + }; + + // A kill-on-close job without JOB_OBJECT_LIMIT_BREAKAWAY_OK, like `cargo test` uses. + let job = unsafe { CreateJobObjectW(std::ptr::null(), std::ptr::null()) }; + assert!(!job.is_null()); + let job = unsafe { OwnedHandle::from_raw_handle(job) }; + let mut limits = unsafe { std::mem::zeroed::() }; + limits.BasicLimitInformation.LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE; + assert_ne!( + unsafe { + SetInformationJobObject( + job.as_raw_handle(), + JobObjectExtendedLimitInformation, + (&raw const limits).cast(), + std::mem::size_of::() as u32, + ) + }, + 0 + ); + + let mut child = std::process::Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "runtime::spawn::tests::windows_detached_job_child", + "--nocapture", + ]) + .env("MSB_TEST_DETACHED_JOB_CHILD", "1") + .stdin(std::process::Stdio::piped()) + .stdout(std::process::Stdio::piped()) + .spawn() + .unwrap(); + let assigned = + unsafe { AssignProcessToJobObject(job.as_raw_handle(), child.as_raw_handle()) }; + if assigned == 0 { + let err = std::io::Error::last_os_error(); + let _ = child.kill(); + panic!("assign the test child to its job: {err}"); + } + child.stdin.take().unwrap().write_all(b"\n").unwrap(); + + let stdout = std::io::BufReader::new(child.stdout.take().unwrap()); + let mut lines = Vec::new(); + let mut grandchild_pid = None; + for line in stdout.lines() { + let line = line.unwrap(); + if let Some(pid) = line.strip_prefix("grandchild-pid=") { + grandchild_pid = Some(pid.parse::().unwrap()); + } + lines.push(line); + } + assert!(child.wait().unwrap().success(), "{}", lines.join("\n")); + let grandchild_pid = grandchild_pid.expect("detached spawn reported no process"); + + // The fallback leaves the detached process in the job, which closing the job ends. + let grandchild = + unsafe { OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, 0, grandchild_pid) }; + assert!(!grandchild.is_null()); + let grandchild = unsafe { OwnedHandle::from_raw_handle(grandchild) }; + let mut in_job = 0; + assert_ne!( + unsafe { IsProcessInJob(grandchild.as_raw_handle(), job.as_raw_handle(), &mut in_job) }, + 0 + ); + assert_ne!(in_job, 0); + } + #[cfg(windows)] fn windows_handle_flags(handle: super::HANDLE) -> u32 { let mut flags = 0; diff --git a/sdk/rust/lib/sandbox/builder.rs b/sdk/rust/lib/sandbox/builder.rs index d1dddf4f8..dafefbbb0 100644 --- a/sdk/rust/lib/sandbox/builder.rs +++ b/sdk/rust/lib/sandbox/builder.rs @@ -413,10 +413,10 @@ impl SandboxBuilder { /// Expose hardware virtualization to the guest so it can run its own VMs through `/dev/kvm`. /// - /// Supported on macOS hosts with an Apple M3 or newer chip and macOS 15 or newer, and on - /// Linux x86_64 hosts whose KVM module has nested virtualization enabled. Creating or - /// starting the sandbox fails on other hosts; `runtime::nested_virt_host_support` checks - /// first. `modify().nested_virt(..)` changes it for an existing sandbox. + /// Supported on macOS hosts with an Apple M3 or newer chip and macOS 15 or newer, on Linux + /// x86_64 hosts whose KVM module has nested virtualization enabled, and on Windows x86_64 + /// hosts with Windows 11 or Windows Server 2022 or newer. Creating or starting the sandbox + /// fails on other hosts; `runtime::nested_virt_host_support` checks first. `modify().nested_virt(..)` changes it for an existing sandbox. /// /// The guest can then reach the host hypervisor's nested virtualization support, which adds /// to what code in the sandbox can attack on the host. Enable it only where VMs are needed. diff --git a/sdk/rust/tests/nested_virt.rs b/sdk/rust/tests/nested_virt.rs index 10b97cfc3..4861d04f3 100644 --- a/sdk/rust/tests/nested_virt.rs +++ b/sdk/rust/tests/nested_virt.rs @@ -1,13 +1,14 @@ //! Integration tests for opt-in nested virtualization. //! -//! These tests require KVM (or libkrun on macOS). The `#[msb_test]` +//! These tests require KVM (or libkrun on macOS and Windows). The `#[msb_test]` //! attribute marks them `#[ignore]`, so plain `cargo test --workspace` //! skips them. Run them via: //! //! cargo nextest run -p microsandbox --test nested_virt --run-ignored=only //! -//! Set `MSB_TEST_EXPECT_NESTED_VIRT=1` where the host has nested KVM, as CI does, so the test -//! fails instead of checking only the refusal if the host check wrongly refuses it. +//! Set `MSB_TEST_EXPECT_NESTED_VIRT=1` where the host supports nesting, as CI does with nested +//! KVM, so the test fails instead of checking only the refusal if the host check wrongly +//! refuses it. use microsandbox::runtime::nested_virt_host_support; use microsandbox::{MicrosandboxError, Operation, Sandbox, Snapshot}; @@ -45,7 +46,7 @@ fn host_support() -> Result<(), String> { if let Err(reason) = &support && std::env::var_os("MSB_TEST_EXPECT_NESTED_VIRT").is_some() { - panic!("the host has nested KVM, but the host check refused it: {reason}"); + panic!("the host is expected to support nesting, but the host check refused it: {reason}"); } support }