Add a per-disk direct I/O option for the cloud-hypervisor backend

[vincepri]

The cloud-hypervisor backend never set DiskConfig.direct, so every virtio-blk disk image went through the host page cache, with no way to change that.

A virtio-blk mount can now carry chDiskDirect=true or chDiskDirect=false in its runtime options, in the same way as the vzDiskImageCachingMode and vzDiskImageSynchronizationMode keys of the Virtualization backend. Mount.chDiskConfig(id:) reads the key into DiskConfig.direct. With true, cloud-hypervisor opens the image with O_DIRECT, so guest I/O to that disk bypasses the host page cache.

The option is per disk. For example, many VMs can boot from the same read-only rootfs and share one cached copy of it, while the read-write volume of each VM uses O_DIRECT:

let rootfs = Mount.block(format: "ext4", source: "/images/shared.ext4", destination: "/", options: ["ro"])
let volume = Mount.block(
    format: "ext4",
    source: "/volumes/vm1.ext4",
    destination: "/data",
    runtimeOptions: ["chDiskDirect=true"]
)

• The key applies to boot disks and to hotplugged disks. Hotplug now builds its DiskConfig with chDiskConfig(id:), so a hotplugged disk gets the same options as a boot disk.
• A mount without the key sends no direct field, so the VM config does not change.
• chDiskConfig(id:) ignores other runtime options, such as the vz keys. It now throws if chDiskDirect is not true or false.
• The Virtualization backend rejects keys that it does not know, so do not set chDiskDirect on a mount that is also given to that backend.

How it was tested

• New unit tests (CHDirectDiskIOTests) check the VM config at boot and the vm.add-disk request on hotplug. They cover these cases:
  • a disk without the key has no direct field;
  • in one VM, a read-only rootfs without the key stays cached and a volume with chDiskDirect=true gets direct: true;
  • chDiskDirect=false is sent as false;
  • an invalid value fails the start or the hotplug, and the hotplug sends nothing;
  • vz keys are ignored.
• Linux (arm64): swift format lint --strict, swift build --build-tests -Xswiftc -warnings-as-errors, and swift test. All 798 tests pass.
• macOS: the same build and swift test. All tests pass, except 8 registry tests that need network access to public registries, which my test machine does not have.

[egernst]

Is there a scenario where we'd want to have different behavior for volumes attached using virtio-blk and the underlying rootfs?

[vincepri]

Good point, it makes more sense per disk. I changed it.

There is no VM-wide flag now. Each virtio-blk mount can carry chDiskDirect=true (or false) in its runtime options, the same way the Virtualization backend takes vzDiskImageCachingMode. Without the key nothing changes: the disk goes through the page cache, as before.

For your example, the 15 VMs all get the same rootfs mount without the key, so the host keeps one cached copy of that image for all of them. Each VM then gets its own volume with chDiskDirect=true:

let rootfs = Mount.block(format: "ext4", source: "/images/shared.ext4", destination: "/", options: ["ro"])
let volume = Mount.block(format: "ext4", source: "/volumes/vm1.ext4", destination: "/data",
                         runtimeOptions: ["chDiskDirect=true"])

This works for boot disks and for hotplugged disks.

I left out a VM-level default. Every disk already comes from a Mount that the caller builds, so a default would only add a second way to set the same thing, plus a rule for which one wins.