Repository navigation
ateworker-microvm: add tmpfs golden slot pool and zero-copy guest RAM restore - #3018
Open
M.Besher Massri (besher-massri) wants to merge 3 commits into
Conversation
…inting Add opt-in differential memory-ranges checkpointing and reconstruction in cmd/ateworker-microvm/internal/ch/diff.go behind ATEWORKER_MICROVM_DIFF_SNAPSHOT=1 and ATEWORKER_MICROVM_DELTA_FORMAT=compact. - Cache the golden base memory-ranges under golden-base-cache/<baseID>/memory-ranges via sparse copy when diffSnapshotEnabled() is true. - Support in-place sparse hole punching via ch.DeflateAgainstBase (unlinking shared hard links first) and compact ADIF v1 extent files (memory-ranges.diff-meta and memory-ranges.diff-data) via ch.CreateDeltaAgainstBase and ch.RestoreFromDelta. - Reconstruct or overlay differential checkpoints onto the cached golden base in reconstructDeltaIfNeeded before Cloud Hypervisor vm.restore, staging preserved delta snapshots into deltaRestoreStagingDir(actorDirs) and recording vmmRestoreDir on runningActor so subsequent OnDemand checkpoints merge against the reconstructed image. Benchmark results (5 runs): goos: linux goarch: amd64 pkg: go.ate.dev/substrate/cmd/ateworker-microvm/internal/ch cpu: Intel(R) Xeon(R) CPU @ 2.20GHz BenchmarkCreateDeltaAgainstBase_167MB-8 9 114859910 ns/op 1524.57 MB/s 1726848 B/op 32 allocs/op BenchmarkCreateDeltaAgainstBase_167MB-8 9 114792135 ns/op 1525.47 MB/s 1726833 B/op 32 allocs/op BenchmarkCreateDeltaAgainstBase_167MB-8 10 118248260 ns/op 1480.89 MB/s 1726833 B/op 32 allocs/op BenchmarkCreateDeltaAgainstBase_167MB-8 9 111826200 ns/op 1565.93 MB/s 1726832 B/op 32 allocs/op BenchmarkCreateDeltaAgainstBase_167MB-8 9 115618577 ns/op 1514.57 MB/s 1726833 B/op 32 allocs/op BenchmarkApplyMemoryRangesDelta/FullCopy_167MB-8 8 138421251 ns/op 1265.07 MB/s 263384 B/op 15 allocs/op BenchmarkApplyMemoryRangesDelta/FullCopy_167MB-8 7 148894746 ns/op 1176.08 MB/s 263384 B/op 15 allocs/op BenchmarkApplyMemoryRangesDelta/FullCopy_167MB-8 7 143559131 ns/op 1219.79 MB/s 263384 B/op 15 allocs/op BenchmarkApplyMemoryRangesDelta/FullCopy_167MB-8 7 149522165 ns/op 1171.15 MB/s 263384 B/op 15 allocs/op BenchmarkApplyMemoryRangesDelta/FullCopy_167MB-8 7 164919411 ns/op 1061.80 MB/s 263384 B/op 15 allocs/op BenchmarkApplyMemoryRangesDelta/DifferentialOverlayOnSlot_167MB-8 51 21933966 ns/op 7983.61 MB/s 382946 B/op 23 allocs/op BenchmarkApplyMemoryRangesDelta/DifferentialOverlayOnSlot_167MB-8 55 24105871 ns/op 7264.30 MB/s 382944 B/op 23 allocs/op BenchmarkApplyMemoryRangesDelta/DifferentialOverlayOnSlot_167MB-8 55 22115341 ns/op 7918.13 MB/s 382944 B/op 23 allocs/op BenchmarkApplyMemoryRangesDelta/DifferentialOverlayOnSlot_167MB-8 56 23774223 ns/op 7365.63 MB/s 382948 B/op 23 allocs/op BenchmarkApplyMemoryRangesDelta/DifferentialOverlayOnSlot_167MB-8 55 24766747 ns/op 7070.46 MB/s 382946 B/op 23 allocs/op
…rse delta creation Add ch.UseFileBackedMemory, ch.CanonicalizeFileBackedSnapshot, and ch.CreateSparseDeltaAgainstBase in cmd/ateworker-microvm/internal/ch. - Rewrite memfd Cloud Hypervisor snapshots to map guest RAM directly from a shared file in ch.UseFileBackedMemory and empty the memory-manager memory_ranges table. - Restore file-backed Cloud Hypervisor snapshots back to canonical memfd format in ch.CanonicalizeFileBackedSnapshot. - Add ch.CreateSparseDeltaAgainstBase to extract modified 4 KiB pages into a sparse delta file without mutating the live guest RAM file. Benchmark results (5 runs): goos: linux goarch: amd64 pkg: go.ate.dev/substrate/cmd/ateworker-microvm/internal/ch cpu: Intel(R) Xeon(R) CPU @ 2.20GHz BenchmarkUseFileBackedMemoryAndCanonicalize-8 277 4276027 ns/op 1277436 B/op 1287 allocs/op BenchmarkUseFileBackedMemoryAndCanonicalize-8 252 4101383 ns/op 1277537 B/op 1288 allocs/op BenchmarkUseFileBackedMemoryAndCanonicalize-8 316 4819291 ns/op 1277414 B/op 1287 allocs/op BenchmarkUseFileBackedMemoryAndCanonicalize-8 273 5125697 ns/op 1278087 B/op 1289 allocs/op BenchmarkUseFileBackedMemoryAndCanonicalize-8 253 4099876 ns/op 1277150 B/op 1287 allocs/op BenchmarkCreateSparseDeltaAgainstBase_167MB-8 9 129497786 ns/op 1352.24 MB/s 526237 B/op 22 allocs/op BenchmarkCreateSparseDeltaAgainstBase_167MB-8 9 128073021 ns/op 1367.28 MB/s 526237 B/op 22 allocs/op BenchmarkCreateSparseDeltaAgainstBase_167MB-8 9 122069111 ns/op 1434.53 MB/s 526237 B/op 22 allocs/op BenchmarkCreateSparseDeltaAgainstBase_167MB-8 9 128366494 ns/op 1364.16 MB/s 526237 B/op 22 allocs/op BenchmarkCreateSparseDeltaAgainstBase_167MB-8 9 125505439 ns/op 1395.26 MB/s 526237 B/op 22 allocs/op PASS ok go.ate.dev/substrate/cmd/ateworker-microvm/internal/ch 21.540s
… restore Add a pre-populated tmpfs golden memory-ranges slot pool in cmd/ateworker-microvm/slot_pool.go behind ATEWORKER_MICROVM_TMPFS_SLOTS=1 and ATEWORKER_MICROVM_ZERO_COPY_RESTORE=1. - Mirror the golden base memory-ranges file into a private tmpfs mount (/run/ateworker-golden-slots, mounted with huge=within_size) and maintain pre-populated 0600 slot files per golden base ID. - On RestoreWorkload, claim a ready tmpfs slot in O(1) time, apply the actor's differential extent overlay in place while recording modified byte ranges, and invoke ch.UseFileBackedMemory so Cloud Hypervisor maps the slot directly as guest RAM via mmap(MAP_SHARED). - On CheckpointWorkload, extract modified pages against the immutable tmpfs golden base via ch.CreateSparseDeltaAgainstBase and canonicalize the checkpoint directory via ch.CanonicalizeFileBackedSnapshot. - When the microVM exits, reset only the dirtied byte ranges back to the golden base before returning the slot to the pool. Benchmark results (5 runs): goos: linux goarch: amd64 pkg: go.ate.dev/substrate/cmd/ateworker-microvm cpu: Intel(R) Xeon(R) CPU @ 2.20GHz BenchmarkSlotPool_OverlayAndReset_167MB-8 56 25139426 ns/op 6965.64 MB/s 526100 B/op 22 allocs/op BenchmarkSlotPool_OverlayAndReset_167MB-8 52 27284322 ns/op 6418.05 MB/s 526098 B/op 22 allocs/op BenchmarkSlotPool_OverlayAndReset_167MB-8 54 22808633 ns/op 7677.45 MB/s 526096 B/op 22 allocs/op BenchmarkSlotPool_OverlayAndReset_167MB-8 54 23572031 ns/op 7428.81 MB/s 526098 B/op 22 allocs/op BenchmarkSlotPool_OverlayAndReset_167MB-8 56 23184013 ns/op 7553.14 MB/s 526096 B/op 22 allocs/op BenchmarkFullBaseCopyAndOverlay_167MB-8 8 172954603 ns/op 1012.47 MB/s 526584 B/op 28 allocs/op BenchmarkFullBaseCopyAndOverlay_167MB-8 8 172168267 ns/op 1017.10 MB/s 526586 B/op 28 allocs/op BenchmarkFullBaseCopyAndOverlay_167MB-8 8 149779096 ns/op 1169.14 MB/s 526550 B/op 28 allocs/op BenchmarkFullBaseCopyAndOverlay_167MB-8 7 146948272 ns/op 1191.66 MB/s 526584 B/op 28 allocs/op BenchmarkFullBaseCopyAndOverlay_167MB-8 9 146780070 ns/op 1193.02 MB/s 526584 B/op 28 allocs/op PASS ok go.ate.dev/substrate/cmd/ateworker-microvm 22.745s
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why is this needed?
Note: This PR is stacked on #3014 (ateworker-microvm: support file-backed guest memory snapshots and sparse delta creation) and #3011 (ateworker-microvm: add golden-base differential memory-ranges checkpointing).
To restore microVMs from file-backed guest memory without copying the full base memory image on every restore, ateworker-microvm needs a pool of pre-populated golden base memory-ranges slot files in tmpfs. With a slot pool, each restoring microVM can claim an isolated 0600 slot file in O(1) time, apply its differential memory delta in place, map the slot file directly as guest RAM in Cloud Hypervisor via mmap(MAP_SHARED), and reset only the dirtied byte ranges back to the golden base when the microVM suspends or terminates.
Changes
Benchmark Results
In-tree microbenchmarks in cmd/ateworker-microvm/slot_pool_test.go compare the two host-side guest RAM preparation paths on a 384 MiB sparse guest memory-ranges image with 167 MiB of populated golden-base pages and 7.15 MB of modified delta pages across 20 dirty regions:
AI use: AI-assisted, I reviewed and tested the changes.