Repository navigation
objectstorage: write snapshot parts as sized zstd frames and pool upload buffers - #2835
Merged
Max Smythe (maxsmythe) merged 4 commits intoOct 20, 2026
Merged
Conversation
Add BenchmarkWriteSparsePart (one 64 MiB part, and a 512 MiB snapshot written as concurrent parts like PutSparseFile) and BenchmarkDecodeSparse (readSparseZstd on the stream this build writes and on the older one-frame-per-part layout). They report throughput, allocations, peak heap, CPU cores used, compression ratio and frame layout, and only call functions every variant of this path has, so the same file can measure encoder and decoder changes against each other. TestSparseSnapshotFormatsDecode checks both layouts decode byte-exact.
Contributor
Author
Each sparse upload part was compressed as one streaming zstd frame with no content size, through fresh buffers on every upload. A frame like that can only be decoded serially, so a parallel decoder gets nothing to split. Write parts with framedZstd instead: it compresses on the caller's goroutine into independent single-segment frames of at most parZstdChunk input bytes, each declaring its size. Unlike running parZstd with one worker per part, it holds one chunk, one frame and one encoder, which keeps a part inside the 32 MiB serialWorkerMemory the upload budget reserves (about 26 MiB measured, vs about 77 MiB for parZstd with one worker), with no goroutines. Chunk and frame buffers are now pooled and shared by parZstd workers and framedZstd, and the parts' 1 MiB copy buffer is pooled too, so uploads stop allocating those from scratch. Encoders stay per worker and per part: an Encoder makes an internal channel on first use, which would tie a pooled one to whichever synctest bubble used it first. The snapshot format is unchanged: concatenated zstd frames still decode through the existing serial reader. newSerialZstd has no callers left and is removed.
Nishanth Kotla (Nishanth29)
force-pushed
the
zstd-framed-parts
branch
from
October 18, 2026 19:27
923fbeb to
f9fc00b
Compare
2 tasks done
Bowei Du (bowei)
added
area/storage
kind/feature
Haowei Cai (Roy) (roycaihw)
approved these changes
Oct 19, 2026
Haowei Cai (Roy) (roycaihw)
left a comment
Haowei Cai (Roy) (roycaihw)
left a comment
Contributor
There was a problem hiding this comment.
Left a nit. Otherwise LGTM
Haowei Cai (Roy) (roycaihw)
approved these changes
Oct 19, 2026
Haowei Cai (Roy) (roycaihw)
left a comment
Haowei Cai (Roy) (roycaihw)
left a comment
Contributor
There was a problem hiding this comment.
LGTM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #2553. Related to #2502 and #2708.
Large GCS snapshots (
PutSparseFile→writeSparsePart) previously compressed each 64 MiB part as one unsized streaming zstd frame with freshly allocated buffers.#2502'sparZstdReaderonly parallelizes frames whose size is declared or bounded at≤ 16 MiB; a part written as one streaming frame of 64 MiB or more is neither, so restore goes serial from the first part. Meanwhile, switching parts to the worker-poolparZstdwith one worker uses ~77 MiB peak per part (2.4× over#2708's 32 MiBserialWorkerMemorybudget).This PR:
parZstdforworkers == 1: no goroutines; each≤ 8 MiBchunk is compressed on the caller's goroutine into a frame that declares its size.writeSparsePartusesnewParZstd(dst, 1), and small files get it too.newSerialZstd.Writeand a secondCloseafterClosereturnos.ErrClosed.BenchmarkWriteSparsePart,BenchmarkDecodeSparse, and format tests, including one that pins the output byte for byte to per-chunkEncodeAll.Results (
-count=10, 512 MiB populated / 2 GiB sparse)main#2502as-is#2502: 254 ms, 3.8×)ConcurrentSnapshotUploads(shared_512MiB):B/op/ peak heap¹#2502: sized frames decode straight into a buffer of the declared size (runtime.memmovedrops from 40% to 9% of decode CPU).all_80_admitted)¹: batch time roughly halves (57.6 → ~29 ms) and peak heap drops about a third (1,994 → 1,330 MiB).sweperf, 150u onc3-highcpu-192-metal, with#2502): microVMrestore_golden_download−31% at p50 (1,095 → 752 ms) and −63% at p95 (3,419 → 1,255 ms);restore_p50_s−29% (945 → 669 ms).¹ This PR's upload numbers are from a later session on the same machine than the
mainand#2502columns, so treat the deltas as approximate.Compatibility & follow-up
Snapshot format is unchanged; old and new snapshots decode byte-exact on both readers.
#2502'snewParZstd(dst, 1)inwriteSparsePartlands as-is; onlyTestReadZstdFrameBoundsUnsizedSmallFramesneeds to build its unsized frame withzstd.NewWriter(nil).EncodeAll.Follow-up (now that
#2848is merged): 1-worker uploads could reserveserialWorkerMemoryinstead ofparallelWorkerMemory.Tests pass
Appropriate changes to documentation are included in the PR (none needed)