Skip to content

Omit redundant fixed-array lengths in Cereal, YAS and Boost.Serialization - #755

Merged
liuzicheng1987 merged 1 commit into
munich-data-engineering:mainfrom
Altren:feat/fixed-array-without-size
Oct 25, 2026
Merged

liuzicheng1987 merged 1 commit into
munich-data-engineering:mainfrom
Altren:feat/fixed-array-without-size

Conversation

@Altren

@Altren Altren commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Fixed-size arrays currently carry a redundant length prefix in the Cereal, YAS and Boost.Serialization adapters, even though the extent is already part of the C++ type. Serialize std::array and C arrays without that prefix, including nested and empty fixed-size arrays. This saves one length field per fixed array (8 bytes in the 64-bit binary configurations).

Add an optional read_fixed_array adapter API and matching writer operations. Fixed and dynamic archive reads share the element-reading loop; fixed reads take their extent from the type, while dynamic reads consume their existing length prefix. Cereal and YAS input array handles remain pointer-only. Map keys, variant tags and element parsing through processors are preserved, and formats without the new API retain their existing array representation.

Compatibility: this changes the fixed-array wire format in these three adapters. Existing archives need to be read with the previous version and rewritten. Readers and writers must agree on the array type and extent; without the prefix, an extent mismatch cannot be reliably detected.

This is extremely useful for data storages like std::vector<Eigen::Vector2d> and custom reflectors, that represent such vectors as std::array<double, N>. Reducing stored size by 33% and improving both read and write performance by 30% for Cereal and Boost and 10% for YAS

@liuzicheng1987

Copy link
Copy Markdown
Collaborator

@Altren thanks!

@liuzicheng1987
liuzicheng1987 merged commit ea0c104 into munich-data-engineering:main Oct 25, 2026
176 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants