Skip to content

The deleted flag stops holding a word of its own - #222

Merged
matrixarkai merged 1 commit into
mainfrom
perf/the-deleted-flag-rides-the-versions-top-bit
Oct 25, 2026
Merged

matrixarkai merged 1 commit into
mainfrom
perf/the-deleted-flag-rides-the-versions-top-bit

Conversation

@matrixarkai

Copy link
Copy Markdown
Owner

MetadataRecord was 56 bytes: two sixteen-byte handles, two u64 stamps, and a bool that
alignment gave a whole eight-byte word to. The version is a per-key mutation counter that cannot
reach 2^63, so the flag rides its top bit.

was now
MetadataRecord 56 48
the metadata service's share 112 104
the metadata line 298 290 (fixed term 164,240 → 164,232)

The wire is unchanged, which is what makes it a representation change

append_metadata_record and the snapshot encoder both write the version and the flag as separate
u64s, and they still do — through version() and deleted(). No snapshot, no proof and no
checksum sees a different byte, so there is no stamp to bump.

The boundary is ruled out, not made unlikely

A version at 2^63 would flip the flag. set_version masks to MAX_VERSION and version() masks
on the way out, so a value at the bound is capped rather than corrupting the flag, and a
debug_assert names it. Reaching 2^63 versions of one key is not a rate any store sustains — but
"unlikely" is not "cannot", and a shared word has to rule the failure out.

The compiler enumerated the sites, because a regex would have been wrong

version and deleted are field names on ObjectMetadata, MetadataChange and
MetadataMutation as well, so a sweep over .version would have rewritten all of them. Making
the two fields private turns the compiler into the enumerator: every site it reports is a
MetadataRecord site and nothing else is.
84 of them were patched from the error locations —
file, line and column — bottom-up, so earlier edits could not shift later columns.

And seven struct literals, where the order is the format

Two of them read their fields straight off a snapshot cursor:

version: cursor.read_u64()?,
update_time_ms: cursor.read_u64()?,
deleted: cursor.read_bool()?,

A struct literal evaluates its fields in source order and a call evaluates its arguments left to
right, so MetadataRecord::new takes them in the order the literal listed them and reads the same
bytes in the same order. Reordering those arguments would decode a different record and compile
perfectly
, which is why the signature is the literal's order and the comment says so.

A fourth stale mirror, and the reason nothing caught it

measure_what_a_one_word_text_handle_would_save mirrors this record to price a thinner handle, and
its mirror still spelled the version and the flag as separate fields — so it measured 56 against a
real 48. Its own control assertion compares the mirror to size_of::<MetadataRecord>() and would
have failed on the first run. It is #[ignore]d, so there was no first run.

That is the fourth stale mirror in this effort, all the same shape: a by-construction measurement
is only as good as its control, and an #[ignore]d control is run by nothing.

What is left, and what the compiler says about each

104  today
 97  the kind out of the stored key -- every object record's key begins `object/`,
     seven characters of kind tag stored once per record
 96  an eight-byte value slot
 89  both

Safe Rust cannot hold text-or-digest in eight bytes, and that is measured rather than argued:
Box<Box<str>> is a thin pointer and NonZeroU64 is one word, but their niches are the same
value
, so an enum over them is sixteen bytes either way. Only a hand-tagged word reaches eight,
and that needs unsafe to read a pointer back out. The route that does not is splitting the
collection by kind — which also makes the object/ prefix implicit.

The allocation header on the key's characters is not a candidate, and the tree refuses it:
an_object_key_costs_two_bytes_a_character_however_many_places_name_it measures resident copies as
the slope of metadata against key length, and the bounded change log names those same characters
— so owning them costs a byte per character per holder, which for a 1,024-byte key is a megabyte.

The figures, each re-measured

amplification rows 1 KiB 3.33x → 3.32x; 4 KiB and 64 KiB unchanged
striped against replicated 1.98x → 1.97x; 3.47x → 3.46x
metadata an object at 100 / 1,000 / 4,000 / 8,000 498 / 454 / 331 / 310
delete trade holding 1,771,625, emptied 2,240,638, 26.4%
the share table objects 178 (61.4%), records 104 (35.9%), keys 8 (2.8%)

Measured

  • Suite: 412 passing on the default feature set, 433 with every feature, the same 21 failures in both and name for name the baseline in tools/known_failures.txt
  • cargo build --all-targets --locked and --all-features, RUSTFLAGS=-D warnings — green
  • cargo doc --no-deps --locked and --all-features, RUSTDOCFLAGS=-D warnings — green

🤖 Generated with Claude Code

`MetadataRecord` was 56 bytes: two sixteen-byte handles, two `u64` stamps, and a
`bool` that alignment gave a whole eight-byte word to. The version is a per-key
mutation counter that cannot reach 2^63, so the flag rides its top bit.

  MetadataRecord                 56 -> 48
  the metadata service's share  112 -> 104
  the metadata line             298 -> 290, fixed term 164,240 -> 164,232

THE WIRE IS UNCHANGED, WHICH IS WHAT MAKES IT A REPRESENTATION CHANGE

`append_metadata_record` and the snapshot encoder both write the version and the
flag as separate `u64`s, and they still do -- through `version()` and `deleted()`.
No snapshot, no proof and no checksum sees a different byte, so there is no stamp
to bump.

THE BOUNDARY IS RULED OUT, NOT MADE UNLIKELY

A version at 2^63 would flip the flag. `set_version` masks to `MAX_VERSION` and
`version()` masks on the way out, so a value at the bound is CAPPED rather than
corrupting the flag, and a `debug_assert` names it. Reaching 2^63 versions of one
key is not a rate any store sustains, but "unlikely" is not the same as "cannot",
and a shared word has to rule the failure out.

THE COMPILER ENUMERATED THE SITES, BECAUSE A REGEX WOULD HAVE BEEN WRONG

`version` and `deleted` are field names on `ObjectMetadata`, `MetadataChange` and
`MetadataMutation` as well. A sweep over `.version` would have rewritten all of
them. Making the two fields private turns the compiler into the enumerator: every
site it reports is a `MetadataRecord` site and nothing else is. 84 of them were
patched from the error locations -- file, line and column -- bottom-up so earlier
edits could not shift later columns.

AND SEVEN STRUCT LITERALS, WHERE THE ORDER IS THE FORMAT

Two of them read their fields straight off a snapshot cursor:

    version: cursor.read_u64()?,
    update_time_ms: cursor.read_u64()?,
    deleted: cursor.read_bool()?,

A struct literal evaluates its fields in source order and a call evaluates its
arguments left to right, so `MetadataRecord::new` takes them in the order the
literal listed them and reads the same bytes in the same order. Reordering those
arguments would decode a different record and compile perfectly, which is why the
constructor's signature is the literal's order and the comment says so.

A FOURTH STALE MIRROR, AND THE REASON NOTHING CAUGHT IT

`measure_what_a_one_word_text_handle_would_save` mirrors this record to price a
thinner handle, and its mirror still spelled the version and the flag as separate
fields -- so it measured 56 against a real 48. Its own control assertion compares
the mirror to `size_of::<MetadataRecord>()` and would have failed on the first
run. It is `#[ignore]`d, so there was no first run.

That is the fourth stale mirror in this effort. They are all the same shape: a
by-construction measurement is only as good as its control, and an `#[ignore]`d
control is run by nothing.

WHAT IS LEFT, AND WHAT THE COMPILER SAYS ABOUT EACH

  104  today
   97  the kind out of the stored key -- every object record's key begins
       `object/`, seven characters of kind tag stored once per record
   96  an eight-byte value slot

SAFE RUST CANNOT HOLD TEXT-OR-DIGEST IN EIGHT BYTES, and that is measured rather
than argued: `Box<Box<str>>` is a thin pointer and `NonZeroU64` is one word, but
their niches are the SAME VALUE, so an enum over them is sixteen bytes either
way. Only a hand-tagged word reaches eight, and that needs `unsafe` to read a
pointer back out. The route that does not is splitting the collection by kind,
which also makes the `object/` prefix implicit.

The allocation header on the key's characters is NOT a candidate and the tree
refuses it: `an_object_key_costs_two_bytes_a_character_however_many_places_name_it`
measures resident copies as the SLOPE of metadata against key length, and the
bounded change log names those same characters -- so owning them costs a byte per
character per holder, which for a 1,024-byte key is a megabyte.

THE FIGURES, EACH RE-MEASURED

  1 KiB row  3.33x -> 3.32x       4 KiB and 64 KiB rows unchanged
  striped    1.98x -> 1.97x       replicated 3.47x -> 3.46x
  metadata an object at 100 / 1,000 / 4,000 / 8,000   498 / 454 / 331 / 310
  delete trade   holding 1,771,625, emptied 2,240,638, 26.4% larger
  the share table    objects 178 (61.4%), records 104 (35.9%), keys 8 (2.8%)

MEASURED

Suite: 412 passing on the default feature set and 433 with every
feature, with the same 21 failures in both, name for name the baseline in
tools/known_failures.txt. Green on cargo build --all-targets for both feature sets with
warnings denied, and on cargo doc for both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@matrixarkai
matrixarkai merged commit d08ce98 into main Oct 25, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants