From f2b8e0b696873aee86c3f9c1535d7cdba1090ff8 Mon Sep 17 00:00:00 2001 From: Deeproute Date: Fri, 23 Oct 2026 19:49:02 -0700 Subject: [PATCH] A metadata record is 200 bytes, and 72 of them buy a replication property The metaserver's share of the metadata line is 200 bytes an object. This prices every part of it, by construction, and records which parts are available: 16 the map key's `SharedText` handle 16 `MetadataRecord.key` -- a SECOND handle to the same characters 16 `RecordValue` -- a tag and a box 24 version, update_time_ms, deleted 56 the key's characters, `object/{bucket}/{key}`: 40 chars + a 16-byte Arc header 72 the boxed `ObjectProjection` --- 200 WHAT A RECORD'S VALUE ACTUALLY IS Nothing reads a projection's FIELDS. Its whole surface is `as_text`, `len`, `is_empty` and an equality that compares the rendered characters -- which the type says itself: two values that spell the same thing are the same value. So the projection is not a restatement of the object sitting beside its real copy. IT IS THE RECORD'S VALUE, held as thirteen fields instead of the 227 characters they render to. WHY IT CANNOT BECOME A LOCATOR A record whose value located the object instead of describing it would be far smaller -- a handle and a few scalars, twenty-four bytes against eighty-eight. That shape is not available here, and the reason is not taste. A record's rendered text goes into the snapshot checksum and into the transaction proofs: for (key, record) in &self.records { append_string(&mut material, key); append_string(&mut material, &record.value.as_text()); and the function that turns a record into bytes is `append_metadata_record(out: &mut Vec, record: &MetadataRecord)` -- a pure function of a record, with no store in reach and no business having one. If an object record rendered its text by consulting the storage node, the checksum over a set of records would become a function of the records AND whatever the local storage node currently said. TWO REPLICAS WHOSE STORAGE NODES DIFFER BY ONE IN-FLIGHT WRITE WOULD HASH DIFFERENT MATERIAL FOR IDENTICAL RECORDS. That is the one failure a replicated log exists to rule out, and this tree refuses trailing bytes in a command for exactly the same reason. So the 72 bytes buy the property that A RECORD'S PROOF IS A FUNCTION OF THE RECORD ALONE. They are load-bearing. WHAT IS AVAILABLE, MEASURED a. the content type as an index 8 BLOCKED b. the three counts as 16-bit 0 c. the retention stamps behind a pointer 8 d. all three together 24 e. the clean subset, b and c only 8 (a) is blocked the way it is blocked on the object side: the shared content-type table is bounded by a COUNT of 256, an object past the bound owns its own string, and a `u16` cannot say which -- an enum over the two cases is sixteen bytes again. Without it the other two collapse no further padding than (c) does alone, so the clean figure is 8 and not 16. (b) saving nothing on its own is worth stating: three `u32` counts narrowed to `u16` free twelve bytes of field and zero bytes of struct, because they were already sitting in padding. the clean narrowing 200 -> 192 with the duplicate key handle and its Arc header dropped -> 160 AND THAT SECOND ONE IS NOT A MIGRATION, which I had wrong until I read the encoder. The SNAPSHOT writes the MAP key, not the record's: for (key, record) in &self.records { append_string(&mut out, key); append_string(&mut out, &record.value.as_text()); Only the transaction proof writes `record.key`, through `append_metadata_record(out, record)`, and it writes the key THEN the value -- so a signature taking the key from the caller instead of the record produces BYTE-IDENTICAL material. What it costs is the 32 `Vec` signatures that carry a record without its key, which is a refactor and not a format change. THE ARC HEADER IS A CONSEQUENCE, NOT A CHOICE The key's characters cost 56 for 40 characters: sixteen of that is an `Arc` header, paid because `MetadataRecord` holds a second handle to the same allocation. A key nothing shares wants `Box`, which charges no header. The header and the duplicate are one item, not two. MEASURED Suite: 407 passing on the default feature set and 428 with every feature, with the same 21 failures in both, name for name the baseline in tools/known_failures.txt. Green on cargo build --all-targets for both feature sets with warnings denied, and on cargo doc for both. Co-Authored-By: Claude Opus 5 --- src/tests.rs | 177 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 177 insertions(+) diff --git a/src/tests.rs b/src/tests.rs index f10901b..d904bc5 100644 --- a/src/tests.rs +++ b/src/tests.rs @@ -36853,6 +36853,183 @@ fn measure_what_a_one_word_text_handle_costs_to_read() { ); } +#[test] +#[ignore = "a measurement, not an assertion: cargo test -- --ignored --nocapture"] +fn measure_the_route_to_under_a_hundred_bytes_a_metadata_record() { + // The metaserver's share of the line is 200 bytes an object, against a per-key record of + // TWENTY-FOUR in the time-series engine beside it. The difference is what the record IS: + // there it locates the data -- an address, a size, a deadline, flags -- and here it + // RESTATES the object, thirteen fields of it, behind a box. + // + // 16 map key handle + 16 duplicate key handle + 16 RecordValue + 24 scalars + // + 56 key characters + 72 boxed projection = 200 + // + // Mirrors sized by the compiler, controls asserted equal to the real types first. + use std::mem::size_of; + + assert_eq!( + size_of::() + size_of::(), + 72, + "the record plus the map key's handle should be the 72 the document publishes as \ + meta.records.struct_inline" + ); + assert_eq!( + size_of::(), + 72, + "and the boxed projection should be the 72 published as value_capacity" + ); + + #[allow(dead_code)] + struct ProjectionControl { + version: u64, + length: u64, + etag_digest_bits: u64, + has_etag: bool, + content_type: SharedText, + metadata_count: u32, + storage_class: StorageClass, + restore_until_ms: u64, + deleted: bool, + extent_count: u32, + tag_count: u32, + legal_hold: bool, + retention_until_ms: u64, + } + let control = size_of::(); + assert_eq!( + control, + size_of::(), + "the projection control has drifted, so nothing below means anything" + ); + + // (a) THE CONTENT TYPE AS AN INDEX. Already interned behind a table bounded by a count, + // and still held as a fat pointer. + #[allow(dead_code)] + struct IndexedContentType { + version: u64, + length: u64, + etag_digest_bits: u64, + has_etag: bool, + content_type: u16, + metadata_count: u32, + storage_class: StorageClass, + restore_until_ms: u64, + deleted: bool, + extent_count: u32, + tag_count: u32, + legal_hold: bool, + retention_until_ms: u64, + } + + // (b) THE THREE COUNTS ARE NOT 32-BIT QUANTITIES. A user-metadata entry count, a tag + // count and an extent count are each bounded far below 65,536 by the store's own limits. + #[allow(dead_code)] + struct NarrowCounts { + version: u64, + length: u64, + etag_digest_bits: u64, + has_etag: bool, + content_type: SharedText, + metadata_count: u16, + storage_class: StorageClass, + restore_until_ms: u64, + deleted: bool, + extent_count: u16, + tag_count: u16, + legal_hold: bool, + retention_until_ms: u64, + } + + // (c) THE TWO RETENTION STAMPS AND THE HOLD ARE RARE. An object under neither a + // restore nor a retention nor a legal hold pays sixteen bytes to say so -- the same + // shape `ObjectLock` already uses on the object side, whose doc says an object under + // none of the three should pay eight rather than twenty-five. + #[allow(dead_code)] + struct Retention { + restore_until_ms: u64, + retention_until_ms: u64, + legal_hold: bool, + } + #[allow(dead_code)] + struct BoxedRetention { + version: u64, + length: u64, + etag_digest_bits: u64, + has_etag: bool, + content_type: SharedText, + metadata_count: u32, + storage_class: StorageClass, + deleted: bool, + extent_count: u32, + tag_count: u32, + retention: Option>, + } + + // (d) ALL THREE. + #[allow(dead_code)] + struct Everything { + version: u64, + length: u64, + etag_digest_bits: u64, + has_etag: bool, + content_type: u16, + metadata_count: u16, + storage_class: StorageClass, + deleted: bool, + extent_count: u16, + tag_count: u16, + retention: Option>, + } + + // (e) THE CLEAN SUBSET: (b) and (c), without the content type. (a) needs the shared + // content-type table to be addressable by index, and it is bounded by a COUNT of 256 + // with an object past the bound owning its own string -- which a `u16` cannot + // represent, and an enum over the two cases is sixteen bytes again. So (a) is not + // available, and this is what is. + #[allow(dead_code)] + struct CleanSubset { + version: u64, + length: u64, + etag_digest_bits: u64, + has_etag: bool, + content_type: SharedText, + metadata_count: u16, + storage_class: StorageClass, + deleted: bool, + extent_count: u16, + tag_count: u16, + retention: Option>, + } + let rows: [(&str, usize); 5] = [ + ("a. the content type as an index (BLOCKED: the table is bounded by a count)", + size_of::()), + ("b. the three counts as 16-bit", size_of::()), + ("c. the retention stamps behind a pointer", size_of::()), + ("d. all three together", size_of::()), + ("e. the CLEAN subset, b and c only", size_of::()), + ]; + println!("METAREC control={control} record={} handle={}", + size_of::(), size_of::()); + for (what, size) in rows { + println!("METAREC {what}: {control} -> {size}, saves {}", control as i64 - size as i64); + } + + // WHERE THE METASERVER LANDS. The projection is the only part of the 200 that is not a + // migration: the duplicate key handle is encoded into the snapshot and the record travels + // without its map, and the Arc header exists only to serve that duplicate. + let now = 16 + 16 + 16 + 24 + 56 + control; + let after = 16 + 16 + 16 + 24 + 56 + size_of::(); + println!("METAREC metaserver now={now} after_the_projection={after}"); + assert_eq!(now, 200, "the metaserver share should be the 200 published, and it is {now}"); + // AND WHAT IT WOULD TAKE TO REACH A HUNDRED, stated so the next reader does not have to + // re-derive it: dropping the duplicate handle and its Arc header is -32 and a migration, + // which would leave 168 - (control - Everything). + println!( + "METAREC with the migration too={}", + after - 32 + ); +} + #[test] #[ignore = "a measurement, not an assertion: cargo test -- --ignored --nocapture"] fn measure_the_route_to_under_a_hundred_bytes_an_object() {