The deleted flag stops holding a word of its own

README.md

@@ -238,18 +238,18 @@ At three replicas with the metadata counted, measured at 26-character keys:
 |---:|---:|---:|---:|
 | 64 KiB | 1,000 | **3.01x** | 3.00x |
 | 4 KiB | 4,000 | **3.08x** | 3.00x |
-| 1 KiB | 8,000 | **3.33x** | 3.02x |
+| 1 KiB | 8,000 | **3.32x** | 3.02x |
 
 The data term is exactly **3.00x** -- the replication floor for a store that
 copies every page to every replica -- so everything above it is metadata, which
-is **298 bytes an object plus 164,240 fixed** whatever the object size.
+is **290 bytes an object plus 164,232 fixed** whatever the object size.
 
 Those bytes are not all in one place. **186 of them are a storage node's own** -- the
-object's entry in its map, and its key in the payload index -- and **112 are the metadata
+object's entry in its map, and its key in the payload index -- and **104 are the metadata
 service's**, which keeps its own record of every object with its own versioning, deletion
 markers and change log. This crate models both services in one process, so the audit counts
 both; a deployment runs them on different machines, where a storage node carries 186 bytes
-an object rather than 298.
+an object rather than 290.
 
 Neither map keys its entries by a copy of what the entry already holds. `objects` is ordered by
 the identity inside each `ObjectMetadata` and `meta.records` by the key inside each record, so
@@ -282,9 +282,9 @@ which is more hardware rather than less space.
 replicas to eight and fails if any of them undercuts that.
 
 Counted with the metadata, a thousand objects of 1 KiB measure
-**1.98x striped against 3.47x replicated**, both read out of `footprint_audit()` on a
+**1.97x striped against 3.46x replicated**, both read out of `footprint_audit()` on a
 store nothing has read back yet. What separates the total from the 1.5x data term is the
-metadata line above, which striping does not change: the same 298 bytes an object either
+metadata line above, which striping does not change: the same 290 bytes an object either
 way. Striping moves the payload and nothing else.
 
 Both numbers are for the same object count, so the comparison is between two
@@ -363,8 +363,8 @@ Writing 500 objects of 1 KiB and then deleting every one leaves the store
 
 | | bytes |
 |---|---:|
-| holding 500 objects | 1,775,633 |
-| after deleting all 500 | 2,244,646 |
+| holding 500 objects | 1,771,625 |
+| after deleting all 500 | 2,240,638 |
 
 The payload index is given back in full. What replaces it is **822 bytes of
 idempotency receipt per delete** across three replicas, bounded only by

docs/footprint.md

@@ -22,7 +22,7 @@ Measured at three replicas, 26-character keys, with the metadata counted:
 |---:|---:|---:|---:|
 | 64 KiB | 1,000 | **3.01x** | 3.00x |
 | 4 KiB | 4,000 | **3.08x** | 3.00x |
-| 1 KiB | 8,000 | **3.33x** | 3.02x |
+| 1 KiB | 8,000 | **3.32x** | 3.02x |
 
 Reproduce any row with the measurement harness:
 
@@ -67,15 +67,15 @@ are bounded rather than per object:
 
 | objects | 100 | 1,000 | 4,000 | 8,000 |
 |---:|---:|---:|---:|---:|
-| metadata per object | 506 B | 462 B | 339 B | **318 B** |
+| metadata per object | 498 B | 454 B | 331 B | **310 B** |
 
 Both change feeds cap at 1,024 records, so together they are **163,866 bytes**
 once full -- the two intercepts in the table below, which is where that figure is
 checked rather than derived from a percentage of a moving total. Everything else
 is exactly linear, which makes the total a line once the feeds are full:
 
 ```
-metadata = 298 bytes an object + 164,240 bytes fixed      (remainder: zero)
+metadata = 290 bytes an object + 164,232 bytes fixed      (remainder: zero)
 ```
 
 reproducing every total above exactly, at every count where the feeds have
@@ -103,13 +103,13 @@ three store sizes and fails unless every one is a line and the fixed terms sum
 to the total's — so a leg that starts depending on the object count, or a fixed
 byte that stops being named, is caught rather than absorbed.
 
-That 298 is two items and a rounding error:
+That 290 is two items and a rounding error:
 
 | | per object | share |
 |---|---:|---:|
-| `objects` — the structured metadata | 178 B | 59.7% |
-| `meta.records` — the metadata digest record | 112 B | 37.6% |
-| payload index keys | 8 B | 2.7% |
+| `objects` — the structured metadata | 178 B | 61.4% |
+| `meta.records` — the metadata digest record | 104 B | 35.9% |
+| payload index keys | 8 B | 2.8% |
 
 Those two halves no longer hold the same facts twice. `objects` is a binary struct —
 136 bytes inline, with the object id, length, etag, content type, storage class
@@ -164,7 +164,7 @@ shared between holders is charged once per allocation — the characters and the
 two counts in front of them — so `mapkey_capacity` against `mapkey_len` is that
 header rather than slack.
 
-### What the 298 bytes are, and what each remaining part would cost to remove
+### What the 290 bytes are, and what each remaining part would cost to remove
 
 The slope is not one thing. Every leg below is fitted from the same two object counts the
 published line is fitted from, and they sum to it exactly:
@@ -175,11 +175,11 @@ objects.mapkey_inline           0  the collection is ordered by the identity the
 objects.mapkey_heap            42  the key's characters: ONE allocation, shared with the value
 objects.metadata_inline       136  ObjectMetadata itself, one extent held inline
 objects.extents_bytes           0  the one extent is INLINE above, so it owns no heap
-meta.records.struct_inline     56  MetadataRecord, and no map key beside it
+meta.records.struct_inline     48  MetadataRecord: the deleted flag rides the version
 meta.records.mapkey_capacity   56  the record key's characters: ONE allocation, ONE holder
 meta.records.value_capacity     0  the value is an eight-byte digest, held INLINE above
 payloads.key_bytes              8  the payload index's key
-METADATA LINE TOTAL          298
+METADATA LINE TOTAL          290
 ```
 
 `the_published_metadata_decomposition_sums_to_the_line` measures each one and fails if this
@@ -209,7 +209,7 @@ characters, sixteen bytes once per allocation.
 
 #### The reductions that are left, and what each one costs
 
-Each is priced against the 298. Two rows have come **off** this list, and both came off the
+Each is priced against the 290. Two rows have come **off** this list, and both came off the
 same way -- by being measured instead of argued. Narrowing an extent's offset and length to 32
 bits was priced at 8 bytes an object and a migration, until the migration turned out not to
 exist: the snapshot still writes two `u64` and only the in-memory form is narrow. Interning
@@ -219,7 +219,7 @@ decision and one representation.
 
 | | saves | what it costs |
 |---|---:|---|
-| The metadata records are a second store of the same objects | **112** | It is the whole of `meta.records`. They no longer RESTATE the objects map — a record holds a name, two stamps and a digest, and the digest is what lets it prove agreement without a second copy. What is left is a separate metadata service's own store, with its own versioning, deletion markers and change log, modelled in this process; removing it means collapsing two services into one, which is a design decision and not a footprint one. |
+| The metadata records are a second store of the same objects | **104** | It is the whole of `meta.records`. They no longer RESTATE the objects map — a record holds a name, two stamps and a digest, and the digest is what lets it prove agreement without a second copy. What is left is a separate metadata service's own store, with its own versioning, deletion markers and change log, modelled in this process; removing it means collapsing two services into one, which is a design decision and not a footprint one. |
 | A one-word text handle | **32**, or **16** without unsafe code | `SharedText` is `Arc<str>`, a fat pointer, so every holder spends two words to name one allocation. Six holders an object carry it -- two keying `objects`, three inside `ObjectMetadata` and one in a metadata record -- so one word each is **48 bytes of inline saving**, measured by construction rather than multiplied, because a narrowed field can leave its padding behind instead of closing up. The allocation pays part of it back by carrying the length the handle stops carrying: one word per allocation, two allocations an object, **net 32**. The shape that needs no unsafe code is `Arc<Box<str>>`, which puts the length in a second allocation instead and nets **16**. Both add a dependent load to a string read, and THAT LOAD IS NOW PRICED. Measured in release over 200,000 names visited in a shuffled order -- shuffled because a constant stride is the pattern a prefetcher exists to hide -- the thin handle costs **2.6x to 3.1x on a length read** and **1.1x to 1.3x on a character read**, four to nine nanoseconds either way. The length case is the whole story: `Arc<str>` carries the length IN THE HANDLE and touches no heap at all, where `Arc<Box<str>>` must reach the box first. The comparison favours the thin handle, whose index is half the size and so has the better locality, and it still loses. `measure_what_a_one_word_text_handle_costs_to_read` prints it. So the trade is 16 bytes an object -- 4.8% of the metadata line -- against roughly tripling the cost of asking a name its length, and the saving is largest for the small objects that do the most string reads per stored byte. |
 
 **Both figures above are measurements now, and the difference has mattered three times.**

src/internal/codec/metadata_object_proofs/shard_integrity_routes/checkpoint_integrity.rs

@@ -135,8 +135,8 @@ pub(crate) fn metadata_integrity_plan_checksum(plan: &MetadataIntegrityPlan) ->
     append_u64(&mut material, plan.compactable_tombstones.len() as u64);
     for record in &plan.compactable_tombstones {
         append_string(&mut material, &record.key);
-        append_u64(&mut material, record.version);
-        append_u64(&mut material, record.deleted as u64);
+        append_u64(&mut material, record.version());
+        append_u64(&mut material, record.deleted() as u64);
     }
     (etag(&material), material.len() as u64)
 }

src/internal/codec/metadata_object_proofs/transaction_proofs.rs

@@ -17,9 +17,9 @@ pub(crate) fn append_metadata_record(out: &mut Vec<u8>, record: &MetadataRecord)
     // that could not tell them apart would be hashing less than the record.
     append_u64(out, record.value.kind_tag());
     append_string(out, &record.value.as_text());
-    append_u64(out, record.version);
+    append_u64(out, record.version());
     append_u64(out, record.update_time_ms);
-    append_u64(out, record.deleted as u64);
+    append_u64(out, record.deleted() as u64);
 }
 
 pub(crate) fn append_metadata_transaction(out: &mut Vec<u8>, transaction: &MetadataTransaction) {

src/internal/codec/snapshot_journal_codec/snapshot_read_metadata.rs

@@ -7,13 +7,17 @@ pub(crate) fn read_metadata_record(
     let key = cursor.read_string()?;
     let value_kind = if has_value_kind { cursor.read_u64()? } else { 0 };
     let value_text = cursor.read_string()?;
-    Ok(MetadataRecord {
-        key: key.into(),
-        value: RecordValue::from_kind_tag(value_kind, &value_text)?,
-        version: cursor.read_u64()?,
-        update_time_ms: cursor.read_u64()?,
-        deleted: cursor.read_bool()?,
-    })
+    // THE ORDER IS THE FORMAT. These three read from the cursor, and a call evaluates its
+    // arguments left to right exactly as a struct literal evaluated its fields -- so this
+    // reads the same bytes in the same order. Reordering them would decode a different
+    // record and still compile.
+    Ok(MetadataRecord::new(
+        key.into(),
+        RecordValue::from_kind_tag(value_kind, &value_text)?,
+        cursor.read_u64()?,
+        cursor.read_u64()?,
+        cursor.read_bool()?,
+    ))
 }
 
 pub(crate) fn read_metadata_lease(

src/protocol/metadata/leases_maintenance/lease_records.rs

@@ -8,9 +8,76 @@ pub struct MetadataRecord {
     /// A projection of the object at this key, or the text any other record
     /// carries. Sixteen bytes inline either way -- see `RecordValue`.
     pub value: RecordValue,
-    pub version: u64,
+    /// The version, with the deleted flag in its top bit.
+    ///
+    /// # WHY THESE SHARE A WORD
+    ///
+    /// The version is a per-key mutation counter and `deleted` is one bit, but alignment gave
+    /// the bool a whole eight-byte word after two `u64`s -- 8 bytes an object for one bit.
+    /// The top bit of a counter that cannot reach 2^63 is free space.
+    ///
+    /// The WIRE IS UNCHANGED: `append_metadata_record` and the snapshot encoder both write
+    /// the version and the flag as separate `u64`s, and they still do, through the accessors.
+    /// A representation change.
+    ///
+    /// Read them with [`MetadataRecord::version`] and [`MetadataRecord::deleted`].
+    version_and_deleted: u64,
     pub update_time_ms: u64,
-    pub deleted: bool,
+}
+
+impl MetadataRecord {
+    /// The flag's bit. The version is masked off it on the way in and out, so a version that
+    /// somehow reached this bound would be capped rather than flipping the flag -- which is
+    /// the failure a shared word has to rule out, not merely make unlikely.
+    const DELETED_BIT: u64 = 1 << 63;
+    /// The largest version the shared word can hold. Reaching it would take more writes to
+    /// one key than a store can perform in any span of time worth modelling, which is why
+    /// the top bit is free space rather than a range being borrowed.
+    pub const MAX_VERSION: u64 = Self::DELETED_BIT - 1;
+
+    pub fn new(
+        key: SharedText,
+        value: RecordValue,
+        version: u64,
+        update_time_ms: u64,
+        deleted: bool,
+    ) -> Self {
+        let mut record = Self {
+            key,
+            value,
+            version_and_deleted: 0,
+            update_time_ms,
+        };
+        record.set_version(version);
+        record.set_deleted(deleted);
+        record
+    }
+
+    pub fn version(&self) -> u64 {
+        self.version_and_deleted & Self::MAX_VERSION
+    }
+
+    pub fn deleted(&self) -> bool {
+        self.version_and_deleted & Self::DELETED_BIT != 0
+    }
+
+    pub fn set_version(&mut self, version: u64) {
+        debug_assert!(
+            version <= Self::MAX_VERSION,
+            "a version past {} cannot share a word with the deleted flag",
+            Self::MAX_VERSION
+        );
+        self.version_and_deleted =
+            (self.version_and_deleted & Self::DELETED_BIT) | version.min(Self::MAX_VERSION);
+    }
+
+    pub fn set_deleted(&mut self, deleted: bool) {
+        if deleted {
+            self.version_and_deleted |= Self::DELETED_BIT;
+        } else {
+            self.version_and_deleted &= Self::MAX_VERSION;
+        }
+    }
 }
 
 /// A metadata record ordered and identified BY ITS OWN KEY, so the collection holding it
@@ -102,12 +169,12 @@ impl MetadataRecord {
         let mut fields = BTreeMap::new();
         fields.insert(format!("{field_prefix}key"), self.key.to_string());
         fields.insert(format!("{field_prefix}value"), self.value.as_text().into_owned());
-        fields.insert(format!("{field_prefix}version"), self.version.to_string().into());
+        fields.insert(format!("{field_prefix}version"), self.version().to_string().into());
         fields.insert(
             format!("{field_prefix}update_time_ms"),
             self.update_time_ms.to_string().into(),
         );
-        fields.insert(format!("{field_prefix}deleted"), self.deleted.to_string().into());
+        fields.insert(format!("{field_prefix}deleted"), self.deleted().to_string().into());
         fields
     }
 }

src/protocol/metadata/snapshot_decode.inc.rs

@@ -112,13 +112,15 @@ impl MetaSnapshot {
                 0
             };
             let value_text = cursor.read_string()?;
-            let record = MetadataRecord {
-                key: key.clone(),
-                value: RecordValue::from_kind_tag(value_kind, &value_text)?,
-                version: cursor.read_u64()?,
-                update_time_ms: cursor.read_u64()?,
-                deleted: cursor.read_bool()?,
-            };
+            // THE ORDER IS THE FORMAT -- see `read_metadata_record`. These three read from
+            // the cursor and must stay in this order.
+            let record = MetadataRecord::new(
+                key.clone(),
+                RecordValue::from_kind_tag(value_kind, &value_text)?,
+                cursor.read_u64()?,
+                cursor.read_u64()?,
+                cursor.read_bool()?,
+            );
             records.insert(key, record);
         }
         let mut leases = BTreeMap::new();

src/protocol/metadata/snapshot_encode.inc.rs

@@ -36,9 +36,9 @@ impl MetaSnapshot {
             // whether it is a digest or a text that happens to spell like one.
             append_u64(&mut out, record.value.kind_tag());
             append_string(&mut out, &record.value.as_text());
-            append_u64(&mut out, record.version);
+            append_u64(&mut out, record.version());
             append_u64(&mut out, record.update_time_ms);
-            append_u64(&mut out, record.deleted as u64);
+            append_u64(&mut out, record.deleted() as u64);
         }
         append_u64(&mut out, self.leases.len() as u64);
         for (namespace, lease) in &self.leases {

src/protocol/metadata/snapshot_install_helpers.inc.rs

@@ -89,7 +89,7 @@ pub(crate) fn metadata_snapshot_install_report(
     let deleted_record_count = snapshot
         .records
         .values()
-        .filter(|record| record.deleted)
+        .filter(|record| record.deleted())
         .count() as u64;
     let last_applied_index = snapshot.next_commit_index.saturating_sub(1);
     MetadataSnapshotInstallReport {

src/protocol/metadata/snapshot_validation/checksum.rs

@@ -27,9 +27,9 @@ impl MetaSnapshot {
         for (key, record) in &self.records {
             append_string(&mut material, key);
             append_string(&mut material, &record.value.as_text());
-            append_u64(&mut material, record.version);
+            append_u64(&mut material, record.version());
             append_u64(&mut material, record.update_time_ms);
-            append_u64(&mut material, record.deleted as u64);
+            append_u64(&mut material, record.deleted() as u64);
         }
         for (namespace, lease) in &self.leases {
             append_string(&mut material, namespace);

src/protocol/metadata/snapshot_validation/validation.rs

@@ -40,7 +40,7 @@ impl MetaSnapshot {
         }
         for (key, record) in &self.records {
             validate_meta_key(key)?;
-            if record.key != *key || record.version == 0 {
+            if record.key != *key || record.version() == 0 {
                 return Err(ObjectError::Corruption(
                     "metadata snapshot record is invalid".to_string(),
                 ));
@@ -184,7 +184,7 @@ impl MetaSnapshot {
             }
             for record in &receipt.records {
                 validate_meta_key(&record.key)?;
-                if record.version == 0 {
+                if record.version() == 0 {
                     return Err(ObjectError::Corruption(
                         "metadata snapshot batch receipt record is invalid".to_string(),
                     ));
@@ -247,7 +247,7 @@ impl MetaSnapshot {
             }
             for record in &receipt.records {
                 validate_meta_key(&record.key)?;
-                if record.version == 0 {
+                if record.version() == 0 {
                     return Err(ObjectError::Corruption(
                         "metadata snapshot transaction receipt record is invalid".to_string(),
                     ));