The projection's three holds go behind one pointer

.github/workflows/ci.yml

@@ -498,3 +498,12 @@ jobs:
             | tee /tmp/guard.log
           grep -q '^test result: ok. 1 passed' /tmp/guard.log \
             || { echo "::error::every_metadata_leg_is_a_line_and_the_intercepts_are_named did not run"; exit 1; }
+
+      - name: A projection holds nothing for an object under no hold
+        run: |
+          set -o pipefail
+          cargo test --locked --lib \
+            tests::a_projection_holds_nothing_for_an_object_under_no_hold -- --exact \
+            | tee /tmp/guard.log
+          grep -q '^test result: ok. 1 passed' /tmp/guard.log \
+            || { echo "::error::a_projection_holds_nothing_for_an_object_under_no_hold did not run"; exit 1; }

CONTRIBUTING.md

@@ -55,9 +55,9 @@ Two jobs, both on a pinned toolchain:
 | job | what it does |
 |---|---|
 | **build & docs** | `cargo build --all-targets --locked` and `--all-features`, then `cargo doc --no-deps --locked` and `--all-features`, all with warnings denied |
-| **format and rollback guards** | 46 individual tests, each run by name with `--exact` |
+| **format and rollback guards** | 47 individual tests, each run by name with `--exact` |
 
-**CI does not run the suite.** It compiles everything and runs those 46 guards. A green
+**CI does not run the suite.** It compiles everything and runs those 47 guards. A green
 tick means the tree builds and those guards hold; it does not mean the suite passed. Run
 `tools/suite_gate.sh` yourself.
 

README.md

@@ -237,19 +237,19 @@ At three replicas with the metadata counted, measured at 26-character keys:
 | object size | objects | amplification | data-only figure |
 |---:|---:|---:|---:|
 | 64 KiB | 1,000 | **3.01x** | 3.00x |
-| 4 KiB | 4,000 | **3.12x** | 3.00x |
+| 4 KiB | 4,000 | **3.11x** | 3.00x |
 | 1 KiB | 8,000 | **3.44x** | 3.02x |
 
 The data term is exactly **3.00x** -- the replication floor for a store that
 copies every page to every replica -- so everything above it is metadata, which
-is **418 bytes an object plus 164,256 fixed** whatever the object size.
+is **410 bytes an object plus 164,256 fixed** whatever the object size.
 
 Those bytes are not all in one place. **218 of them are a storage node's own** -- the
-object's entry in its map, and its key in the payload index -- and **200 are the metadata
+object's entry in its map, and its key in the payload index -- and **192 are the metadata
 service's**, which keeps its own record of every object with its own versioning, deletion
 markers and change log. This crate models both services in one process, so the audit counts
 both; a deployment runs them on different machines, where a storage node carries 218 bytes
-an object rather than 418.
+an object rather than 410.
 `every_metadata_leg_is_a_line_and_the_intercepts_are_named` measures both shares and fails
 if either stops being what this file publishes.
 `footprint_audit()` names every resident byte and its breakdown sums to its own
@@ -269,9 +269,9 @@ which is more hardware rather than less space.
 replicas to eight and fails if any of them undercuts that.
 
 Counted with the metadata, a thousand objects of 1 KiB measure
-**2.10x striped against 3.58x replicated**, both read out of `footprint_audit()` on a
+**2.09x striped against 3.58x replicated**, both read out of `footprint_audit()` on a
 store nothing has read back yet. What separates the total from the 1.5x data term is the
-metadata line above, which striping does not change: the same 418 bytes an object either
+metadata line above, which striping does not change: the same 410 bytes an object either
 way. Striping moves the payload and nothing else.
 
 Both numbers are for the same object count, so the comparison is between two
@@ -346,12 +346,12 @@ cargo test --release --lib measure_what_a_mutation_batch_snapshot_costs -- --ign
 #### A delete does not reclaim, it trades
 
 Writing 500 objects of 1 KiB and then deleting every one leaves the store
-**25.5% larger** than when it held them:
+**25.6% larger** than when it held them:
 
 | | bytes |
 |---|---:|
-| holding 500 objects | 1,835,649 |
-| after deleting all 500 | 2,304,662 |
+| holding 500 objects | 1,831,649 |
+| after deleting all 500 | 2,300,662 |
 
 The payload index is given back in full. What replaces it is **822 bytes of
 idempotency receipt per delete** across three replicas, bounded only by

docs/footprint.md

@@ -21,7 +21,7 @@ Measured at three replicas, 26-character keys, with the metadata counted:
 | object size | objects | amplification | data-only figure |
 |---:|---:|---:|---:|
 | 64 KiB | 1,000 | **3.01x** | 3.00x |
-| 4 KiB | 4,000 | **3.12x** | 3.00x |
+| 4 KiB | 4,000 | **3.11x** | 3.00x |
 | 1 KiB | 8,000 | **3.44x** | 3.02x |
 
 Reproduce any row with the measurement harness:
@@ -67,14 +67,14 @@ are bounded rather than per object:
 
 | objects | 100 | 1,000 | 4,000 | 8,000 |
 |---:|---:|---:|---:|---:|
-| metadata per object | 626 B | 582 B | 459 B | **438 B** |
+| metadata per object | 618 B | 574 B | 451 B | **430 B** |
 
 Both change feeds cap at 1,024 records, so together they are 30.0% of metadata
 at a hundred objects and 4.1% at eight thousand. Everything else is exactly
 linear, which makes the total a line once the feeds are full:
 
 ```
-metadata = 418 bytes an object + 164,256 bytes fixed      (remainder: zero)
+metadata = 410 bytes an object + 164,256 bytes fixed      (remainder: zero)
 ```
 
 reproducing every total above exactly, at every count where the feeds have
@@ -102,13 +102,13 @@ three store sizes and fails unless every one is a line and the fixed terms sum
 to the total's — so a leg that starts depending on the object count, or a fixed
 byte that stops being named, is caught rather than absorbed.
 
-That 418 is two items and a rounding error:
+That 410 is two items and a rounding error:
 
 | | per object | share |
 |---|---:|---:|
-| `objects` — the structured metadata | 210 B | 50.2% |
-| `meta.records` — the metadata projection record | 200 B | 47.8% |
-| payload index keys | 8 B | 1.9% |
+| `objects` — the structured metadata | 210 B | 51.2% |
+| `meta.records` — the metadata projection record | 192 B | 46.8% |
+| payload index keys | 8 B | 2.0% |
 
 Those two halves hold the **same facts twice**. `objects` is a binary struct —
 144 bytes inline, with the object id, length, etag, content type, storage class
@@ -157,7 +157,7 @@ shared between holders is charged once per allocation — the characters and the
 two counts in front of them — so `mapkey_capacity` against `mapkey_len` is that
 header rather than slack.
 
-### What the 418 bytes are, and what each remaining part would cost to remove
+### What the 410 bytes are, and what each remaining part would cost to remove
 
 The slope is not one thing. Every leg below is fitted from the same two object counts the
 published line is fitted from, and they sum to it exactly:
@@ -170,9 +170,9 @@ objects.metadata_inline       136  ObjectMetadata itself, one extent held inline
 objects.extents_bytes           0  the one extent is INLINE above, so it owns no heap
 meta.records.struct_inline     72  the map key's holder, plus MetadataRecord
 meta.records.mapkey_capacity   56  the record key's characters: ONE allocation, shared
-meta.records.value_capacity    72  the projection the record holds
+meta.records.value_capacity    64  the projection, its three holds behind a pointer
 payloads.key_bytes              8  the payload index's key
-METADATA LINE TOTAL          418
+METADATA LINE TOTAL          410
 ```
 
 `the_published_metadata_decomposition_sums_to_the_line` measures each one and fails if this
@@ -192,7 +192,7 @@ characters, sixteen bytes once per allocation.
 
 #### The reductions that are left, and what each one costs
 
-Each is priced against the 418. Two rows have come **off** this list, and both came off the
+Each is priced against the 410. Two rows have come **off** this list, and both came off the
 same way -- by being measured instead of argued. Narrowing an extent's offset and length to 32
 bits was priced at 8 bytes an object and a migration, until the migration turned out not to
 exist: the snapshot still writes two `u64` and only the in-memory form is narrow. Interning

src/internal/codec/control_storage_proofs/root_usage_projection.rs

@@ -194,12 +194,24 @@ pub struct ObjectProjection {
     pub content_type: SharedText,
     pub metadata_count: u32,
     pub storage_class: StorageClass,
-    pub restore_until_ms: u64,
     pub deleted: bool,
     pub extent_count: u32,
     pub tag_count: u32,
-    pub legal_hold: bool,
+    /// A restore deadline, a retention deadline and a legal hold -- behind one pointer,
+    /// because most objects are under none of the three and paid sixteen bytes to say so.
+    ///
+    /// `None` and an all-zero box would spell the SAME TEXT, and a derived `PartialEq`
+    /// would call them different values. So `from_metadata` never builds the second, and
+    /// `a_projection_with_no_holds_is_the_same_value_as_one_with_empty_holds` asserts it.
+    holds: Option<Box<ProjectedHolds>>,
+}
+
+/// The three deadlines and the hold, for an object that has any of them.
+#[derive(Clone, Debug, Default, Eq, PartialEq)]
+pub struct ProjectedHolds {
+    pub restore_until_ms: u64,
     pub retention_until_ms: u64,
+    pub legal_hold: bool,
 }
 
 impl ObjectProjection {
@@ -212,12 +224,47 @@ impl ObjectProjection {
             content_type: metadata.content_type.clone(),
             metadata_count: metadata.user_metadata.len() as u32,
             storage_class: metadata.storage_class.clone(),
-            restore_until_ms: metadata.lock.restore_until_ms,
             deleted: metadata.deleted,
             extent_count: metadata.extents.len() as u32,
             tag_count: metadata.tags.len() as u32,
-            legal_hold: metadata.lock.legal_hold,
-            retention_until_ms: metadata.lock.retention_until_ms,
+            // ONLY WHEN THERE IS SOMETHING TO HOLD. An all-zero box spells the same text
+            // as no box, and a derived `PartialEq` would call the two different values.
+            holds: (metadata.lock.restore_until_ms != 0
+                || metadata.lock.retention_until_ms != 0
+                || metadata.lock.legal_hold)
+                .then(|| {
+                    Box::new(ProjectedHolds {
+                        restore_until_ms: metadata.lock.restore_until_ms,
+                        retention_until_ms: metadata.lock.retention_until_ms,
+                        legal_hold: metadata.lock.legal_hold,
+                    })
+                }),
+        }
+    }
+
+    /// When a restore expires, or zero for an object under none.
+    pub fn restore_until_ms(&self) -> u64 {
+        self.holds.as_deref().map_or(0, |holds| holds.restore_until_ms)
+    }
+
+    /// When a retention expires, or zero for an object under none.
+    pub fn retention_until_ms(&self) -> u64 {
+        self.holds
+            .as_deref()
+            .map_or(0, |holds| holds.retention_until_ms)
+    }
+
+    /// Whether a legal hold is in force.
+    pub fn legal_hold(&self) -> bool {
+        self.holds.as_deref().is_some_and(|holds| holds.legal_hold)
+    }
+
+    /// What the holds own on the heap, which the audit needs: an object under none of the
+    /// three owes nothing for them.
+    pub(crate) fn holds_heap_bytes(&self) -> u64 {
+        match self.holds {
+            Some(_) => std::mem::size_of::<ProjectedHolds>() as u64,
+            None => 0,
         }
     }
 
@@ -238,12 +285,12 @@ impl ObjectProjection {
             self.content_type,
             self.metadata_count,
             self.storage_class,
-            self.restore_until_ms,
+            self.restore_until_ms(),
             self.deleted,
             self.extent_count,
             self.tag_count,
-            self.legal_hold,
-            self.retention_until_ms
+            self.legal_hold(),
+            self.retention_until_ms()
         )
     }
 }
@@ -336,7 +383,13 @@ impl RecordValue {
     pub fn heap_bytes(&self) -> u64 {
         match self {
             RecordValue::Text(text) => text.len() as u64,
-            RecordValue::Projection(_) => std::mem::size_of::<ObjectProjection>() as u64,
+            // PLUS WHAT THE HOLDS OWN. An object under a restore, a retention or a legal
+            // hold keeps them behind a pointer, and charging only the struct would report a
+            // saving those objects did not get.
+            RecordValue::Projection(projection) => {
+                (std::mem::size_of::<ObjectProjection>() as u64)
+                    .saturating_add(projection.holds_heap_bytes())
+            }
         }
     }