StorageReplication.md

StorageReplication

Status: current implementation under development verification. Full solution, process recovery, Aspire SDK/MCP, coverage and endurance qualification remain open in status.json.

Decisions: ADR-136, ADR-143, ADR-146 and ADR-003.

Ownership and execution

Microsoft Orleans owns membership, discovery, failure detection, the distributed grain directory, placement, activation movement, scheduling and communication. Each silo-local Grain Service owns that silo's ZoneTree files (with ZoneTree's write-ahead log), locks and ordered apply gate. Grains call those services through typed Orleans calls; moving an activation never moves a file or open storage handle.

The public API enters one ordinary request grain per admitted operation. One commit grain per persisted storage incarnation orders canonical commands across logical tenant, database and partition identities into batches and requests operations from the required silo-local owners. Every queued command joins the next batch. Each owner applies a batch as one process-persistent native ZoneTree transaction; disk sync is periodic work owned by ZoneTree. Each command keeps its own canonical sequence and the outcome it would have alone. Intensive reads and background batches use stateless workers. Every operation retains fresh persisted authorization, bounded resources, cancellation and joined cleanup.

KeyLoad:SiloCount and KeyLoad:ReplicaCount are independent positive startup options. Copies cannot exceed configured or available silos. There is no artificial maximum silo count. membership supplies active silo addresses; the current implementation orders their names and selects the configured number for a write. This selection does not rebalance existing files or recover a lost data set onto new empty silos.

Writes, reads and retries

A write succeeds only after every configured copy returns its persisted canonical outcome. CommitReceipt.PersistedCopies records that actual count; Durability records the local persistence barrier. Buffered reports zero persisted copies. A stopped required copy blocks successful writes even when other copies can still serve reads.

Only the selected result copy returns full command outcomes over Orleans. Other copies return the actual applied prefix's canonical sequences and success/error-code decisions. The coordinator compares those decisions and positions before attaching the observed copy count to the selected original result. Missing full results or mismatched decisions leave the write outcome unknown. Copy repair uses the same compact acknowledgement. Every copy still persists its complete command outcome for replay; this transport shape removes duplicate response payloads rather than any durable outcome or copy check.

The canonical record is a execution ManagedCode.Communication.Command<DatabaseCommand>. The original command ID, verified subject, timestamp, input and prepared choices are retained with the execution Result<object> and canonical sequence in one local atomic transaction. A retry must match the original authorized operation and keeps its original result and sequence. A timeout after effects may have started is an unknown outcome, resolved using that same command ID.

If an unfinished batch is present on only some copies, the ordinary commit owner periodically completes the missing commands from the last batch that the newest copy retained. KeyLoad:SynchronizationInterval is a positive, validated timer interval with a five-second default. The timer runs only for isolated storage with several configured copies. It shares the existing command drain: background copy completion and new canonical writes never apply concurrently, while read barriers can continue on an advanced copy. Timer disposal and activation shutdown join the original drain. The next write or a stable-ID retry can complete the same work through this one copy path. Gaps larger than the last batch and unrelated divergent prefixes return RecoveryRequired; a successful response never hides that condition. General restoration of an empty or lagging data copy is not qualified by this implementation.

Reads select an available storage owner with the highest observed canonical cut, preferring the executing silo when cuts match. The commit grain caches the selected owners and their canonical position per membership version and re-reads silo cuts only after a membership change or a failed copy. A requested minimum token must be reached locally before reading, and current persisted credentials, resource permissions and projection rules still apply. Observing a cut alone does not prove equal data or a distributed snapshot.

A request on a single configured isolated silo with only that local silo active in Orleans membership runs beside its local storage owner directly. The published storage session still admits and joins the request. Shared storage and multi-silo deployments resolve the canonical owner through the commit grain.

Interleaved owner discovery adopts the already-published owner set. Completing a batch advances that same set and fences older discoveries; a different cached set is invalidated. A concurrent read cannot leave the next command using a cached canonical sequence from before the acknowledged batch.

After completing any retained copy gap, batch preparation prefers the commit grain's local silo when that silo is already a selected copy at the same canonical cut. This avoids sending the full command batch to another silo and back solely for preparation. The selected copy set is unchanged, and every selected owner still applies the same prepared commands before acknowledgement. An absent or lagging local copy cannot replace a required advanced copy.

Requirements and acceptance

Requirement	Observable acceptance	Current evidence boundary
REQ-REP-001: all copy calls use typed Orleans Grain Services and membership.	AC-REP-001: public SDK/MCP operations on the Aspire topology return the same authorized model data through each serving node.	FunctionalDatabaseGateTests owns the model flows; exact-source runtime qualification remains open.
REQ-REP-002: each owner confirms only after ordered atomic apply and its selected ZoneTree persistence barrier.	AC-REP-002: process interruption and cold reopen preserve acknowledged effects and their original outcomes; malformed complete records fail closed.	Real local process recovery and replicated-cluster evidence are separate gates.
REQ-REP-003: every configured copy acknowledges before a write succeeds.	AC-REP-003: stopping a required copy leaves earlier seeded data readable but rejects ordinary and duplicate submissions; restart permits healthy operations with exact copy counts.	ClusterTests.AtomicBatchSurvivesSiloRestartAndMissingCopiesRejectWrites; not yet qualified on the rewritten source.
REQ-REP-004: retained original commands and canonical sequence remain consistent across retries.	AC-REP-004: the same original command completes an interrupted application without duplicate effects; unrelated prefix gaps explicitly refuse further writes.	Requires actual fault injection and independent data/outcome oracles.
REQ-REP-005: every serving owner enforces current persisted authorization.	AC-REP-005: unauthorized and protected-field SDK/MCP reads refuse before and after restart, with unchanged records and healthy follow-up.	Authorization owns these assertions.
REQ-REP-006: admission, cancellation and shutdown bound all admitted work.	AC-REP-006: saturation explicitly refuses excess work; cancellation settles retained identities, joins producers and preserves earlier acknowledged data.	Unit ownership mechanisms and real cluster fault flows remain distinct.
REQ-REP-007: startup validates independent positive silo/copy counts without an artificial maximum.	AC-REP-007: 1/1, 2/1, 2/2, 3/2, 5/3 and 5/5 are valid counts; zero counts and copies exceeding silos refuse before resources start.	Configuration acceptance does not qualify a topology's runtime or durability.
REQ-REP-008: concurrent canonical commands share one process-persistent native commit per copy.	AC-REP-008: commands of one batch receive their own ordered sequences and the outcomes they would have alone. A retry inside a batch replays the first outcome without another sequence. A command that fails outside its own persisted outcome commits first in the next batch. Business time never decreases along a batch. Reset discards only the current command's writes. A copy that missed the last batch completes from its retained commands.	GroupCommitTests and ActionStagingTests own the engine and store contracts; the cluster batch path and its throughput are qualified by the Aspire suites and the benchmark cells.
REQ-REP-009: retained interrupted copy work is completed without requiring a new write.	AC-REP-009: interrupt one actual copy RPC, observe the background repair RPC, keep a read barrier available while repair is held, then verify the original document and sequence on both copies, exact retry and healthy continuation with joined cleanup.	CopySynchronizationTests.InterruptedCopyCompletesWithoutAnotherWriteAndReadBarriersContinueDuringRepair; local and delivered-source qualification remain separate.
REQ-REP-010: preparation prefers a selected local copy only at the same canonical cut.	AC-REP-010: the actual prepare call targets the coordinator's silo when it is a selected current copy; both copies retain exact data, revision and replayed outcome, and a healthy command follows. A different incarnation, lagging cut or unselected local silo cannot replace an advanced required owner.	StoragePreparationLocalityTests separates owner-selection checks from a real two-host command, replay and follow-up. Aspire throughput and delivered-source qualification remain separate.
REQ-REP-011: secondary copy responses omit duplicated outcome payloads while retaining the original copy-agreement checks.	AC-REP-011: a real two-host batch returns one full outcome and one compact acknowledgement with the same persisted decision; exact bodies, revisions, replay and healthy continuation hold. Wrong sequence, decision, prefix or a missing selected outcome return UnknownWriteOutcome after the real commit, then an exact retry resolves it without duplicate effects.	StorageBatchAcknowledgementTests and StorageBatchAcknowledgementFailureTests; local, delivered-source and throughput qualification remain separate.

Source and verification

• src/KeyLoad.Clustering/StorageReplication/Grains/DatabaseCommitGrain.cs owns ordered copy calls; StorageReplication/Execution/StorageCommitQueue.cs joins foreground batches and timer-driven completion on that same activation.
• src/KeyLoad.Clustering/StorageReplication/GrainServices/ owns addressing of silo-local storage services.
• src/KeyLoad/StoragePartitions/GrainServices/PartitionStorageGrainService.cs owns each silo's storage lifetime.
• build/KeyLoad.AppHost/TestInfrastructure/ owns topology and fault commands.
• Integration tests use actual SDK and official MCP clients. Process tests use actual owned child processes. Neither source presence nor a focused unit pass establishes distributed recovery.

Process-kill recovery does not prove power-loss durability. Device persistence, coverage and endurance need their own original evidence. See AspireOwnedFaultCommands and FunctionalDevelopmentTopology.