Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CONTEXT-MAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@
- [Stack Encrypt](./packages/stack-encrypt/CONTEXT.md) — encrypts values under
per-value ZeroKMS data keys and derives searchable terms from them, for Rust
and, through a WASI guest, Go.
- [Migrate](./languages/typescript/packages/migrate/CONTEXT.md) — moves an
existing column between plaintext and encrypted form, resumably, for the
Stack CLI and customer workers.

## Relationships

Expand Down
60 changes: 60 additions & 0 deletions languages/typescript/packages/migrate/CONTEXT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Migrate

Moves an existing column of a customer's database between plaintext and encrypted form without downtime, tracking every step so an interrupted move can resume. Used by the `stash encrypt *` and `stash decrypt *` commands and by customers who embed it in their own workers.

## Language

### The move

**Column migration**:
One direction applied to one column, from the first dual-write confirmation to the drop of the old column.
_Avoid_: lifecycle, rollout, reverse lifecycle

**Direction**:
Which way a column migration moves data: **forward** (plaintext to encrypted) or **reverse** (encrypted to plaintext).
_Avoid_: mode, encrypt/decrypt flow

**Source column**:
The column a column migration reads from and eventually drops.

**Target column**:
The column a column migration fills and keeps.

**Twin**:
The other column of the pair, in either direction. In forward it is the encrypted column; in reverse it is the plaintext column.
_Avoid_: plaintext twin, shadow column

**Phase**:
The recorded stage a column migration is in, such as dual-writing, backfilling, cut over, or dropped.
_Avoid_: state, status

**Dual-writing**:
The phase in which the application writes every change to both the source and target columns.

**Abort**:
Abandoning a column migration that has not been dropped, leaving the source column as the column of record. Distinct from a reverse column migration, which starts only after a forward one is dropped.
_Avoid_: rollback, revert

**Rollout**:
The work before dual-writing is confirmed: adding the target column and deploying dual-write code. Not a name for the whole column migration.

### Filling the target

**Backfill pass**:
One logical sweep that fills the target column from the source column. It may span several runs through resumes.
_Avoid_: backfill (alone, when the pass is meant), job

**Run**:
One invocation of a backfill, from the CLI or a worker. A run either starts a backfill pass or resumes one.
_Avoid_: pass (when one invocation is meant)

**Forced pass**:
A backfill pass that overwrites values already in the target column instead of skipping them.

**Obligation**:
A condition recorded on a column migration that must be met before a later phase can be entered. An open obligation to complete a forced pass arises when a forced pass starts, or when a reverse column migration begins on a twin that already holds values; while it is open, every run is a forced pass.
_Avoid_: acknowledgement, hazard flag

**Waiver**:
An operator's recorded assertion that a twin's existing values are already current, which closes the up-front obligation of a reverse column migration without a forced pass. It cannot close the obligation of a forced pass that has started.
_Avoid_: confirmation, acknowledgement
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
status: accepted
---

# A forced pass is an obligation on the column migration, enforced by migrate

Whether the target column must be overwritten is a fact about the target column's contents, not about one run. It is therefore recorded once, on the column migration, as an obligation. An obligation opens when a forced pass starts (in either direction) or when a reverse column migration begins on a twin that already holds values. It closes only when a forced pass completes, or, for the up-front reverse case, when the operator records a waiver asserting the existing values are current. A waiver cannot close the obligation of a forced pass that has already started, because by then the target column is part overwritten. While an obligation is open, every run is a forced pass, whatever the caller passes. `@cipherstash/migrate` enforces this rule itself, not the `stash` CLI, because customers embed `runBackfill` and `runDecryptBackfill` in their own workers and would otherwise bypass it.

## Considered Options

- **`force` as a per-run option, checked when resuming** (what was first built for reverse). An interrupted forced pass that failed before its first checkpoint left no cursor, so the next plain run started a fresh, unforced pass. That kept stale values in the twin, and the drop then removed the only current copy.
- **Refuse unforced runs while a forced pass is incomplete.** This is safe, but every caller has to remember the flag, and that is how the per-run model came about.
- **Enforce the rule in the CLI only.** Embedded workers would skip it.

## Consequences

- At most one column migration is open per column pair. A reverse column migration cannot start until the forward one is dropped. Abandoning a forward column migration that is still in progress is a separate operation.
- Events in `cs_migrations` carry the pass mode, and the obligation's opening and closing are recorded as events. The shape of `details` is effectively public, so changes to it go in a changeset.
Loading