diff --git a/CONTEXT-MAP.md b/CONTEXT-MAP.md index af93fdfbd..97375a919 100644 --- a/CONTEXT-MAP.md +++ b/CONTEXT-MAP.md @@ -7,6 +7,9 @@ - [Stack Encrypt](./packages/stack-encrypt/CONTEXT.md) — encrypts values under per-value ZeroKMS data keys and derives searchable terms from them, for Rust and, through a WASI guest, Go. +- [Migrate](./languages/typescript/packages/migrate/CONTEXT.md) — moves an + existing column between plaintext and encrypted form, resumably, for the + Stack CLI and customer workers. ## Relationships diff --git a/languages/typescript/packages/migrate/CONTEXT.md b/languages/typescript/packages/migrate/CONTEXT.md new file mode 100644 index 000000000..2de507ec5 --- /dev/null +++ b/languages/typescript/packages/migrate/CONTEXT.md @@ -0,0 +1,60 @@ +# Migrate + +Moves an existing column of a customer's database between plaintext and encrypted form without downtime, tracking every step so an interrupted move can resume. Used by the `stash encrypt *` and `stash decrypt *` commands and by customers who embed it in their own workers. + +## Language + +### The move + +**Column migration**: +One direction applied to one column, from the first dual-write confirmation to the drop of the old column. +_Avoid_: lifecycle, rollout, reverse lifecycle + +**Direction**: +Which way a column migration moves data: **forward** (plaintext to encrypted) or **reverse** (encrypted to plaintext). +_Avoid_: mode, encrypt/decrypt flow + +**Source column**: +The column a column migration reads from and eventually drops. + +**Target column**: +The column a column migration fills and keeps. + +**Twin**: +The other column of the pair, in either direction. In forward it is the encrypted column; in reverse it is the plaintext column. +_Avoid_: plaintext twin, shadow column + +**Phase**: +The recorded stage a column migration is in, such as dual-writing, backfilling, cut over, or dropped. +_Avoid_: state, status + +**Dual-writing**: +The phase in which the application writes every change to both the source and target columns. + +**Abort**: +Abandoning a column migration that has not been dropped, leaving the source column as the column of record. Distinct from a reverse column migration, which starts only after a forward one is dropped. +_Avoid_: rollback, revert + +**Rollout**: +The work before dual-writing is confirmed: adding the target column and deploying dual-write code. Not a name for the whole column migration. + +### Filling the target + +**Backfill pass**: +One logical sweep that fills the target column from the source column. It may span several runs through resumes. +_Avoid_: backfill (alone, when the pass is meant), job + +**Run**: +One invocation of a backfill, from the CLI or a worker. A run either starts a backfill pass or resumes one. +_Avoid_: pass (when one invocation is meant) + +**Forced pass**: +A backfill pass that overwrites values already in the target column instead of skipping them. + +**Obligation**: +A condition recorded on a column migration that must be met before a later phase can be entered. An open obligation to complete a forced pass arises when a forced pass starts, or when a reverse column migration begins on a twin that already holds values; while it is open, every run is a forced pass. +_Avoid_: acknowledgement, hazard flag + +**Waiver**: +An operator's recorded assertion that a twin's existing values are already current, which closes the up-front obligation of a reverse column migration without a forced pass. It cannot close the obligation of a forced pass that has started. +_Avoid_: confirmation, acknowledgement diff --git a/languages/typescript/packages/migrate/docs/adr/0001-forced-pass-is-an-obligation.md b/languages/typescript/packages/migrate/docs/adr/0001-forced-pass-is-an-obligation.md new file mode 100644 index 000000000..3d00c34c9 --- /dev/null +++ b/languages/typescript/packages/migrate/docs/adr/0001-forced-pass-is-an-obligation.md @@ -0,0 +1,18 @@ +--- +status: accepted +--- + +# A forced pass is an obligation on the column migration, enforced by migrate + +Whether the target column must be overwritten is a fact about the target column's contents, not about one run. It is therefore recorded once, on the column migration, as an obligation. An obligation opens when a forced pass starts (in either direction) or when a reverse column migration begins on a twin that already holds values. It closes only when a forced pass completes, or, for the up-front reverse case, when the operator records a waiver asserting the existing values are current. A waiver cannot close the obligation of a forced pass that has already started, because by then the target column is part overwritten. While an obligation is open, every run is a forced pass, whatever the caller passes. `@cipherstash/migrate` enforces this rule itself, not the `stash` CLI, because customers embed `runBackfill` and `runDecryptBackfill` in their own workers and would otherwise bypass it. + +## Considered Options + +- **`force` as a per-run option, checked when resuming** (what was first built for reverse). An interrupted forced pass that failed before its first checkpoint left no cursor, so the next plain run started a fresh, unforced pass. That kept stale values in the twin, and the drop then removed the only current copy. +- **Refuse unforced runs while a forced pass is incomplete.** This is safe, but every caller has to remember the flag, and that is how the per-run model came about. +- **Enforce the rule in the CLI only.** Embedded workers would skip it. + +## Consequences + +- At most one column migration is open per column pair. A reverse column migration cannot start until the forward one is dropped. Abandoning a forward column migration that is still in progress is a separate operation. +- Events in `cs_migrations` carry the pass mode, and the obligation's opening and closing are recorded as events. The shape of `details` is effectively public, so changes to it go in a changeset.