feat(cli): reverse migration (stash decrypt) for moving columns back to plaintext

.changeset/stash-decrypt-reverse-migration.md

@@ -0,0 +1,8 @@
+---
+'stash': minor
+'@cipherstash/migrate': minor
+---
+
+`stash decrypt status|plan|backfill|drop` moves encrypted columns back to plaintext — the encrypt lifecycle run in reverse, with the same safety rails. `decrypt plan` adds a nullable plaintext twin per EQL v3 column with its type mapped back from the domain (`text*` → `text`, `timestamp*` → `timestamptz`, `json*` → `jsonb`, …): Drizzle schema lines for `drizzle-kit generate`, or a reviewed SQL migration elsewhere. `decrypt backfill` decrypts in resumable, checkpointed chunks, one `bulkDecrypt` (one ZeroKMS request) per chunk, default 500 rows. It refuses a plaintext column that a forward encryption started from and never dropped: only one migration may be open on a pair of columns, so finish or abandon the forward one first. When that column was already removed outside stash, `decrypt plan` finds it missing and records the forward migration as dropped (marked as observed), so the reverse migration can start. Its first run also refuses a plaintext column that already holds values, because those values may be stale and the backfill only fills NULLs: pass `--force` to overwrite them from the ciphertext or `--confirm-plaintext-current` to keep them. Once `--force` is chosen, or any forced run starts, every later run is forced until a forced pass completes, so an interrupted overwrite is always finished. The application keeps writing both columns until the drop migration is applied. Values bound to an end-user lock context cannot be decrypted with workspace credentials; they are skipped, counted, left NULL and reported, and `decrypt drop` refuses while any remain. `decrypt drop` re-verifies coverage at apply time, drops the indexes built on the column's EQL terms, drops the column, and with `--uninstall-eql` removes EQL v3 when it was the last encrypted column in the database.
+
+`@cipherstash/migrate`: `runBackfill` now runs on a shared chunked engine (`runChunkedMigration`), which also drives the new `runDecryptBackfill`. Forward behaviour and recorded event details are unchanged. Adds `runDecryptBackfill` (resumes after a failed run; `PlaintextWriteError` keeps decrypted values out of write errors; chunk size capped at `MAX_DECRYPT_CHUNK_SIZE`), `columnHasValues`, `countUndecrypted`, `fetchUndecryptedPage`, `plaintextTypeForEqlDomain`, `columnSqlType`, `listColumnIndexes`, `primaryKeyColumns`, `countEqlColumnsInDatabase`, `removeManifestColumn`, and the reverse event/phase vocabulary (`decrypt_started`, `decrypt_checkpoint`, `decrypted`, `encrypted_dropped`; `decrypt-dual-writing`, `decrypting`, `decrypted`, `encrypted-dropped`), recorded against the encrypted column. `MigrationPhase` and `MigrationEvent` gain those members, so an exhaustive `switch` over them needs the new cases.

languages/typescript/packages/cli/README.md

@@ -42,7 +42,7 @@ npx stash auth login
                             └── npx stash status   ← where am I?
 ```
 
-`stash` covers authentication, initialization, EQL install/upgrade/status, schema introspection, and the staged EQL v3 encryption rollout.
+`stash` covers authentication, initialization, EQL install/upgrade/status, schema introspection, the staged EQL v3 encryption rollout, and its reverse: moving encrypted columns back to plaintext (`stash decrypt`).
 
 ---
 
@@ -66,7 +66,7 @@ export default defineConfig({
 
 The CLI loads `.env` files automatically before reading the config, so `process.env` references work without extra setup. The config file is resolved by walking up from the current working directory.
 
-Commands that consume `stash.config.ts`: `eql install`, `eql upgrade`, `eql validate`, `eql status`, `db test-connection`, `schema build`, and `encrypt *`.
+Commands that consume `stash.config.ts`: `eql install`, `eql upgrade`, `eql validate`, `eql status`, `db test-connection`, `schema build`, `encrypt *`, and `decrypt *`.
 
 ---
 
@@ -310,6 +310,40 @@ Reads `databaseUrl` from `stash.config.ts`.
 
 ---
 
+### `npx stash decrypt` — moving columns back to plaintext
+
+Leaving CipherStash is the encrypt lifecycle run backwards, with the same safety rails. Decryption is never charged.
+
+```
+decrypt plan      add a plaintext twin per encrypted column (type mapped back from its EQL domain)
+  → deploy dual-writes (app writes the encrypted column AND its plaintext twin)
+decrypt backfill  decrypt every row into the twin — resumable, one ZeroKMS request per chunk
+  → switch reads to the plaintext column and deploy (keep writing BOTH columns)
+decrypt drop      generate the migration that drops the encrypted column and its EQL indexes
+  → apply it, then stop writing the encrypted column
+decrypt status    where each column stands
+```
+
+```bash
+npx stash decrypt plan                                                  # read-only overview
+npx stash decrypt plan --table users                                    # generate the twins
+npx stash decrypt backfill --table users --column email_encrypted
+npx stash decrypt drop --table users --column email_encrypted [--uninstall-eql]
+npx stash decrypt status
+```
+
+`--column` always names the column being migrated away from — the encrypted one here. The plaintext twin defaults to the pairing the reverse migration in progress recorded, then the original name recorded in `.cipherstash/migrations.json`, else the column name without `_encrypted`, else `<column>_plaintext`; override with `--plaintext-column`. Plaintext types: `text*` → `text`, `integer*`/`smallint*`/`bigint*`/`numeric*`/`real*` → the same type, `double*` → `double precision`, `boolean` → `boolean`, `date*` → `date`, `timestamp*` → `timestamptz` (EQL timestamps are instants), `json`/`json_search` → `jsonb`. The backfill casts to whatever type the twin actually has, so you can widen or narrow it in the migration.
+
+**Keep dual-writing until the drop is applied.** Unlike the forward rollout, the application keeps writing both the encrypted column and the plaintext column until the `decrypt drop` migration has run. `decrypt backfill --force` and the drop's coverage check depend on it: if writes to the encrypted column stop early, a value cleared in the plaintext column keeps its old ciphertext, and a re-run of the backfill writes the old value back.
+
+**A forward encryption that was never dropped cannot be reversed.** The backfill refuses a plaintext column that is the source of a forward encryption not yet dropped: finish it with `encrypt drop`, or abandon it by stopping writes to the encrypted column and dropping it. **A plaintext column that already holds values is refused on the first run.** The backfill only fills NULLs, so it refuses such a column until you pass `--force` (overwrite it from the ciphertext; every later run is then forced until a forced pass completes) or `--confirm-plaintext-current` (keep it; only if it was written on every change). `decrypt plan` warns about the same columns.
+
+**Values bound to a lock context are not decrypted.** A value encrypted with an end-user lock context can only be decrypted as that user. The backfill skips those rows, counts them, leaves the twin NULL and reports them; `decrypt drop` refuses while any remain, because dropping the encrypted column would destroy their only copy. Decrypt them in your application on each user's behalf, then re-run the backfill.
+
+The backfill loads your encryption client exactly as `encrypt backfill` does (client file or Prisma Next contract), so export the `CS_*` credentials of the environment that wrote the data.
+
+---
+
 ## Migration mode
 
 Use `eql migration` to add the EQL v3 installation to your migration history instead of applying it directly. The install then ships to every environment through the same migrate step as the rest of your schema.

languages/typescript/packages/cli/scripts/e2e-encrypt.sh

@@ -1,5 +1,6 @@
 #!/usr/bin/env bash
-# End-to-end smoke test for `stash encrypt`.
+# End-to-end smoke test for `stash encrypt`, then `stash decrypt` back to
+# plaintext — the full round trip, verified against the original values.
 #
 # Requires a local Postgres you have superuser on. Creates & destroys
 # `stash_e2e_test`. Requires CipherStash credentials in the environment
@@ -54,5 +55,54 @@ echo "==> 5. Switch application reads to email_encrypted (manual deploy step)"
 
 echo "==> 6. Generate plaintext drop migration"
 "$STASH" encrypt drop --table users --column email --migrations-dir "$(pwd)/drizzle"
+DROP_MIGRATION=$(ls -t "$(pwd)/drizzle"/*_drop_users_email.sql | head -1)
+psql "$DATABASE_URL" -f "$DROP_MIGRATION" >/dev/null
+ORIGINAL=$(psql "$DATABASE_URL" -At -c "SELECT md5(string_agg('user-' || id || '@example.com', ',' ORDER BY id)) FROM users")
+
+# ---------------------------------------------------------------------------
+# Reverse lifecycle: move the column back to plaintext (stash decrypt).
+# ---------------------------------------------------------------------------
+
+echo "==> 7. Plan the plaintext twin (users.email_encrypted → email text)"
+"$STASH" decrypt plan
+"$STASH" decrypt plan --table users --column email_encrypted --migrations-dir "$(pwd)/drizzle"
+ADD_MIGRATION=$(ls -t "$(pwd)/drizzle"/*_add_plaintext_users_email.sql | head -1)
+psql "$DATABASE_URL" -f "$ADD_MIGRATION" >/dev/null
+
+echo "==> 8. Decrypt backfill with interrupt/resume (reverse dual-writes confirmed via flag)"
+"$STASH" decrypt backfill --table users --column email_encrypted --chunk-size 500 --confirm-dual-writes-deployed &
+PID=$!
+sleep 2
+kill -INT "$PID" || true
+wait "$PID" || true
+"$STASH" decrypt backfill --table users --column email_encrypted
+
+REMAINING=$(psql "$DATABASE_URL" -At -c "SELECT count(*) FROM users WHERE email_encrypted IS NOT NULL AND email IS NULL")
+if [ "$REMAINING" != "0" ]; then
+  echo "FAIL: ${REMAINING} rows still undecrypted" >&2
+  exit 1
+fi
+ROUNDTRIP=$(psql "$DATABASE_URL" -At -c "SELECT md5(string_agg(email, ',' ORDER BY id)) FROM users")
+if [ "$ROUNDTRIP" != "$ORIGINAL" ]; then
+  echo "FAIL: decrypted plaintext does not match the original values" >&2
+  exit 1
+fi
+echo "OK: all 5000 rows decrypted back to their original plaintext"
+
+echo "==> 9. Reverse status"
+"$STASH" decrypt status
+
+echo "==> 10. Switch application reads to email (manual deploy step)"
+
+echo "==> 11. Generate and apply the encrypted-column drop"
+"$STASH" decrypt drop --table users --column email_encrypted --migrations-dir "$(pwd)/drizzle"
+DECRYPT_DROP=$(ls -t "$(pwd)/drizzle"/*_drop_encrypted_users_email_encrypted.sql | head -1)
+psql "$DATABASE_URL" -f "$DECRYPT_DROP" >/dev/null
+GONE=$(psql "$DATABASE_URL" -At -c "SELECT count(*) FROM information_schema.columns WHERE table_name = 'users' AND column_name = 'email_encrypted'")
+if [ "$GONE" != "0" ]; then
+  echo "FAIL: email_encrypted still exists" >&2
+  exit 1
+fi
+echo "OK: users is plaintext again"
 
 echo "==> Done."

languages/typescript/packages/cli/src/bin/main.ts

@@ -128,6 +128,11 @@ Commands:
   encrypt backfill     Resumably encrypt plaintext into the encrypted column
   encrypt drop         Generate a migration to drop the plaintext column
 
+  decrypt status       Show per-column reverse-migration status (phase, progress, undecryptable)
+  decrypt plan         Plan (or generate) the plaintext twins for encrypted columns
+  decrypt backfill     Resumably decrypt ciphertext into the plaintext column
+  decrypt drop         Generate a migration to drop the encrypted column
+
   env                  Mint deployment credentials and print them as env vars
 
 Options:
@@ -466,6 +471,92 @@ async function runEncryptCommand(
   }
 }
 
+async function runDecryptCommand(
+  sub: string | undefined,
+  flags: Record<string, boolean>,
+  values: Record<string, string>,
+) {
+  switch (sub) {
+    case 'status': {
+      const { decryptStatusCommand } = await requireStack(
+        () => import('../commands/decrypt/status.js'),
+      )
+      await decryptStatusCommand()
+      break
+    }
+    case 'plan': {
+      const { decryptPlanCommand } = await requireStack(
+        () => import('../commands/decrypt/plan.js'),
+      )
+      await decryptPlanCommand({
+        table: values.table,
+        column: values.column,
+        plaintextColumn: values['plaintext-column'],
+        migrationsDir: values['migrations-dir'],
+        dryRun: flags['dry-run'],
+      })
+      break
+    }
+    case 'backfill': {
+      const table = requireValue(values, 'table')
+      const column = requireValue(values, 'column')
+      const { decryptBackfillCommand } = await requireStack(
+        () => import('../commands/decrypt/backfill.js'),
+      )
+      // Through requireStack: the bound comes from @cipherstash/migrate,
+      // which imports @cipherstash/stack.
+      const { InvalidDecryptChunkSizeError, parseDecryptChunkSize } =
+        await requireStack(
+          () => import('../commands/decrypt/lib/chunk-size.js'),
+        )
+      let chunkSize: number | undefined
+      if (values['chunk-size'] !== undefined) {
+        // Parsed here, where the raw text is still in hand, so the error can
+        // quote exactly what was typed.
+        try {
+          chunkSize = parseDecryptChunkSize(values['chunk-size'])
+        } catch (error) {
+          if (!(error instanceof InvalidDecryptChunkSizeError)) throw error
+          p.log.error(error.message)
+          throw new CliExit(1)
+        }
+      }
+      await decryptBackfillCommand({
+        table,
+        column,
+        plaintextColumn: values['plaintext-column'],
+        pkColumn: values['pk-column'],
+        chunkSize,
+        confirmDualWritesDeployed: flags['confirm-dual-writes-deployed'],
+        force: flags.force,
+        confirmPlaintextCurrent: flags['confirm-plaintext-current'],
+        continueOnUndecryptable: flags['continue-on-undecryptable'],
+      })
+      break
+    }
+    case 'drop': {
+      const table = requireValue(values, 'table')
+      const column = requireValue(values, 'column')
+      const { decryptDropCommand } = await requireStack(
+        () => import('../commands/decrypt/drop.js'),
+      )
+      await decryptDropCommand({
+        table,
+        column,
+        plaintextColumn: values['plaintext-column'],
+        migrationsDir: values['migrations-dir'],
+        uninstallEql: flags['uninstall-eql'],
+      })
+      break
+    }
+    default:
+      p.log.error(`${messages.decrypt.unknownSubcommand}: ${sub ?? '(none)'}`)
+      console.log()
+      console.log(HELP)
+      throw new CliExit(1)
+  }
+}
+
 function requireValue(values: Record<string, string>, key: string): string {
   const v = values[key]
   if (!v) {
@@ -610,6 +701,9 @@ async function dispatch(
     case 'encrypt':
       await runEncryptCommand(subcommand, flags, values)
       break
+    case 'decrypt':
+      await runDecryptCommand(subcommand, flags, values)
+      break
     case 'schema':
       await runSchemaCommand(subcommand, flags, values)
       break