diff --git a/.changeset/stash-decrypt-reverse-migration.md b/.changeset/stash-decrypt-reverse-migration.md new file mode 100644 index 000000000..3bff0401a --- /dev/null +++ b/.changeset/stash-decrypt-reverse-migration.md @@ -0,0 +1,8 @@ +--- +'stash': minor +'@cipherstash/migrate': minor +--- + +`stash decrypt status|plan|backfill|drop` moves encrypted columns back to plaintext — the encrypt lifecycle run in reverse, with the same safety rails. `decrypt plan` adds a nullable plaintext twin per EQL v3 column with its type mapped back from the domain (`text*` → `text`, `timestamp*` → `timestamptz`, `json*` → `jsonb`, …): Drizzle schema lines for `drizzle-kit generate`, or a reviewed SQL migration elsewhere. `decrypt backfill` decrypts in resumable, checkpointed chunks, one `bulkDecrypt` (one ZeroKMS request) per chunk, default 500 rows. It refuses a plaintext column that a forward encryption started from and never dropped: only one migration may be open on a pair of columns, so finish or abandon the forward one first. When that column was already removed outside stash, `decrypt plan` finds it missing and records the forward migration as dropped (marked as observed), so the reverse migration can start. Its first run also refuses a plaintext column that already holds values, because those values may be stale and the backfill only fills NULLs: pass `--force` to overwrite them from the ciphertext or `--confirm-plaintext-current` to keep them. Once `--force` is chosen, or any forced run starts, every later run is forced until a forced pass completes, so an interrupted overwrite is always finished. The application keeps writing both columns until the drop migration is applied. Values bound to an end-user lock context cannot be decrypted with workspace credentials; they are skipped, counted, left NULL and reported, and `decrypt drop` refuses while any remain. `decrypt drop` re-verifies coverage at apply time, drops the indexes built on the column's EQL terms, drops the column, and with `--uninstall-eql` removes EQL v3 when it was the last encrypted column in the database. + +`@cipherstash/migrate`: `runBackfill` now runs on a shared chunked engine (`runChunkedMigration`), which also drives the new `runDecryptBackfill`. Forward behaviour and recorded event details are unchanged. Adds `runDecryptBackfill` (resumes after a failed run; `PlaintextWriteError` keeps decrypted values out of write errors; chunk size capped at `MAX_DECRYPT_CHUNK_SIZE`), `columnHasValues`, `countUndecrypted`, `fetchUndecryptedPage`, `plaintextTypeForEqlDomain`, `columnSqlType`, `listColumnIndexes`, `primaryKeyColumns`, `countEqlColumnsInDatabase`, `removeManifestColumn`, and the reverse event/phase vocabulary (`decrypt_started`, `decrypt_checkpoint`, `decrypted`, `encrypted_dropped`; `decrypt-dual-writing`, `decrypting`, `decrypted`, `encrypted-dropped`), recorded against the encrypted column. `MigrationPhase` and `MigrationEvent` gain those members, so an exhaustive `switch` over them needs the new cases. diff --git a/languages/typescript/packages/cli/README.md b/languages/typescript/packages/cli/README.md index cf68abf9f..c051e2625 100644 --- a/languages/typescript/packages/cli/README.md +++ b/languages/typescript/packages/cli/README.md @@ -42,7 +42,7 @@ npx stash auth login └── npx stash status ← where am I? ``` -`stash` covers authentication, initialization, EQL install/upgrade/status, schema introspection, and the staged EQL v3 encryption rollout. +`stash` covers authentication, initialization, EQL install/upgrade/status, schema introspection, the staged EQL v3 encryption rollout, and its reverse: moving encrypted columns back to plaintext (`stash decrypt`). --- @@ -66,7 +66,7 @@ export default defineConfig({ The CLI loads `.env` files automatically before reading the config, so `process.env` references work without extra setup. The config file is resolved by walking up from the current working directory. -Commands that consume `stash.config.ts`: `eql install`, `eql upgrade`, `eql validate`, `eql status`, `db test-connection`, `schema build`, and `encrypt *`. +Commands that consume `stash.config.ts`: `eql install`, `eql upgrade`, `eql validate`, `eql status`, `db test-connection`, `schema build`, `encrypt *`, and `decrypt *`. --- @@ -310,6 +310,40 @@ Reads `databaseUrl` from `stash.config.ts`. --- +### `npx stash decrypt` — moving columns back to plaintext + +Leaving CipherStash is the encrypt lifecycle run backwards, with the same safety rails. Decryption is never charged. + +``` +decrypt plan add a plaintext twin per encrypted column (type mapped back from its EQL domain) + → deploy dual-writes (app writes the encrypted column AND its plaintext twin) +decrypt backfill decrypt every row into the twin — resumable, one ZeroKMS request per chunk + → switch reads to the plaintext column and deploy (keep writing BOTH columns) +decrypt drop generate the migration that drops the encrypted column and its EQL indexes + → apply it, then stop writing the encrypted column +decrypt status where each column stands +``` + +```bash +npx stash decrypt plan # read-only overview +npx stash decrypt plan --table users # generate the twins +npx stash decrypt backfill --table users --column email_encrypted +npx stash decrypt drop --table users --column email_encrypted [--uninstall-eql] +npx stash decrypt status +``` + +`--column` always names the column being migrated away from — the encrypted one here. The plaintext twin defaults to the pairing the reverse migration in progress recorded, then the original name recorded in `.cipherstash/migrations.json`, else the column name without `_encrypted`, else `_plaintext`; override with `--plaintext-column`. Plaintext types: `text*` → `text`, `integer*`/`smallint*`/`bigint*`/`numeric*`/`real*` → the same type, `double*` → `double precision`, `boolean` → `boolean`, `date*` → `date`, `timestamp*` → `timestamptz` (EQL timestamps are instants), `json`/`json_search` → `jsonb`. The backfill casts to whatever type the twin actually has, so you can widen or narrow it in the migration. + +**Keep dual-writing until the drop is applied.** Unlike the forward rollout, the application keeps writing both the encrypted column and the plaintext column until the `decrypt drop` migration has run. `decrypt backfill --force` and the drop's coverage check depend on it: if writes to the encrypted column stop early, a value cleared in the plaintext column keeps its old ciphertext, and a re-run of the backfill writes the old value back. + +**A forward encryption that was never dropped cannot be reversed.** The backfill refuses a plaintext column that is the source of a forward encryption not yet dropped: finish it with `encrypt drop`, or abandon it by stopping writes to the encrypted column and dropping it. **A plaintext column that already holds values is refused on the first run.** The backfill only fills NULLs, so it refuses such a column until you pass `--force` (overwrite it from the ciphertext; every later run is then forced until a forced pass completes) or `--confirm-plaintext-current` (keep it; only if it was written on every change). `decrypt plan` warns about the same columns. + +**Values bound to a lock context are not decrypted.** A value encrypted with an end-user lock context can only be decrypted as that user. The backfill skips those rows, counts them, leaves the twin NULL and reports them; `decrypt drop` refuses while any remain, because dropping the encrypted column would destroy their only copy. Decrypt them in your application on each user's behalf, then re-run the backfill. + +The backfill loads your encryption client exactly as `encrypt backfill` does (client file or Prisma Next contract), so export the `CS_*` credentials of the environment that wrote the data. + +--- + ## Migration mode Use `eql migration` to add the EQL v3 installation to your migration history instead of applying it directly. The install then ships to every environment through the same migrate step as the rest of your schema. diff --git a/languages/typescript/packages/cli/scripts/e2e-encrypt.sh b/languages/typescript/packages/cli/scripts/e2e-encrypt.sh index 893f51ee4..baf91ca58 100755 --- a/languages/typescript/packages/cli/scripts/e2e-encrypt.sh +++ b/languages/typescript/packages/cli/scripts/e2e-encrypt.sh @@ -1,5 +1,6 @@ #!/usr/bin/env bash -# End-to-end smoke test for `stash encrypt`. +# End-to-end smoke test for `stash encrypt`, then `stash decrypt` back to +# plaintext — the full round trip, verified against the original values. # # Requires a local Postgres you have superuser on. Creates & destroys # `stash_e2e_test`. Requires CipherStash credentials in the environment @@ -54,5 +55,54 @@ echo "==> 5. Switch application reads to email_encrypted (manual deploy step)" echo "==> 6. Generate plaintext drop migration" "$STASH" encrypt drop --table users --column email --migrations-dir "$(pwd)/drizzle" +DROP_MIGRATION=$(ls -t "$(pwd)/drizzle"/*_drop_users_email.sql | head -1) +psql "$DATABASE_URL" -f "$DROP_MIGRATION" >/dev/null +ORIGINAL=$(psql "$DATABASE_URL" -At -c "SELECT md5(string_agg('user-' || id || '@example.com', ',' ORDER BY id)) FROM users") + +# --------------------------------------------------------------------------- +# Reverse lifecycle: move the column back to plaintext (stash decrypt). +# --------------------------------------------------------------------------- + +echo "==> 7. Plan the plaintext twin (users.email_encrypted → email text)" +"$STASH" decrypt plan +"$STASH" decrypt plan --table users --column email_encrypted --migrations-dir "$(pwd)/drizzle" +ADD_MIGRATION=$(ls -t "$(pwd)/drizzle"/*_add_plaintext_users_email.sql | head -1) +psql "$DATABASE_URL" -f "$ADD_MIGRATION" >/dev/null + +echo "==> 8. Decrypt backfill with interrupt/resume (reverse dual-writes confirmed via flag)" +"$STASH" decrypt backfill --table users --column email_encrypted --chunk-size 500 --confirm-dual-writes-deployed & +PID=$! +sleep 2 +kill -INT "$PID" || true +wait "$PID" || true +"$STASH" decrypt backfill --table users --column email_encrypted + +REMAINING=$(psql "$DATABASE_URL" -At -c "SELECT count(*) FROM users WHERE email_encrypted IS NOT NULL AND email IS NULL") +if [ "$REMAINING" != "0" ]; then + echo "FAIL: ${REMAINING} rows still undecrypted" >&2 + exit 1 +fi +ROUNDTRIP=$(psql "$DATABASE_URL" -At -c "SELECT md5(string_agg(email, ',' ORDER BY id)) FROM users") +if [ "$ROUNDTRIP" != "$ORIGINAL" ]; then + echo "FAIL: decrypted plaintext does not match the original values" >&2 + exit 1 +fi +echo "OK: all 5000 rows decrypted back to their original plaintext" + +echo "==> 9. Reverse status" +"$STASH" decrypt status + +echo "==> 10. Switch application reads to email (manual deploy step)" + +echo "==> 11. Generate and apply the encrypted-column drop" +"$STASH" decrypt drop --table users --column email_encrypted --migrations-dir "$(pwd)/drizzle" +DECRYPT_DROP=$(ls -t "$(pwd)/drizzle"/*_drop_encrypted_users_email_encrypted.sql | head -1) +psql "$DATABASE_URL" -f "$DECRYPT_DROP" >/dev/null +GONE=$(psql "$DATABASE_URL" -At -c "SELECT count(*) FROM information_schema.columns WHERE table_name = 'users' AND column_name = 'email_encrypted'") +if [ "$GONE" != "0" ]; then + echo "FAIL: email_encrypted still exists" >&2 + exit 1 +fi +echo "OK: users is plaintext again" echo "==> Done." diff --git a/languages/typescript/packages/cli/src/bin/main.ts b/languages/typescript/packages/cli/src/bin/main.ts index a74b99a0e..47485afc5 100644 --- a/languages/typescript/packages/cli/src/bin/main.ts +++ b/languages/typescript/packages/cli/src/bin/main.ts @@ -128,6 +128,11 @@ Commands: encrypt backfill Resumably encrypt plaintext into the encrypted column encrypt drop Generate a migration to drop the plaintext column + decrypt status Show per-column reverse-migration status (phase, progress, undecryptable) + decrypt plan Plan (or generate) the plaintext twins for encrypted columns + decrypt backfill Resumably decrypt ciphertext into the plaintext column + decrypt drop Generate a migration to drop the encrypted column + env Mint deployment credentials and print them as env vars Options: @@ -466,6 +471,92 @@ async function runEncryptCommand( } } +async function runDecryptCommand( + sub: string | undefined, + flags: Record, + values: Record, +) { + switch (sub) { + case 'status': { + const { decryptStatusCommand } = await requireStack( + () => import('../commands/decrypt/status.js'), + ) + await decryptStatusCommand() + break + } + case 'plan': { + const { decryptPlanCommand } = await requireStack( + () => import('../commands/decrypt/plan.js'), + ) + await decryptPlanCommand({ + table: values.table, + column: values.column, + plaintextColumn: values['plaintext-column'], + migrationsDir: values['migrations-dir'], + dryRun: flags['dry-run'], + }) + break + } + case 'backfill': { + const table = requireValue(values, 'table') + const column = requireValue(values, 'column') + const { decryptBackfillCommand } = await requireStack( + () => import('../commands/decrypt/backfill.js'), + ) + // Through requireStack: the bound comes from @cipherstash/migrate, + // which imports @cipherstash/stack. + const { InvalidDecryptChunkSizeError, parseDecryptChunkSize } = + await requireStack( + () => import('../commands/decrypt/lib/chunk-size.js'), + ) + let chunkSize: number | undefined + if (values['chunk-size'] !== undefined) { + // Parsed here, where the raw text is still in hand, so the error can + // quote exactly what was typed. + try { + chunkSize = parseDecryptChunkSize(values['chunk-size']) + } catch (error) { + if (!(error instanceof InvalidDecryptChunkSizeError)) throw error + p.log.error(error.message) + throw new CliExit(1) + } + } + await decryptBackfillCommand({ + table, + column, + plaintextColumn: values['plaintext-column'], + pkColumn: values['pk-column'], + chunkSize, + confirmDualWritesDeployed: flags['confirm-dual-writes-deployed'], + force: flags.force, + confirmPlaintextCurrent: flags['confirm-plaintext-current'], + continueOnUndecryptable: flags['continue-on-undecryptable'], + }) + break + } + case 'drop': { + const table = requireValue(values, 'table') + const column = requireValue(values, 'column') + const { decryptDropCommand } = await requireStack( + () => import('../commands/decrypt/drop.js'), + ) + await decryptDropCommand({ + table, + column, + plaintextColumn: values['plaintext-column'], + migrationsDir: values['migrations-dir'], + uninstallEql: flags['uninstall-eql'], + }) + break + } + default: + p.log.error(`${messages.decrypt.unknownSubcommand}: ${sub ?? '(none)'}`) + console.log() + console.log(HELP) + throw new CliExit(1) + } +} + function requireValue(values: Record, key: string): string { const v = values[key] if (!v) { @@ -610,6 +701,9 @@ async function dispatch( case 'encrypt': await runEncryptCommand(subcommand, flags, values) break + case 'decrypt': + await runDecryptCommand(subcommand, flags, values) + break case 'schema': await runSchemaCommand(subcommand, flags, values) break diff --git a/languages/typescript/packages/cli/src/cli/registry.ts b/languages/typescript/packages/cli/src/cli/registry.ts index 746c68e1c..1e44415e3 100644 --- a/languages/typescript/packages/cli/src/cli/registry.ts +++ b/languages/typescript/packages/cli/src/cli/registry.ts @@ -87,6 +87,12 @@ const COLUMN_FLAG: Flag = { value: '', description: 'Target column.', } +const PLAINTEXT_COLUMN_FLAG: Flag = { + name: '--plaintext-column', + value: '', + description: + "The plaintext twin. Default: the pairing the reverse migration in progress recorded, else the original name from .cipherstash/migrations.json, else the column name without its '_encrypted' suffix, else '_plaintext'.", +} /** * The registry. Ordered as it should render in help. Populated from the existing @@ -634,6 +640,148 @@ export const registry: CommandGroup[] = [ }, ], }, + { + title: 'Decrypt', + commands: [ + { + // Zero-arg like `encrypt status` — it reads every EQL v3 column. + name: 'decrypt status', + summary: + 'Show per-column reverse-migration status (phase, progress, undecryptable)', + }, + { + name: 'decrypt plan', + summary: 'Plan (or generate) the plaintext twins for encrypted columns', + long: [ + 'Step one of moving encrypted columns back to plaintext. With no flags,', + 'lists every EQL v3 column in the current schema, the plaintext twin it', + 'would reverse into (type mapped back from its EQL domain), and the next', + 'command to run. With --table, produces the change that adds the twins:', + 'Drizzle schema lines for drizzle-kit generate, or a reviewed SQL', + 'migration elsewhere. Twins are added nullable; nothing is applied.', + ].join('\n'), + examples: [ + 'decrypt plan', + 'decrypt plan --table users', + 'decrypt plan --table users --column email_encrypted --dry-run', + ], + flags: [ + { + ...TABLE_FLAG, + description: 'Generate for this table (omit for the overview).', + }, + { + ...COLUMN_FLAG, + description: + 'One encrypted column on --table (default: every EQL v3 column on it).', + }, + PLAINTEXT_COLUMN_FLAG, + { + name: '--migrations-dir', + value: '', + description: + 'Directory to write the migration into (non-Drizzle projects).', + }, + DRY_RUN_FLAG, + ], + }, + { + name: 'decrypt backfill', + summary: 'Resumably decrypt ciphertext into the plaintext column', + long: [ + 'The reverse of `encrypt backfill`, on the same engine: chunked,', + 'resumable, idempotent, one bulkDecrypt (one ZeroKMS request) per chunk.', + 'Values that cannot be decrypted with these credentials, typically ones', + 'bound to an end-user lock context, are skipped, counted, left NULL and', + 'reported; `decrypt drop` refuses while any remain.', + '', + 'It refuses a plaintext column that a forward encryption started from', + 'and never dropped: finish or abandon that migration first. The first', + 'run also refuses a plaintext column that already holds values: the', + 'backfill only fills NULLs, so stale values would survive. Pass --force', + 'to overwrite them from the ciphertext (every later run is then forced', + 'until a forced pass completes), or --confirm-plaintext-current to keep', + 'them. Keep writing both columns until the drop is applied.', + ].join('\n'), + examples: [ + 'decrypt backfill --table users --column email_encrypted', + 'decrypt backfill --table users --column email_encrypted --plaintext-column email', + ], + flags: [ + TABLE_FLAG, + { + ...COLUMN_FLAG, + description: 'The encrypted (EQL v3) column to decrypt.', + }, + PLAINTEXT_COLUMN_FLAG, + { + name: '--pk-column', + value: '', + description: 'Primary-key column used to page through rows.', + }, + { + name: '--chunk-size', + value: '', + description: + 'Rows decrypted per batch — one ZeroKMS request each. A whole number from 1 to 32767.', + default: '500', + }, + { + name: '--confirm-dual-writes-deployed', + description: + 'Assert the app writes both the encrypted and plaintext columns (safety gate).', + }, + { + name: '--force', + description: + 'Re-decrypt rows whose plaintext is already set and overwrite it. Safe only while dual-writes are live.', + }, + { + name: '--confirm-plaintext-current', + description: + 'First run only: keep the values a plaintext column already holds, asserting they are current.', + }, + { + name: '--continue-on-undecryptable', + description: + 'Carry on when the first batch decrypts nothing, recording those rows as skipped.', + }, + ], + }, + { + name: 'decrypt drop', + summary: 'Generate a migration to drop the encrypted column', + long: [ + 'Runs from the `decrypted` phase. Verifies live coverage (no row with', + 'ciphertext set and plaintext NULL), then writes a migration that', + 're-checks coverage at apply time, drops the indexes built on the', + "column's EQL terms, and drops the column. Not applied for you.", + ].join('\n'), + examples: [ + 'decrypt drop --table users --column email_encrypted', + 'decrypt drop --table users --column email_encrypted --uninstall-eql', + ], + flags: [ + TABLE_FLAG, + { + ...COLUMN_FLAG, + description: 'The encrypted (EQL v3) column to drop.', + }, + PLAINTEXT_COLUMN_FLAG, + { + name: '--migrations-dir', + value: '', + description: 'Directory to write the drop migration into.', + }, + { + name: '--uninstall-eql', + description: + 'Also uninstall EQL v3 (its schemas and public.eql_v3_* domains) in the same migration — only when this is the last EQL v3 column in the database (re-checked at apply time).', + }, + ], + }, + ], + }, { title: 'Deployment', commands: [ diff --git a/languages/typescript/packages/cli/src/commands/decrypt/__tests__/chunk-size.test.ts b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/chunk-size.test.ts new file mode 100644 index 000000000..8743009f1 --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/chunk-size.test.ts @@ -0,0 +1,47 @@ +/** + * `decrypt backfill --chunk-size`: what counts as a chunk size. The upper + * bound is the one that matters most here: each row binds two parameters in + * the chunk's write, and Postgres caps a statement at 65535, so a larger + * chunk would fail only after its ZeroKMS request. + */ + +import { MAX_DECRYPT_CHUNK_SIZE } from '@cipherstash/migrate' +import { describe, expect, it } from 'vitest' +import { + InvalidDecryptChunkSizeError, + parseDecryptChunkSize, +} from '../lib/chunk-size.js' + +describe('parseDecryptChunkSize', () => { + it('caps at 32767 rows, two bound parameters each', () => { + expect(MAX_DECRYPT_CHUNK_SIZE).toBe(32767) + expect(MAX_DECRYPT_CHUNK_SIZE * 2).toBeLessThanOrEqual(65535) + }) + + it.each([ + ['500', 500], + ['1', 1], + [' 42 ', 42], + ['32767', 32767], + ])('accepts %j as %d', (input, expected) => { + expect(parseDecryptChunkSize(input)).toBe(expected) + }) + + it.each(['abc', '', '0', '-5', '1.5', '1e3', '0x10', 'Infinity', '32768'])( + 'rejects the text %j', + (input) => { + expect(() => parseDecryptChunkSize(input)).toThrow( + InvalidDecryptChunkSizeError, + ) + }, + ) + + it('quotes what was typed and says why there is a ceiling', () => { + expect(() => parseDecryptChunkSize('1e3')).toThrow( + '--chunk-size must be a whole number of rows from 1 to 32767 (got "1e3").', + ) + expect(() => parseDecryptChunkSize('32768')).toThrow( + 'from 1 to 32767 (got "32768"). Each row binds two parameters in the write, and Postgres allows 65535 per statement.', + ) + }) +}) diff --git a/languages/typescript/packages/cli/src/commands/decrypt/__tests__/decrypt-backfill.test.ts b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/decrypt-backfill.test.ts new file mode 100644 index 000000000..b94bcd367 --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/decrypt-backfill.test.ts @@ -0,0 +1,513 @@ +/** + * `stash decrypt backfill` — the command shell around `runDecryptBackfill`. + * The engine itself is covered against real Postgres in + * `@cipherstash/migrate`'s `decrypt.integration.test.ts`; this pins what the + * command adds: the twin-must-exist guard, the reverse dual-write gate, the + * undecryptable-row report, and which errors are allowed to print verbatim. + */ + +import { + type DecryptBackfillResult, + PlaintextWriteError, + UndecryptableBatchError, +} from '@cipherstash/migrate' +import { beforeEach, describe, expect, it, vi } from 'vitest' +import type { DecryptPair, TwinHazard } from '../lib/resolve-pair.js' + +const releaseMock = vi.hoisted(() => vi.fn()) +vi.mock('pg', () => ({ + default: { + Pool: class { + connect = vi.fn(async () => ({ release: releaseMock, query: vi.fn() })) + end = vi.fn(async () => {}) + }, + }, +})) + +const migrateMocks = vi.hoisted(() => ({ + installMigrationsSchema: vi.fn(async () => {}), + appendEvent: vi.fn(async () => {}), + runDecryptBackfill: vi.fn( + async (): Promise => ({ + resumed: false, + rowsProcessed: 10, + rowsTotal: 10, + rowsDecrypted: 10, + rowsSkipped: 0, + skippedByCode: {}, + nullPlaintextPks: [], + completed: true, + }), + ), +})) +vi.mock('@cipherstash/migrate', async (importOriginal) => ({ + ...(await importOriginal()), + ...migrateMocks, +})) + +const resolveMock = vi.hoisted(() => vi.fn()) +const hazardMock = vi.hoisted(() => + vi.fn(async (): Promise => null), +) +vi.mock('../lib/resolve-pair.js', async (importOriginal) => ({ + ...(await importOriginal()), + resolveDecryptPair: resolveMock, + assessExistingTwin: hazardMock, +})) + +const decryptionClient = vi.hoisted(() => ({ bulkDecrypt: vi.fn() })) +vi.mock('@/commands/encrypt/context.js', () => ({ + loadEncryptionContext: vi.fn(async () => ({ + client: decryptionClient, + tables: new Map(), + })), + requireTable: vi.fn(), +})) +vi.mock('@/commands/encrypt/lib/db-readers.js', () => ({ + detectPkColumn: vi.fn(async () => 'id'), +})) +vi.mock('@clack/prompts', () => ({ + intro: vi.fn(), + outro: vi.fn(), + note: vi.fn(), + confirm: vi.fn(async () => true), + isCancel: vi.fn(() => false), + log: { + info: vi.fn(), + success: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + step: vi.fn(), + }, +})) +vi.mock('@/config/index.js', () => ({ + loadStashConfig: vi.fn(async () => ({ databaseUrl: 'postgres://test' })), +})) +vi.mock('@/commands/init/utils.js', () => ({ + detectPackageManager: vi.fn(() => 'npm'), + runnerCommand: vi.fn((_pm: string, ref: string) => `npx ${ref}`), +})) + +import * as p from '@clack/prompts' +import { decryptBackfillCommand, skippedReport } from '../backfill.js' + +function pair(overrides: Partial = {}): DecryptPair { + return { + table: 'users', + encryptedColumn: 'email_encrypted', + domain: 'eql_v3_text_search', + mapping: { domain: 'eql_v3_text_search', family: 'text', sqlType: 'text' }, + plaintextColumn: 'email', + via: 'convention', + plaintextSqlType: 'text', + state: null, + // A recorded reverse event means a lifecycle in progress, unless a test + // says otherwise. + lifecycleActive: overrides.state !== undefined && overrides.state !== null, + forwardState: null, + ...overrides, + } +} + +const reverseInFlight = { + id: '1', + tableName: 'users', + columnName: 'email_encrypted', + event: 'decrypt_checkpoint' as const, + phase: 'decrypting' as const, + cursorValue: '5', + rowsProcessed: 5, + rowsTotal: 10, + details: { plaintextColumn: 'email' }, + createdAt: new Date(0), +} + +function spyExit() { + return vi + .spyOn(process, 'exit') + .mockImplementation((() => undefined) as never) +} + +describe('decrypt backfill', () => { + beforeEach(() => { + vi.clearAllMocks() + resolveMock.mockResolvedValue(pair({ state: reverseInFlight })) + hazardMock.mockResolvedValue(null) + vi.mocked(p.confirm).mockResolvedValue(true) + }) + + it('decrypts into the twin with its declared type, 500 rows per ZeroKMS request by default', async () => { + resolveMock.mockResolvedValue( + pair({ + state: reverseInFlight, + plaintextSqlType: 'character varying(64)', + }), + ) + + await decryptBackfillCommand({ table: 'users', column: 'email_encrypted' }) + + expect(migrateMocks.runDecryptBackfill).toHaveBeenCalledWith( + expect.objectContaining({ + decryptionClient, + tableName: 'users', + encryptedColumn: 'email_encrypted', + plaintextColumn: 'email', + plaintextSqlType: 'character varying(64)', + pkColumn: 'id', + chunkSize: 500, + }), + ) + }) + + it('refuses to start before the plaintext twin exists, pointing at decrypt plan', async () => { + resolveMock.mockResolvedValue(pair({ plaintextSqlType: null })) + const exitSpy = spyExit() + + await decryptBackfillCommand({ table: 'users', column: 'email_encrypted' }) + + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining( + 'stash decrypt plan --table users --column email_encrypted', + ), + ) + expect(migrateMocks.runDecryptBackfill).not.toHaveBeenCalled() + expect(exitSpy).toHaveBeenCalledWith(1) + exitSpy.mockRestore() + }) + + it('non-interactive first run: refuses without --confirm-dual-writes-deployed', async () => { + resolveMock.mockResolvedValue(pair({ state: null })) + + await decryptBackfillCommand({ table: 'users', column: 'email_encrypted' }) + + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('--confirm-dual-writes-deployed'), + ) + expect(migrateMocks.appendEvent).not.toHaveBeenCalled() + expect(migrateMocks.runDecryptBackfill).not.toHaveBeenCalled() + }) + + it('records the reverse dual-write gate, with the pairing, on first acceptance', async () => { + resolveMock.mockResolvedValue(pair({ state: null })) + + await decryptBackfillCommand({ + table: 'users', + column: 'email_encrypted', + confirmDualWritesDeployed: true, + }) + + expect(migrateMocks.appendEvent).toHaveBeenCalledWith(expect.anything(), { + tableName: 'users', + columnName: 'email_encrypted', + event: 'decrypt_dual_writing', + phase: 'decrypt-dual-writing', + details: { plaintextColumn: 'email' }, + }) + expect(migrateMocks.runDecryptBackfill).toHaveBeenCalled() + }) + + it('reports undecryptable rows as a lock-context limitation, with codes and the drop consequence', async () => { + migrateMocks.runDecryptBackfill.mockResolvedValueOnce({ + resumed: false, + rowsProcessed: 10, + rowsTotal: 10, + rowsDecrypted: 7, + rowsSkipped: 3, + skippedByCode: { UNKNOWN: 3 }, + nullPlaintextPks: [], + completed: true, + }) + + await decryptBackfillCommand({ table: 'users', column: 'email_encrypted' }) + + const warning = vi + .mocked(p.log.warn) + .mock.calls.map(([message]) => message) + .find((message) => message.includes('could not be decrypted')) + expect(warning).toContain('3 row(s) of users.email_encrypted') + expect(warning).toContain('3 × UNKNOWN') + expect(warning).toContain('lock context') + expect(warning).toContain('`stash decrypt drop` refuses') + expect(p.outro).toHaveBeenCalledWith( + 'Decrypt backfill complete. 7 rows decrypted, 3 skipped.', + ) + }) + + it('prints an UndecryptableBatchError verbatim — it carries counts and codes only', async () => { + migrateMocks.runDecryptBackfill.mockRejectedValueOnce( + new UndecryptableBatchError(500, { UNKNOWN: 500 }), + ) + const exitSpy = spyExit() + + await decryptBackfillCommand({ table: 'users', column: 'email_encrypted' }) + + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('None of the 500 row(s) in the first batch'), + ) + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('--continue-on-undecryptable'), + ) + expect(exitSpy).toHaveBeenCalledWith(1) + exitSpy.mockRestore() + }) + + it('suppresses the text of any other error', async () => { + migrateMocks.runDecryptBackfill.mockRejectedValueOnce( + new Error('upstream said: alice@example.com'), + ) + const exitSpy = spyExit() + + await decryptBackfillCommand({ table: 'users', column: 'email_encrypted' }) + + const messages = vi.mocked(p.log.error).mock.calls.map(([m]) => m) + expect(messages.join('\n')).not.toContain('alice@example.com') + expect(messages).toContain( + 'Decrypt backfill failed (Error). Re-run with diagnostic logging if you need details.', + ) + exitSpy.mockRestore() + }) + + it('refuses a twin whose forward migration was never dropped, even with --force, recording nothing', async () => { + resolveMock.mockResolvedValue( + pair({ + state: null, + forwardState: { + ...reverseInFlight, + columnName: 'email', + event: 'backfilled', + phase: 'backfilled', + details: null, + }, + }), + ) + const exitSpy = spyExit() + + await decryptBackfillCommand({ + table: 'users', + column: 'email_encrypted', + confirmDualWritesDeployed: true, + force: true, + }) + + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining("still open at 'backfilled'"), + ) + expect(migrateMocks.appendEvent).not.toHaveBeenCalled() + expect(migrateMocks.runDecryptBackfill).not.toHaveBeenCalled() + expect(exitSpy).toHaveBeenCalledWith(1) + exitSpy.mockRestore() + }) + + describe('a plaintext twin that already holds values (first run)', () => { + const prepopulated: TwinHazard = { kind: 'prepopulated' } + + beforeEach(() => { + resolveMock.mockResolvedValue(pair({ state: null })) + }) + + it('refuses it, recording nothing and decrypting nothing', async () => { + hazardMock.mockResolvedValue(prepopulated) + const exitSpy = spyExit() + + await decryptBackfillCommand({ + table: 'users', + column: 'email_encrypted', + confirmDualWritesDeployed: true, + }) + + const message = vi.mocked(p.log.error).mock.calls[0]?.[0] ?? '' + expect(message).toContain('users.email already holds values') + expect(message).toContain('--force') + expect(message).toContain('--confirm-plaintext-current') + expect(migrateMocks.appendEvent).not.toHaveBeenCalled() + expect(migrateMocks.runDecryptBackfill).not.toHaveBeenCalled() + expect(exitSpy).toHaveBeenCalledWith(1) + exitSpy.mockRestore() + }) + + it('--force overwrites it from the ciphertext, and the gate records the choice', async () => { + hazardMock.mockResolvedValue(prepopulated) + + await decryptBackfillCommand({ + table: 'users', + column: 'email_encrypted', + confirmDualWritesDeployed: true, + force: true, + }) + + expect(migrateMocks.appendEvent).toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ + event: 'decrypt_dual_writing', + details: { + plaintextColumn: 'email', + existingPlaintext: 'prepopulated', + acknowledgedWith: 'force', + }, + }), + ) + expect(migrateMocks.runDecryptBackfill).toHaveBeenCalledWith( + expect.objectContaining({ force: true }), + ) + }) + + it('--confirm-plaintext-current keeps it and fills only the NULLs', async () => { + hazardMock.mockResolvedValue(prepopulated) + + await decryptBackfillCommand({ + table: 'users', + column: 'email_encrypted', + confirmDualWritesDeployed: true, + confirmPlaintextCurrent: true, + }) + + expect(p.log.warn).toHaveBeenCalledWith( + expect.stringContaining('Proceeding under --confirm-plaintext-current'), + ) + expect(migrateMocks.appendEvent).toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ + details: expect.objectContaining({ + existingPlaintext: 'prepopulated', + acknowledgedWith: 'confirm-plaintext-current', + }), + }), + ) + expect(migrateMocks.runDecryptBackfill).toHaveBeenCalledWith( + expect.not.objectContaining({ force: true }), + ) + }) + + it('checks the twin before the --force prompt, so cancelling it records nothing', async () => { + hazardMock.mockResolvedValue(prepopulated) + vi.mocked(p.confirm).mockResolvedValueOnce(false) + + await decryptBackfillCommand({ + table: 'users', + column: 'email_encrypted', + confirmDualWritesDeployed: true, + force: true, + }) + + // Had the gate's event been recorded first, the next run would treat + // the twin's values as this lifecycle's own and skip the check. + expect(hazardMock).toHaveBeenCalled() + expect(migrateMocks.appendEvent).not.toHaveBeenCalled() + expect(migrateMocks.runDecryptBackfill).not.toHaveBeenCalled() + }) + }) + + it('a finished earlier lifecycle does not pass the dual-write gate', async () => { + resolveMock.mockResolvedValue( + pair({ + state: { + ...reverseInFlight, + event: 'encrypted_dropped', + phase: 'encrypted-dropped', + }, + lifecycleActive: false, + }), + ) + + await decryptBackfillCommand({ table: 'users', column: 'email_encrypted' }) + + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('--confirm-dual-writes-deployed'), + ) + expect(migrateMocks.runDecryptBackfill).not.toHaveBeenCalled() + }) + + it('--force says it is safe only while dual-writes are live', async () => { + await decryptBackfillCommand({ + table: 'users', + column: 'email_encrypted', + force: true, + }) + + expect(p.log.warn).toHaveBeenCalledWith( + expect.stringContaining('Safe only while dual-writes are live'), + ) + const warnings = vi.mocked(p.log.warn).mock.calls.map(([m]) => m) + expect(warnings.join('\n')).not.toContain('not destructive') + }) + + it('tells the application to keep writing both columns until the drop is applied', async () => { + await decryptBackfillCommand({ table: 'users', column: 'email_encrypted' }) + + const steps = vi + .mocked(p.note) + .mock.calls.find(([, title]) => + String(title).startsWith('Next steps'), + )?.[0] + expect(steps).toContain( + 'Keep writing BOTH email_encrypted and email on every insert and update until the drop migration is applied', + ) + expect(steps).not.toMatch(/stop writing email_encrypted once reads/) + }) + + it('prints a PlaintextWriteError verbatim — it carries names and a SQLSTATE only', async () => { + migrateMocks.runDecryptBackfill.mockRejectedValueOnce( + new PlaintextWriteError('22P02', 'email', 'integer', undefined), + ) + const exitSpy = spyExit() + + await decryptBackfillCommand({ table: 'users', column: 'email_encrypted' }) + + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('"email" (integer): SQLSTATE 22P02'), + ) + exitSpy.mockRestore() + }) + + it('reports rows that decrypt to null apart from lock-context failures, with their keys', async () => { + migrateMocks.runDecryptBackfill.mockResolvedValueOnce({ + resumed: false, + rowsProcessed: 10, + rowsTotal: 10, + rowsDecrypted: 7, + rowsSkipped: 3, + skippedByCode: { UNKNOWN: 1, NULL_PLAINTEXT: 2 }, + nullPlaintextPks: ['4', '9'], + completed: true, + }) + + await decryptBackfillCommand({ table: 'users', column: 'email_encrypted' }) + + const warning = + vi + .mocked(p.log.warn) + .mock.calls.map(([message]) => message) + .find((message) => message.includes('could not be decrypted')) ?? '' + const [lockBound, nulls] = warning.split('\n\n') + expect(lockBound).toContain('1 row(s) of users.email_encrypted') + expect(lockBound).toContain('1 × UNKNOWN') + expect(lockBound).not.toContain('NULL_PLAINTEXT') + expect(nulls).toContain('2 row(s) of users.email_encrypted') + expect(nulls).toContain('decrypts to an empty (null) value') + expect(nulls).toContain('not a credentials or lock-context problem') + expect(nulls).toContain('Primary keys: 4, 9') + }) +}) + +describe('skippedReport', () => { + it('does not blame credentials for skips carried over from an earlier run whose codes it does not know', () => { + // Run 1 skipped 10 NULL_PLAINTEXT rows and was interrupted. Run 2 resumes: + // `rowsSkipped` is restored cumulatively from the checkpoint, while + // `skippedByCode` and `nullPlaintextPks` cover this run only, so nothing + // here says those 10 rows failed to decrypt. + const report = skippedReport( + 'users', + { + encryptedColumn: 'email_encrypted', + plaintextColumn: 'email', + plaintextSqlType: 'text', + }, + { rowsSkipped: 10, skippedByCode: {}, nullPlaintextPks: [] }, + ) + + expect(report).not.toMatch( + /10 row\(s\) of users\.email_encrypted could not be decrypted with these credentials/, + ) + expect(report).not.toContain('lock context') + }) +}) diff --git a/languages/typescript/packages/cli/src/commands/decrypt/__tests__/decrypt-drop-plan.test.ts b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/decrypt-drop-plan.test.ts new file mode 100644 index 000000000..d1f34abd9 --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/decrypt-drop-plan.test.ts @@ -0,0 +1,634 @@ +/** + * `stash decrypt drop` and `stash decrypt plan` — the two commands that + * generate migrations for the reverse lifecycle. + * + * `decrypt drop` is the one irreversible step: once the encrypted column is + * gone, any row whose plaintext twin was never filled has lost its only copy. + * So these tests pin its gates (phase, recorded pairing, live coverage) and + * the apply-time re-check the generated SQL carries, the same way + * `encrypt-v3.test.ts` pins `encrypt drop`. + */ + +import type { ColumnIndexInfo, MigrationStateRow } from '@cipherstash/migrate' +import { beforeEach, describe, expect, it, vi } from 'vitest' +import type { DecryptPair } from '../lib/resolve-pair.js' + +const queryMock = vi.hoisted(() => + vi.fn(async (_sql: string): Promise<{ rows: unknown[] }> => ({ rows: [] })), +) +vi.mock('pg', () => ({ + default: { + Client: class { + connect = vi.fn(async () => {}) + end = vi.fn(async () => {}) + query = queryMock + }, + }, +})) + +const migrateMocks = vi.hoisted(() => ({ + countUndecrypted: vi.fn(async () => 0), + listColumnIndexes: vi.fn(async (): Promise => []), + countEqlColumnsInDatabase: vi.fn(async () => 1), + appendEvent: vi.fn(async () => {}), + removeManifestColumn: vi.fn(async () => true), + columnHasValues: vi.fn(async () => false), + readManifest: vi.fn(async (): Promise => null), + listEncryptedColumns: vi.fn(async () => [ + { column: 'email_encrypted', domain: 'eql_v3_text_search', version: 3 }, + ]), +})) +vi.mock('@cipherstash/migrate', async (importOriginal) => ({ + ...(await importOriginal()), + ...migrateMocks, +})) + +const resolveMock = vi.hoisted(() => vi.fn()) +vi.mock('../lib/resolve-pair.js', async (importOriginal) => ({ + ...(await importOriginal()), + resolveDecryptPair: resolveMock, +})) + +vi.mock('@clack/prompts', () => ({ + intro: vi.fn(), + outro: vi.fn(), + note: vi.fn(), + confirm: vi.fn(async () => true), + isCancel: vi.fn(() => false), + log: { + info: vi.fn(), + success: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + step: vi.fn(), + }, +})) +vi.mock('@/config/index.js', () => ({ + loadStashConfig: vi.fn(async () => ({ databaseUrl: 'postgres://test' })), +})) +vi.mock('@/commands/encrypt/context.js', () => ({ + loadEncryptionContext: vi.fn(), + requireTable: vi.fn(), +})) +const detectMocks = vi.hoisted(() => ({ + detectDrizzle: vi.fn(() => false), + detectPrismaNext: vi.fn(() => false), +})) +vi.mock('@/commands/db/detect.js', () => detectMocks) +vi.mock('@/commands/init/utils.js', () => ({ + detectPackageManager: vi.fn(() => 'npm'), + runnerCommand: vi.fn((_pm: string, ref: string) => `npx ${ref}`), +})) +vi.mock('@/commands/encrypt/drizzle-helper.js', () => ({ + scaffoldDrizzleMigration: vi.fn(async ({ name }: { name: string }) => ({ + path: `drizzle/${name}.sql`, + })), +})) +const writeFileMock = vi.hoisted(() => vi.fn()) +// Only the default export's writers are faked (migration-file.ts uses them); +// the named exports stay real because `@cipherstash/eql/sql` reads the real +// uninstall bundle through them — the `--uninstall-eql` test embeds it. +vi.mock('node:fs', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + default: { ...actual, mkdirSync: vi.fn(), writeFileSync: writeFileMock }, + } +}) + +import * as p from '@clack/prompts' +import { BackfillConfigError } from '@/commands/encrypt/backfill.js' +import { decryptDropCommand } from '../drop.js' +import { buildAddPlaintextSql, decryptPlanCommand } from '../plan.js' + +function pair(overrides: Partial = {}): DecryptPair { + return { + table: 'users', + encryptedColumn: 'email_encrypted', + domain: 'eql_v3_text_search', + mapping: { domain: 'eql_v3_text_search', family: 'text', sqlType: 'text' }, + plaintextColumn: 'email', + via: 'recorded', + plaintextSqlType: 'text', + state: decrypted('email'), + // A recorded reverse event means a lifecycle in progress, unless a test + // says otherwise. + lifecycleActive: 'state' in overrides ? overrides.state !== null : true, + forwardState: null, + ...overrides, + } +} + +/** A forward event recorded under the plaintext twin's name. */ +function forward(phase: MigrationStateRow['phase']): MigrationStateRow { + return { + id: '3', + tableName: 'users', + columnName: 'email', + event: phase === 'backfilled' ? 'backfilled' : 'dual_writing', + phase, + cursorValue: null, + rowsProcessed: null, + rowsTotal: null, + details: null, + createdAt: new Date(0), + } +} + +function decrypted(plaintextColumn?: string): MigrationStateRow { + return { + id: '9', + tableName: 'users', + columnName: 'email_encrypted', + event: 'decrypted', + phase: 'decrypted', + cursorValue: '100', + rowsProcessed: 100, + rowsTotal: 100, + details: plaintextColumn ? { plaintextColumn, rowsSkipped: 0 } : {}, + createdAt: new Date(0), + } +} + +function spyExit() { + return vi + .spyOn(process, 'exit') + .mockImplementation((() => undefined) as never) +} + +const writtenSql = () => writeFileMock.mock.calls[0]?.[1] as string | undefined + +describe('decrypt drop', () => { + beforeEach(() => { + vi.clearAllMocks() + resolveMock.mockResolvedValue(pair()) + migrateMocks.countUndecrypted.mockResolvedValue(0) + migrateMocks.listColumnIndexes.mockResolvedValue([]) + migrateMocks.countEqlColumnsInDatabase.mockResolvedValue(1) + migrateMocks.readManifest.mockResolvedValue(null) + }) + + it('verifies coverage against the RECORDED twin and drops the encrypted column', async () => { + await decryptDropCommand({ table: 'users', column: 'email_encrypted' }) + + expect(migrateMocks.countUndecrypted).toHaveBeenCalledWith( + expect.anything(), + 'users', + 'email_encrypted', + 'email', + ) + const sql = writtenSql() ?? '' + expect(sql).toContain('LOCK TABLE "users" IN ACCESS EXCLUSIVE MODE') + expect(sql).toContain('"email_encrypted" IS NOT NULL') + expect(sql).toContain('"email" IS NULL') + expect(sql).toContain('RAISE EXCEPTION') + expect(sql).toMatch( + /EXECUTE 'ALTER TABLE "users" DROP COLUMN "email_encrypted"'/, + ) + // The plaintext twin is never touched. + expect(sql).not.toMatch(/DROP COLUMN "email"'/) + expect(migrateMocks.appendEvent).toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ + tableName: 'users', + columnName: 'email_encrypted', + event: 'encrypted_dropped', + phase: 'encrypted-dropped', + details: expect.objectContaining({ plaintextColumn: 'email' }), + }), + ) + }) + + it('drops the EQL-term indexes explicitly, inside the same atomic block, and names them', async () => { + migrateMocks.listColumnIndexes.mockResolvedValue([ + { + schema: 'public', + name: 'users_email_ord', + definition: + 'CREATE INDEX users_email_ord ON public.users USING btree (eql_v3.ord_term(email_encrypted))', + }, + ]) + + await decryptDropCommand({ table: 'users', column: 'email_encrypted' }) + + const sql = writtenSql() ?? '' + const indexAt = sql.indexOf( + `EXECUTE 'DROP INDEX IF EXISTS "public"."users_email_ord"'`, + ) + expect(indexAt).toBeGreaterThan(sql.indexOf('RAISE EXCEPTION')) + expect(indexAt).toBeLessThan(sql.indexOf(`EXECUTE 'ALTER TABLE`)) + expect(p.log.info).toHaveBeenCalledWith( + expect.stringContaining('public.users_email_ord'), + ) + }) + + it('refuses while any row still has ciphertext and no plaintext — naming lock contexts as a cause', async () => { + migrateMocks.countUndecrypted.mockResolvedValue(3) + const exitSpy = spyExit() + + await decryptDropCommand({ table: 'users', column: 'email_encrypted' }) + + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('3 row(s)'), + ) + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('lock context'), + ) + // A value cleared in the plaintext column after writes to the encrypted + // column stopped must not be "fixed" by a backfill that restores it. + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining( + 'Do NOT re-run the backfill for these: it would write the old value back', + ), + ) + expect(writeFileMock).not.toHaveBeenCalled() + expect(migrateMocks.appendEvent).not.toHaveBeenCalled() + expect(exitSpy).toHaveBeenCalledWith(1) + exitSpy.mockRestore() + }) + + it("requires the reverse phase 'decrypted'", async () => { + resolveMock.mockResolvedValue( + pair({ state: { ...decrypted('email'), phase: 'decrypting' } }), + ) + const exitSpy = spyExit() + + await decryptDropCommand({ table: 'users', column: 'email_encrypted' }) + + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining("Must be 'decrypted'"), + ) + expect(migrateMocks.countUndecrypted).not.toHaveBeenCalled() + expect(writeFileMock).not.toHaveBeenCalled() + expect(exitSpy).toHaveBeenCalledWith(1) + exitSpy.mockRestore() + }) + + it('refuses without a RECORDED pairing, even when a twin name could be guessed', async () => { + resolveMock.mockResolvedValue(pair({ state: decrypted(undefined) })) + const exitSpy = spyExit() + + await decryptDropCommand({ table: 'users', column: 'email_encrypted' }) + + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('no plaintext column is recorded'), + ) + expect(writeFileMock).not.toHaveBeenCalled() + expect(exitSpy).toHaveBeenCalledWith(1) + exitSpy.mockRestore() + }) + + it('refuses when the recorded twin no longer exists', async () => { + resolveMock.mockResolvedValue(pair({ plaintextSqlType: null })) + const exitSpy = spyExit() + + await decryptDropCommand({ table: 'users', column: 'email_encrypted' }) + + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('no longer exists'), + ) + expect(writeFileMock).not.toHaveBeenCalled() + exitSpy.mockRestore() + }) + + it('--uninstall-eql: refused while other EQL v3 columns remain', async () => { + migrateMocks.countEqlColumnsInDatabase.mockResolvedValue(4) + const exitSpy = spyExit() + + await decryptDropCommand({ + table: 'users', + column: 'email_encrypted', + uninstallEql: true, + }) + + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('3 other EQL v3 column(s) remain'), + ) + expect(writeFileMock).not.toHaveBeenCalled() + expect(exitSpy).toHaveBeenCalledWith(1) + exitSpy.mockRestore() + }) + + it('--uninstall-eql: on the last column, embeds the real uninstall SQL behind an apply-time re-check', async () => { + await decryptDropCommand({ + table: 'users', + column: 'email_encrypted', + uninstallEql: true, + }) + + const sql = writtenSql() ?? '' + const dropColumnAt = sql.indexOf(`EXECUTE 'ALTER TABLE`) + expect(dropColumnAt).toBeGreaterThan(0) + const recheckAt = sql.indexOf('SELECT count(*) INTO remaining_eql') + const uninstallAt = sql.indexOf( + `EXECUTE 'DROP SCHEMA IF EXISTS eql_v3 CASCADE'`, + ) + expect(recheckAt).toBeGreaterThan(dropColumnAt) + expect(uninstallAt).toBeGreaterThan(recheckAt) + expect(sql).toContain( + `EXECUTE 'DROP SCHEMA IF EXISTS eql_v3_internal CASCADE'`, + ) + // The public column domains survive the uninstall script by design (they + // sit outside the disposable schemas); a full exit removes them too, + // without CASCADE. + const domainsAt = sql.indexOf( + `EXECUTE 'DROP DOMAIN IF EXISTS ' || leftover.name`, + ) + expect(domainsAt).toBeGreaterThan(uninstallAt) + expect(sql).not.toMatch(/DROP DOMAIN[^;]*CASCADE/) + // Everything stays inside the one DO block. + expect(sql.indexOf('END\n$stash_decrypt_drop$;')).toBeGreaterThan( + uninstallAt, + ) + }) + + it('keeps the forward manifest entry until the migration is applied, and says to remove it then', async () => { + migrateMocks.readManifest.mockResolvedValue({ + version: 1, + tables: { + users: [{ column: 'email', encryptedColumn: 'email_encrypted' }], + }, + }) + + await decryptDropCommand({ table: 'users', column: 'email_encrypted' }) + + // Generating is not applying: the apply-time check can still abort. + expect(migrateMocks.removeManifestColumn).not.toHaveBeenCalled() + const next = vi + .mocked(p.note) + .mock.calls.find(([, title]) => title === 'Next')?.[0] + expect(next).toContain( + 'Remove the users.email entry from .cipherstash/migrations.json', + ) + expect(next).toContain( + 'Keep writing BOTH email_encrypted and email until it is applied', + ) + }) + + it('quotes unusual identifiers through both SQL layers and sanitises the filename', async () => { + resolveMock.mockResolvedValue( + pair({ + table: 'tenant.weird"table', + encryptedColumn: "o'brien_encrypted", + plaintextColumn: 'o"brien', + state: decrypted('o"brien'), + }), + ) + + await decryptDropCommand({ + table: 'tenant.weird"table', + column: "o'brien_encrypted", + }) + + const [filePath, sql] = writeFileMock.mock.calls[0] as [string, string] + expect(sql).toContain('LOCK TABLE "tenant"."weird""table"') + expect(sql).toContain(`"o'brien_encrypted" IS NOT NULL`) + expect(sql).toContain( + `EXECUTE 'ALTER TABLE "tenant"."weird""table" DROP COLUMN "o''brien_encrypted"'`, + ) + expect(filePath).toMatch( + /_drop_encrypted_tenant_weird_table_o_brien_encrypted\.sql$/, + ) + }) +}) + +describe('decrypt plan', () => { + beforeEach(() => { + vi.clearAllMocks() + detectMocks.detectDrizzle.mockReturnValue(false) + detectMocks.detectPrismaNext.mockReturnValue(false) + migrateMocks.columnHasValues.mockResolvedValue(false) + resolveMock.mockResolvedValue( + pair({ plaintextSqlType: null, state: null, via: 'convention' }), + ) + }) + + it('builds nullable ADD COLUMN IF NOT EXISTS with the type mapped from the domain', () => { + const sql = buildAddPlaintextSql('users', [ + { + encryptedColumn: 'email_encrypted', + domain: 'eql_v3_text_search', + plaintextColumn: 'email', + mapping: { sqlType: 'text' }, + }, + { + encryptedColumn: 'created_encrypted', + domain: 'eql_v3_timestamp_ord', + plaintextColumn: 'created', + mapping: { sqlType: 'timestamptz' }, + }, + ]) + expect(sql).toContain( + 'ALTER TABLE "users" ADD COLUMN IF NOT EXISTS "email" text;', + ) + expect(sql).toContain( + 'ALTER TABLE "users" ADD COLUMN IF NOT EXISTS "created" timestamptz;', + ) + expect(sql).not.toMatch(/NOT NULL/) + }) + + it('writes the twin migration for a non-Drizzle project', async () => { + await decryptPlanCommand({ table: 'users' }) + + const [filePath, sql] = writeFileMock.mock.calls[0] as [string, string] + expect(filePath).toMatch(/_add_plaintext_users_email\.sql$/) + expect(sql).toContain('ADD COLUMN IF NOT EXISTS "email" text') + }) + + it('Drizzle: prints the schema lines for drizzle-kit generate and writes nothing', async () => { + detectMocks.detectDrizzle.mockReturnValue(true) + + await decryptPlanCommand({ table: 'users' }) + + expect(writeFileMock).not.toHaveBeenCalled() + expect(p.note).toHaveBeenCalledWith( + expect.stringContaining("email: text('email'),"), + 'Drizzle schema change', + ) + }) + + it('--dry-run prints the SQL and writes nothing', async () => { + await decryptPlanCommand({ table: 'users', dryRun: true }) + + expect(writeFileMock).not.toHaveBeenCalled() + expect(p.note).toHaveBeenCalledWith( + expect.stringContaining('ADD COLUMN IF NOT EXISTS "email" text'), + 'Migration (dry run — not written)', + ) + }) + + it('adds nothing for an empty twin that already exists', async () => { + resolveMock.mockResolvedValue(pair({ state: null })) + + await decryptPlanCommand({ table: 'users' }) + + expect(writeFileMock).not.toHaveBeenCalled() + expect(p.log.info).toHaveBeenCalledWith( + expect.stringContaining('already exists'), + ) + expect(p.log.warn).not.toHaveBeenCalled() + }) + + it('warns, with the backfill refusal text, when an existing twin already holds values', async () => { + resolveMock.mockResolvedValue(pair({ state: null })) + migrateMocks.columnHasValues.mockResolvedValueOnce(true) + + await decryptPlanCommand({ table: 'users' }) + + expect(writeFileMock).not.toHaveBeenCalled() + const warning = vi.mocked(p.log.warn).mock.calls[0]?.[0] ?? '' + expect(warning).toContain('users.email already holds values') + expect(warning).toContain('--force') + expect(warning).toContain('--confirm-plaintext-current') + expect(p.note).toHaveBeenCalledWith( + expect.stringContaining( + 'stash decrypt backfill --table users --column email_encrypted <--force | --confirm-plaintext-current>', + ), + 'Next', + ) + }) + + it('warns that the backfill will refuse a twin whose forward migration was never dropped', async () => { + resolveMock.mockResolvedValue( + pair({ state: null, forwardState: forward('backfilled') }), + ) + migrateMocks.columnHasValues.mockResolvedValueOnce(true) + + await decryptPlanCommand({ table: 'users' }) + + const warning = vi.mocked(p.log.warn).mock.calls[0]?.[0] ?? '' + expect(warning).toContain('`stash decrypt backfill` will refuse') + expect(warning).toContain("still open at 'backfilled'") + expect(warning).toContain('stash encrypt drop') + }) + + it('closes an open forward migration whose source column was removed outside stash', async () => { + resolveMock.mockResolvedValue( + pair({ + plaintextSqlType: null, + state: null, + via: 'convention', + forwardState: forward('backfilled'), + }), + ) + + await decryptPlanCommand({ table: 'users' }) + + expect(migrateMocks.appendEvent).toHaveBeenCalledWith(expect.anything(), { + tableName: 'users', + columnName: 'email', + event: 'dropped', + phase: 'dropped', + details: { observedAbsent: true, recordedBy: 'stash decrypt plan' }, + }) + expect(p.log.info).toHaveBeenCalledWith( + expect.stringContaining("still open at 'backfilled'"), + ) + // The twin is still planned. + const [, sql] = writeFileMock.mock.calls[0] as [string, string] + expect(sql).toContain('ADD COLUMN IF NOT EXISTS "email" text') + }) + + it('--dry-run records nothing for a removed forward source', async () => { + resolveMock.mockResolvedValue( + pair({ + plaintextSqlType: null, + state: null, + via: 'convention', + forwardState: forward('backfilled'), + }), + ) + + await decryptPlanCommand({ table: 'users', dryRun: true }) + + expect(migrateMocks.appendEvent).not.toHaveBeenCalled() + expect(p.log.info).toHaveBeenCalledWith( + expect.stringContaining('would record'), + ) + }) + + it('records nothing for a missing twin with no open forward migration', async () => { + for (const forwardState of [null, forward('dropped')]) { + vi.clearAllMocks() + resolveMock.mockResolvedValue( + pair({ + plaintextSqlType: null, + state: null, + via: 'convention', + forwardState, + }), + ) + + await decryptPlanCommand({ table: 'users' }) + + expect(migrateMocks.appendEvent).not.toHaveBeenCalled() + } + }) + + it('does not second-guess a twin the reverse lifecycle in progress filled', async () => { + resolveMock.mockResolvedValue( + pair({ state: { ...decrypted('email'), phase: 'decrypting' } }), + ) + migrateMocks.columnHasValues.mockResolvedValue(true) + + await decryptPlanCommand({ table: 'users' }) + + expect(migrateMocks.columnHasValues).not.toHaveBeenCalled() + expect(p.log.warn).not.toHaveBeenCalled() + }) + + it('overview: one line per EQL v3 column, surfacing pairings it cannot resolve', async () => { + // `fetchPhysicalColumns`' information_schema read. + queryMock.mockResolvedValueOnce({ + rows: [ + { table_name: 'users', column_name: 'id', domain_name: null }, + { + table_name: 'users', + column_name: 'email_encrypted', + domain_name: 'eql_v3_text_search', + }, + { + table_name: 'users', + column_name: 'note_encrypted', + domain_name: 'eql_v3_text', + }, + ], + }) + resolveMock + .mockResolvedValueOnce( + pair({ plaintextSqlType: null, state: null, via: 'convention' }), + ) + .mockRejectedValueOnce( + new BackfillConfigError('users.note is itself an EQL v3 column'), + ) + + await decryptPlanCommand() + + const overview = vi + .mocked(p.note) + .mock.calls.find(([, title]) => title === 'Reverse plan')?.[0] + expect(overview).toContain( + '+ users.email_encrypted (eql_v3_text_search) → email text', + ) + expect(overview).toContain( + '! users.note_encrypted: users.note is itself an EQL v3 column', + ) + expect(writeFileMock).not.toHaveBeenCalled() + }) + + it('refuses a domain with no known plaintext type instead of inventing one', async () => { + resolveMock.mockResolvedValue( + pair({ mapping: null, domain: 'eql_v3_uuid_eq', plaintextSqlType: null }), + ) + const exitSpy = spyExit() + + await decryptPlanCommand({ table: 'users', column: 'email_encrypted' }) + + expect(p.log.error).toHaveBeenCalledWith( + expect.stringContaining('No plaintext type is known'), + ) + expect(writeFileMock).not.toHaveBeenCalled() + expect(exitSpy).toHaveBeenCalledWith(1) + exitSpy.mockRestore() + }) +}) diff --git a/languages/typescript/packages/cli/src/commands/decrypt/__tests__/decrypt-sql.live.test.ts b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/decrypt-sql.live.test.ts new file mode 100644 index 000000000..56626885c --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/decrypt-sql.live.test.ts @@ -0,0 +1,171 @@ +/** + * Live-Postgres coverage for the SQL `stash decrypt plan` and + * `stash decrypt drop` generate. + * + * The unit tests read the generated text; this suite EXECUTES it. The drop + * migration nests identifiers inside an `EXECUTE '…'` string inside a + * dollar-quoted `DO` block, so a quoting slip that every string assertion + * accepts would still fail here — and an atomicity slip (a drop that runs + * before its own coverage check has raised) is only observable on a real + * server. + * + * Gated on STASH_TEST_DATABASE_URL so the default `pnpm test` stays green + * without a database. Locally: + * + * docker compose -f local/docker-compose.postgres.yml up -d --wait + * export STASH_TEST_DATABASE_URL=postgres://cipherstash:password@localhost:55432/cipherstash + * + * Needs no EQL install: `jsonb` stands in for the encrypted column (the SQL + * under test never looks at its type), and the `--uninstall-eql` gate is + * exercised with a probe domain in this suite's own schema and a harmless + * stand-in statement, so the shared database's real EQL is never touched. + */ + +import { + afterAll, + beforeAll, + beforeEach, + describe, + expect, + it, + vi, +} from 'vitest' + +vi.mock('@/commands/encrypt/context.js', () => ({ + loadEncryptionContext: vi.fn(), + requireTable: vi.fn(), +})) + +import type pg from 'pg' +import { buildDecryptDropSql } from '../drop.js' +import { buildAddPlaintextSql } from '../plan.js' + +const DATABASE_URL = process.env.STASH_TEST_DATABASE_URL +const describeLive = DATABASE_URL ? describe : describe.skip + +const SCHEMA = 'stash_decrypt_live' +const TABLE = `${SCHEMA}.weird"users` +const QUOTED_TABLE = `"${SCHEMA}"."weird""users"` + +describeLive('decrypt plan/drop SQL — live Postgres', () => { + let client: pg.Client + + beforeAll(async () => { + const { default: pgModule } = await import('pg') + client = new pgModule.Client({ connectionString: DATABASE_URL }) + await client.connect() + }) + + afterAll(async () => { + await client.query(`DROP SCHEMA IF EXISTS ${SCHEMA} CASCADE`) + await client.end().catch(() => undefined) + }) + + beforeEach(async () => { + await client.query(`DROP SCHEMA IF EXISTS ${SCHEMA} CASCADE`) + await client.query(`CREATE SCHEMA ${SCHEMA}`) + await client.query(` + CREATE TABLE ${QUOTED_TABLE} ( + id int PRIMARY KEY, + "o'brien_encrypted" jsonb + ) + `) + await client.query( + `INSERT INTO ${QUOTED_TABLE} (id, "o'brien_encrypted") + SELECT g, jsonb_build_object('v', 3, 'c', 'ct-' || g) + FROM generate_series(1, 5) AS g`, + ) + // An EQL-shaped functional index: it references the column only through + // its expression. + await client.query( + `CREATE INDEX "weird idx" ON ${QUOTED_TABLE} (("o'brien_encrypted"->>'c'))`, + ) + }) + + async function columns(): Promise { + const { rows } = await client.query<{ attname: string }>( + `SELECT attname FROM pg_attribute + WHERE attrelid = $1::regclass AND attnum > 0 AND NOT attisdropped + ORDER BY attnum`, + [QUOTED_TABLE], + ) + return rows.map((row) => row.attname) + } + + async function addTwinAndFill(skipId?: number) { + await client.query( + buildAddPlaintextSql(TABLE, [ + { + encryptedColumn: "o'brien_encrypted", + domain: 'eql_v3_text', + plaintextColumn: 'o"brien', + mapping: { sqlType: 'text' }, + }, + ]), + ) + await client.query( + `UPDATE ${QUOTED_TABLE} SET "o""brien" = "o'brien_encrypted"->>'c' + WHERE id IS DISTINCT FROM $1`, + [skipId ?? null], + ) + } + + const dropSql = (uninstallStatements: string[] = []) => + buildDecryptDropSql({ + table: TABLE, + encryptedColumn: "o'brien_encrypted", + plaintextColumn: 'o"brien', + indexes: [{ schema: SCHEMA, name: 'weird idx', definition: '' }], + uninstallStatements, + }) + + it('plan: the twin migration applies, and re-applies as a no-op', async () => { + await addTwinAndFill() + await addTwinAndFill() + expect(await columns()).toEqual(['id', "o'brien_encrypted", 'o"brien']) + }) + + it('drop: removes the index and the encrypted column, keeps the plaintext', async () => { + await addTwinAndFill() + + await client.query(dropSql()) + + expect(await columns()).toEqual(['id', 'o"brien']) + const { rows } = await client.query( + `SELECT 1 FROM pg_class WHERE relname = 'weird idx'`, + ) + expect(rows).toHaveLength(0) + }) + + it('drop: an undecrypted row aborts the whole migration — nothing is dropped', async () => { + await addTwinAndFill(3) + + await expect(client.query(dropSql())).rejects.toThrow( + /refusing to drop .* 1 row\(s\) have ciphertext set/, + ) + expect(await columns()).toEqual(['id', "o'brien_encrypted", 'o"brien']) + const { rows } = await client.query( + `SELECT 1 FROM pg_class WHERE relname = 'weird idx'`, + ) + expect(rows).toHaveLength(1) + }) + + it('drop --uninstall-eql: another EQL v3 column anywhere aborts it, leaving the column in place', async () => { + await addTwinAndFill() + await client.query(`CREATE DOMAIN ${SCHEMA}.eql_v3_probe AS jsonb`) + await client.query( + `CREATE TABLE ${SCHEMA}.other (id int, secret ${SCHEMA}.eql_v3_probe)`, + ) + + await expect( + client.query(dropSql([`CREATE TABLE ${SCHEMA}.uninstall_ran (x int)`])), + ).rejects.toThrow( + /refusing to uninstall EQL — \d+ EQL v3 column\(s\) still exist/, + ) + expect(await columns()).toEqual(['id', "o'brien_encrypted", 'o"brien']) + const { rows } = await client.query( + `SELECT to_regclass('${SCHEMA}.uninstall_ran') AS ran`, + ) + expect(rows[0]?.ran).toBeNull() + }) +}) diff --git a/languages/typescript/packages/cli/src/commands/decrypt/__tests__/decrypt-status.test.ts b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/decrypt-status.test.ts new file mode 100644 index 000000000..956a7933b --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/decrypt-status.test.ts @@ -0,0 +1,174 @@ +/** + * `stash decrypt status` must name the same plaintext twin `decrypt plan` and + * `decrypt backfill` would, and flag a twin those commands would refuse. It + * once derived the name itself and disagreed with them on an edge case. + */ + +import type { MigrationStateRow } from '@cipherstash/migrate' +import { beforeEach, describe, expect, it, vi } from 'vitest' + +vi.mock('pg', () => ({ + default: { + Client: class { + connect = vi.fn(async () => {}) + end = vi.fn(async () => {}) + query = vi.fn(async () => ({ rows: [] })) + }, + }, +})) + +const readers = vi.hoisted(() => ({ + latestByColumnSafe: vi.fn( + async (): Promise> => new Map(), + ), + fetchPhysicalColumns: vi.fn( + async (): Promise>> => new Map(), + ), +})) +vi.mock('@/commands/encrypt/lib/db-readers.js', () => readers) + +vi.mock('@cipherstash/migrate', async (importOriginal) => ({ + ...(await importOriginal()), + readManifest: vi.fn(async () => null), +})) +vi.mock('@clack/prompts', () => ({ + intro: vi.fn(), + outro: vi.fn(), + note: vi.fn(), + log: { info: vi.fn(), error: vi.fn(), warn: vi.fn(), success: vi.fn() }, +})) +vi.mock('@/config/index.js', () => ({ + loadStashConfig: vi.fn(async () => ({ databaseUrl: 'postgres://test' })), +})) +vi.mock('@/commands/encrypt/context.js', () => ({ + loadEncryptionContext: vi.fn(), + requireTable: vi.fn(), +})) +vi.mock('@/commands/init/utils.js', () => ({ + detectPackageManager: vi.fn(() => 'npm'), + runnerCommand: vi.fn((_pm: string, ref: string) => `npx ${ref}`), +})) + +import * as p from '@clack/prompts' +import { decryptStatusCommand } from '../status.js' + +function event( + overrides: Partial & + Pick, +): MigrationStateRow { + return { + tableName: 'users', + cursorValue: null, + rowsProcessed: null, + rowsTotal: null, + details: null, + createdAt: new Date(0), + ...overrides, + } +} + +/** The rendered status table, one string per data row. */ +async function statusRows(): Promise { + await decryptStatusCommand() + const table = + vi + .mocked(p.note) + .mock.calls.find( + ([, title]) => title === 'Reverse migration status', + )?.[0] ?? '' + return table.split('\n').slice(1) +} + +describe('decrypt status', () => { + beforeEach(() => { + vi.clearAllMocks() + readers.latestByColumnSafe.mockResolvedValue(new Map()) + }) + + it('derives the twin name with the same rule as plan, even for a column named _encrypted', async () => { + readers.fetchPhysicalColumns.mockResolvedValue( + new Map([ + [ + 'users', + new Map([ + ['_encrypted', 'eql_v3_text'], + ['email_encrypted', 'eql_v3_text_search'], + ]), + ], + ]), + ) + + const rows = await statusRows() + + expect(rows.find((row) => row.includes('_encrypted '))).toContain( + '_encrypted_plaintext (text, to add)', + ) + expect(rows.find((row) => row.includes('email_encrypted'))).toContain( + 'email (text, to add)', + ) + }) + + it('flags a derived twin that is itself an EQL column, which plan and backfill refuse', async () => { + readers.fetchPhysicalColumns.mockResolvedValue( + new Map([ + [ + 'users', + new Map([ + ['email_encrypted', 'eql_v3_text_search'], + ['email', 'eql_v3_text_eq'], + ]), + ], + ]), + ) + + const rows = await statusRows() + + const row = rows.find((r) => r.startsWith('users email_encrypted')) ?? '' + expect(row).toContain('plaintext-col-is-eql') + expect(row).not.toContain('plaintext-col-missing') + }) + + it('shows a column whose earlier reverse lifecycle finished as starting over', async () => { + readers.fetchPhysicalColumns.mockResolvedValue( + new Map([ + [ + 'users', + new Map([ + ['email_encrypted', 'eql_v3_text_search'], + ['email', null], + ]), + ], + ]), + ) + readers.latestByColumnSafe.mockResolvedValue( + new Map([ + [ + 'users.email_encrypted', + event({ + id: '10', + columnName: 'email_encrypted', + event: 'encrypted_dropped', + phase: 'encrypted-dropped', + details: { plaintextColumn: 'old_email' }, + }), + ], + [ + 'users.old_email', + event({ + id: '12', + columnName: 'old_email', + event: 'dual_writing', + phase: 'dual-writing', + }), + ], + ]), + ) + + const rows = await statusRows() + + const row = rows.find((r) => r.startsWith('users email_encrypted')) ?? '' + expect(row).not.toContain('encrypted-dropped') + expect(row).not.toContain('old_email') + expect(row).toMatch(/email_encrypted\s+eql_v3_text_search\s+email\s/) + }) +}) diff --git a/languages/typescript/packages/cli/src/commands/decrypt/__tests__/domain-mapping.test.ts b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/domain-mapping.test.ts new file mode 100644 index 000000000..256cf6d59 --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/domain-mapping.test.ts @@ -0,0 +1,99 @@ +/** + * The reverse type mapping and the uninstall embedding, checked against the + * REAL `@cipherstash/eql` bundle this CLI ships with — not a hand-copied + * list. A domain added to EQL without a plaintext mapping fails here, before a + * user's `stash decrypt plan` meets it. + */ + +import { readInstallSql, readUninstallSql } from '@cipherstash/eql/sql' +import { plaintextTypeForEqlDomain } from '@cipherstash/migrate' +import { describe, expect, it, vi } from 'vitest' + +vi.mock('@/commands/encrypt/context.js', () => ({ + loadEncryptionContext: vi.fn(), + requireTable: vi.fn(), +})) + +import { splitUninstallSql } from '../drop.js' +import { drizzleColumnLine, drizzleImports } from '../lib/orm-hints.js' + +/** + * Every COLUMN domain the bundle creates: `public.eql_v3_*`. (`eql_v3.query_*` + * needles and `eql_v3_internal.*` term types are never a column's type.) + */ +const columnDomains = [ + ...readInstallSql().matchAll(/CREATE DOMAIN public\.(eql_v3_[a-z0-9_]+)/g), +] + .map((match) => match[1] as string) + .filter((domain, index, all) => all.indexOf(domain) === index) + +// SteVec fragment: the type of ONE element inside a searchable JSON +// document's `sv` array, not of a column. The catalog says so; mapping it to a +// plaintext type would invent one. +const NOT_A_COLUMN_TYPE = new Set(['eql_v3_json_entry']) + +describe('reverse type mapping vs the shipped EQL bundle', () => { + it('found the bundle’s column domains', () => { + expect(columnDomains.length).toBeGreaterThan(40) + expect(columnDomains).toContain('eql_v3_text_search') + expect(columnDomains).toContain('eql_v3_json_search') + }) + + it.each(columnDomains.filter((domain) => !NOT_A_COLUMN_TYPE.has(domain)))( + '%s maps back to a plaintext type', + (domain) => { + expect(plaintextTypeForEqlDomain(domain)).not.toBeNull() + }, + ) + + it('refuses the SteVec fragment domain', () => { + for (const domain of NOT_A_COLUMN_TYPE) { + expect(columnDomains).toContain(domain) + expect(plaintextTypeForEqlDomain(domain)).toBeNull() + } + }) +}) + +describe('splitUninstallSql', () => { + it('splits the shipped uninstall script into its DROP SCHEMA statements', () => { + expect(splitUninstallSql(readUninstallSql())).toEqual([ + 'DROP SCHEMA IF EXISTS eql_v3 CASCADE', + 'DROP SCHEMA IF EXISTS eql_v3_internal CASCADE', + ]) + }) + + it('refuses a script it cannot split safely rather than mangling it', () => { + expect(() => + splitUninstallSql("DO $$ BEGIN RAISE NOTICE 'x;y'; END $$;"), + ).toThrow(/not in the shape this CLI can embed safely/) + }) +}) + +describe('Drizzle hints', () => { + it('declares each twin with the builder for its mapped type, nullable', () => { + expect(drizzleColumnLine('email', 'text')).toBe("email: text('email'),") + expect(drizzleColumnLine('account_id', 'bigint')).toBe( + "account_id: bigint('account_id', { mode: 'bigint' }),", + ) + expect(drizzleColumnLine('created_at', 'timestamptz')).toBe( + "created_at: timestamp('created_at', { withTimezone: true }),", + ) + expect(drizzleColumnLine('score', 'double precision')).toBe( + "score: doublePrecision('score'),", + ) + }) + + it('quotes awkward names safely', () => { + expect(drizzleColumnLine("o'brien-x", 'text')).toBe( + `"o'brien-x": text('o\\'brien-x'),`, + ) + }) + + it('lists each import once, sorted', () => { + expect(drizzleImports(['text', 'jsonb', 'text', 'timestamptz'])).toEqual([ + 'jsonb', + 'text', + 'timestamp', + ]) + }) +}) diff --git a/languages/typescript/packages/cli/src/commands/decrypt/__tests__/resolve-pair.test.ts b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/resolve-pair.test.ts new file mode 100644 index 000000000..88d78c157 --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/__tests__/resolve-pair.test.ts @@ -0,0 +1,423 @@ +/** + * Pins how `stash decrypt` pairs an encrypted column with its plaintext twin. + * + * The pairing is what every later step trusts: the backfill fills that column, + * and `decrypt drop`'s coverage gate verifies it before destroying the + * ciphertext. A pairing that silently drifted — a flag naming a different + * column mid-lifecycle — would let that gate pass against the wrong column. + */ + +import type { + EncryptedColumnInfo, + MigrationStateRow, +} from '@cipherstash/migrate' +import { beforeEach, describe, expect, it, vi } from 'vitest' + +// Only the catalog/state probes are replaced; the domain mapping and phase +// predicates stay real so a rename on either side fails loudly. +const probes = vi.hoisted(() => ({ + listEncryptedColumns: vi.fn(async (): Promise => []), + columnExists: vi.fn(async () => true), + detectColumnEqlVersion: vi.fn(async (): Promise<2 | 3 | null> => null), + columnSqlType: vi.fn(async (): Promise => null), + columnHasValues: vi.fn(async () => false), + progress: vi.fn( + async ( + _client: unknown, + _table: string, + _column: string, + ): Promise | null> => null, + ), + readManifest: vi.fn(async (): Promise => null), +})) +vi.mock('@cipherstash/migrate', async (importOriginal) => ({ + ...(await importOriginal()), + ...probes, +})) +vi.mock('@/commands/encrypt/context.js', () => ({ + loadEncryptionContext: vi.fn(), + requireTable: vi.fn(), +})) + +import type pg from 'pg' +import { BackfillConfigError } from '@/commands/encrypt/backfill.js' +import { + assessExistingTwin, + type DecryptPair, + derivePlaintextName, + openForwardMigrationMessage, + resolveDecryptPair, + twinHazardMessage, +} from '../lib/resolve-pair.js' + +const db = {} as pg.ClientBase +const EMAIL_ENC: EncryptedColumnInfo = { + column: 'email_encrypted', + domain: 'eql_v3_text_search', + version: 3, +} + +function reverseState(plaintextColumn: string): Partial { + return { + event: 'decrypt_checkpoint', + phase: 'decrypting', + details: { plaintextColumn }, + } +} + +async function failure(promise: Promise): Promise { + const error = await promise.catch((e: unknown) => e) + expect(error).toBeInstanceOf(BackfillConfigError) + return error as Error +} + +describe('resolveDecryptPair', () => { + beforeEach(() => { + vi.clearAllMocks() + probes.listEncryptedColumns.mockResolvedValue([EMAIL_ENC]) + probes.detectColumnEqlVersion.mockResolvedValue(null) + probes.columnSqlType.mockResolvedValue(null) + probes.columnHasValues.mockResolvedValue(false) + probes.progress.mockResolvedValue(null) + probes.readManifest.mockResolvedValue(null) + }) + + /** `progress` answers per column: reverse history under the encrypted + * column, forward history under the plaintext name. */ + function history(byColumn: Record>) { + probes.progress.mockImplementation( + async (_client, _table, column) => byColumn[column] ?? null, + ) + } + + it('reads the forward history recorded under the twin’s name', async () => { + history({ email: { id: '3', event: 'backfilled', phase: 'backfilled' } }) + + const pair = await resolveDecryptPair(db, { + table: 'users', + encryptedColumn: 'email_encrypted', + }) + expect(pair.forwardState?.phase).toBe('backfilled') + expect(pair.lifecycleActive).toBe(false) + }) + + it('a drop generated in THIS lifecycle (not yet applied, or aborted) keeps it active', async () => { + history({ + email_encrypted: { + id: '10', + event: 'encrypted_dropped', + phase: 'encrypted-dropped', + details: { plaintextColumn: 'contact_email' }, + }, + // Older than the drop: the forward history that preceded the reversal. + contact_email: { id: '4', event: 'backfilled', phase: 'backfilled' }, + }) + + const pair = await resolveDecryptPair(db, { + table: 'users', + encryptedColumn: 'email_encrypted', + }) + expect(pair.lifecycleActive).toBe(true) + expect(pair).toMatchObject({ + plaintextColumn: 'contact_email', + via: 'recorded', + }) + }) + + it('a lifecycle that finished before the column was encrypted again neither binds nor counts as active', async () => { + history({ + email_encrypted: { + id: '10', + event: 'encrypted_dropped', + phase: 'encrypted-dropped', + details: { plaintextColumn: 'contact_email' }, + }, + // A forward rollout AFTER the drop re-created the encrypted column. + contact_email: { id: '12', event: 'dual_writing', phase: 'dual-writing' }, + }) + + const pair = await resolveDecryptPair(db, { + table: 'users', + encryptedColumn: 'email_encrypted', + plaintextColumn: 'email', + }) + expect(pair.lifecycleActive).toBe(false) + expect(pair).toMatchObject({ plaintextColumn: 'email', via: 'flag' }) + }) + + it('maps the domain back to its plaintext type', async () => { + const pair = await resolveDecryptPair(db, { + table: 'users', + encryptedColumn: 'email_encrypted', + }) + expect(pair.domain).toBe('eql_v3_text_search') + expect(pair.mapping?.sqlType).toBe('text') + }) + + it('prefers the original name the forward manifest recorded', async () => { + probes.readManifest.mockResolvedValue({ + version: 1, + tables: { + users: [ + { column: 'email_address', encryptedColumn: 'email_encrypted' }, + ], + }, + }) + + const pair = await resolveDecryptPair(db, { + table: 'users', + encryptedColumn: 'email_encrypted', + }) + expect(pair).toMatchObject({ + plaintextColumn: 'email_address', + via: 'manifest', + }) + }) + + it('falls back to the _encrypted convention, then to _plaintext', async () => { + const conventional = await resolveDecryptPair(db, { + table: 'users', + encryptedColumn: 'email_encrypted', + }) + expect(conventional).toMatchObject({ + plaintextColumn: 'email', + via: 'convention', + }) + + probes.listEncryptedColumns.mockResolvedValue([ + { column: 'secret', domain: 'eql_v3_text', version: 3 }, + ]) + const fallback = await resolveDecryptPair(db, { + table: 'users', + encryptedColumn: 'secret', + }) + expect(fallback).toMatchObject({ + plaintextColumn: 'secret_plaintext', + via: 'fallback', + }) + }) + + it('reuses the pairing an earlier reverse run recorded', async () => { + probes.progress.mockResolvedValue(reverseState('contact_email')) + + const pair = await resolveDecryptPair(db, { + table: 'users', + encryptedColumn: 'email_encrypted', + }) + expect(pair).toMatchObject({ + plaintextColumn: 'contact_email', + via: 'recorded', + }) + }) + + it('refuses a --plaintext-column that contradicts the recorded pairing', async () => { + probes.progress.mockResolvedValue(reverseState('contact_email')) + + const error = await failure( + resolveDecryptPair(db, { + table: 'users', + encryptedColumn: 'email_encrypted', + plaintextColumn: 'email', + }), + ) + expect(error.message).toContain( + 'already being reversed into "contact_email"', + ) + expect(error.message).toContain('coverage check') + }) + + it('ignores forward history under the same key — only reverse events pair', async () => { + probes.progress.mockResolvedValue({ + event: 'backfilled', + phase: 'backfilled', + details: { plaintextColumn: 'not-a-reverse-record' }, + }) + + const pair = await resolveDecryptPair(db, { + table: 'users', + encryptedColumn: 'email_encrypted', + }) + expect(pair.state).toBeNull() + expect(pair.via).toBe('convention') + }) + + it('names a missing source column, listing the EQL columns that do exist', async () => { + probes.columnExists.mockResolvedValue(false) + + const error = await failure( + resolveDecryptPair(db, { table: 'users', encryptedColumn: 'email' }), + ) + expect(error.message).toContain('Column email does not exist on users') + expect(error.message).toContain('email_encrypted') + }) + + it('refuses a source that is plaintext already', async () => { + probes.columnExists.mockResolvedValue(true) + + const error = await failure( + resolveDecryptPair(db, { table: 'users', encryptedColumn: 'email' }), + ) + expect(error.message).toContain('is not an EQL v3 column') + }) + + it('refuses a target that is itself encrypted', async () => { + probes.detectColumnEqlVersion.mockResolvedValue(3) + + const error = await failure( + resolveDecryptPair(db, { + table: 'users', + encryptedColumn: 'email_encrypted', + plaintextColumn: 'other_encrypted', + }), + ) + expect(error.message).toContain('is itself an EQL v3 column') + }) + + it('reports the twin’s declared type once it exists', async () => { + probes.columnSqlType.mockResolvedValue('character varying(255)') + + const pair = await resolveDecryptPair(db, { + table: 'users', + encryptedColumn: 'email_encrypted', + }) + expect(pair.plaintextSqlType).toBe('character varying(255)') + }) +}) + +describe('derivePlaintextName', () => { + it('strips the _encrypted convention', () => { + expect(derivePlaintextName('email_encrypted')).toEqual({ + name: 'email', + via: 'convention', + }) + }) + + it('never derives an empty name from a column called exactly _encrypted', () => { + expect(derivePlaintextName('_encrypted')).toEqual({ + name: '_encrypted_plaintext', + via: 'fallback', + }) + expect(derivePlaintextName('secret')).toEqual({ + name: 'secret_plaintext', + via: 'fallback', + }) + }) +}) + +describe('assessExistingTwin', () => { + const db = {} as pg.ClientBase + + function twin(overrides: Partial = {}): DecryptPair { + return { + table: 'users', + encryptedColumn: 'email_encrypted', + domain: 'eql_v3_text_search', + mapping: null, + plaintextColumn: 'email', + via: 'manifest', + plaintextSqlType: 'text', + state: null, + lifecycleActive: false, + forwardState: null, + ...overrides, + } + } + + function forwardAt(phase: MigrationStateRow['phase']): MigrationStateRow { + return { + id: '3', + tableName: 'users', + columnName: 'email', + event: 'backfilled', + phase, + cursorValue: null, + rowsProcessed: null, + rowsTotal: null, + details: null, + createdAt: new Date(0), + } + } + + beforeEach(() => { + vi.clearAllMocks() + probes.columnHasValues.mockResolvedValue(false) + }) + + it('trusts a twin the reverse lifecycle in progress filled, without scanning it', async () => { + probes.columnHasValues.mockResolvedValue(true) + + expect(await assessExistingTwin(db, twin({ lifecycleActive: true }))).toBe( + null, + ) + expect(probes.columnHasValues).not.toHaveBeenCalled() + }) + + it('has nothing to check before the twin exists', async () => { + expect( + await assessExistingTwin(db, twin({ plaintextSqlType: null })), + ).toBeNull() + }) + + it('accepts an empty twin with no forward history', async () => { + expect(await assessExistingTwin(db, twin())).toBeNull() + }) + + it('flags a twin that already holds values', async () => { + probes.columnHasValues.mockResolvedValue(true) + + expect(await assessExistingTwin(db, twin())).toEqual({ + kind: 'prepopulated', + }) + }) + + it.each([ + 'schema-added', + 'dual-writing', + 'backfilling', + 'backfilled', + ] as const)( + 'refuses the source column of a forward migration still open at %s', + (phase) => { + expect( + openForwardMigrationMessage(twin({ forwardState: forwardAt(phase) })), + ).toContain(`still open at '${phase}'`) + }, + ) + + it("does not refuse once the forward migration reached 'dropped'", () => { + expect( + openForwardMigrationMessage(twin({ forwardState: forwardAt('dropped') })), + ).toBeNull() + expect(openForwardMigrationMessage(twin())).toBeNull() + }) + + it("a forward lifecycle at 'dropped' is no longer the source, but its values still count", async () => { + expect( + await assessExistingTwin( + db, + twin({ forwardState: forwardAt('dropped') }), + ), + ).toBeNull() + + // `dropped` is recorded when the drop is generated; if it was never + // applied the old column is still there, full. + probes.columnHasValues.mockResolvedValue(true) + expect( + await assessExistingTwin( + db, + twin({ forwardState: forwardAt('dropped') }), + ), + ).toEqual({ kind: 'prepopulated' }) + }) + + it('explains both ways forward, with names only', () => { + const message = twinHazardMessage(twin()) + expect(message).toContain('users.email already holds values') + expect(message).toContain( + '--force overwrite every row of email with the decrypted ciphertext', + ) + expect(message).toContain( + '--confirm-plaintext-current keep the existing values', + ) + }) +}) diff --git a/languages/typescript/packages/cli/src/commands/decrypt/backfill.ts b/languages/typescript/packages/cli/src/commands/decrypt/backfill.ts new file mode 100644 index 000000000..21339f5a3 --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/backfill.ts @@ -0,0 +1,458 @@ +import { + appendEvent, + type DecryptBackfillResult, + formatCodes, + installMigrationsSchema, + PlaintextWriteError, + qualifyTable, + quoteIdent, + runDecryptBackfill, + UndecryptableBatchError, +} from '@cipherstash/migrate' +import * as p from '@clack/prompts' +import pg from 'pg' +import { BackfillConfigError } from '@/commands/encrypt/backfill.js' +import { loadEncryptionContext } from '@/commands/encrypt/context.js' +import { detectPkColumn } from '@/commands/encrypt/lib/db-readers.js' +import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' +import { loadStashConfig } from '@/config/index.js' +import { buildPgClientConfig, explainTlsError } from '@/db/client.js' +import { + assessExistingTwin, + type DecryptPair, + openForwardMigrationMessage, + resolveDecryptPair, + type TwinHazard, + twinHazardMessage, +} from './lib/resolve-pair.js' + +/** Default rows per chunk — and so per `bulkDecrypt` ZeroKMS request. */ +export const DEFAULT_DECRYPT_CHUNK_SIZE = 500 + +/** + * Options accepted by `stash decrypt backfill`. Same flags, same meaning, as + * `stash encrypt backfill` — with the roles of the two columns swapped: + * `--column` is the column being migrated AWAY from, which in this direction + * is the encrypted one. + */ +export interface DecryptBackfillCommandOptions { + /** Physical table, e.g. `users` (`schema.table` supported). */ + table: string + /** The EQL v3 column to decrypt, e.g. `email_encrypted`. */ + column: string + /** + * The plaintext twin to fill. Defaults to the pairing a previous run + * recorded, else the forward manifest's original name, else the + * `_encrypted` convention stripped. It must already exist — add it + * with `stash decrypt plan`. + */ + plaintextColumn?: string + /** Override primary-key detection. */ + pkColumn?: string + /** + * Rows per chunk, and so per `bulkDecrypt` call — one ZeroKMS request per + * chunk. Default {@link DEFAULT_DECRYPT_CHUNK_SIZE}; at most + * `MAX_DECRYPT_CHUNK_SIZE` (32767), because each row binds two parameters + * in the chunk's write. Validated by the caller with + * `parseDecryptChunkSize` (`./lib/chunk-size.ts`), where the raw flag text + * is still in hand; the handler trusts it. + */ + chunkSize?: number + /** + * Assert the application already writes BOTH the encrypted column and its + * plaintext twin on every insert/update. Required (or prompted) on the + * first run, as for the forward backfill. + */ + confirmDualWritesDeployed?: boolean + /** + * Re-decrypt rows whose plaintext is already set and overwrite it (drift + * recovery, or replacing plaintext that predates the reverse lifecycle). + * Safe only while dual-writes are live: a row the application wrote to the + * plaintext column alone gets the older ciphertext's value back. + */ + force?: boolean + /** + * Assert that a plaintext twin which already holds values before the first + * reverse run is current, and keep those values. Without it (or `force`) + * the first run refuses such a twin; see {@link TwinHazard}. + */ + confirmPlaintextCurrent?: boolean + /** + * Keep going when the first chunk decrypts nothing (see + * `UndecryptableBatchError`) — for leading rows that really are all + * lock-context bound, or a re-run where only skipped rows remain. + */ + continueOnUndecryptable?: boolean +} + +/** + * CLI handler for `stash decrypt backfill` — the reverse of + * `stash encrypt backfill`, on the same engine (`@cipherstash/migrate`'s + * chunked runner): keyset pagination, one transaction per chunk with its + * checkpoint, SIGINT/SIGTERM finishing the current chunk, resume on re-run, + * and the `plaintext IS NULL` idempotency guard. + * + * Each chunk is ONE `bulkDecrypt` call, so ZeroKMS sees one request per chunk. + * Decryption is never charged. + * + * Values that cannot be decrypted with these credentials — typically values + * bound to an end-user lock context, which only that user can decrypt — are + * skipped, counted, left NULL, and reported. There is deliberately no bypass; + * see the lock-context limitation in the `stash-cli` docs. + * + * On the first run, a twin that already holds values (or is the source of a + * forward encryption that was never dropped) is refused unless the operator + * passes `--force` (overwrite it from the ciphertext) or + * `--confirm-plaintext-current` (keep it). The backfill only fills NULLs, so + * trusting stale values silently would let `decrypt drop` destroy the only + * current copy. + */ +export async function decryptBackfillCommand( + options: DecryptBackfillCommandOptions, +) { + p.intro(runnerCommand(detectPackageManager(), 'stash decrypt backfill')) + + const chunkSize = options.chunkSize ?? DEFAULT_DECRYPT_CHUNK_SIZE + + const stashConfig = await loadStashConfig() + // The same client loading `encrypt backfill` does — the user's encryption + // client file, or the Prisma Next contract — so decryption runs as exactly + // the credentials and keyset the application resolves. + const ctx = await loadEncryptionContext() + + const pool = new pg.Pool({ + ...buildPgClientConfig(stashConfig.databaseUrl), + max: 2, + }) + + const controller = new AbortController() + const onSignal = () => { + p.log.warn('Interrupt received; finishing current chunk and exiting.') + controller.abort() + } + + let db: pg.PoolClient | undefined + let exitCode = 0 + try { + process.on('SIGINT', onSignal) + process.on('SIGTERM', onSignal) + try { + db = await pool.connect() + } catch (error) { + const tlsExplanation = explainTlsError(error, stashConfig.databaseUrl) + if (tlsExplanation) throw new BackfillConfigError(tlsExplanation) + throw error + } + + try { + await installMigrationsSchema(db) + } catch (error) { + throw new BackfillConfigError( + 'Could not create the `cipherstash.cs_migrations` tracking schema, which the reverse backfill needs to checkpoint progress.\n\n' + + `Cause: ${error instanceof Error ? error.message : String(error)}\n\n` + + 'This database user needs CREATE on the database (to create the `cipherstash` schema) and on that schema. Grant it, or have someone run `stash eql install` once, then re-run.', + ) + } + + const pair = await resolveDecryptPair(db, { + table: options.table, + encryptedColumn: options.column, + plaintextColumn: options.plaintextColumn, + }) + const plaintextSqlType = requirePlaintextTwin(pair) + const pkColumn = + options.pkColumn ?? (await detectPkColumn(db, options.table)) + + p.log.info( + `${options.table}.${pair.encryptedColumn} is ${pair.domain} — reversing into ${pair.plaintextColumn} (${plaintextSqlType}). Lifecycle: backfill → switch the app to ${pair.plaintextColumn} by name → stash decrypt drop.`, + ) + + const openForward = openForwardMigrationMessage(pair) + if (openForward !== null) throw new BackfillConfigError(openForward) + + // Before anything is recorded: once the dual-write gate writes its event, + // later runs treat the twin's values as this lifecycle's own. + const hazard = await assessExistingTwin(db, pair) + if (hazard !== null) { + if (!options.force && !options.confirmPlaintextCurrent) { + throw new BackfillConfigError(twinHazardMessage(pair)) + } + if (!options.force) { + p.log.warn( + `Proceeding under --confirm-plaintext-current: keeping the values already in ${options.table}.${pair.plaintextColumn} and decrypting only the rows where it is NULL. You are asserting those values are current.`, + ) + } + } + + if (options.force) { + p.log.warn( + `--force will re-decrypt every row of ${options.table}.${pair.encryptedColumn} and overwrite the ${pair.plaintextColumn} values that already exist. Safe only while dual-writes are live: any row your application wrote to ${pair.plaintextColumn} alone (a newer value, or one cleared on purpose) gets the older ciphertext's value back.`, + ) + const proceed = await p.confirm({ + message: 'Continue with --force?', + initialValue: false, + }) + if (p.isCancel(proceed) || !proceed) { + p.outro('Aborted.') + return + } + } + + const okToProceed = await ensureReverseDualWritesDeployed(db, { + pair, + confirmFlag: options.confirmDualWritesDeployed === true, + acknowledgedHazard: + hazard === null + ? undefined + : { + hazard, + via: options.force ? 'force' : 'confirm-plaintext-current', + }, + }) + if (!okToProceed) { + p.outro('Aborted.') + return + } + + p.log.info( + `Decrypting ${options.table}.${pair.encryptedColumn} → ${pair.plaintextColumn} (pk: ${pkColumn}, chunk: ${chunkSize} rows = one ZeroKMS request).`, + ) + + let lastLogged = 0 + const result = await runDecryptBackfill({ + db, + decryptionClient: ctx.client, + tableName: options.table, + encryptedColumn: pair.encryptedColumn, + plaintextColumn: pair.plaintextColumn, + plaintextSqlType, + pkColumn, + chunkSize, + signal: controller.signal, + force: options.force, + continueOnUndecryptable: options.continueOnUndecryptable, + onProgress: (progress) => { + if ( + progress.rowsProcessed - lastLogged >= 5000 || + progress.rowsProcessed === progress.rowsTotal + ) { + p.log.step( + `${progress.rowsProcessed.toLocaleString()}/${progress.rowsTotal.toLocaleString()} rows${progress.rowsSkipped > 0 ? ` (${progress.rowsSkipped.toLocaleString()} undecryptable, skipped)` : ''}`, + ) + lastLogged = progress.rowsProcessed + } + }, + }) + + // The closing count, when the last chunk did not already print it (rows + // inserted during the run move the total after it was counted). + if (result.completed && lastLogged !== result.rowsProcessed) { + p.log.step( + `${result.rowsProcessed.toLocaleString()} rows${result.rowsSkipped > 0 ? ` (${result.rowsSkipped.toLocaleString()} skipped)` : ''}`, + ) + } + + if (result.rowsSkipped > 0) { + p.log.warn(skippedReport(options.table, pair, result)) + } + + if (!result.completed) { + p.log.warn( + `Stopped before completion. ${result.rowsProcessed.toLocaleString()} rows processed. Re-run to resume.`, + ) + p.outro('Paused.') + return + } + + p.note( + cutoverSteps(options.table, pair, result.rowsSkipped), + 'Next steps (switch reads to plaintext)', + ) + p.outro( + `Decrypt backfill complete. ${result.rowsDecrypted.toLocaleString()} rows decrypted${result.rowsSkipped > 0 ? `, ${result.rowsSkipped.toLocaleString()} skipped` : ''}.`, + ) + } catch (error) { + if ( + error instanceof BackfillConfigError || + error instanceof UndecryptableBatchError || + error instanceof PlaintextWriteError + ) { + // Author-controlled diagnostics carrying only names, counts and error + // codes — never a value — so they print verbatim. + p.log.error(error.message) + } else { + // Generic message only, exactly as `encrypt backfill`: upstream errors + // are not ours to vouch for, and a decrypt path is the last place to + // risk echoing row data. + p.log.error( + `Decrypt backfill failed${error instanceof Error && /^[\w. -]+$/.test(error.name) ? ` (${error.name})` : ''}. Re-run with diagnostic logging if you need details.`, + ) + } + exitCode = 1 + } finally { + process.off('SIGINT', onSignal) + process.off('SIGTERM', onSignal) + db?.release() + await pool.end() + } + if (exitCode) process.exit(exitCode) +} + +/** + * The twin must exist before anything is decrypted into it — the CLI never + * adds columns behind the user's migration tooling. Returns its declared type. + */ +function requirePlaintextTwin(pair: DecryptPair): string { + if (pair.plaintextSqlType !== null) return pair.plaintextSqlType + const suggestion = pair.mapping + ? ` (${pair.mapping.sqlType}, mapped from ${pair.domain})` + : '' + throw new BackfillConfigError( + `The plaintext column ${pair.table}.${pair.plaintextColumn}${suggestion} does not exist yet. Generate and apply it first:\n stash decrypt plan --table ${pair.table} --column ${pair.encryptedColumn}${pair.via === 'flag' ? ` --plaintext-column ${pair.plaintextColumn}` : ''}\nIf the twin has a different name, pass --plaintext-column .`, + ) +} + +/** + * The reverse dual-write gate — the mirror of `encrypt backfill`'s. Rows the + * application writes DURING the backfill must land in both columns, or they + * end up encrypted-only and the drop coverage check later refuses. Records a + * `decrypt_dual_writing` event on first acceptance so re-runs skip the prompt. + * + * Only the reverse lifecycle IN PROGRESS passes the gate: an + * `encrypted-dropped` event from an earlier lifecycle (the column was dropped + * and later encrypted again) asks again. + */ +async function ensureReverseDualWritesDeployed( + db: pg.PoolClient, + opts: { + pair: DecryptPair + confirmFlag: boolean + acknowledgedHazard?: { + hazard: TwinHazard + via: 'force' | 'confirm-plaintext-current' + } + }, +): Promise { + const { pair } = opts + if (pair.lifecycleActive) return true + + const isInteractive = + Boolean(process.stdin.isTTY) && process.env.CI !== 'true' + const target = `${pair.table}.${pair.encryptedColumn}` + + if (opts.confirmFlag) { + p.log.warn( + `Proceeding under --confirm-dual-writes-deployed. You are asserting that every code path that writes ${target} now also writes ${pair.plaintextColumn}. If that is not true, rows written during the backfill stay encrypted-only and you will need to re-run with --force later to recover.`, + ) + } else if (isInteractive) { + p.log.info( + `${target} has no reverse-migration events yet — the backfill requires your application to already write both \`${pair.encryptedColumn}\` (ciphertext) and \`${pair.plaintextColumn}\` (plaintext) on every insert/update.`, + ) + const ok = await p.confirm({ + message: `Has the dual-write code been deployed for ${target} → ${pair.plaintextColumn}?`, + initialValue: false, + }) + if (p.isCancel(ok) || !ok) return false + } else { + p.log.error( + `${target} has no recorded reverse \`dual-writing\` transition. Re-run with --confirm-dual-writes-deployed once your application is writing to both columns.`, + ) + return false + } + + await appendEvent(db, { + tableName: pair.table, + columnName: pair.encryptedColumn, + event: 'decrypt_dual_writing', + phase: 'decrypt-dual-writing', + details: { + plaintextColumn: pair.plaintextColumn, + // A twin that held values before this lifecycle: what was found and + // which flag the operator chose. `acknowledgedWith: 'force'` opens the + // forced-pass obligation `runDecryptBackfill` enforces on every run. + ...(opts.acknowledgedHazard + ? { + existingPlaintext: opts.acknowledgedHazard.hazard.kind, + acknowledgedWith: opts.acknowledgedHazard.via, + } + : {}), + }, + }) + p.log.success(`${target} marked as 'decrypt-dual-writing' in cs_migrations.`) + return true +} + +/** + * The warning printed when rows were skipped. Two different situations, with + * different remedies, so they are reported apart: + * + * - rows that did not decrypt (typically lock-context bound): only the end + * user's identity can decrypt them; + * - `NULL_PLAINTEXT` rows, whose ciphertext decrypts to an empty value that a + * non-JSON column can only store as the same NULL that means "not + * decrypted yet". Lock contexts have nothing to do with them. + * + * `rowsSkipped` is cumulative across resumed runs, while `skippedByCode` and + * `nullPlaintextPks` cover this run only. Only what this run's codes show is + * attributed to a cause; skips carried over from an earlier, interrupted run + * are reported apart, without guessing which kind they were. + */ +export function skippedReport( + table: string, + pair: Pick< + DecryptPair, + 'encryptedColumn' | 'plaintextColumn' | 'plaintextSqlType' + >, + result: Pick< + DecryptBackfillResult, + 'rowsSkipped' | 'skippedByCode' | 'nullPlaintextPks' + >, +): string { + const { NULL_PLAINTEXT: nullThisRun = 0, ...failedCodes } = + result.skippedByCode + const failed = Object.values(failedCodes).reduce((sum, n) => sum + n, 0) + const carriedOver = Math.max(0, result.rowsSkipped - nullThisRun - failed) + const paragraphs: string[] = [] + + if (carriedOver > 0) { + paragraphs.push( + `${carriedOver.toLocaleString()} row(s) of ${table}.${pair.encryptedColumn} were skipped by an earlier run of this backfill, before this run resumed from its checkpoint, and were left with ${pair.plaintextColumn} NULL. This run did not revisit them, so it cannot say why they were skipped: the earlier run's report, or the per-code tallies in its cipherstash.cs_migrations events, name the cause. \`stash decrypt drop\` refuses while any of them still holds ciphertext.`, + ) + } + + if (failed > 0) { + const codes = ` This run: ${formatCodes(failedCodes)}.` + paragraphs.push( + `${failed.toLocaleString()} row(s) of ${table}.${pair.encryptedColumn} could not be decrypted with these credentials and were left with ${pair.plaintextColumn} NULL.${codes} The usual cause is a value bound to an end-user lock context: only that user's identity can decrypt it, and the CLI has no way to act as them. Decrypt those rows in your application on behalf of each user (writing ${pair.plaintextColumn}), then re-run this backfill. \`stash decrypt drop\` refuses while any of them remain, because dropping ${pair.encryptedColumn} would destroy their only copy.`, + ) + } + + if (nullThisRun > 0) { + const sample = result.nullPlaintextPks + const more = + nullThisRun > sample.length ? ` (first ${sample.length} shown)` : '' + paragraphs.push( + `${nullThisRun.toLocaleString()} row(s) of ${table}.${pair.encryptedColumn} hold ciphertext that decrypts to an empty (null) value. ${pair.plaintextColumn}${pair.plaintextSqlType ? ` (${pair.plaintextSqlType})` : ''} can only store that as NULL, which is also what "not decrypted yet" looks like, so they were left NULL. This is not a credentials or lock-context problem, and re-running will not change it. \`stash decrypt drop\` refuses while they hold ciphertext. Once you have confirmed in your application that these values really are empty, clear their ciphertext and generate the drop:\n UPDATE ${qualifyTable(table)} SET ${quoteIdent(pair.encryptedColumn)} = NULL WHERE ${quoteIdent(pair.plaintextColumn)} IS NULL AND IN (...);\nPrimary keys${more}: ${sample.join(', ')}`, + ) + } + + return paragraphs.join('\n\n') +} + +function cutoverSteps( + table: string, + pair: DecryptPair, + rowsSkipped: number, +): string { + const drop = `stash decrypt drop --table ${table} --column ${pair.encryptedColumn}` + return [ + `1. Point your application's reads at ${pair.plaintextColumn} and deploy:`, + ` - Drizzle: query ${pair.plaintextColumn} directly (no decryptModel).`, + ` - Prisma Next: read the plain ${pair.plaintextColumn} field instead of the cipherstash.* one.`, + ` - Other: select ${pair.plaintextColumn} instead of decrypting ${pair.encryptedColumn}.`, + ` Keep writing BOTH ${pair.encryptedColumn} and ${pair.plaintextColumn} on every insert and update until the drop migration is applied. If writes to ${pair.encryptedColumn} stop early, a value cleared in ${pair.plaintextColumn} leaves its old ciphertext behind: \`decrypt drop\` then refuses, and a re-run of this backfill (or --force) writes the old value back.`, + rowsSkipped > 0 + ? `2. Resolve the ${rowsSkipped.toLocaleString()} undecryptable row(s) first (see the warning above), then: ${drop}` + : `2. Generate the encrypted-column drop: ${drop}`, + ].join('\n') +} diff --git a/languages/typescript/packages/cli/src/commands/decrypt/drop.ts b/languages/typescript/packages/cli/src/commands/decrypt/drop.ts new file mode 100644 index 000000000..c64839a53 --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/drop.ts @@ -0,0 +1,355 @@ +import { readUninstallSql } from '@cipherstash/eql/sql' +import { + appendEvent, + type ColumnIndexInfo, + countEqlColumnsInDatabase, + countUndecrypted, + EQL_COLUMN_PREDICATE_SQL, + listColumnIndexes, + qualifyTable, + quoteIdent, +} from '@cipherstash/migrate' +import * as p from '@clack/prompts' +import { BackfillConfigError } from '@/commands/encrypt/backfill.js' +import { + buildMigrationStem, + sqlLiteral, + writeMigrationFile, +} from '@/commands/encrypt/lib/migration-file.js' +import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' +import { loadStashConfig } from '@/config/index.js' +import { createPgClient } from '@/db/client.js' +import { + manifestPlaintextFor, + recordedPlaintextColumn, + resolveDecryptPair, +} from './lib/resolve-pair.js' + +/** + * Options accepted by `stash decrypt drop`. Generates — never applies — a + * migration that drops the encrypted column once its plaintext twin is + * complete, mirroring `stash encrypt drop`. + */ +export interface DecryptDropOptions { + /** Physical table, e.g. `users`. */ + table: string + /** The EQL v3 column to drop, e.g. `email_encrypted`. */ + column: string + /** + * The plaintext twin. Optional: the reverse backfill records the pairing, + * and a flag that disagrees with that record is refused. + */ + plaintextColumn?: string + /** Directory for the generated `.sql` (non-Drizzle). Default `./drizzle`. */ + migrationsDir?: string + /** + * Also uninstall EQL v3 in the same migration — only when this is the LAST + * EQL v3 column in the whole database, re-checked at apply time. + */ + uninstallEql?: boolean +} + +/** + * CLI handler for `stash decrypt drop` — the reverse of `stash encrypt drop`, + * and the one irreversible step of the reverse lifecycle. + * + * Gates, in order, each failing closed: + * + * 1. The column is an EQL v3 column. + * 2. Its reverse lifecycle is at `decrypted` — the backfill finished. + * 3. The plaintext twin is the one that backfill RECORDED, so coverage is + * checked against the column that was actually filled (a flag naming a + * different, fully-populated column would pass the check vacuously). + * 4. Live coverage: no row has ciphertext set and plaintext NULL. Rows the + * backfill skipped as undecryptable (lock contexts) fail this, by design. + * + * The migration then re-verifies coverage at APPLY time under the same + * ACCESS EXCLUSIVE lock the drop takes, drops every index that depends on the + * encrypted column EXPLICITLY (the EQL-term functional indexes) and names + * them here, then drops the column — all inside one `DO` block, so a failed + * check drops nothing even under a non-transactional runner. With + * `--uninstall-eql`, the EQL v3 schemas are removed in the same block when, + * and only when, no EQL v3 column remains anywhere in the database. + * + * Generating records `encrypted_dropped` (as `encrypt drop` records + * `dropped`), which stops a second migration being generated for the same + * column. It does NOT touch `.cipherstash/migrations.json`: the migration has + * not run yet, and its apply-time check can still abort it. The user removes + * the entry once it is applied; until then `stash encrypt status` reports the + * column as `reversed` rather than as drift. + */ +export async function decryptDropCommand(options: DecryptDropOptions) { + p.intro(runnerCommand(detectPackageManager(), 'stash decrypt drop')) + + const config = await loadStashConfig() + const client = createPgClient(config.databaseUrl) + let exitCode = 0 + + try { + await client.connect() + + const pair = await resolveDecryptPair(client, { + table: options.table, + encryptedColumn: options.column, + plaintextColumn: options.plaintextColumn, + }) + + const requiredPhase = 'decrypted' + if (pair.state?.phase !== requiredPhase) { + p.log.error( + `Cannot generate the drop migration: ${options.table}.${options.column} is in reverse phase '${pair.state?.phase ?? '—'}'. Must be '${requiredPhase}' — run \`stash decrypt backfill --table ${options.table} --column ${options.column}\` to completion, switch the application's reads to the plaintext column, then drop.`, + ) + exitCode = 1 + return + } + + // `resolveDecryptPair` already refused a flag that contradicts the + // record; here the record itself is required, because it is the only + // positive evidence of which column the backfill filled. + const plaintextColumn = recordedPlaintextColumn(pair.state) + if (plaintextColumn === undefined) { + p.log.error( + `${options.table}.${options.column} reached '${requiredPhase}' but no plaintext column is recorded for it, so this command cannot verify which column holds the data. Re-run \`stash decrypt backfill --table ${options.table} --column ${options.column} --plaintext-column \` (it records the pairing; already-decrypted rows are not rewritten), then drop.`, + ) + exitCode = 1 + return + } + if (pair.plaintextSqlType === null) { + p.log.error( + `The recorded plaintext column ${options.table}.${plaintextColumn} no longer exists. Refusing to drop ${options.column}, which may now hold the only copy of this data.`, + ) + exitCode = 1 + return + } + + const undecrypted = await countUndecrypted( + client, + options.table, + options.column, + plaintextColumn, + ) + if (undecrypted > 0) { + p.log.error( + `Refusing to generate the drop migration: ${undecrypted} row(s) in ${options.table} have "${options.column}" set but "${plaintextColumn}" NULL — dropping "${options.column}" would permanently destroy that data. Possible causes:\n - rows written to "${options.column}" alone since the backfill (dual-writes not deployed everywhere): re-run\n stash decrypt backfill --table ${options.table} --column ${options.column}\n - values the backfill could not decrypt (typically bound to an end-user lock context, or reported as NULL_PLAINTEXT): see the backfill's report. Decrypt lock-bound rows in your application as each user and write "${plaintextColumn}", then re-run the backfill.\n - values your application CLEARED in "${plaintextColumn}" after it stopped writing "${options.column}". Do NOT re-run the backfill for these: it would write the old value back. Restore dual-writes, then set "${options.column}" to NULL on those rows.\nThen generate the drop again.`, + ) + exitCode = 1 + return + } + p.log.success( + `Verified: no rows with "${options.column}" set and "${plaintextColumn}" NULL.`, + ) + + const indexes = await listColumnIndexes( + client, + options.table, + options.column, + ) + if (indexes.length > 0) { + p.log.info( + `The migration also drops ${indexes.length} index(es) built on ${options.column} (EQL search terms are meaningless once the column is gone):\n${indexes.map((index) => ` - ${index.schema}.${index.name}`).join('\n')}`, + ) + } + + let uninstallStatements: string[] = [] + if (options.uninstallEql) { + const remaining = (await countEqlColumnsInDatabase(client)) - 1 + if (remaining > 0) { + throw new BackfillConfigError( + `--uninstall-eql refused: ${remaining} other EQL v3 column(s) remain in this database. Uninstalling EQL drops its schemas with CASCADE, which would delete those columns and their data. Reverse them first, or drop the flag.`, + ) + } + uninstallStatements = splitUninstallSql(readUninstallSql()) + p.log.info( + `This is the last EQL v3 column in the database, so the migration also uninstalls EQL v3 — its schemas and the public.eql_v3_* column domains — re-checked at apply time. \`cipherstash.cs_migrations\` is kept as the audit trail.`, + ) + } + + const sql = buildDecryptDropSql({ + table: options.table, + encryptedColumn: options.column, + plaintextColumn, + indexes, + uninstallStatements, + }) + + const { filePath, nextStep, drizzleScaffolded } = await writeMigrationFile({ + stem: buildMigrationStem('drop_encrypted', options.table, options.column), + sql, + migrationsDir: options.migrationsDir, + }) + + await appendEvent(client, { + tableName: options.table, + columnName: options.column, + event: 'encrypted_dropped', + phase: 'encrypted-dropped', + details: { + plaintextColumn, + migrationFile: filePath, + drizzleScaffolded, + droppedIndexes: indexes.map((index) => `${index.schema}.${index.name}`), + uninstallEql: uninstallStatements.length > 0, + }, + }) + + // The forward manifest entry stays until the migration is APPLIED: this + // file has not run, and its apply-time coverage check can still abort it. + // `stash encrypt status` shows the entry as `reversed` meanwhile. + const manifestColumn = await manifestPlaintextFor( + options.table, + options.column, + ) + + p.log.success(`Migration written to ${filePath}`) + p.note( + [ + nextStep, + '', + `Keep writing BOTH ${options.column} and ${plaintextColumn} until it is applied. The migration re-checks coverage when it runs and drops nothing if a row has ciphertext without plaintext.`, + '', + `Once applied, remove ${options.column} from your schema:`, + ` - Drizzle: delete the ${options.column} column from the table definition. If the next \`drizzle-kit generate\` emits a DROP COLUMN for it, delete that statement — this migration already drops it.`, + ` - Prisma Next: remove the cipherstash.* field from the contract and re-emit it.`, + ` - Remove the table/column from your encryption client schema if nothing else on it is encrypted.`, + ...(manifestColumn !== undefined + ? [ + ` - Remove the ${options.table}.${manifestColumn} entry from .cipherstash/migrations.json: it is no longer an encryption target.`, + ] + : []), + ' - Then stop writing the encrypted column.', + ].join('\n'), + 'Next', + ) + p.outro('Encrypted-column drop migration generated.') + } catch (error) { + p.log.error( + error instanceof Error ? error.message : 'Drop generation failed.', + ) + exitCode = 1 + } finally { + await client.end() + // In `finally` for the same reason as `encrypt drop`: the guards above + // `return` from inside `try`, which would skip a trailing exit. + if (exitCode) process.exit(exitCode) + } +} + +/** + * Split `@cipherstash/eql`'s uninstall SQL into single statements so each can + * run through `EXECUTE` inside the drop's `DO` block. Line comments are + * removed first. The file is a short list of `DROP SCHEMA … CASCADE;` + * statements; anything that would make naive `;` splitting unsafe (a dollar + * quote or a string literal) is refused rather than mangled. + */ +export function splitUninstallSql(sql: string): string[] { + const stripped = sql + .split('\n') + .map((line) => line.replace(/--.*$/, '')) + .join('\n') + if (/[$']/.test(stripped)) { + throw new BackfillConfigError( + 'The installed @cipherstash/eql uninstall script is not in the shape this CLI can embed safely (it contains quoted text). Drop --uninstall-eql and run its uninstall SQL by hand after this migration.', + ) + } + return stripped + .split(';') + .map((statement) => statement.replace(/\s+/g, ' ').trim()) + .filter((statement) => statement.length > 0) +} + +/** + * Build the encrypted-column drop. As with `stash encrypt drop`, the CLI's + * coverage check goes stale the moment the file is written, so the migration + * re-checks under the lock DROP COLUMN takes anyway and aborts — dropping + * nothing — if any row gained ciphertext without plaintext in between. Every + * DDL statement runs via `EXECUTE` inside the one `DO` block, which keeps the + * check and the drops atomic even under runners that do not wrap files in a + * transaction (plain `psql -f`). + * + * Identifiers are quoted for SQL (`quoteIdent`) and then embedded in the + * `EXECUTE` string literal (`sqlLiteral`), so unusual names survive both + * layers. + */ +export function buildDecryptDropSql(opts: { + table: string + encryptedColumn: string + plaintextColumn: string + indexes: readonly ColumnIndexInfo[] + uninstallStatements: readonly string[] +}): string { + const lit = sqlLiteral + const qualifiedTable = qualifyTable(opts.table) + const quotedEncrypted = quoteIdent(opts.encryptedColumn) + const quotedPlaintext = quoteIdent(opts.plaintextColumn) + + const dropIndexes = opts.indexes.map( + (index) => + ` EXECUTE '${lit(`DROP INDEX IF EXISTS ${quoteIdent(index.schema)}.${quoteIdent(index.name)}`)}';`, + ) + + const uninstall = + opts.uninstallStatements.length > 0 + ? ` + + -- --uninstall-eql: only when no EQL v3 column remains ANYWHERE. The + -- uninstall drops the EQL schemas with CASCADE, which would take any + -- remaining encrypted column (and its data) with it. + SELECT count(*) INTO remaining_eql + FROM pg_attribute a + JOIN pg_type t ON t.oid = a.atttypid + JOIN pg_class c ON c.oid = a.attrelid + WHERE ${EQL_COLUMN_PREDICATE_SQL}; + + IF remaining_eql > 0 THEN + RAISE EXCEPTION 'stash decrypt drop: refusing to uninstall EQL — % EQL v3 column(s) still exist. Nothing was dropped.', remaining_eql; + END IF; + +${opts.uninstallStatements.map((statement) => ` EXECUTE '${lit(statement)}';`).join('\n')} + + -- The column domains (public.eql_v3_*) deliberately live outside the + -- disposable EQL schemas, so that upgrades never touch encrypted columns — + -- which also means the uninstall script leaves them behind. No column uses + -- them any more (checked above), so remove them too. No CASCADE: anything + -- else still depending on one fails this migration instead of being + -- dropped silently. + FOR leftover IN + SELECT format('%I.%I', n.nspname, t.typname) AS name + FROM pg_type t + JOIN pg_namespace n ON n.oid = t.typnamespace + WHERE t.typtype = 'd' + AND n.nspname = 'public' + AND t.typname LIKE 'eql\\_v3\\_%' + LOOP + EXECUTE 'DROP DOMAIN IF EXISTS ' || leftover.name; + END LOOP;` + : '' + + return `-- Generated by stash decrypt drop +-- Drops the encrypted column ${qualifiedTable}.${quotedEncrypted} now that its +-- plaintext twin ${quotedPlaintext} is complete. +-- Coverage is re-verified here, at apply time: the check stash ran at +-- generation time cannot see rows written after it. + +DO $stash_decrypt_drop$ +DECLARE + undecrypted bigint;${opts.uninstallStatements.length > 0 ? '\n remaining_eql bigint;\n leftover record;' : ''} +BEGIN + -- The lock DROP COLUMN takes anyway, acquired up front so the + -- count-then-drop below is atomic against concurrent writes. + LOCK TABLE ${qualifiedTable} IN ACCESS EXCLUSIVE MODE; + + SELECT count(*) INTO undecrypted + FROM ${qualifiedTable} + WHERE ${quotedEncrypted} IS NOT NULL + AND ${quotedPlaintext} IS NULL; + + IF undecrypted > 0 THEN + RAISE EXCEPTION 'stash decrypt drop: refusing to drop %.% — % row(s) have ciphertext set but % NULL. Dropping now would permanently destroy that data. Re-run: stash decrypt backfill, then regenerate this migration.', + '${lit(qualifiedTable)}', '${lit(opts.encryptedColumn)}', undecrypted, '${lit(opts.plaintextColumn)}'; + END IF; +${dropIndexes.length > 0 ? `\n -- Indexes built on the encrypted column's EQL terms.\n${dropIndexes.join('\n')}\n` : ''} + EXECUTE '${lit(`ALTER TABLE ${qualifiedTable} DROP COLUMN ${quotedEncrypted}`)}';${uninstall} +END +$stash_decrypt_drop$; +` +} diff --git a/languages/typescript/packages/cli/src/commands/decrypt/lib/chunk-size.ts b/languages/typescript/packages/cli/src/commands/decrypt/lib/chunk-size.ts new file mode 100644 index 000000000..f637173de --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/lib/chunk-size.ts @@ -0,0 +1,47 @@ +import { MAX_DECRYPT_CHUNK_SIZE } from '@cipherstash/migrate' + +/** + * `decrypt backfill --chunk-size` validation. + * + * `Number(...)` accepts `1e3`, `0x10` and `Infinity`, and has no ceiling. The + * ceiling matters here: each decrypted row binds two parameters in the + * chunk's write, and Postgres caps a statement at 65535 parameters, so a + * chunk above {@link MAX_DECRYPT_CHUNK_SIZE} rows would fail at the write, + * after its ZeroKMS request had already been made. + * + * The shape rule (plain decimal digits, a positive safe integer) is the one + * `encrypt backfill` adopts in #1194 (`parseChunkSize` in + * `src/cli/chunk-size.ts`). When that lands, this function should delegate + * the shape check to it and keep only the maximum. + */ + +/** + * Thrown for a chunk size that is not a whole number from 1 to + * {@link MAX_DECRYPT_CHUNK_SIZE}. The message names the flag and quotes what + * was received; it carries no row data, so it prints verbatim. + */ +export class InvalidDecryptChunkSizeError extends Error { + constructor(message: string) { + super(message) + this.name = 'InvalidDecryptChunkSizeError' + } +} + +/** + * Parse a `--chunk-size` flag's raw argv text, so the error can quote exactly + * what was typed. + */ +export function parseDecryptChunkSize(input: string): number { + const value = /^\s*\d+\s*$/.test(input) ? Number(input) : Number.NaN + + if ( + !Number.isSafeInteger(value) || + value < 1 || + value > MAX_DECRYPT_CHUNK_SIZE + ) { + throw new InvalidDecryptChunkSizeError( + `--chunk-size must be a whole number of rows from 1 to ${MAX_DECRYPT_CHUNK_SIZE} (got ${JSON.stringify(input)}). Each row binds two parameters in the write, and Postgres allows 65535 per statement.`, + ) + } + return value +} diff --git a/languages/typescript/packages/cli/src/commands/decrypt/lib/orm-hints.ts b/languages/typescript/packages/cli/src/commands/decrypt/lib/orm-hints.ts new file mode 100644 index 000000000..c4caa8f5b --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/lib/orm-hints.ts @@ -0,0 +1,59 @@ +import type { PlaintextSqlType } from '@cipherstash/migrate' + +/** + * The `drizzle-orm/pg-core` column builder for each plaintext type the reverse + * mapping can produce. Total over {@link PlaintextSqlType} — a type added to + * the mapping without a builder here fails to compile. + * + * `bigint` uses `mode: 'bigint'` because that is what the encrypted column + * decrypted to (a JS `bigint`, the full i64 range); `mode: 'number'` would + * silently lose precision above 2^53. `timestamptz` keeps the time zone for the + * reason the mapping chose it: the encrypted value was an instant. + */ +const DRIZZLE_BUILDERS: Record< + PlaintextSqlType, + { fn: string; options?: string } +> = { + text: { fn: 'text' }, + integer: { fn: 'integer' }, + smallint: { fn: 'smallint' }, + bigint: { fn: 'bigint', options: "{ mode: 'bigint' }" }, + numeric: { fn: 'numeric' }, + real: { fn: 'real' }, + 'double precision': { fn: 'doublePrecision' }, + boolean: { fn: 'boolean' }, + date: { fn: 'date' }, + timestamptz: { fn: 'timestamp', options: '{ withTimezone: true }' }, + jsonb: { fn: 'jsonb' }, +} + +/** + * The Drizzle schema line that declares a plaintext twin, e.g. + * `email: text('email'),`. Nullable on purpose: existing rows have no + * plaintext until `stash decrypt backfill` fills them. + */ +export function drizzleColumnLine( + column: string, + sqlType: PlaintextSqlType, +): string { + const { fn, options } = DRIZZLE_BUILDERS[sqlType] + const key = /^[A-Za-z_$][A-Za-z0-9_$]*$/.test(column) + ? column + : JSON.stringify(column) + const args = [singleQuoted(column), options] + .filter((arg): arg is string => arg !== undefined) + .join(', ') + return `${key}: ${fn}(${args}),` +} + +/** A TypeScript single-quoted string literal for `value`. */ +function singleQuoted(value: string): string { + return `'${value.replace(/\\/g, '\\\\').replace(/'/g, "\\'")}'` +} + +/** The `drizzle-orm/pg-core` imports a set of plaintext types needs. */ +export function drizzleImports(types: Iterable): string[] { + return [ + ...new Set([...types].map((type) => DRIZZLE_BUILDERS[type].fn)), + ].sort() +} diff --git a/languages/typescript/packages/cli/src/commands/decrypt/lib/resolve-pair.ts b/languages/typescript/packages/cli/src/commands/decrypt/lib/resolve-pair.ts new file mode 100644 index 000000000..1657df59f --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/lib/resolve-pair.ts @@ -0,0 +1,344 @@ +import { + columnExists, + columnHasValues, + columnSqlType, + detectColumnEqlVersion, + isReverseMigrationPhase, + listEncryptedColumns, + type MigrationStateRow, + type PlaintextTypeMapping, + plaintextTypeForEqlDomain, + readManifest, +} from '@cipherstash/migrate' +import type pg from 'pg' +import { BackfillConfigError } from '@/commands/encrypt/backfill.js' +import { progressSafe } from '@/commands/encrypt/lib/db-readers.js' + +/** + * How the plaintext twin's name was decided. Destructive steps trust only the + * first two: `flag` is the user saying so, `recorded` is the pairing the + * reverse backfill actually ran against. The rest are proposals. + */ +export type PlaintextResolution = + | 'flag' + | 'recorded' + | 'manifest' + | 'convention' + | 'fallback' + +/** One encrypted column and the plaintext twin it reverses into. */ +export interface DecryptPair { + table: string + /** The EQL v3 column being reversed — the source. */ + encryptedColumn: string + /** Its unqualified domain, e.g. `eql_v3_text_search`. */ + domain: string + /** + * The plaintext type the domain maps back to, or `null` for a domain this + * release cannot map (a SteVec fragment, or a domain newer than the CLI). + */ + mapping: PlaintextTypeMapping | null + /** The plaintext twin — the target. */ + plaintextColumn: string + via: PlaintextResolution + /** + * The twin's declared type (`format_type`) when it already exists, else + * `null`. Present means `stash decrypt plan` has nothing left to add. + */ + plaintextSqlType: string | null + /** + * Latest REVERSE event for this encrypted column, if any. It can belong to + * an earlier, finished lifecycle; {@link lifecycleActive} says whether it + * is the one in progress. + */ + state: MigrationStateRow | null + /** Whether {@link state} belongs to the reverse lifecycle in progress. */ + lifecycleActive: boolean + /** + * Latest FORWARD event recorded under the plaintext twin's name: the + * forward lifecycle that encrypted it, when there was one. Forward events + * are keyed by the plaintext column, so this is where they live. + */ + forwardState: MigrationStateRow | null +} + +/** + * The reverse lifecycle's state for one encrypted column: its latest event, + * but only when that event belongs to the reverse lifecycle. Reverse events + * are keyed by the encrypted column precisely so forward history never shows + * up here, but a hand-written or legacy row under the same key must not be + * mistaken for reverse progress either. + */ +export async function reverseState( + client: pg.ClientBase, + table: string, + encryptedColumn: string, +): Promise { + // `cs_migrations` not installed yet reads as no reverse history, which is + // a normal state for `plan`/`status` on a fresh project. + const last = await progressSafe(client, table, encryptedColumn) + return last && isReverseMigrationPhase(last.phase) ? last : null +} + +/** + * The forward lifecycle's state for a plaintext column: its latest event, but + * only when that event is a forward one. `null` when there is no history or + * no `cs_migrations` table yet. + */ +export async function forwardState( + client: pg.ClientBase, + table: string, + plaintextColumn: string, +): Promise { + const last = await progressSafe(client, table, plaintextColumn) + return last && !isReverseMigrationPhase(last.phase) ? last : null +} + +/** + * Whether the latest reverse event belongs to the lifecycle in progress, as + * opposed to an earlier one that finished. + * + * `decrypt-dual-writing`, `decrypting` and `decrypted` are always current. + * `encrypted-dropped` is recorded when the drop migration is GENERATED, and + * the encrypted column still exists (it is being resolved), so it means one + * of two things: the migration has not been applied, or its apply-time check + * aborted. That is still the same lifecycle. Or the column was dropped and + * later encrypted again: then the forward lifecycle that re-encrypted it has + * recorded events under the twin's name AFTER the drop. Event ids are + * monotonic, so the order tells them apart. + */ +export function isActiveReverseLifecycle( + state: MigrationStateRow | null, + forwardOfRecordedTwin: MigrationStateRow | null, +): boolean { + if (state === null) return false + if (state.phase !== 'encrypted-dropped') return true + return ( + forwardOfRecordedTwin === null || + BigInt(forwardOfRecordedTwin.id) < BigInt(state.id) + ) +} + +/** + * The plaintext name derived from the encrypted column's own name when + * nothing better is known: the `_encrypted` convention stripped, else + * `_plaintext`, so a derived name never collides with the + * encrypted column itself. Shared by every command that shows a twin name. + */ +export function derivePlaintextName(encryptedColumn: string): { + name: string + via: 'convention' | 'fallback' +} { + const suffix = '_encrypted' + return encryptedColumn.endsWith(suffix) && + encryptedColumn.length > suffix.length + ? { name: encryptedColumn.slice(0, -suffix.length), via: 'convention' } + : { name: `${encryptedColumn}_plaintext`, via: 'fallback' } +} + +/** `details.plaintextColumn` from a reverse event, when it recorded one. */ +export function recordedPlaintextColumn( + state: MigrationStateRow | null, +): string | undefined { + const value = state?.details?.plaintextColumn + return typeof value === 'string' && value.length > 0 ? value : undefined +} + +/** + * Resolve an encrypted column and the plaintext twin it reverses into. + * + * The SOURCE must be an EQL v3 column — named explicitly, never guessed — + * because the reverse lifecycle starts from what exists. The TARGET's name is + * decided in this order: + * + * 1. `--plaintext-column`, when given; + * 2. the pairing the reverse lifecycle IN PROGRESS recorded in + * `cs_migrations` — a flag that disagrees with it is refused rather than + * silently switching which column the lifecycle fills (and which column + * `decrypt drop`'s coverage gate then checks). A pairing from an earlier, + * finished lifecycle is not binding (see {@link isActiveReverseLifecycle}); + * 3. the forward manifest's entry whose `encryptedColumn` is this column — + * the original plaintext name, which is what most reversals want back; + * 4. the `_encrypted` convention, stripped; + * 5. `_plaintext`, so a name that cannot be derived never + * collides with the encrypted column itself. + * + * Throws {@link BackfillConfigError} (printed verbatim by the commands) for a + * source that is missing or not EQL v3, and for a target that is itself an + * EQL column. + */ +export async function resolveDecryptPair( + client: pg.ClientBase, + opts: { + table: string + encryptedColumn: string + plaintextColumn?: string + }, +): Promise { + const { table, encryptedColumn } = opts + + const candidates = await listEncryptedColumns(client, table) + const source = candidates.find((c) => c.column === encryptedColumn) + if (!source) { + if (!(await columnExists(client, table, encryptedColumn))) { + throw new BackfillConfigError( + `Column ${encryptedColumn} does not exist on ${table}. Pass the ENCRYPTED column with --column (the EQL v3 column you want to move back to plaintext).${candidates.length > 0 ? ` EQL v3 columns on ${table}: ${candidates.map((c) => c.column).join(', ')}.` : ''}`, + ) + } + throw new BackfillConfigError( + `${table}.${encryptedColumn} is not an EQL v3 column, so there is nothing to decrypt. Pass the ENCRYPTED column with --column.${candidates.length > 0 ? ` EQL v3 columns on ${table}: ${candidates.map((c) => c.column).join(', ')}.` : ''}`, + ) + } + + const state = await reverseState(client, table, encryptedColumn) + const recordedTwin = recordedPlaintextColumn(state) + const recordedForward = + recordedTwin === undefined + ? null + : await forwardState(client, table, recordedTwin) + const lifecycleActive = isActiveReverseLifecycle(state, recordedForward) + const recorded = lifecycleActive ? recordedTwin : undefined + + let plaintextColumn: string + let via: PlaintextResolution + if (opts.plaintextColumn !== undefined) { + if (recorded !== undefined && recorded !== opts.plaintextColumn) { + throw new BackfillConfigError( + `${table}.${encryptedColumn} is already being reversed into "${recorded}" (recorded in cipherstash.cs_migrations), but --plaintext-column is "${opts.plaintextColumn}". Refusing to switch targets mid-lifecycle: the coverage check before the encrypted column is dropped would then verify the wrong column. Drop the flag to continue with "${recorded}".`, + ) + } + plaintextColumn = opts.plaintextColumn + via = 'flag' + } else if (recorded !== undefined) { + plaintextColumn = recorded + via = 'recorded' + } else { + const fromManifest = await manifestPlaintextFor(table, encryptedColumn) + if (fromManifest) { + plaintextColumn = fromManifest + via = 'manifest' + } else { + ;({ name: plaintextColumn, via } = derivePlaintextName(encryptedColumn)) + } + } + + if (plaintextColumn === encryptedColumn) { + throw new BackfillConfigError( + `The plaintext column cannot be the encrypted column itself (${encryptedColumn}). Pass --plaintext-column .`, + ) + } + if ((await detectColumnEqlVersion(client, table, plaintextColumn)) !== null) { + throw new BackfillConfigError( + `${table}.${plaintextColumn} is itself an EQL v3 column, so it cannot hold plaintext. Pass --plaintext-column to choose another target.`, + ) + } + + return { + table, + encryptedColumn, + domain: source.domain, + mapping: plaintextTypeForEqlDomain(source.domain), + plaintextColumn, + via, + plaintextSqlType: await columnSqlType(client, table, plaintextColumn), + state, + lifecycleActive, + forwardState: + plaintextColumn === recordedTwin + ? recordedForward + : await forwardState(client, table, plaintextColumn), + } +} + +/** + * Why a plaintext twin that already exists cannot be trusted as-is when a + * reverse lifecycle starts: it already holds values (`prepopulated`), from + * wherever. The reverse backfill only fills NULLs, so it would keep those + * values, record `decrypted`, and let `decrypt drop` destroy the only current + * copy, in the encrypted column. (A twin that is the source of a forward + * migration still open is not a hazard to acknowledge but a refusal: + * {@link openForwardMigrationMessage}.) + */ +export type TwinHazard = { kind: 'prepopulated' } + +/** + * Check a twin before the FIRST reverse run trusts it. Returns `null` when + * there is nothing to check (the twin does not exist yet, or a reverse + * lifecycle is already in progress, so its values came from this lifecycle's + * own backfill and dual-writes) or nothing wrong. + */ +export async function assessExistingTwin( + client: pg.ClientBase, + pair: DecryptPair, +): Promise { + if (pair.lifecycleActive || pair.plaintextSqlType === null) return null + const hasValues = await columnHasValues( + client, + pair.table, + pair.plaintextColumn, + ) + return hasValues ? { kind: 'prepopulated' } : null +} + +/** + * Why no reverse migration may start on this pair yet, or `null`: the twin is + * the source column of a forward migration into the encrypted column that was + * never dropped. At most one column migration is open per column pair, so the + * forward one must be finished (`stash encrypt drop`) or abandoned first. + * Names and phases only, never a value. + */ +export function openForwardMigrationMessage( + pair: Pick< + DecryptPair, + 'table' | 'encryptedColumn' | 'plaintextColumn' | 'forwardState' + >, +): string | null { + const forward = pair.forwardState + if (forward === null || forward.phase === 'dropped') return null + return `${pair.table}.${pair.plaintextColumn} is the source of a forward encryption into ${pair.encryptedColumn} that is still open at '${forward.phase}' in cipherstash.cs_migrations. A reverse migration cannot start while it is open. Finish it with \`stash encrypt drop\` first, or, to abandon encryption instead, stop writing ${pair.encryptedColumn}, switch reads back to ${pair.plaintextColumn} and drop ${pair.encryptedColumn}: while dual-writes are live, ${pair.plaintextColumn} is still current and there is nothing to decrypt.` +} + +/** The two ways past a {@link TwinHazard}, as `decrypt backfill` flags. */ +export const TWIN_HAZARD_FLAGS = { + overwrite: '--force', + keep: '--confirm-plaintext-current', +} as const + +/** + * The refusal (in `decrypt backfill`) or warning (in `decrypt plan`) for a + * {@link TwinHazard}. Names and phases only, never a value. + */ +export function twinHazardMessage( + pair: Pick, +): string { + const twin = `${pair.table}.${pair.plaintextColumn}` + const source = `${pair.table}.${pair.encryptedColumn}` + return [ + `${twin} already holds values, and no reverse migration is in progress for ${source}. Nothing records where they came from or whether they are current.`, + `The reverse backfill only fills rows where ${pair.plaintextColumn} is NULL, so it would keep those values, mark the column decrypted, and \`stash decrypt drop\` would then drop ${pair.encryptedColumn}: the only current copy.`, + 'Choose one, once the application writes BOTH columns on every insert and update:', + ` ${TWIN_HAZARD_FLAGS.overwrite} overwrite every row of ${pair.plaintextColumn} with the decrypted ciphertext (the safe choice when in doubt).`, + ` ${TWIN_HAZARD_FLAGS.keep} keep the existing values. Only when the application has written ${pair.plaintextColumn} on every insert and update for the whole time ${pair.encryptedColumn} existed.`, + ].join('\n') +} + +/** + * The forward manifest's plaintext name for this encrypted column: the entry + * whose recorded `encryptedColumn` is it, or — for entries written before that + * field existed — whose `_encrypted` convention names it. + */ +export async function manifestPlaintextFor( + table: string, + encryptedColumn: string, +): Promise { + const manifest = await readManifest() + const entries = manifest?.tables[table] ?? [] + const entry = + entries.find((e) => e.encryptedColumn === encryptedColumn) ?? + entries.find( + (e) => + e.encryptedColumn === undefined && + `${e.column}_encrypted` === encryptedColumn, + ) + return entry?.column +} diff --git a/languages/typescript/packages/cli/src/commands/decrypt/plan.ts b/languages/typescript/packages/cli/src/commands/decrypt/plan.ts new file mode 100644 index 000000000..3ef9cc43f --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/plan.ts @@ -0,0 +1,421 @@ +import { + appendEvent, + classifyEqlDomain, + listEncryptedColumns, + type PlaintextSqlType, + qualifyTable, + quoteIdent, +} from '@cipherstash/migrate' +import * as p from '@clack/prompts' +import { detectDrizzle, detectPrismaNext } from '@/commands/db/detect.js' +import { BackfillConfigError } from '@/commands/encrypt/backfill.js' +import { fetchPhysicalColumns } from '@/commands/encrypt/lib/db-readers.js' +import { + buildMigrationStem, + writeMigrationFile, +} from '@/commands/encrypt/lib/migration-file.js' +import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' +import { loadStashConfig } from '@/config/index.js' +import { createPgClient } from '@/db/client.js' +import { drizzleColumnLine, drizzleImports } from './lib/orm-hints.js' +import { + assessExistingTwin, + type DecryptPair, + openForwardMigrationMessage, + resolveDecryptPair, + TWIN_HAZARD_FLAGS, + type TwinHazard, + twinHazardMessage, +} from './lib/resolve-pair.js' + +/** + * Options accepted by `stash decrypt plan`. With no `table` the command is a + * read-only overview; with one it generates the plaintext-twin migration. + */ +export interface DecryptPlanOptions { + /** Generate for this table. Omit for the read-only overview. */ + table?: string + /** + * One ENCRYPTED column on `table`. Omit to plan every EQL v3 column on it. + */ + column?: string + /** Name the plaintext twin (single column only). */ + plaintextColumn?: string + /** Directory for the generated `.sql` (non-Drizzle). Default `./drizzle`. */ + migrationsDir?: string + /** Print the SQL and the schema change; write nothing. */ + dryRun?: boolean +} + +/** + * CLI handler for `stash decrypt plan` — step one of moving encrypted columns + * back to plaintext. + * + * Without `--table` it mirrors `stash encrypt plan`: a read-only list of every + * EQL v3 column in the current schema, the plaintext twin each would reverse + * into (type mapped back from its domain), where its reverse lifecycle stands, + * and the next command to run. + * + * With `--table` it produces the change that ADDS the plaintext twins: + * + * - **Drizzle** — the schema lines to add, for `drizzle-kit generate` to turn + * into a migration. Deliberately not a `--custom` migration: a hand-written + * ADD COLUMN never reaches drizzle-kit's snapshot, so the next `generate` + * after the schema edit would emit the same ADD COLUMN again and fail at + * migrate time with "column already exists". + * - **Everything else** — a reviewed `.sql` migration (`ADD COLUMN IF NOT + * EXISTS`), written but not applied, exactly as `stash encrypt drop` + * delivers its migration. Prisma Next projects also get the contract change + * to make. + * + * Twins are added NULLABLE: existing rows have no plaintext until + * `stash decrypt backfill` fills them. Restore a NOT NULL afterwards if the + * original column had one. + * + * A twin that already exists is checked the way `decrypt backfill`'s first + * run checks it: when it is the source of a forward encryption that was never + * dropped ({@link openForwardMigrationMessage}), or holds values + * ({@link assessExistingTwin}), plan warns with the same text the backfill + * refuses with. Plan writes nothing for such a twin, so it warns rather than + * fails. + */ +export async function decryptPlanCommand(options: DecryptPlanOptions = {}) { + p.intro(runnerCommand(detectPackageManager(), 'stash decrypt plan')) + + const config = await loadStashConfig() + const client = createPgClient(config.databaseUrl) + let exitCode = 0 + + try { + await client.connect() + + if (options.table === undefined) { + await renderOverview(client) + return + } + + if (options.plaintextColumn !== undefined && options.column === undefined) { + throw new BackfillConfigError( + '--plaintext-column names ONE twin, so it needs --column too.', + ) + } + + const targets = + options.column !== undefined + ? [options.column] + : (await listEncryptedColumns(client, options.table)).map( + (c) => c.column, + ) + if (targets.length === 0) { + p.log.info(`${options.table} has no EQL v3 columns. Nothing to plan.`) + p.outro('Nothing to plan.') + return + } + + const pairs: DecryptPair[] = [] + for (const encryptedColumn of targets) { + pairs.push( + await resolveDecryptPair(client, { + table: options.table, + encryptedColumn, + plaintextColumn: options.plaintextColumn, + }), + ) + } + + const unmappable = pairs.filter((pair) => pair.mapping === null) + if (unmappable.length > 0) { + throw new BackfillConfigError( + `No plaintext type is known for ${unmappable.map((pair) => `${options.table}.${pair.encryptedColumn} (${pair.domain})`).join(', ')}. Add the plaintext column by hand with the type its values should have, then run \`stash decrypt backfill\` — the backfill casts to whatever type the column declares.`, + ) + } + + const existing = pairs.filter((pair) => pair.plaintextSqlType !== null) + const hazards = new Map() + for (const pair of existing) { + const openForward = openForwardMigrationMessage(pair) + if (openForward !== null) { + p.log.warn( + `\`stash decrypt backfill\` will refuse ${options.table}.${pair.encryptedColumn}: ${openForward}`, + ) + continue + } + const hazard = await assessExistingTwin(client, pair) + if (hazard === null) { + p.log.info( + `${options.table}.${pair.plaintextColumn} already exists (${pair.plaintextSqlType}) — nothing to add for ${pair.encryptedColumn}.`, + ) + continue + } + hazards.set(pair, hazard) + p.log.warn( + `${options.table}.${pair.plaintextColumn} already exists (${pair.plaintextSqlType}), but it cannot be trusted as the plaintext twin of ${pair.encryptedColumn} as it stands. \`stash decrypt backfill\` will refuse it until you choose:\n${twinHazardMessage(pair)}`, + ) + } + const toAdd = pairs.filter( + (pair): pair is MappedPair => + pair.plaintextSqlType === null && pair.mapping !== null, + ) + + // A twin that does not exist cannot be the live source of a forward + // encryption, so a forward lifecycle still open under its name lost its + // source column outside stash (an ORM migration, say). Plan is the only + // step that can see that: once the twin below is added, the column is + // back and indistinguishable from a forward source that was never + // dropped, which `decrypt backfill` refuses. So the forward lifecycle is + // closed here, recorded as observed rather than as a drop stash generated. + for (const pair of toAdd) { + const openForward = openForwardMigrationMessage(pair) + if (openForward === null || pair.forwardState === null) continue + const what = `${options.table}.${pair.plaintextColumn} does not exist, but the forward encryption into ${pair.encryptedColumn} is still open at '${pair.forwardState.phase}' in cipherstash.cs_migrations: the column was removed outside stash.` + if (options.dryRun) { + p.log.info( + `${what} Without --dry-run, plan would record that forward migration as dropped so \`stash decrypt backfill\` can start.`, + ) + continue + } + await appendEvent(client, { + tableName: options.table, + columnName: pair.plaintextColumn, + event: 'dropped', + phase: 'dropped', + details: { observedAbsent: true, recordedBy: 'stash decrypt plan' }, + }) + p.log.info(`${what} Recorded it as dropped.`) + } + + for (const pair of toAdd) { + p.log.step( + `${options.table}.${pair.encryptedColumn} (${pair.domain}) → ${pair.plaintextColumn} ${pair.mapping.sqlType}${describeVia(pair)}`, + ) + } + + if (toAdd.length === 0) { + p.note(nextSteps(options.table, pairs, hazards), 'Next') + p.outro('Plaintext twins already in place.') + return + } + + const cwd = process.cwd() + if (detectDrizzle(cwd)) { + const lines = toAdd.map((pair) => + drizzleColumnLine(pair.plaintextColumn, pair.mapping.sqlType), + ) + const imports = drizzleImports(toAdd.map((pair) => pair.mapping.sqlType)) + p.note( + [ + `Add to the ${options.table} table in your Drizzle schema (nullable — existing rows are empty until the backfill runs):`, + '', + ...lines.map((line) => ` ${line}`), + '', + `Import from 'drizzle-orm/pg-core': ${imports.join(', ')}`, + '', + `Then generate and apply the migration:\n ${runnerCommand(detectPackageManager(), 'drizzle-kit generate')}\n ${runnerCommand(detectPackageManager(), 'drizzle-kit migrate')}`, + '', + 'Keep the encrypted column declared, and keep writing it, until the `stash decrypt drop` migration is applied. The application dual-writes both columns from now until then.', + ].join('\n'), + 'Drizzle schema change', + ) + p.note(nextSteps(options.table, pairs, hazards), 'Next') + p.outro( + 'Plan ready. No files written (drizzle-kit generates the migration).', + ) + return + } + + const sql = buildAddPlaintextSql(options.table, toAdd) + if (detectPrismaNext(cwd)) { + p.note( + toAdd + .map( + (pair) => + ` ${pair.plaintextColumn}: a plain (non-cipherstash) field mapping to Postgres ${pair.mapping.sqlType}, optional`, + ) + .join('\n'), + 'Prisma Next contract change', + ) + } + + if (options.dryRun) { + p.note(sql, 'Migration (dry run — not written)') + p.outro('Dry run complete.') + return + } + + const { filePath, nextStep } = await writeMigrationFile({ + stem: buildMigrationStem( + 'add_plaintext', + options.table, + ...toAdd.map((pair) => pair.plaintextColumn), + ), + sql, + migrationsDir: options.migrationsDir, + }) + p.log.success(`Migration written to ${filePath}`) + p.note(`${nextStep}\n\n${nextSteps(options.table, pairs, hazards)}`, 'Next') + p.outro('Plaintext-twin migration generated.') + } catch (error) { + p.log.error(error instanceof Error ? error.message : 'Plan failed.') + exitCode = 1 + } finally { + await client.end() + if (exitCode) process.exit(exitCode) + } +} + +/** A pair whose domain maps to a known plaintext type. */ +type MappedPair = DecryptPair & { + mapping: NonNullable +} + +/** + * `ALTER TABLE … ADD COLUMN IF NOT EXISTS` for each twin. `IF NOT EXISTS` + * keeps a re-applied or hand-raced migration from failing; the nullable + * column is what lets it apply to a populated table without a default. + */ +export function buildAddPlaintextSql( + table: string, + pairs: ReadonlyArray<{ + encryptedColumn: string + domain: string + plaintextColumn: string + mapping: { sqlType: PlaintextSqlType } + }>, +): string { + const qualifiedTable = qualifyTable(table) + const statements = pairs.map( + (pair) => + `-- ${pair.encryptedColumn} (${pair.domain}) decrypts to ${pair.mapping.sqlType}\nALTER TABLE ${qualifiedTable} ADD COLUMN IF NOT EXISTS ${quoteIdent(pair.plaintextColumn)} ${pair.mapping.sqlType};`, + ) + return `-- Generated by stash decrypt plan +-- Adds a NULLABLE plaintext twin for each encrypted column on ${qualifiedTable}. +-- Existing rows stay NULL until \`stash decrypt backfill\` fills them. + +${statements.join('\n\n')} +` +} + +function describeVia(pair: DecryptPair): string { + switch (pair.via) { + case 'flag': + return '' + case 'recorded': + return ' (recorded by an earlier reverse run)' + case 'manifest': + return ' (original name, from .cipherstash/migrations.json)' + case 'convention': + return ' (from the _encrypted convention; override with --plaintext-column)' + case 'fallback': + return ' (no original name known; override with --plaintext-column)' + } +} + +function nextSteps( + table: string, + pairs: readonly DecryptPair[], + hazards: ReadonlyMap = new Map(), +): string { + const runner = (command: string) => + runnerCommand(detectPackageManager(), command) + const pinTarget = (pair: DecryptPair) => + pair.via === 'flag' ? ` --plaintext-column ${pair.plaintextColumn}` : '' + const choice = (pair: DecryptPair) => + hazards.has(pair) + ? ` <${TWIN_HAZARD_FLAGS.overwrite} | ${TWIN_HAZARD_FLAGS.keep}>` + : '' + return [ + '1. Apply the migration, then deploy application code that writes BOTH the encrypted column and its plaintext twin on every insert/update.', + '2. Backfill the plaintext:', + ...pairs.map( + (pair) => + ` ${runner(`stash decrypt backfill --table ${table} --column ${pair.encryptedColumn}${pinTarget(pair)}${choice(pair)}`)}`, + ), + '3. Switch reads to the plaintext column(s) and deploy, still writing BOTH columns, then generate the encrypted-column drop (keep dual-writing until it is applied):', + ...pairs.map( + (pair) => + ` ${runner(`stash decrypt drop --table ${table} --column ${pair.encryptedColumn}`)}`, + ), + ].join('\n') +} + +/** + * Read-only overview: every EQL v3 column in the current schema, its proposed + * or recorded plaintext twin, and where its reverse lifecycle stands. Mirrors + * `stash encrypt plan`'s `+` / `→` / `✓` lines. + */ +async function renderOverview( + client: ReturnType, +): Promise { + const physical = await fetchPhysicalColumns(client) + const lines: string[] = [] + const runner = (command: string) => + runnerCommand(detectPackageManager(), command) + + for (const [table, columns] of physical) { + for (const [column, domain] of columns) { + if (domain === null || classifyEqlDomain(domain) !== 3) continue + const key = `${table}.${column}` + let pair: DecryptPair + try { + pair = await resolveDecryptPair(client, { + table, + encryptedColumn: column, + }) + } catch (error) { + // A pairing that cannot be resolved (say, the derived twin name is + // itself an EQL column) is exactly what the user needs to see here. + if (!(error instanceof BackfillConfigError)) throw error + lines.push(` ! ${key}: ${error.message}`) + continue + } + const target = `${pair.plaintextColumn} ${pair.plaintextSqlType ?? pair.mapping?.sqlType ?? '(unknown type)'}` + // An `encrypted-dropped` from an earlier, finished lifecycle is history, + // not where this column stands now. + const phase = pair.lifecycleActive ? pair.state?.phase : undefined + + if (phase === 'encrypted-dropped') { + lines.push( + ` ✓ ${key}: encrypted-dropped (apply the generated drop migration)`, + ) + } else if (phase === 'decrypted') { + lines.push( + ` → ${key} → ${pair.plaintextColumn}: decrypted → switch reads to ${pair.plaintextColumn}, then ${runner(`stash decrypt drop --table ${table} --column ${column}`)}`, + ) + } else if (phase !== undefined) { + lines.push( + ` → ${key} → ${pair.plaintextColumn}: ${phase} → ${runner(`stash decrypt backfill --table ${table} --column ${column}`)}`, + ) + } else if ( + pair.forwardState !== null && + pair.forwardState.phase !== 'dropped' + ) { + lines.push( + ` ! ${key} → ${target}: forward encryption still open at '${pair.forwardState.phase}' — finish or abandon it first, see ${runner(`stash decrypt plan --table ${table} --column ${column}`)}`, + ) + } else if (pair.plaintextSqlType !== null) { + const hazard = await assessExistingTwin(client, pair) + lines.push( + hazard === null + ? ` → ${key} → ${target}: twin exists → ${runner(`stash decrypt backfill --table ${table} --column ${column}`)}` + : ` ! ${key} → ${target}: twin already holds values; the backfill needs ${TWIN_HAZARD_FLAGS.overwrite} or ${TWIN_HAZARD_FLAGS.keep} — see ${runner(`stash decrypt plan --table ${table} --column ${column}`)}`, + ) + } else if (pair.mapping === null) { + lines.push( + ` ! ${key} (${pair.domain}): no known plaintext type — add the twin by hand`, + ) + } else { + lines.push( + ` + ${key} (${pair.domain}) → ${target}: add the plaintext twin — ${runner(`stash decrypt plan --table ${table} --column ${column}`)}`, + ) + } + } + } + + if (lines.length === 0) { + p.log.info('No EQL v3 columns in the current schema. Nothing to reverse.') + p.outro('Nothing to plan.') + return + } + p.note(lines.join('\n'), 'Reverse plan') + p.outro('Done.') +} diff --git a/languages/typescript/packages/cli/src/commands/decrypt/status.ts b/languages/typescript/packages/cli/src/commands/decrypt/status.ts new file mode 100644 index 000000000..74ce3985e --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/decrypt/status.ts @@ -0,0 +1,257 @@ +import { + classifyEqlDomain, + isReverseMigrationPhase, + type MigrationStateRow, + plaintextTypeForEqlDomain, +} from '@cipherstash/migrate' +import * as p from '@clack/prompts' +import { + fetchPhysicalColumns, + latestByColumnSafe, +} from '@/commands/encrypt/lib/db-readers.js' +import { formatTable } from '@/commands/encrypt/lib/table-format.js' +import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' +import { loadStashConfig } from '@/config/index.js' +import { createPgClient } from '@/db/client.js' +import { + derivePlaintextName, + isActiveReverseLifecycle, + manifestPlaintextFor, + recordedPlaintextColumn, +} from './lib/resolve-pair.js' + +interface Row { + table: string + column: string + domain: string + plaintext: string + phase: string + progress: string + skipped: string + flags: string +} + +const STATUS_HEADERS: Row = { + table: 'TABLE', + column: 'COLUMN', + domain: 'DOMAIN', + plaintext: 'PLAINTEXT', + phase: 'PHASE', + progress: 'PROGRESS', + skipped: 'UNDECRYPTABLE', + flags: 'FLAGS', +} + +/** + * CLI handler for `stash decrypt status` — the reverse twin of + * `stash encrypt status`. One row per EQL v3 column in the current schema, + * plus any column whose reverse lifecycle is on record but whose encrypted + * column is already gone, merging: + * + * - `information_schema.columns` — which columns are encrypted, their domain, + * and whether the plaintext twin exists yet; + * - the latest REVERSE event per encrypted column in `cs_migrations` — phase, + * progress, and the count of values the backfill could not decrypt. + * + * The twin name comes from the same rules `decrypt plan` and `decrypt + * backfill` use (recorded pairing of the lifecycle in progress, then the + * manifest, then {@link derivePlaintextName}), so the three commands never + * show different names for one column. A twin that is itself an EQL column, + * which `plan` and `backfill` refuse, is flagged `plaintext-col-is-eql`. + * + * Read-only, and runs no per-row coverage queries (as `encrypt status`). + */ +export async function decryptStatusCommand() { + p.intro(runnerCommand(detectPackageManager(), 'stash decrypt status')) + + const config = await loadStashConfig() + const client = createPgClient(config.databaseUrl) + let exitCode = 0 + + try { + await client.connect() + + const [stateMap, physicalCols] = await Promise.all([ + latestByColumnSafe(client), + fetchPhysicalColumns(client), + ]) + + const rows: Row[] = [] + const seen = new Set() + + for (const [table, columns] of physicalCols) { + for (const [column, domain] of columns) { + if (domain === null || classifyEqlDomain(domain) !== 3) continue + const key = `${table}.${column}` + seen.add(key) + const latest = reverseOnly(stateMap.get(`${table}.${column}`)) + const recordedTwin = recordedPlaintextColumn(latest) + const active = isActiveReverseLifecycle( + latest, + recordedTwin === undefined + ? null + : forwardOnly(stateMap.get(`${table}.${recordedTwin}`)), + ) + // A finished lifecycle's events are history: this column starts over. + const state = active ? latest : null + const plaintext = + (active ? recordedTwin : undefined) ?? + (await manifestPlaintextFor(table, column)) ?? + derivePlaintextName(column).name + const twinDomain = columns.get(plaintext) + rows.push( + renderRow({ + table, + column, + domain, + plaintext, + state, + plaintextExists: columns.has(plaintext), + plaintextIsEql: + twinDomain !== undefined && + twinDomain !== null && + classifyEqlDomain(twinDomain) !== null, + mappedType: plaintextTypeForEqlDomain(domain)?.sqlType, + }), + ) + } + } + + // Reverse lifecycles whose encrypted column is no longer visible — most + // often `encrypted-dropped` after the migration was applied. + for (const [key, row] of stateMap) { + if (seen.has(key)) continue + const state = reverseOnly(row) + if (!state) continue + const lastDot = key.lastIndexOf('.') + const table = key.slice(0, lastDot) + const column = key.slice(lastDot + 1) + const plaintext = recordedPlaintextColumn(state) ?? '—' + rows.push( + renderRow({ + table, + column, + domain: '—', + plaintext, + state, + plaintextExists: physicalCols.get(table)?.has(plaintext) ?? false, + encryptedGone: true, + }), + ) + } + + if (rows.length === 0) { + p.log.info( + 'No EQL v3 columns and no reverse migrations on record. Nothing to show.', + ) + p.outro('Nothing to show.') + return + } + + p.note(formatTable(STATUS_HEADERS, rows), 'Reverse migration status') + p.outro('Done.') + } catch (error) { + p.log.error( + error instanceof Error ? error.message : 'Failed to read status.', + ) + exitCode = 1 + } finally { + await client.end() + } + if (exitCode) process.exit(exitCode) +} + +function reverseOnly( + row: MigrationStateRow | undefined, +): MigrationStateRow | null { + return row && isReverseMigrationPhase(row.phase) ? row : null +} + +function forwardOnly( + row: MigrationStateRow | undefined, +): MigrationStateRow | null { + return row && !isReverseMigrationPhase(row.phase) ? row : null +} + +function renderRow(input: { + table: string + column: string + domain: string + plaintext: string + state: MigrationStateRow | null + plaintextExists: boolean + /** The twin's name is taken by an EQL column, so it cannot hold plaintext. */ + plaintextIsEql?: boolean + /** The plaintext type the domain maps to — shown while the twin is absent. */ + mappedType?: string + /** The encrypted column is no longer in the catalog. */ + encryptedGone?: boolean +}): Row { + const { state } = input + const flags: string[] = [] + if (input.plaintextIsEql) { + flags.push('plaintext-col-is-eql') + } else if (!input.plaintextExists && input.plaintext !== '—') { + flags.push('plaintext-col-missing') + } + if (input.encryptedGone && state?.phase !== 'encrypted-dropped') { + flags.push('encrypted-col-missing') + } + if (!input.encryptedGone && input.mappedType === undefined) { + flags.push('unmapped-domain') + } + + const skipped = skippedCount(state) + const plaintext = + input.plaintextExists || input.plaintext === '—' + ? input.plaintext + : `${input.plaintext} (${input.mappedType ?? '?'}, to add)` + + return { + table: input.table, + column: input.column, + domain: input.domain, + plaintext, + phase: state?.phase ?? '—', + progress: formatProgress(state), + skipped: skipped === null ? '—' : skipped.toLocaleString(), + flags: flags.join(', '), + } +} + +/** Cumulative undecryptable count recorded on the latest reverse event. */ +function skippedCount(state: MigrationStateRow | null): number | null { + const value = state?.details?.rowsSkipped + return typeof value === 'number' && Number.isFinite(value) ? value : null +} + +/** + * Phase-aware PROGRESS framing, matching `encrypt status`: + * + * decrypt-dual-writing — gate passed, no backfill yet. + * decrypting — rowsProcessed/rowsTotal (an `error` event keeps + * this phase, so a failed run shows where it stopped). + * decrypted — complete; not a degenerate `0/0` ratio. + * encrypted-dropped — drop migration generated. + */ +function formatProgress(state: MigrationStateRow | null): string { + switch (state?.phase) { + case 'decrypt-dual-writing': + return '(awaiting backfill)' + case 'decrypting': { + if (state.rowsTotal === null) return '—' + const processed = state.rowsProcessed ?? 0 + const pct = + state.rowsTotal > 0 + ? Math.floor((processed / state.rowsTotal) * 100) + : 100 + return `${processed}/${state.rowsTotal} (${pct}%)${state.event === 'error' ? ' (last run failed)' : ''}` + } + case 'decrypted': + return '(backfill complete)' + case 'encrypted-dropped': + return '(dropped)' + default: + return '—' + } +} diff --git a/languages/typescript/packages/cli/src/commands/encrypt/__tests__/status-reversed.test.ts b/languages/typescript/packages/cli/src/commands/encrypt/__tests__/status-reversed.test.ts new file mode 100644 index 000000000..2512c79b1 --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/encrypt/__tests__/status-reversed.test.ts @@ -0,0 +1,115 @@ +/** + * `stash decrypt drop` no longer removes the column's entry from + * `.cipherstash/migrations.json` when it generates the drop (the migration's + * apply-time check can still abort). Until the user removes the entry, the + * encrypted column's absence is intended, and `stash encrypt status` must say + * so rather than report it as drift. + */ + +import type { MigrationStateRow } from '@cipherstash/migrate' +import { beforeEach, describe, expect, it, vi } from 'vitest' + +vi.mock('pg', () => ({ + default: { + Client: class { + connect = vi.fn(async () => {}) + end = vi.fn(async () => {}) + query = vi.fn(async () => ({ rows: [] })) + }, + }, +})) + +const readers = vi.hoisted(() => ({ + latestByColumnSafe: vi.fn( + async (): Promise> => new Map(), + ), + fetchActiveEqlConfig: vi.fn(async () => new Map()), + fetchPhysicalColumns: vi.fn( + async (): Promise>> => + new Map([['users', new Map([['email', null]])]]), + ), +})) +vi.mock('../lib/db-readers.js', () => readers) + +vi.mock('@cipherstash/migrate', async (importOriginal) => ({ + ...(await importOriginal()), + readManifest: vi.fn(async () => ({ + version: 1, + tables: { + users: [ + { + column: 'email', + castAs: 'text', + indexes: [], + targetPhase: 'dropped', + encryptedColumn: 'email_encrypted', + }, + ], + }, + })), +})) +vi.mock('@clack/prompts', () => ({ + intro: vi.fn(), + outro: vi.fn(), + note: vi.fn(), + log: { info: vi.fn(), error: vi.fn(), warn: vi.fn(), success: vi.fn() }, +})) +vi.mock('@/config/index.js', () => ({ + loadStashConfig: vi.fn(async () => ({ databaseUrl: 'postgres://test' })), +})) +vi.mock('@/commands/init/utils.js', () => ({ + detectPackageManager: vi.fn(() => 'npm'), + runnerCommand: vi.fn((_pm: string, ref: string) => `npx ${ref}`), +})) + +import * as p from '@clack/prompts' +import { statusCommand } from '../status.js' + +async function emailRow(): Promise { + await statusCommand() + const table = + vi + .mocked(p.note) + .mock.calls.find( + ([, title]) => title === 'Column migration status', + )?.[0] ?? '' + return table.split('\n').find((line) => line.startsWith('users')) ?? '' +} + +describe('encrypt status for a reversed column', () => { + beforeEach(() => { + vi.clearAllMocks() + }) + + it('reports a missing encrypted column as drift when nothing reversed it', async () => { + readers.latestByColumnSafe.mockResolvedValue(new Map()) + + expect(await emailRow()).toContain('encrypted-col-missing') + }) + + it("reports it as reversed once `decrypt drop` recorded 'encrypted-dropped'", async () => { + readers.latestByColumnSafe.mockResolvedValue( + new Map([ + [ + 'users.email_encrypted', + { + id: '9', + tableName: 'users', + columnName: 'email_encrypted', + event: 'encrypted_dropped', + phase: 'encrypted-dropped', + cursorValue: null, + rowsProcessed: null, + rowsTotal: null, + details: { plaintextColumn: 'email' }, + createdAt: new Date(0), + }, + ], + ]), + ) + + const row = await emailRow() + expect(row).toContain('reversed (remove from migrations.json)') + expect(row).not.toContain('encrypted-col-missing') + }) +}) diff --git a/languages/typescript/packages/cli/src/commands/encrypt/backfill.ts b/languages/typescript/packages/cli/src/commands/encrypt/backfill.ts index 21d73a5fc..370025d6b 100644 --- a/languages/typescript/packages/cli/src/commands/encrypt/backfill.ts +++ b/languages/typescript/packages/cli/src/commands/encrypt/backfill.ts @@ -19,6 +19,7 @@ import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { loadStashConfig } from '@/config/index.js' import { buildPgClientConfig, explainTlsError } from '@/db/client.js' import { loadEncryptionContext, requireTable } from './context.js' +import { detectPkColumn } from './lib/db-readers.js' /** * Options accepted by `stash encrypt backfill`. Each field maps 1:1 to a @@ -482,40 +483,6 @@ function buildPlaintextCoercer(castAs: string | undefined): { return { transform, castAs } } -async function detectPkColumn( - db: pg.PoolClient, - tableName: string, -): Promise { - const [schema, table] = tableName.includes('.') - ? tableName.split('.') - : [null, tableName] - - const result = await db.query<{ column_name: string }>( - `SELECT a.attname AS column_name - FROM pg_index i - JOIN pg_attribute a ON a.attrelid = i.indrelid AND a.attnum = ANY(i.indkey) - JOIN pg_class c ON c.oid = i.indrelid - JOIN pg_namespace n ON n.oid = c.relnamespace - WHERE i.indisprimary - AND c.relname = $1 - AND ($2::text IS NULL OR n.nspname = $2) - ORDER BY a.attnum ASC`, - [table, schema], - ) - - if (result.rows.length === 0) { - throw new Error( - `Could not detect a primary key on ${tableName}. Pass --pk-column .`, - ) - } - if (result.rows.length > 1) { - throw new Error( - `${tableName} has a composite primary key; composite keys are not yet supported. Pass --pk-column to override.`, - ) - } - return result.rows[0]?.column_name -} - /** * Confirm the application is dual-writing before running backfill, and * record the `dual_writing` event in `cs_migrations` so subsequent runs diff --git a/languages/typescript/packages/cli/src/commands/encrypt/drop.ts b/languages/typescript/packages/cli/src/commands/encrypt/drop.ts index 7ade0fabc..72bb16db7 100644 --- a/languages/typescript/packages/cli/src/commands/encrypt/drop.ts +++ b/languages/typescript/packages/cli/src/commands/encrypt/drop.ts @@ -1,5 +1,3 @@ -import fs from 'node:fs' -import path from 'node:path' import { appendEvent, countUnencrypted, @@ -9,11 +7,14 @@ import { setManifestTargetPhase, } from '@cipherstash/migrate' import * as p from '@clack/prompts' -import { detectDrizzle } from '@/commands/db/detect.js' import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' import { loadStashConfig } from '@/config/index.js' import { createPgClient } from '@/db/client.js' -import { scaffoldDrizzleMigration } from './drizzle-helper.js' +import { + buildMigrationStem, + sqlLiteral, + writeMigrationFile, +} from './lib/migration-file.js' import { explainUnresolved, resolveColumnLifecycle } from './lib/resolve-eql.js' /** @@ -156,39 +157,15 @@ export async function dropCommand(options: DropCommandOptions) { options.column, encryptedColumn, ) - const migrationStem = buildMigrationStem(options.table, plaintextToDrop) - - const cwd = process.cwd() - const migrationsDir = options.migrationsDir ?? 'drizzle' - const isDrizzle = detectDrizzle(cwd) - let filePath: string - let nextStep: string - - if (isDrizzle) { - // Scaffold via drizzle-kit so the migration is journaled + snapshotted. - // Without this, the file ships but `drizzle-kit migrate` never picks it - // up because the journal doesn't reference it. - const result = await scaffoldDrizzleMigration({ - name: migrationStem, - outDir: migrationsDir, - sql: dropSql, - }) - filePath = result.path - nextStep = `Apply with your usual Drizzle migrate command:\n ${runnerCommand(detectPackageManager(), 'drizzle-kit migrate')}` - } else { - // Non-Drizzle fallback: emit a timestamped SQL file the user - // applies with whichever migration tool they're running. - const dirAbs = path.resolve(cwd, migrationsDir) - fs.mkdirSync(dirAbs, { recursive: true }) - const ts = new Date() - .toISOString() - .replace(/[-:.TZ]/g, '') - .slice(0, 14) - const fileName = `${ts}_${migrationStem}.sql` - filePath = path.join(dirAbs, fileName) - fs.writeFileSync(filePath, dropSql, 'utf-8') - nextStep = `Review the migration, then apply with your migration tool:\n - prisma migrate deploy\n - psql -f ${fileName}` - } + const { + filePath, + nextStep, + drizzleScaffolded: isDrizzle, + } = await writeMigrationFile({ + stem: buildMigrationStem('drop', options.table, plaintextToDrop), + sql: dropSql, + migrationsDir: options.migrationsDir, + }) await appendEvent(client, { tableName: options.table, @@ -242,7 +219,7 @@ function buildV3DropSql( plaintextColumn: string, encryptedColumn: string, ): string { - const lit = (s: string) => s.replace(/'/g, "''") + const lit = sqlLiteral const qualifiedTable = qualifyTable(table) const quotedPlaintext = quoteIdent(plaintextColumn) const quotedEncrypted = quoteIdent(encryptedColumn) @@ -274,14 +251,3 @@ END $stash_drop$; ` } - -/** Filesystem- and drizzle-kit-safe name derived from untrusted identifiers. */ -function buildMigrationStem(...identifiers: string[]): string { - const sanitized = identifiers - .map((identifier) => - identifier.replace(/[^A-Za-z0-9_-]+/g, '_').replace(/^_+|_+$/g, ''), - ) - .filter(Boolean) - .join('_') - return `drop_${sanitized || 'column'}` -} diff --git a/languages/typescript/packages/cli/src/commands/encrypt/lib/__tests__/db-readers.live.test.ts b/languages/typescript/packages/cli/src/commands/encrypt/lib/__tests__/db-readers.live.test.ts new file mode 100644 index 000000000..0e3ddee70 --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/encrypt/lib/__tests__/db-readers.live.test.ts @@ -0,0 +1,69 @@ +/** + * Live-Postgres coverage for `detectPkColumn`, shared by `encrypt backfill` + * and `decrypt backfill`. + * + * An unqualified table name must resolve the way Postgres resolves it — the + * first match on `search_path` — not match `relname` in every schema. On + * Supabase `public.users` and `auth.users` both have PK `id`; a catalog query + * that ignores `search_path` sees two PK rows and reports a composite key. + * + * Gated on STASH_TEST_DATABASE_URL so the default `pnpm test` stays green + * without a database. + */ + +import type pg from 'pg' +import { afterAll, beforeAll, describe, expect, it } from 'vitest' +import { detectPkColumn } from '../db-readers.js' + +const DATABASE_URL = process.env.STASH_TEST_DATABASE_URL +const describeLive = DATABASE_URL ? describe : describe.skip + +const SCHEMA_A = 'stash_pk_live_a' +const SCHEMA_B = 'stash_pk_live_b' + +describeLive('detectPkColumn — live Postgres', () => { + let client: pg.Client + + beforeAll(async () => { + const { default: pgMod } = await import('pg') + client = new pgMod.Client({ connectionString: DATABASE_URL }) + await client.connect() + await client.query(` + DROP SCHEMA IF EXISTS ${SCHEMA_A} CASCADE; + DROP SCHEMA IF EXISTS ${SCHEMA_B} CASCADE; + CREATE SCHEMA ${SCHEMA_A}; + CREATE SCHEMA ${SCHEMA_B}; + CREATE TABLE ${SCHEMA_A}.users (id bigint PRIMARY KEY, email text); + CREATE TABLE ${SCHEMA_B}.users (uid uuid PRIMARY KEY, email text); + CREATE TABLE ${SCHEMA_A}."User" (pk_mixed bigint PRIMARY KEY); + `) + }) + + afterAll(async () => { + await client.query(` + DROP SCHEMA IF EXISTS ${SCHEMA_A} CASCADE; + DROP SCHEMA IF EXISTS ${SCHEMA_B} CASCADE; + `) + await client.end().catch(() => undefined) + }) + + it('resolves an unqualified name through search_path, ignoring a same-named table in another schema', async () => { + // Only SCHEMA_A is on the path, as `public` is on Supabase while + // `auth.users` is not. + await client.query(`SET search_path TO ${SCHEMA_A}`) + await expect(detectPkColumn(client, 'users')).resolves.toBe('id') + }) + + it('still resolves a schema-qualified name to that schema', async () => { + await client.query(`SET search_path TO ${SCHEMA_A}`) + await expect(detectPkColumn(client, `${SCHEMA_B}.users`)).resolves.toBe( + 'uid', + ) + }) + + it('keeps the name case-exact (a Prisma-style "User"), as REGCLASS_SQL does', async () => { + // A bare to_regclass($1) would case-fold 'User' to 'user' and miss it. + await client.query(`SET search_path TO ${SCHEMA_A}`) + await expect(detectPkColumn(client, 'User')).resolves.toBe('pk_mixed') + }) +}) diff --git a/languages/typescript/packages/cli/src/commands/encrypt/lib/db-readers.ts b/languages/typescript/packages/cli/src/commands/encrypt/lib/db-readers.ts index f0e214ec1..cf19a12b6 100644 --- a/languages/typescript/packages/cli/src/commands/encrypt/lib/db-readers.ts +++ b/languages/typescript/packages/cli/src/commands/encrypt/lib/db-readers.ts @@ -1,6 +1,23 @@ -import { latestByColumn } from '@cipherstash/migrate' +import { + latestByColumn, + type MigrationStateRow, + primaryKeyColumns, + progress, +} from '@cipherstash/migrate' import type pg from 'pg' +/** + * Whether `err` says `cipherstash.cs_migrations` (or its schema) does not + * exist yet: the pre-install case, which every rollout reader treats as "no + * events" so the commands stay readable on a fresh project. + */ +function isMissingMigrationsTable(err: unknown): boolean { + return ( + err instanceof Error && + /cs_migrations|schema "cipherstash"/i.test(err.message) + ) +} + /** * `latestByColumn` from `@cipherstash/migrate`, but tolerant of the * pre-install case where `cipherstash.cs_migrations` doesn't exist. @@ -9,20 +26,29 @@ import type pg from 'pg' */ export async function latestByColumnSafe( client: pg.ClientBase, -): Promise< - ReturnType extends Promise ? T : never -> { +): Promise>> { try { - return (await latestByColumn(client)) as Awaited< - ReturnType - > + return await latestByColumn(client) } catch (err) { - if ( - err instanceof Error && - /cs_migrations|schema "cipherstash"/i.test(err.message) - ) { - return new Map() as Awaited> - } + if (isMissingMigrationsTable(err)) return new Map() + throw err + } +} + +/** + * `progress` from `@cipherstash/migrate` — the latest event for one column — + * but `null` rather than a throw when `cipherstash.cs_migrations` doesn't + * exist yet, as {@link latestByColumnSafe} does. + */ +export async function progressSafe( + client: pg.ClientBase, + table: string, + column: string, +): Promise { + try { + return await progress(client, table, column) + } catch (err) { + if (isMissingMigrationsTable(err)) return null throw err } } @@ -128,3 +154,27 @@ export async function fetchPhysicalColumns( } return out } + +/** + * The table's single-column primary key, for keyset pagination. Throws an + * author-facing error for a table with no PK or a composite one — both + * backfill directions then ask for `--pk-column`. + */ +export async function detectPkColumn( + db: pg.ClientBase, + tableName: string, +): Promise { + const columns = await primaryKeyColumns(db, tableName) + const [only] = columns + if (only === undefined) { + throw new Error( + `Could not detect a primary key on ${tableName}. Pass --pk-column .`, + ) + } + if (columns.length > 1) { + throw new Error( + `${tableName} has a composite primary key; composite keys are not yet supported. Pass --pk-column to override.`, + ) + } + return only +} diff --git a/languages/typescript/packages/cli/src/commands/encrypt/lib/migration-file.ts b/languages/typescript/packages/cli/src/commands/encrypt/lib/migration-file.ts new file mode 100644 index 000000000..5b0ef6317 --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/encrypt/lib/migration-file.ts @@ -0,0 +1,88 @@ +import fs from 'node:fs' +import path from 'node:path' +import { detectDrizzle } from '@/commands/db/detect.js' +import { detectPackageManager, runnerCommand } from '@/commands/init/utils.js' +import { scaffoldDrizzleMigration } from '../drizzle-helper.js' + +/** Where a generated migration landed, and how to apply it. */ +export interface WrittenMigration { + filePath: string + /** Ready-to-print "how to apply this" guidance for the detected tooling. */ + nextStep: string + /** True when the file was scaffolded (journaled) through drizzle-kit. */ + drizzleScaffolded: boolean +} + +/** + * Write a generated SQL migration the way the user's tooling will pick it up. + * Shared by `stash encrypt drop` and the `stash decrypt` commands, so both + * lifecycle directions land migrations identically. + * + * For Drizzle projects, scaffolds the file via `drizzle-kit generate --custom` + * so it gets the correct sequential prefix and a journal/snapshot entry — + * which is what `drizzle-kit migrate` actually picks up. A hand-named file + * would ship but never run. + * + * Otherwise, writes a timestamped `_.sql` into `migrationsDir` for + * the user to review and apply with their own tool (Prisma migrate, psql, …). + */ +export async function writeMigrationFile(opts: { + stem: string + sql: string + /** Relative to cwd. Default `./drizzle`. */ + migrationsDir?: string +}): Promise { + const cwd = process.cwd() + const migrationsDir = opts.migrationsDir ?? 'drizzle' + + if (detectDrizzle(cwd)) { + const result = await scaffoldDrizzleMigration({ + name: opts.stem, + outDir: migrationsDir, + sql: opts.sql, + }) + return { + filePath: result.path, + nextStep: `Apply with your usual Drizzle migrate command:\n ${runnerCommand(detectPackageManager(), 'drizzle-kit migrate')}`, + drizzleScaffolded: true, + } + } + + const dirAbs = path.resolve(cwd, migrationsDir) + fs.mkdirSync(dirAbs, { recursive: true }) + const ts = new Date() + .toISOString() + .replace(/[-:.TZ]/g, '') + .slice(0, 14) + const fileName = `${ts}_${opts.stem}.sql` + const filePath = path.join(dirAbs, fileName) + fs.writeFileSync(filePath, opts.sql, 'utf-8') + return { + filePath, + nextStep: `Review the migration, then apply with your migration tool:\n - prisma migrate deploy\n - psql -f ${fileName}`, + drizzleScaffolded: false, + } +} + +/** + * Filesystem- and drizzle-kit-safe migration name derived from untrusted + * identifiers, e.g. `buildMigrationStem('drop', 'users', 'email')` → + * `drop_users_email`. + */ +export function buildMigrationStem( + prefix: string, + ...identifiers: string[] +): string { + const sanitized = identifiers + .map((identifier) => + identifier.replace(/[^A-Za-z0-9_-]+/g, '_').replace(/^_+|_+$/g, ''), + ) + .filter(Boolean) + .join('_') + return `${prefix}_${sanitized || 'column'}` +} + +/** Escape a string for inclusion inside a single-quoted SQL literal. */ +export function sqlLiteral(value: string): string { + return value.replace(/'/g, "''") +} diff --git a/languages/typescript/packages/cli/src/commands/encrypt/lib/table-format.ts b/languages/typescript/packages/cli/src/commands/encrypt/lib/table-format.ts new file mode 100644 index 000000000..5c6106893 --- /dev/null +++ b/languages/typescript/packages/cli/src/commands/encrypt/lib/table-format.ts @@ -0,0 +1,26 @@ +/** + * Render rows of string cells as an aligned, two-space-separated text table + * with a header row. Shared by `stash encrypt status` and `stash decrypt + * status` so the two read alike. + * + * `headers` fixes both the column order and the header labels; every row must + * carry the same keys. + */ +export function formatTable( + headers: Readonly>, + rows: ReadonlyArray>>, +): string { + const keys = Object.keys(headers) as K[] + const all = [headers, ...rows] + const widths = new Map(keys.map((key) => [key, 0])) + for (const row of all) { + for (const key of keys) { + widths.set(key, Math.max(widths.get(key) ?? 0, row[key].length)) + } + } + return all + .map((row) => + keys.map((key) => row[key].padEnd(widths.get(key) ?? 0)).join(' '), + ) + .join('\n') +} diff --git a/languages/typescript/packages/cli/src/commands/encrypt/status.ts b/languages/typescript/packages/cli/src/commands/encrypt/status.ts index 8f99593c4..855438562 100644 --- a/languages/typescript/packages/cli/src/commands/encrypt/status.ts +++ b/languages/typescript/packages/cli/src/commands/encrypt/status.ts @@ -13,6 +13,7 @@ import { fetchPhysicalColumns, latestByColumnSafe, } from './lib/db-readers.js' +import { formatTable } from './lib/table-format.js' interface Row { table: string @@ -61,6 +62,8 @@ export async function statusCommand() { for (const column of columns) { const key: `${string}.${string}` = `${tableName}.${column.column}` seen.add(key) + const encryptedName = + column.encryptedColumn ?? `${column.column}_encrypted` rows.push( renderRow({ tableName, @@ -71,6 +74,12 @@ export async function statusCommand() { physicalColumns: physicalCols.get(tableName) ?? new Map(), eqlVersion: column.eqlVersion, encryptedColumn: column.encryptedColumn, + // `stash decrypt drop` records this against the ENCRYPTED + // column; the manifest entry is the user's to remove once that + // migration is applied. + reversed: + stateMap.get(`${tableName}.${encryptedName}`)?.phase === + 'encrypted-dropped', }), ) } @@ -105,7 +114,7 @@ export async function statusCommand() { return } - p.note(formatTable(rows), 'Column migration status') + p.note(formatTable(STATUS_HEADERS, rows), 'Column migration status') p.outro('Done.') } catch (error) { p.log.error( @@ -130,6 +139,11 @@ function renderRow(input: { /** The encrypted counterpart's name from the manifest, when recorded. * `_encrypted` is only the conventional fallback. */ encryptedColumn?: string + /** + * `stash decrypt drop` generated the encrypted column's drop. Its absence + * is then intended, not drift. + */ + reversed?: boolean state: { phase: MigrationPhase rowsProcessed: number | null @@ -183,7 +197,11 @@ function renderRow(input: { // rename that creates `_plaintext`. if (!isV3 && intentIndexes && !eqlColumn) flags.push('not-registered') if (intentIndexes && !physicalColumns.has(encryptedName)) { - flags.push('encrypted-col-missing') + flags.push( + input.reversed + ? 'reversed (remove from migrations.json)' + : 'encrypted-col-missing', + ) } if ( !isV3 && @@ -256,36 +274,12 @@ function formatProgress( } } -function formatTable(rows: Row[]): string { - const headers: Row = { - table: 'TABLE', - column: 'COLUMN', - phase: 'PHASE', - eql: 'EQL', - indexes: 'INDEXES', - progress: 'PROGRESS', - flags: 'FLAGS', - } - const all = [headers, ...rows] - const widths: Record = { - table: 0, - column: 0, - phase: 0, - eql: 0, - indexes: 0, - progress: 0, - flags: 0, - } - for (const row of all) { - for (const key of Object.keys(widths) as (keyof Row)[]) { - widths[key] = Math.max(widths[key], row[key].length) - } - } - return all - .map((row) => - (Object.keys(widths) as (keyof Row)[]) - .map((k) => row[k].padEnd(widths[k])) - .join(' '), - ) - .join('\n') +const STATUS_HEADERS: Row = { + table: 'TABLE', + column: 'COLUMN', + phase: 'PHASE', + eql: 'EQL', + indexes: 'INDEXES', + progress: 'PROGRESS', + flags: 'FLAGS', } diff --git a/languages/typescript/packages/cli/src/commands/init/lib/rollout-state.ts b/languages/typescript/packages/cli/src/commands/init/lib/rollout-state.ts index 773e6887f..cdd50a2b7 100644 --- a/languages/typescript/packages/cli/src/commands/init/lib/rollout-state.ts +++ b/languages/typescript/packages/cli/src/commands/init/lib/rollout-state.ts @@ -1,4 +1,7 @@ -import type { MigrationPhase } from '@cipherstash/migrate' +import { + isReverseMigrationPhase, + type MigrationPhase, +} from '@cipherstash/migrate' import { createPgClient } from '@/db/client.js' import { latestByColumnSafe } from '../../encrypt/lib/db-readers.js' @@ -38,6 +41,10 @@ export function classifyPhase(phase: MigrationPhase | null): ColumnNeeds { if (phase === null) return 'unknown' if (phase === 'schema-added') return 'rollout' if (phase === 'dropped') return 'completed' + // Reverse (`stash decrypt`) phases are recorded against the ENCRYPTED + // column, so a forward (table, plaintext column) lookup never sees one. If + // one does arrive, the forward rollout has nothing left to do for it. + if (isReverseMigrationPhase(phase)) return 'completed' return 'cutover' } diff --git a/languages/typescript/packages/cli/src/commands/status/quest.ts b/languages/typescript/packages/cli/src/commands/status/quest.ts index c69d7a709..af40ac419 100644 --- a/languages/typescript/packages/cli/src/commands/status/quest.ts +++ b/languages/typescript/packages/cli/src/commands/status/quest.ts @@ -217,6 +217,12 @@ function computeDoneMigrate(obs: ColumnObservation): number { // shorter (no cut-over), so its terminal phases map one lower. switch (obs.phase) { case 'dropped': + // Reverse (`stash decrypt`) phases are keyed by the encrypted column and + // never reach this forward lookup; if one does, the forward ladder is done. + case 'decrypt-dual-writing': + case 'decrypting': + case 'decrypted': + case 'encrypted-dropped': return isV3 ? 4 : 5 case 'cut-over': return 4 diff --git a/languages/typescript/packages/cli/src/messages.ts b/languages/typescript/packages/cli/src/messages.ts index 2b0c653ea..a6a5776c0 100644 --- a/languages/typescript/packages/cli/src/messages.ts +++ b/languages/typescript/packages/cli/src/messages.ts @@ -389,6 +389,9 @@ export const messages = { repairJournalUnreadable: (detail: string) => `${detail}\nstash eql repair needs drizzle-kit's meta/_journal.json to tell which migrations have already been applied. Check that --out points at your drizzle-kit output directory.`, }, + decrypt: { + unknownSubcommand: 'Unknown decrypt subcommand', + }, db: { unknownSubcommand: 'Unknown db subcommand', /** Warning shown when a deprecated `db ` alias for `eql ` is used. */ diff --git a/languages/typescript/packages/cli/tests/e2e/command-help.e2e.test.ts b/languages/typescript/packages/cli/tests/e2e/command-help.e2e.test.ts index 182082e7d..795f779fe 100644 --- a/languages/typescript/packages/cli/tests/e2e/command-help.e2e.test.ts +++ b/languages/typescript/packages/cli/tests/e2e/command-help.e2e.test.ts @@ -93,6 +93,64 @@ describe('per-command --help', () => { expect(r.output).toContain('--database-url') }) + it('renders a group listing for `decrypt --help`', async () => { + const r = await run(['decrypt', '--help'], { + env: { npm_config_user_agent: '' }, + }) + expect(r.exitCode).toBe(0) + expect(r.output).toContain('Usage: npx stash decrypt [options]') + expect(r.output).toContain('decrypt status') + expect(r.output).toContain('decrypt plan') + expect(r.output).toContain('decrypt backfill') + expect(r.output).toContain('decrypt drop') + expect(r.output).not.toContain('CipherStash CLI v') + }) + + it('renders full command help for `decrypt backfill --help`', async () => { + const r = await run(['decrypt', 'backfill', '--help'], { + env: { npm_config_user_agent: '' }, + }) + expect(r.exitCode).toBe(0) + expect(r.output).toContain('Usage: npx stash decrypt backfill [options]') + expect(r.output).toContain('--plaintext-column') + expect(r.output).toContain('--chunk-size') + expect(r.output).toContain('(default: 500)') + expect(r.output).toContain('--confirm-dual-writes-deployed') + expect(r.output).toContain('--confirm-plaintext-current') + expect(r.output).toContain('--continue-on-undecryptable') + expect(r.output).toContain('Safe only while dual-writes are live') + }) + + it('refuses a `decrypt backfill --chunk-size` one write cannot bind, before touching the database', async () => { + const r = await run( + [ + 'decrypt', + 'backfill', + '--table', + 'users', + '--column', + 'email_encrypted', + '--chunk-size', + '40000', + ], + { env: { npm_config_user_agent: '' } }, + ) + expect(r.exitCode).toBe(1) + expect(r.output).toContain( + '--chunk-size must be a whole number of rows from 1 to 32767 (got "40000")', + ) + }) + + it('renders full command help for `decrypt drop --help`', async () => { + const r = await run(['decrypt', 'drop', '--help'], { + env: { npm_config_user_agent: '' }, + }) + expect(r.exitCode).toBe(0) + expect(r.output).toContain('Usage: npx stash decrypt drop [options]') + expect(r.output).toContain('--uninstall-eql') + expect(r.output).toContain('--migrations-dir') + }) + it('honours the `-h` short flag after a command', async () => { const r = await run(['eql', 'install', '-h'], { env: { npm_config_user_agent: '' }, diff --git a/languages/typescript/packages/cli/tests/e2e/smoke.e2e.test.ts b/languages/typescript/packages/cli/tests/e2e/smoke.e2e.test.ts index aa21b3486..ab41b64e9 100644 --- a/languages/typescript/packages/cli/tests/e2e/smoke.e2e.test.ts +++ b/languages/typescript/packages/cli/tests/e2e/smoke.e2e.test.ts @@ -30,6 +30,10 @@ describe('stash CLI — non-interactive smoke', () => { expect(r.output).toContain('eql upgrade') expect(r.output).toContain('eql status') expect(r.output).toContain('eql validate') + expect(r.output).toContain('decrypt status') + expect(r.output).toContain('decrypt plan') + expect(r.output).toContain('decrypt backfill') + expect(r.output).toContain('decrypt drop') // The dotenv "injected env" banner regression guard lives in the // dedicated test below — this cwd has no .env file, so a bare // `not.toContain('injected env')` here would pass vacuously. @@ -96,6 +100,14 @@ describe('stash CLI — non-interactive smoke', () => { expect(r.output).toContain('bogus-sub') }) + it('decrypt bogus-sub exits 1 with help', async () => { + const r = render(['decrypt', 'bogus-sub']) + const { exitCode } = await r.exit + expect(exitCode).toBe(1) + expect(r.output).toContain(messages.decrypt.unknownSubcommand) + expect(r.output).toContain('bogus-sub') + }) + it('eql bogus-sub exits 1 with help', async () => { const r = render(['eql', 'bogus-sub']) const { exitCode } = await r.exit diff --git a/languages/typescript/packages/migrate/README.md b/languages/typescript/packages/migrate/README.md index 310a8d150..2472acb35 100644 --- a/languages/typescript/packages/migrate/README.md +++ b/languages/typescript/packages/migrate/README.md @@ -2,7 +2,7 @@ Primitives for safely and resumably migrating existing plaintext columns to concrete EQL v3 `eql_v3_*` domains in production PostgreSQL databases. -The package backs `stash encrypt backfill` and `stash encrypt drop`. You can also embed `runBackfill()` in a worker or cron job when a large migration should not run inside a CLI process. +The package backs `stash encrypt backfill` and `stash encrypt drop`, and the reverse lifecycle `stash decrypt` uses to move columns back to plaintext. You can also embed `runBackfill()` in a worker or cron job when a large migration should not run inside a CLI process. ## Lifecycle @@ -12,6 +12,12 @@ schema-added → dual-writing → backfilling → backfilled → dropped EQL v3 has no configuration table and no rename cut-over. The application switches to the encrypted column by name after backfill, then drops the original plaintext column after verifying coverage. State is tracked in the append-only `cipherstash.cs_migrations` table installed by `stash eql install`. +The reverse lifecycle (`stash decrypt`) runs on the same engine and records its own phases, keyed by the ENCRYPTED column so they never share a timeline with the forward history: + +```text +decrypt-dual-writing → decrypting → decrypted → encrypted-dropped +``` + State readers still accept the legacy `cut_over` event and its `cut-over` phase. Manifest readers retain the `cut-over` target phase and `eqlVersion: 2`. Those fields are kept only so status tools can display existing migration history; this package no longer exports the EQL v2 Proxy configuration or rename primitives. ## API @@ -38,6 +44,31 @@ Runs a chunked, resumable, idempotent plaintext-to-encrypted backfill. For each Pass an initialized EQL v3 `Encryption` client and an EQL v3 `encryptedTable`. The lower-level runner writes the envelope produced by the supplied client; the `stash encrypt backfill` command additionally verifies that the destination is an `eql_v3_*` domain before invoking it. +### `runDecryptBackfill(options)` + +The reverse of `runBackfill`, on the same chunked engine (`runChunkedMigration`): keyset pagination, one transaction per chunk with its `decrypt_checkpoint`, resume from the last checkpoint (or from a failed run's `error` event, which records the last committed position), and the `encrypted IS NOT NULL AND plaintext IS NULL` idempotency guard. Each chunk is decrypted with ONE `bulkDecrypt` call (one ZeroKMS request; default 500 rows, at most `MAX_DECRYPT_CHUNK_SIZE` = 32767 because each row binds two parameters in the write), and each value is cast to the plaintext column's declared type (`columnSqlType`). A failed write throws `PlaintextWriteError`, which carries the SQLSTATE, column and type but never the database's message: a cast error's message contains the decrypted value. + +`force` overwrites plaintext that is already set. It is safe only while the application writes both columns: a row written to the plaintext column alone gets the older ciphertext's value back. A forced pass, once started, is owed: until one completes, every run is forced whether or not `force` is passed (`forceObligationOpen` reports it), so an interrupted forced pass is always finished as one. A `force` run over an unforced pass starts from the first row. The CLI also refuses, on a first run, a plaintext column that already holds values (`columnHasValues`) unless the operator chooses to overwrite or keep them. + +`bulkDecrypt` reports failures per row, so a value that cannot be decrypted with the client's credentials — typically one bound to an end-user lock context — is skipped and counted (`rowsSkipped`, `skippedByCode`) and its plaintext left NULL. `countUndecrypted` keeps counting it, which is what makes the `stash decrypt drop` coverage gate refuse to destroy its only copy. Rows whose ciphertext decrypts to `null` for a non-JSON column are skipped as `NULL_PLAINTEXT`, with their primary keys in `nullPlaintextPks`. A first chunk that decrypts nothing throws `UndecryptableBatchError` before writing (usually credentials for the wrong keyset) unless `continueOnUndecryptable` is set. A decrypted value that is still an EQL envelope fails the chunk (reverse leak guard). + +```ts +import { columnSqlType, runDecryptBackfill } from '@cipherstash/migrate' + +const result = await runDecryptBackfill({ + db, + decryptionClient: encryptionClient, + tableName: 'users', + encryptedColumn: 'email_encrypted', + plaintextColumn: 'email', + plaintextSqlType: (await columnSqlType(db, 'users', 'email')) ?? 'text', + pkColumn: 'id', +}) +console.log(result.rowsDecrypted, result.rowsSkipped) +``` + +`plaintextTypeForEqlDomain(domain)` maps an EQL v3 domain back to its plaintext Postgres type (`eql_v3_text_search` → `text`, `eql_v3_timestamp_ord` → `timestamptz`, `eql_v3_json_search` → `jsonb`, …), returning `null` for anything it does not know. + ### State and manifest helpers `appendEvent`, `progress`, and `latestByColumn` access the migration event log. `readManifest` and `writeManifest` manage the Zod-validated `.cipherstash/migrations.json` intent file. @@ -46,7 +77,7 @@ Pass an initialized EQL v3 `Encryption` client and an EQL v3 `encryptedTable`. T `detectColumnEqlVersion`, `classifyEqlDomain`, `listEncryptedColumns`, and `resolveEncryptedColumn` recognize concrete EQL v3 domains and resolve the encrypted counterpart without relying on a naming convention. A legacy `eql_v2_encrypted` column returns `null`; callers must not treat that as an authorable generation. -`countUnencrypted(client, table, plaintextColumn, encryptedColumn)` counts rows with plaintext set and ciphertext NULL. `stash encrypt drop` uses it before generating the v3 plaintext-drop migration. `countEncrypted` counts populated target rows. Both are full-table scans intended for one-shot verification. +`countUnencrypted(client, table, plaintextColumn, encryptedColumn)` counts rows with plaintext set and ciphertext NULL; `countUndecrypted(client, table, encryptedColumn, plaintextColumn)` is its reverse. `stash encrypt drop` uses it before generating the v3 plaintext-drop migration. `countEncrypted` counts populated target rows. Both are full-table scans intended for one-shot verification. ## Worker example diff --git a/languages/typescript/packages/migrate/src/__tests__/decrypt.integration.test.ts b/languages/typescript/packages/migrate/src/__tests__/decrypt.integration.test.ts new file mode 100644 index 000000000..a13ffdaf4 --- /dev/null +++ b/languages/typescript/packages/migrate/src/__tests__/decrypt.integration.test.ts @@ -0,0 +1,876 @@ +/** + * Integration tests for the REVERSE (decrypt) backfill against a real local + * Postgres — the twin of `backfill.integration.test.ts`, on the same engine. + * + * Skipped unless `PG_TEST_URL` is set: + * + * ``` + * docker compose -f local/docker-compose.postgres.yml up -d --wait + * PG_TEST_URL=postgres://cipherstash:password@localhost:55432/cipherstash \ + * pnpm -F @cipherstash/migrate test decrypt.integration + * ``` + * + * No CipherStash credentials required: a stub `bulkDecrypt` decodes + * deterministic v3-shaped markers, and reports a per-row failure for any + * marker flagged `locked` — the shape a value bound to an end-user lock + * context takes when decrypted with workspace credentials. What is under test + * is everything around the crypto: keyset paging, the per-row skip path, the + * checkpoint/resume contract, the typed write, and the guards. + */ + +import 'dotenv/config' +import pg from 'pg' +import { afterAll, afterEach, beforeAll, describe, expect, it } from 'vitest' +import { columnHasValues, countUndecrypted } from '../cursor.js' +import { + type BulkDecryptItem, + type DecryptionClientLike, + forceObligationOpen, + MAX_DECRYPT_CHUNK_SIZE, + PlaintextWriteError, + runDecryptBackfill, + UndecryptableBatchError, +} from '../decrypt.js' +import { installMigrationsSchema } from '../install.js' +import { appendEvent, latestByColumn, progress } from '../state.js' +import { + columnSqlType, + countEqlColumnsInDatabase, + listColumnIndexes, +} from '../version.js' + +const PG_URL = process.env.PG_TEST_URL +const runIntegration = Boolean(PG_URL) + +const TABLE = 'migrate_decrypt_test.users' + +/** A v3-shaped marker the stub "decrypts" back to `plaintext`. */ +function envelope(plaintext: unknown, locked = false) { + return { + v: 3, + i: { t: 'users', c: 'email_encrypted' }, + c: JSON.stringify({ plaintext }), + ...(locked ? { locked: true } : {}), + } +} + +/** Decodes {@link envelope}; a `locked` marker fails that row only. */ +const stubClient: DecryptionClientLike & { calls: number[] } = { + calls: [], + bulkDecrypt(payloads) { + this.calls.push(payloads.length) + const data: BulkDecryptItem[] = payloads.map((payload) => { + const value = payload.data as { c: string; locked?: boolean } | null + if (value?.locked) { + return { id: payload.id, error: 'lock context required' } + } + return { + id: payload.id, + data: (JSON.parse(value?.c ?? 'null') as { plaintext: unknown }) + .plaintext, + } + }) + return Promise.resolve({ data }) + }, +} + +describe.skipIf(!runIntegration)('runDecryptBackfill (integration)', () => { + let pool: pg.Pool + + beforeAll(async () => { + pool = new pg.Pool({ connectionString: PG_URL, max: 4 }) + const db = await pool.connect() + try { + await db.query('DROP SCHEMA IF EXISTS cipherstash CASCADE') + await db.query('DROP SCHEMA IF EXISTS migrate_decrypt_test CASCADE') + await db.query('CREATE SCHEMA migrate_decrypt_test') + await installMigrationsSchema(db) + } finally { + db.release() + } + }) + + afterAll(async () => { + await pool.end() + }) + + afterEach(async () => { + stubClient.calls = [] + await pool.query(`DROP TABLE IF EXISTS ${TABLE}`) + await pool.query('TRUNCATE cipherstash.cs_migrations') + }) + + /** + * `n` rows of encrypted email; every `lockedEvery`-th row is locked. The + * plaintext twin starts empty, as `stash decrypt plan` leaves it. + */ + async function seed(n: number, lockedEvery = 0, plaintextType = 'text') { + await pool.query(` + CREATE TABLE ${TABLE} ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email_encrypted jsonb, + email ${plaintextType} + ) + `) + const values: unknown[] = [] + const rows: string[] = [] + for (let i = 1; i <= n; i++) { + values.push( + JSON.stringify( + envelope( + `user-${i}@example.com`, + lockedEvery > 0 && i % lockedEvery === 0, + ), + ), + ) + rows.push(`($${values.length}::jsonb)`) + } + await pool.query( + `INSERT INTO ${TABLE} (email_encrypted) VALUES ${rows.join(', ')}`, + values, + ) + } + + async function run( + overrides: Partial[0]> = {}, + ) { + const db = await pool.connect() + try { + return await runDecryptBackfill({ + db, + decryptionClient: stubClient, + tableName: TABLE, + encryptedColumn: 'email_encrypted', + plaintextColumn: 'email', + plaintextSqlType: 'text', + pkColumn: 'id', + chunkSize: 100, + ...overrides, + }) + } finally { + db.release() + } + } + + async function plaintextCount(): Promise { + const { rows } = await pool.query<{ n: string }>( + `SELECT count(*)::text AS n FROM ${TABLE} WHERE email IS NOT NULL`, + ) + return Number(rows[0]?.n ?? 0) + } + + it('decrypts every row, one bulkDecrypt per chunk, keyed by the encrypted column', async () => { + await seed(250) + + const result = await run() + + expect(result.completed).toBe(true) + expect(result.rowsProcessed).toBe(250) + expect(result.rowsDecrypted).toBe(250) + expect(result.rowsSkipped).toBe(0) + // 250 rows at chunk 100 → exactly three ZeroKMS round trips. + expect(stubClient.calls).toEqual([100, 100, 50]) + expect(await plaintextCount()).toBe(250) + const { rows } = await pool.query<{ email: string }>( + `SELECT email FROM ${TABLE} WHERE id = 7`, + ) + expect(rows[0]?.email).toBe('user-7@example.com') + + const db = await pool.connect() + try { + // Recorded under the ENCRYPTED column, never the plaintext name — the + // forward history lives there. + expect(await progress(db, TABLE, 'email')).toBeNull() + const state = await progress(db, TABLE, 'email_encrypted') + expect(state?.event).toBe('decrypted') + expect(state?.phase).toBe('decrypted') + expect(state?.details).toMatchObject({ + plaintextColumn: 'email', + rowsSkipped: 0, + }) + } finally { + db.release() + } + }) + + it('skips and counts values it cannot decrypt (lock contexts), leaving them NULL', async () => { + await seed(200, 10) + + const result = await run() + + expect(result.completed).toBe(true) + expect(result.rowsProcessed).toBe(200) + expect(result.rowsSkipped).toBe(20) + expect(result.rowsDecrypted).toBe(180) + expect(result.skippedByCode).toEqual({ UNKNOWN: 20 }) + expect(await plaintextCount()).toBe(180) + + const db = await pool.connect() + try { + // The coverage gate `stash decrypt drop` relies on still sees them. + expect( + await countUndecrypted(db, TABLE, 'email_encrypted', 'email'), + ).toBe(20) + const state = await progress(db, TABLE, 'email_encrypted') + expect(state?.details).toMatchObject({ rowsSkipped: 20 }) + } finally { + db.release() + } + }) + + it('resumes from a checkpoint after an abort, carrying the skip count forward', async () => { + await seed(500, 50) + + const controller = new AbortController() + const first = await run({ + signal: controller.signal, + onProgress: (p) => { + if (p.rowsProcessed >= 200) controller.abort() + }, + }) + expect(first.completed).toBe(false) + expect(first.rowsProcessed).toBe(200) + expect(first.rowsSkipped).toBe(4) + + const second = await run() + expect(second.resumed).toBe(true) + expect(second.completed).toBe(true) + expect(second.rowsProcessed).toBe(500) + expect(second.rowsSkipped).toBe(10) + expect(await plaintextCount()).toBe(490) + }) + + it('a resumed run counts its total once, so progress ends at 100%', async () => { + await seed(500, 50) + + const controller = new AbortController() + await run({ + signal: controller.signal, + onProgress: (p) => { + if (p.rowsProcessed >= 200) controller.abort() + }, + }) + + // The 4 rows skipped before the cursor are already in rowsProcessed and + // still match the pending predicate; counting them again made the total + // 504 and the run end below 100%. + const seen: Array<{ processed: number; total: number }> = [] + const second = await run({ + onProgress: (p) => + seen.push({ processed: p.rowsProcessed, total: p.rowsTotal }), + }) + expect(second.resumed).toBe(true) + expect(second.rowsTotal).toBe(500) + expect(seen.at(-1)).toEqual({ processed: 500, total: 500 }) + }) + + it('resumes after a failed run from its last committed cursor, not the first primary key', async () => { + await seed(300) + + const controller = new AbortController() + await run({ + signal: controller.signal, + onProgress: (p) => { + if (p.rowsProcessed >= 200) controller.abort() + }, + }) + + // Credentials for the wrong keyset: the first chunk of this run decrypts + // nothing, so it stops before writing and records an `error` event… + const wrongKeyset: DecryptionClientLike = { + bulkDecrypt: (payloads) => + Promise.resolve({ + data: payloads.map((payload) => ({ + id: payload.id, + error: 'no key', + })), + }), + } + await expect(run({ decryptionClient: wrongKeyset })).rejects.toBeInstanceOf( + UndecryptableBatchError, + ) + const db = await pool.connect() + try { + const state = await progress(db, TABLE, 'email_encrypted') + expect(state?.event).toBe('error') + expect(state?.cursorValue).toBe('200') + expect(state?.rowsProcessed).toBe(200) + } finally { + db.release() + } + + // …which carries the committed cursor, so the fixed re-run picks up at + // row 201 instead of re-reading the first 200. + stubClient.calls = [] + const fixed = await run() + expect(fixed.resumed).toBe(true) + expect(fixed.completed).toBe(true) + expect(fixed.rowsProcessed).toBe(300) + expect(stubClient.calls).toEqual([100]) + expect(await plaintextCount()).toBe(300) + }) + + it('a plain run finishes an interrupted --force pass as a forced pass', async () => { + await seed(300) + // Stale plaintext everywhere: the case --force exists for. + await pool.query(`UPDATE ${TABLE} SET email = 'stale@example.com'`) + + const controller = new AbortController() + await run({ + force: true, + signal: controller.signal, + onProgress: (p) => { + if (p.rowsProcessed >= 100) controller.abort() + }, + }) + + // Unforced, this re-run would resume at row 101 and keep rows 101-300 + // stale. The open obligation makes it forced, so it resumes the pass. + const finished = await run() + expect(finished.resumed).toBe(true) + const { rows } = await pool.query<{ n: string }>( + `SELECT count(*)::text AS n FROM ${TABLE} WHERE email = 'stale@example.com'`, + ) + expect(rows[0]?.n).toBe('0') + }) + + it('a stale twin acknowledged with --force at dual-write confirmation is overwritten by a plain run', async () => { + await seed(300) + await pool.query(`UPDATE ${TABLE} SET email = 'stale@example.com'`) + // What `stash decrypt backfill --force` records before its first run. + const db = await pool.connect() + try { + await appendEvent(db, { + tableName: TABLE, + columnName: 'email_encrypted', + event: 'decrypt_dual_writing', + phase: 'decrypt-dual-writing', + details: { + plaintextColumn: 'email', + existingPlaintext: 'prepopulated', + acknowledgedWith: 'force', + }, + }) + } finally { + db.release() + } + + await run() + const { rows } = await pool.query<{ n: string }>( + `SELECT count(*)::text AS n FROM ${TABLE} WHERE email = 'stale@example.com'`, + ) + expect(rows[0]?.n).toBe('0') + }) + + it('a completed forced pass meets the obligation: later plain runs are unforced again', async () => { + await seed(300) + await pool.query(`UPDATE ${TABLE} SET email = 'stale@example.com'`) + expect((await run({ force: true })).completed).toBe(true) + + // The app writes a newer value to the plaintext column alone. A forced + // run would put the older ciphertext's value back over it. + await pool.query( + `UPDATE ${TABLE} SET email = 'newer@example.com' WHERE id = 1`, + ) + await run() + const { rows } = await pool.query<{ email: string }>( + `SELECT email FROM ${TABLE} WHERE id = 1`, + ) + expect(rows[0]?.email).toBe('newer@example.com') + }) + + it('a plain run after a forced pass failed before its first commit still overwrites every row', async () => { + await seed(300) + await pool.query(`UPDATE ${TABLE} SET email = 'stale@example.com'`) + + // The forced pass fails on its first chunk, so no checkpoint and no + // cursor exist: nothing a resume could key on. + const wrongKeyset: DecryptionClientLike = { + bulkDecrypt: (payloads) => + Promise.resolve({ + data: payloads.map((payload) => ({ + id: payload.id, + error: 'no key', + })), + }), + } + await expect( + run({ force: true, decryptionClient: wrongKeyset }), + ).rejects.toBeInstanceOf(UndecryptableBatchError) + + // The forced pass opened an obligation; a plain run still has to meet it. + const finished = await run() + expect(finished.completed).toBe(true) + const { rows } = await pool.query<{ n: string }>( + `SELECT count(*)::text AS n FROM ${TABLE} WHERE email = 'stale@example.com'`, + ) + expect(rows[0]?.n).toBe('0') + }) + + it('a --force run does not resume an unforced pass: it starts from the first row', async () => { + await seed(300) + await pool.query( + `UPDATE ${TABLE} SET email = 'stale@example.com' WHERE id <= 50`, + ) + + const controller = new AbortController() + await run({ + signal: controller.signal, + onProgress: (p) => { + if (p.rowsProcessed >= 100) controller.abort() + }, + }) + + const forced = await run({ force: true }) + expect(forced.resumed).toBe(false) + const { rows } = await pool.query<{ n: string }>( + `SELECT count(*)::text AS n FROM ${TABLE} WHERE email = 'stale@example.com'`, + ) + expect(rows[0]?.n).toBe('0') + }) + + it('never puts a decrypted value in a write error, its message or the error event', async () => { + await pool.query(` + CREATE TABLE ${TABLE} ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email_encrypted jsonb, + email integer + ) + `) + await pool.query( + `INSERT INTO ${TABLE} (email_encrypted) VALUES ($1::jsonb)`, + [JSON.stringify(envelope('secret-not-a-number'))], + ) + + const error = await run({ plaintextSqlType: 'integer' }).catch( + (e: unknown) => e, + ) + + // Postgres's own text for this is + // invalid input syntax for type integer: "secret-not-a-number" + expect(error).toBeInstanceOf(PlaintextWriteError) + const message = error instanceof Error ? error.message : '' + expect(message).not.toContain('secret-not-a-number') + expect(message).toContain('SQLSTATE 22P02') + expect(message).toContain('"email" (integer)') + expect(error instanceof Error ? error.cause : 'n/a').toBeUndefined() + + const db = await pool.connect() + try { + const { rows } = await db.query<{ details: unknown }>( + 'SELECT details FROM cipherstash.cs_migrations', + ) + expect(JSON.stringify(rows)).not.toContain('secret-not-a-number') + const state = await progress(db, TABLE, 'email_encrypted') + expect(state?.event).toBe('error') + } finally { + db.release() + } + expect(await plaintextCount()).toBe(0) + }) + + it.each([0, -1, 1.5, Number.NaN, MAX_DECRYPT_CHUNK_SIZE + 1])( + 'refuses chunk size %s before any request or event', + async (chunkSize) => { + await seed(5) + + await expect(run({ chunkSize })).rejects.toBeInstanceOf(RangeError) + + expect(stubClient.calls).toEqual([]) + const db = await pool.connect() + try { + expect((await latestByColumn(db)).size).toBe(0) + } finally { + db.release() + } + }, + ) + + it('writes a chunk of the maximum size in one statement (2 parameters per row)', async () => { + await seed(MAX_DECRYPT_CHUNK_SIZE) + + const result = await run({ chunkSize: MAX_DECRYPT_CHUNK_SIZE }) + + expect(result.completed).toBe(true) + expect(stubClient.calls).toEqual([MAX_DECRYPT_CHUNK_SIZE]) + expect(await plaintextCount()).toBe(MAX_DECRYPT_CHUNK_SIZE) + }, 60_000) + + it('names the rows whose ciphertext decrypts to null in a scalar column', async () => { + await pool.query(` + CREATE TABLE ${TABLE} ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email_encrypted jsonb, + email integer + ) + `) + await pool.query( + `INSERT INTO ${TABLE} (email_encrypted) VALUES ($1::jsonb), ($2::jsonb)`, + [JSON.stringify(envelope(null)), JSON.stringify(envelope(5))], + ) + + const result = await run({ plaintextSqlType: 'integer' }) + + expect(result.rowsSkipped).toBe(1) + expect(result.skippedByCode).toEqual({ NULL_PLAINTEXT: 1 }) + expect(result.nullPlaintextPks).toEqual(['1']) + }) + + it('tells an empty plaintext twin from one that already holds values', async () => { + await seed(3) + const db = await pool.connect() + try { + expect(await columnHasValues(db, TABLE, 'email')).toBe(false) + await db.query( + `UPDATE ${TABLE} SET email = 'stale@example.com' WHERE id = 2`, + ) + expect(await columnHasValues(db, TABLE, 'email')).toBe(true) + } finally { + db.release() + } + }) + + it('is idempotent: a re-run rewrites nothing and only retries the skipped rows', async () => { + await seed(120, 40) + await run() + const before = await pool.query<{ h: string }>( + `SELECT md5(string_agg(coalesce(email, '∅'), ',' ORDER BY id)) AS h FROM ${TABLE}`, + ) + stubClient.calls = [] + + // A fresh pass (the last event was `decrypted`, not a checkpoint) that + // only sees the 3 still-NULL rows. They are still locked, so its one chunk + // decrypts nothing and is refused by default… + await expect(run()).rejects.toBeInstanceOf(UndecryptableBatchError) + expect(stubClient.calls).toEqual([3]) + + // …and with the explicit opt-in it completes, re-skipping them. + const again = await run({ continueOnUndecryptable: true }) + expect(again.completed).toBe(true) + expect(again.rowsProcessed).toBe(3) + expect(again.rowsSkipped).toBe(3) + expect(again.rowsDecrypted).toBe(0) + const after = await pool.query<{ h: string }>( + `SELECT md5(string_agg(coalesce(email, '∅'), ',' ORDER BY id)) AS h FROM ${TABLE}`, + ) + expect(after.rows[0]?.h).toBe(before.rows[0]?.h) + }, 10_000) + + it('refuses a first batch that decrypts nothing, writing nothing and recording no checkpoint', async () => { + await seed(30, 1) // every row locked + + await expect(run()).rejects.toBeInstanceOf(UndecryptableBatchError) + + expect(await plaintextCount()).toBe(0) + const db = await pool.connect() + try { + const state = await progress(db, TABLE, 'email_encrypted') + expect(state?.event).toBe('error') + expect(state?.cursorValue).toBeNull() + expect(state?.details).toMatchObject({ plaintextColumn: 'email' }) + } finally { + db.release() + } + }) + + it('refuses to write an EQL envelope into the plaintext column (reverse leak guard)', async () => { + await seed(5) + const passthrough: DecryptionClientLike = { + bulkDecrypt: (payloads) => + Promise.resolve({ + data: payloads.map((payload) => ({ + id: payload.id, + data: payload.data, + })), + }), + } + + await expect(run({ decryptionClient: passthrough })).rejects.toThrow( + /EQL envelope instead of plaintext/, + ) + expect(await plaintextCount()).toBe(0) + }) + + it('never overwrites plaintext already written by a dual-writing app, unless forced', async () => { + await seed(10) + await pool.query( + `UPDATE ${TABLE} SET email = 'newer@example.com' WHERE id = 3`, + ) + + await run() + const { rows } = await pool.query<{ email: string }>( + `SELECT email FROM ${TABLE} WHERE id = 3`, + ) + expect(rows[0]?.email).toBe('newer@example.com') + + await run({ force: true }) + const forced = await pool.query<{ email: string }>( + `SELECT email FROM ${TABLE} WHERE id = 3`, + ) + expect(forced.rows[0]?.email).toBe('user-3@example.com') + }) + + // F1: a forced write must not overwrite a dual-write that landed while the + // chunk's ciphertext was out for decryption. + it('a forced pass does not overwrite a concurrent dual-write that landed during bulkDecrypt', async () => { + await seed(10) + // The app dual-writes row 4 — both columns — while the chunk that read its + // old ciphertext is still in its bulkDecrypt round trip. + const racingClient: DecryptionClientLike = { + async bulkDecrypt(payloads) { + await pool.query( + `UPDATE ${TABLE} SET email_encrypted = $1::jsonb, email = $2 WHERE id = 4`, + [JSON.stringify(envelope('newer@example.com')), 'newer@example.com'], + ) + return stubClient.bulkDecrypt(payloads) + }, + } + + await run({ force: true, decryptionClient: racingClient }) + + const { rows } = await pool.query<{ email: string }>( + `SELECT email FROM ${TABLE} WHERE id = 4`, + ) + // The ciphertext now says newer@; plaintext must agree, or `decrypt drop` + // drops the only copy of the newer value. + expect(rows[0]?.email).toBe('newer@example.com') + }) + + // F7: a first chunk whose every row decrypts to null is not a credentials + // problem; decryption worked. + it('does not blame credentials when every row in the first chunk decrypts to null', async () => { + await pool.query(` + CREATE TABLE ${TABLE} ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email_encrypted jsonb, + email integer + ) + `) + await pool.query( + `INSERT INTO ${TABLE} (email_encrypted) VALUES ($1::jsonb), ($2::jsonb), ($3::jsonb)`, + [ + JSON.stringify(envelope(null)), + JSON.stringify(envelope(null)), + JSON.stringify(envelope(7)), + ], + ) + + // Chunk of 2: the first chunk is entirely NULL_PLAINTEXT. + const outcome = await run({ + plaintextSqlType: 'integer', + chunkSize: 2, + }).catch((error: unknown) => error) + + expect(outcome).not.toBeInstanceOf(UndecryptableBatchError) + expect(outcome).toMatchObject({ + completed: true, + skippedByCode: { NULL_PLAINTEXT: 2 }, + nullPlaintextPks: ['1', '2'], + rowsDecrypted: 1, + }) + }) + + // F8: behaviour-preservation pin for forceObligationOpen. The obligation is + // scoped to the latest decrypt_dual_writing; a forced pass left open in an + // earlier lifecycle does not carry into the next one. + it('a forced pass left open in an earlier lifecycle does not force the next one', async () => { + await seed(5) + const db = await pool.connect() + try { + const base = { tableName: TABLE, columnName: 'email_encrypted' } + // Lifecycle 1: acknowledged with force, a forced pass starts and never + // completes. + await appendEvent(db, { + ...base, + event: 'decrypt_dual_writing', + phase: 'decrypt-dual-writing', + details: { plaintextColumn: 'email', acknowledgedWith: 'force' }, + }) + await appendEvent(db, { + ...base, + event: 'decrypt_started', + phase: 'decrypting', + details: { plaintextColumn: 'email', force: true }, + }) + expect(await forceObligationOpen(db, TABLE, 'email_encrypted')).toBe(true) + // Lifecycle 2: a fresh dual-write confirmation over an empty twin. + await appendEvent(db, { + ...base, + event: 'decrypt_dual_writing', + phase: 'decrypt-dual-writing', + details: { plaintextColumn: 'email' }, + }) + expect(await forceObligationOpen(db, TABLE, 'email_encrypted')).toBe( + false, + ) + // An unforced completed event does not close a forced obligation. + await appendEvent(db, { + ...base, + event: 'decrypt_started', + phase: 'decrypting', + details: { plaintextColumn: 'email', force: true }, + }) + await appendEvent(db, { + ...base, + event: 'decrypted', + phase: 'decrypted', + details: { plaintextColumn: 'email' }, + }) + expect(await forceObligationOpen(db, TABLE, 'email_encrypted')).toBe(true) + // A forced completed event does. + await appendEvent(db, { + ...base, + event: 'decrypted', + phase: 'decrypted', + details: { plaintextColumn: 'email', force: true }, + }) + expect(await forceObligationOpen(db, TABLE, 'email_encrypted')).toBe( + false, + ) + } finally { + db.release() + } + }) + + it('ignores a forward checkpoint recorded under the same key — no cross-direction resume', async () => { + await seed(50) + const db = await pool.connect() + try { + await appendEvent(db, { + tableName: TABLE, + columnName: 'email_encrypted', + event: 'backfill_checkpoint', + phase: 'backfilling', + cursorValue: '40', + rowsProcessed: 40, + }) + } finally { + db.release() + } + + const result = await run() + expect(result.resumed).toBe(false) + expect(result.rowsProcessed).toBe(50) + expect(await plaintextCount()).toBe(50) + }) + + it.each([ + { type: 'integer', plaintext: 42, expected: '42' }, + { + type: 'bigint', + plaintext: 9007199254740993n, + expected: '9007199254740993', + }, + { type: 'double precision', plaintext: 0.1, expected: '0.1' }, + { type: 'boolean', plaintext: false, expected: 'false' }, + { type: 'date', plaintext: '2024-02-29', expected: '2024-02-29' }, + { + type: 'timestamptz', + plaintext: '2024-02-29T10:11:12.000Z', + expected: '2024-02-29 10:11:12+00', + }, + { + type: 'jsonb', + plaintext: { a: [1, 'x'], b: null }, + expected: '{"a": [1, "x"], "b": null}', + }, + { + type: 'jsonb', + plaintext: 'a string document', + expected: '"a string document"', + }, + { type: 'character varying(32)', plaintext: 'short', expected: 'short' }, + ])( + 'writes a $type twin with the column’s declared type', + async ({ type, plaintext, expected }) => { + await pool.query(` + CREATE TABLE ${TABLE} ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email_encrypted jsonb, + email ${type} + ) + `) + // bigint plaintext can't round-trip through the JSON marker, so the stub + // is bypassed for this one with a client that returns it natively. + await pool.query( + `INSERT INTO ${TABLE} (email_encrypted) VALUES ($1::jsonb)`, + [JSON.stringify(envelope(null))], + ) + const native: DecryptionClientLike = { + bulkDecrypt: (payloads) => + Promise.resolve({ + data: payloads.map((payload) => ({ + id: payload.id, + data: plaintext, + })), + }), + } + const db = await pool.connect() + let sqlType: string | null + try { + sqlType = await columnSqlType(db, TABLE, 'email') + } finally { + db.release() + } + expect(sqlType).not.toBeNull() + + await run({ + decryptionClient: native, + plaintextSqlType: sqlType as string, + }) + + const reader = await pool.connect() + try { + await reader.query("SET TIME ZONE 'UTC'") + const { rows } = await reader.query<{ v: string }>( + `SELECT email::text AS v FROM ${TABLE}`, + ) + expect(rows[0]?.v).toBe(expected) + } finally { + reader.release() + } + }, + ) + + it('lists plain AND expression indexes over a column — the EQL-term indexes decrypt drop removes', async () => { + await seed(1) + await pool.query( + `CREATE INDEX users_enc_plain ON ${TABLE} (email_encrypted)`, + ) + // The shape of an EQL functional index: the column appears only inside + // the expression, so `indkey` alone would not find it. + await pool.query( + `CREATE INDEX users_enc_expr ON ${TABLE} ((email_encrypted->>'c'))`, + ) + await pool.query(`CREATE INDEX users_other ON ${TABLE} (email)`) + + const db = await pool.connect() + try { + const indexes = await listColumnIndexes(db, TABLE, 'email_encrypted') + expect(indexes.map((index) => index.name)).toEqual([ + 'users_enc_expr', + 'users_enc_plain', + ]) + expect(indexes[0]?.schema).toBe('migrate_decrypt_test') + expect(indexes[0]?.definition).toContain('CREATE INDEX users_enc_expr') + } finally { + db.release() + } + }) + + it('counts EQL v3 domain columns across every schema (the --uninstall-eql gate)', async () => { + const db = await pool.connect() + try { + const baseline = await countEqlColumnsInDatabase(db) + await db.query('CREATE DOMAIN migrate_decrypt_test.eql_v3_probe AS jsonb') + await db.query( + `CREATE TABLE ${TABLE} (id int PRIMARY KEY, a migrate_decrypt_test.eql_v3_probe, b migrate_decrypt_test.eql_v3_probe[], c jsonb)`, + ) + expect(await countEqlColumnsInDatabase(db)).toBe(baseline + 2) + await db.query(`DROP TABLE ${TABLE}`) + await db.query('DROP DOMAIN migrate_decrypt_test.eql_v3_probe') + } finally { + db.release() + } + }) +}) diff --git a/languages/typescript/packages/migrate/src/__tests__/manifest.test.ts b/languages/typescript/packages/migrate/src/__tests__/manifest.test.ts index 41cbd1837..bd4e58016 100644 --- a/languages/typescript/packages/migrate/src/__tests__/manifest.test.ts +++ b/languages/typescript/packages/migrate/src/__tests__/manifest.test.ts @@ -5,6 +5,7 @@ import { describe, expect, it } from 'vitest' import { manifestPath, readManifest, + removeManifestColumn, setManifestTargetPhase, upsertManifestColumn, writeManifest, @@ -268,3 +269,52 @@ describe('setManifestTargetPhase', () => { } }) }) + +describe('removeManifestColumn', () => { + const column = (name: string) => ({ + column: name, + castAs: 'text' as const, + indexes: [], + targetPhase: 'dropped' as const, + }) + + it('removes one column and keeps its siblings', async () => { + const tmp = mkdtempSync(join(tmpdir(), 'cs-manifest-')) + try { + await writeManifest( + { version: 1, tables: { users: [column('email'), column('name')] } }, + tmp, + ) + expect(await removeManifestColumn('users', 'email', tmp)).toBe(true) + const after = await readManifest(tmp) + expect(after?.tables.users?.map((c) => c.column)).toEqual(['name']) + } finally { + rmSync(tmp, { recursive: true, force: true }) + } + }) + + it('deletes the table key when its last column goes', async () => { + const tmp = mkdtempSync(join(tmpdir(), 'cs-manifest-')) + try { + await writeManifest( + { version: 1, tables: { users: [column('email')] } }, + tmp, + ) + await removeManifestColumn('users', 'email', tmp) + expect((await readManifest(tmp))?.tables).toEqual({}) + } finally { + rmSync(tmp, { recursive: true, force: true }) + } + }) + + it('is a no-op (false) without a manifest or an entry', async () => { + const tmp = mkdtempSync(join(tmpdir(), 'cs-manifest-')) + try { + expect(await removeManifestColumn('users', 'email', tmp)).toBe(false) + await writeManifest({ version: 1, tables: {} }, tmp) + expect(await removeManifestColumn('users', 'email', tmp)).toBe(false) + } finally { + rmSync(tmp, { recursive: true, force: true }) + } + }) +}) diff --git a/languages/typescript/packages/migrate/src/__tests__/plaintext-type.test.ts b/languages/typescript/packages/migrate/src/__tests__/plaintext-type.test.ts new file mode 100644 index 000000000..248671c43 --- /dev/null +++ b/languages/typescript/packages/migrate/src/__tests__/plaintext-type.test.ts @@ -0,0 +1,90 @@ +import { describe, expect, it } from 'vitest' +import { formatCodes, plaintextSerializer } from '../decrypt.js' +import { plaintextTypeForEqlDomain } from '../plaintext-type.js' + +describe('plaintextTypeForEqlDomain', () => { + it.each([ + ['eql_v3_text', 'text'], + ['eql_v3_text_eq', 'text'], + ['eql_v3_text_match', 'text'], + ['eql_v3_text_ord', 'text'], + ['eql_v3_text_ord_ore', 'text'], + ['eql_v3_text_ord_ope', 'text'], + ['eql_v3_text_search', 'text'], + ['eql_v3_text_search_ore', 'text'], + ['eql_v3_integer_ord', 'integer'], + ['eql_v3_smallint_eq', 'smallint'], + ['eql_v3_bigint_ord_ore', 'bigint'], + ['eql_v3_numeric', 'numeric'], + ['eql_v3_real_ord', 'real'], + ['eql_v3_double_eq', 'double precision'], + ['eql_v3_boolean', 'boolean'], + ['eql_v3_date_ord', 'date'], + // An instant, not a wall-clock time — see the module docs. + ['eql_v3_timestamp_ord', 'timestamptz'], + ['eql_v3_json', 'jsonb'], + ['eql_v3_json_search', 'jsonb'], + ])('%s → %s', (domain, sqlType) => { + expect(plaintextTypeForEqlDomain(domain)?.sqlType).toBe(sqlType) + }) + + it('accepts the public-qualified form the SDK records', () => { + expect(plaintextTypeForEqlDomain('public.eql_v3_text_search')).toEqual({ + domain: 'eql_v3_text_search', + family: 'text', + sqlType: 'text', + }) + }) + + it.each([ + // SteVec fragments and query needles are never a column's type. + 'eql_v3_json_entry', + 'eql_v3_json_query', + // Legacy v2 has no per-domain plaintext type to recover. + 'eql_v2_encrypted', + // A suffix this release does not know: refuse rather than guess. + 'eql_v3_text_fuzzy', + 'eql_v3_boolean_eq', + // A family this release does not know. + 'eql_v3_uuid', + 'eql_v30_text', + 'text', + ])('refuses %s', (domain) => { + expect(plaintextTypeForEqlDomain(domain)).toBeNull() + }) +}) + +describe('plaintextSerializer', () => { + it('serialises scalars for a scalar column, casting left to Postgres', () => { + const serialize = plaintextSerializer('integer') + expect(serialize(42)).toBe('42') + expect(serialize(9007199254740993n)).toBe('9007199254740993') + expect(serialize(true)).toBe('true') + expect(serialize('x')).toBe('x') + expect(serialize(new Date('2024-02-29T10:11:12Z'))).toBe( + '2024-02-29T10:11:12.000Z', + ) + }) + + it('treats a NULL decrypt of a scalar as unwritable (skipped), never as NULL', () => { + expect(plaintextSerializer('text')(null)).toBeUndefined() + expect(plaintextSerializer('text')(undefined)).toBeUndefined() + }) + + it('keeps JSON meaning for json/jsonb targets, including scalar documents', () => { + const serialize = plaintextSerializer('jsonb') + expect(serialize({ a: 1 })).toBe('{"a":1}') + expect(serialize('doc')).toBe('"doc"') + expect(serialize(null)).toBe('null') + expect(plaintextSerializer(' JSON ')(7)).toBe('7') + }) +}) + +describe('formatCodes', () => { + it('orders by count then code, and says so when there are none', () => { + expect(formatCodes({ UNKNOWN: 1, INVALID_CIPHERTEXT: 3 })).toBe( + '3 × INVALID_CIPHERTEXT, 1 × UNKNOWN', + ) + expect(formatCodes({})).toBe('no error codes reported') + }) +}) diff --git a/languages/typescript/packages/migrate/src/backfill.ts b/languages/typescript/packages/migrate/src/backfill.ts index d20153b5a..09c46dd93 100644 --- a/languages/typescript/packages/migrate/src/backfill.ts +++ b/languages/typescript/packages/migrate/src/backfill.ts @@ -5,8 +5,8 @@ import { fetchUnencryptedPage, qualifyTable, } from './cursor.js' +import { type ChunkedRunEvents, runChunkedMigration } from './engine.js' import { quoteIdent } from './sql.js' -import { appendEvent, type MigrationPhase, progress } from './state.js' // Loose structural types — keep this library decoupled from @cipherstash/stack // so @cipherstash/migrate can be built and tested without pulling the full @@ -297,61 +297,31 @@ export async function runBackfill( ): Promise { const chunkSize = options.chunkSize ?? 1000 const { db, tableName, pkColumn, plaintextColumn, encryptedColumn } = options - const force = options.force ?? false - const rowsTotal = await countUnencrypted( - db, - tableName, - plaintextColumn, - encryptedColumn, - force, - ) + const coerce = options.transformPlaintext ?? ((v: unknown) => v) - const last = await progress(db, tableName, plaintextColumn) - const resumeCursor = - last?.event === 'backfill_checkpoint' ? last.cursorValue : null - const resumed = resumeCursor !== null - const priorProcessed = - last?.event === 'backfill_checkpoint' ? (last.rowsProcessed ?? 0) : 0 - - await appendEvent(db, { + const result = await runChunkedMigration({ + db, tableName, - columnName: plaintextColumn, - event: 'backfill_started', - phase: 'backfilling', - cursorValue: resumeCursor, - rowsProcessed: priorProcessed, - rowsTotal: priorProcessed + rowsTotal, - details: { chunkSize, resumed, ...(force ? { force: true } : {}) }, - }) - - let cursor = resumeCursor - let rowsProcessed = priorProcessed - const rowsTotalWithResumed = priorProcessed + rowsTotal - let chunkIndex = 0 - let completed = false - - try { - while (true) { - if (options.signal?.aborted) break - - const page = await fetchUnencryptedPage(db, { + stateColumn: plaintextColumn, + chunkSize, + signal: options.signal, + events: FORWARD_EVENTS, + startedDetails: force ? { force: true } : {}, + countPending: () => + countUnencrypted(db, tableName, plaintextColumn, encryptedColumn, force), + fetchPage: (after, limit) => + fetchUnencryptedPage(db, { tableName, pkColumn, plaintextColumn, encryptedColumn, - after: cursor, - limit: chunkSize, + after, + limit, force, - }) - - if (page.rows.length === 0) { - completed = true - break - } - - const coerce = options.transformPlaintext ?? ((v: unknown) => v) - const models = page.rows.map((row) => ({ + }), + prepareChunk: async (rows) => { + const models = rows.map((row) => ({ __pk: row.pk, [options.schemaColumnKey]: coerce(row.plaintext), })) @@ -377,7 +347,7 @@ export async function runBackfill( for (const [i, row] of encryptResult.data.entries()) { const value = row[options.schemaColumnKey] if (!isEncryptedPayload(value)) { - const pk = row.__pk ?? page.rows[i]?.pk + const pk = row.__pk ?? rows[i]?.pk // Only the *type* of the offender goes into the error — never the // value. The path that hits this branch is precisely the one // where the encryption client passed plaintext through, so the @@ -391,81 +361,47 @@ export async function runBackfill( } } - await db.query('BEGIN') - try { - await writeEncryptedChunk(db, { - tableName, - pkColumn, - encryptedColumn, - schemaColumnKey: options.schemaColumnKey, - encryptedRows: encryptResult.data, - force, - }) - rowsProcessed += page.rows.length - cursor = page.lastPk - await appendEvent(db, { - tableName, - columnName: plaintextColumn, - event: 'backfill_checkpoint', - phase: 'backfilling', - cursorValue: cursor, - rowsProcessed, - rowsTotal: rowsTotalWithResumed, - details: { chunkIndex, chunkRows: page.rows.length }, - }) - await db.query('COMMIT') - } catch (err) { - await db.query('ROLLBACK').catch(() => {}) - throw err + return { + write: (client) => + writeEncryptedChunk(client, { + tableName, + pkColumn, + encryptedColumn, + schemaColumnKey: options.schemaColumnKey, + encryptedRows: encryptResult.data, + force, + }), } - - options.onProgress?.({ - rowsProcessed, - rowsTotal: rowsTotalWithResumed, - lastPk: cursor, - chunkSize, - chunkIndex, - }) - chunkIndex += 1 - } - - if (completed) { - await appendEvent(db, { - tableName, - columnName: plaintextColumn, - event: 'backfilled', - phase: 'backfilled', - cursorValue: cursor, - rowsProcessed, - rowsTotal: rowsTotalWithResumed, - details: { chunkCount: chunkIndex }, - }) - } - } catch (err) { - await appendEvent(db, { - tableName, - columnName: plaintextColumn, - event: 'error', - phase: 'backfilling', - cursorValue: cursor, - rowsProcessed, - rowsTotal: rowsTotalWithResumed, - details: { - message: err instanceof Error ? err.message : String(err), - chunkIndex, - }, - }) - throw err - } + }, + onProgress: options.onProgress + ? (progress) => + options.onProgress?.({ + rowsProcessed: progress.rowsProcessed, + rowsTotal: progress.rowsTotal, + lastPk: progress.lastPk, + chunkSize: progress.chunkSize, + chunkIndex: progress.chunkIndex, + }) + : undefined, + }) return { - resumed, - rowsProcessed, - rowsTotal: rowsTotalWithResumed, - completed, + resumed: result.resumed, + rowsProcessed: result.rowsProcessed, + rowsTotal: result.rowsTotal, + completed: result.completed, } } +/** The forward lifecycle's `cs_migrations` vocabulary. */ +const FORWARD_EVENTS = { + started: 'backfill_started', + checkpoint: 'backfill_checkpoint', + completed: 'backfilled', + runningPhase: 'backfilling', + completedPhase: 'backfilled', +} as const satisfies ChunkedRunEvents + interface WriteChunkOptions { tableName: string pkColumn: string diff --git a/languages/typescript/packages/migrate/src/cursor.ts b/languages/typescript/packages/migrate/src/cursor.ts index 15b2a7eb3..1045f4cb8 100644 --- a/languages/typescript/packages/migrate/src/cursor.ts +++ b/languages/typescript/packages/migrate/src/cursor.ts @@ -58,38 +58,137 @@ export async function fetchUnencryptedPage( client: ClientBase, opts: KeysetPageOptions, ): Promise> { + const page = await fetchPendingPage(client, { + tableName: opts.tableName, + pkColumn: opts.pkColumn, + sourceColumn: opts.plaintextColumn, + targetColumn: opts.encryptedColumn, + after: opts.after, + limit: opts.limit, + force: opts.force, + }) + return { + rows: page.rows.map((row) => ({ pk: row.pk, plaintext: row.value })), + lastPk: page.lastPk, + } +} + +/** + * Inputs to {@link fetchUndecryptedPage} — the reverse (decrypt) twin of + * {@link KeysetPageOptions}. The roles swap: the ENCRYPTED column is the + * source being read and the plaintext column is the target being filled. + */ +export interface DecryptKeysetPageOptions { + /** Physical table name. Supports `schema.table`. */ + tableName: string + /** Primary-key column used for keyset pagination. */ + pkColumn: string + /** EQL v3 column to read the ciphertext from, e.g. `email_encrypted`. */ + encryptedColumn: string + /** + * Plaintext column to fill, e.g. `email`. Rows where this is already + * non-null are skipped (idempotency guard) unless `force` is set. + */ + plaintextColumn: string + /** Exclusive lower bound on `pkColumn`; `null` starts from the beginning. */ + after: string | null + /** Maximum rows returned per call. */ + limit: number + /** + * When true, drop the ` IS NULL` guard so rows that already + * have plaintext are re-emitted — the reverse twin of the forward `force`. + */ + force?: boolean +} + +/** + * Fetch the next page of rows that still need DECRYPTING: ciphertext set, + * plaintext twin NULL. The exact mirror of {@link fetchUnencryptedPage} — + * same keyset pagination and the same idempotency guard, with source and + * target swapped — because it is the same query. + * + * The ciphertext is read as `::jsonb` so the driver hands back the parsed EQL + * envelope regardless of how it reports the domain's type OID. + */ +export async function fetchUndecryptedPage( + client: ClientBase, + opts: DecryptKeysetPageOptions, +): Promise> { + const page = await fetchPendingPage(client, { + tableName: opts.tableName, + pkColumn: opts.pkColumn, + sourceColumn: opts.encryptedColumn, + targetColumn: opts.plaintextColumn, + after: opts.after, + limit: opts.limit, + force: opts.force, + sourceCast: 'jsonb', + }) + return { + rows: page.rows.map((row) => ({ pk: row.pk, ciphertext: row.value })), + lastPk: page.lastPk, + } +} + +/** + * The one keyset query both directions share: rows whose `source` is set and + * whose `target` is still NULL, after the cursor, in PK order. Forward reads + * plaintext into an empty encrypted column; reverse reads ciphertext into an + * empty plaintext column. + */ +async function fetchPendingPage( + client: ClientBase, + opts: { + tableName: string + pkColumn: string + sourceColumn: string + targetColumn: string + after: string | null + limit: number + force?: boolean + /** Only ever a fixed literal from this module — never caller input. */ + sourceCast?: 'jsonb' + }, +): Promise> { const pk = quoteIdent(opts.pkColumn) - const plain = quoteIdent(opts.plaintextColumn) - const enc = quoteIdent(opts.encryptedColumn) + const source = quoteIdent(opts.sourceColumn) + const target = quoteIdent(opts.targetColumn) const table = qualifyTable(opts.tableName) const params: unknown[] = [] - let where = opts.force - ? `${plain} IS NOT NULL` - : `${plain} IS NOT NULL AND ${enc} IS NULL` + let where = pendingPredicate(source, target, opts.force ?? false) if (opts.after !== null) { params.push(opts.after) where += ` AND ${pk} > $${params.length}` } params.push(opts.limit) const limitParam = `$${params.length}` + const selected = opts.sourceCast ? `${source}::${opts.sourceCast}` : source const sql = ` - SELECT ${pk}::text AS pk, ${plain} AS plaintext + SELECT ${pk}::text AS pk, ${selected} AS value FROM ${table} WHERE ${where} ORDER BY ${pk} ASC LIMIT ${limitParam} ` - const result = await client.query<{ pk: string; plaintext: unknown }>( - sql, - params, - ) + const result = await client.query<{ pk: string; value: unknown }>(sql, params) const rows = result.rows const lastPk = rows.length > 0 ? rows[rows.length - 1]?.pk : null return { rows, lastPk } } +/** `source IS NOT NULL [AND target IS NULL]` over already-quoted identifiers. */ +function pendingPredicate( + quotedSource: string, + quotedTarget: string, + force: boolean, +): string { + return force + ? `${quotedSource} IS NOT NULL` + : `${quotedSource} IS NOT NULL AND ${quotedTarget} IS NULL` +} + /** * Count rows that still need encryption: `plaintext IS NOT NULL AND * encrypted IS NULL`. Called once at the start of a backfill to compute @@ -108,18 +207,97 @@ export async function countUnencrypted( encryptedColumn: string, force = false, ): Promise { - const plain = quoteIdent(plaintextColumn) - const enc = quoteIdent(encryptedColumn) + return countPending( + client, + tableName, + plaintextColumn, + encryptedColumn, + force, + ) +} + +/** + * Count rows that still need DECRYPTING: `encrypted IS NOT NULL AND + * plaintext IS NULL`. The reverse twin of {@link countUnencrypted}, used for + * the reverse backfill's `rowsTotal` and as the coverage gate before + * `stash decrypt drop` removes the encrypted column — any row it counts would + * lose its only copy of the data. + * + * Rows that cannot be decrypted (for example, values bound to an end-user + * lock context) stay counted here after a backfill, by design: they still + * have no plaintext. + */ +export async function countUndecrypted( + client: ClientBase, + tableName: string, + encryptedColumn: string, + plaintextColumn: string, + force = false, + after: KeysetBound | null = null, +): Promise { + return countPending( + client, + tableName, + encryptedColumn, + plaintextColumn, + force, + after, + ) +} + +/** + * An exclusive lower bound on the primary key, in the same form the keyset + * page query takes: count only rows the pass has not reached yet. + */ +export interface KeysetBound { + pkColumn: string + /** The cursor value (`pk::text`) to count strictly after. */ + pk: string +} + +async function countPending( + client: ClientBase, + tableName: string, + sourceColumn: string, + targetColumn: string, + force: boolean, + after: KeysetBound | null = null, +): Promise { const table = qualifyTable(tableName) - const where = force - ? `${plain} IS NOT NULL` - : `${plain} IS NOT NULL AND ${enc} IS NULL` + let where = pendingPredicate( + quoteIdent(sourceColumn), + quoteIdent(targetColumn), + force, + ) + const params: unknown[] = [] + if (after !== null) { + params.push(after.pk) + where += ` AND ${quoteIdent(after.pkColumn)} > $${params.length}` + } const result = await client.query<{ count: string }>( `SELECT count(*)::text AS count FROM ${table} WHERE ${where}`, + params, ) return Number(result.rows[0]?.count ?? 0) } +/** + * Whether any row has `column` set. A plaintext twin that already holds + * values before the reverse lifecycle starts is the case `stash decrypt` + * refuses by default: the reverse backfill only fills NULLs, so whatever is + * there would be kept, current or not. `EXISTS` stops at the first match. + */ +export async function columnHasValues( + client: ClientBase, + tableName: string, + column: string, +): Promise { + const result = await client.query<{ populated: boolean }>( + `SELECT EXISTS (SELECT 1 FROM ${qualifyTable(tableName)} WHERE ${quoteIdent(column)} IS NOT NULL) AS populated`, + ) + return result.rows[0]?.populated === true +} + /** * Count rows whose encrypted column is populated: `encrypted IS NOT NULL`. * diff --git a/languages/typescript/packages/migrate/src/decrypt.ts b/languages/typescript/packages/migrate/src/decrypt.ts new file mode 100644 index 000000000..01059a178 --- /dev/null +++ b/languages/typescript/packages/migrate/src/decrypt.ts @@ -0,0 +1,661 @@ +import { isEncryptedPayload } from '@cipherstash/stack' +import type { ClientBase, PoolClient } from 'pg' +import { + countUndecrypted, + fetchUndecryptedPage, + qualifyTable, +} from './cursor.js' +import { + type ChunkedRunEvents, + type ChunkedRunProgress, + runChunkedMigration, +} from './engine.js' +import { quoteIdent } from './sql.js' + +/** + * One position-stable entry of a bulk-decrypt result: the plaintext, or a + * per-row failure. Structural, so this package stays decoupled from + * `@cipherstash/stack`'s generics — `EncryptionClient.bulkDecrypt` resolves to + * exactly this shape (`{ data } | { error, code?, … }`). + */ +export type BulkDecryptItem = + | { id?: string; data: unknown; error?: undefined } + | { id?: string; error: unknown; code?: unknown; data?: undefined } + +/** Whole-call result of a bulk decrypt (`@byteslice/result` convention). */ +export type BulkDecryptResult = + | { failure?: undefined; data: BulkDecryptItem[] } + | { failure: { message: string }; data?: undefined } + +/** + * Minimal surface of `@cipherstash/stack`'s `EncryptionClient` the reverse + * backfill needs: `bulkDecrypt`, which decrypts a whole chunk in ONE ZeroKMS + * round trip and reports failures per row instead of failing the batch. + * + * Per-row failure is the property this runner depends on. A value bound to an + * end-user lock context can only be decrypted as that user; with workspace + * credentials it fails on its own, and the rest of the chunk still decrypts. + * `bulkDecryptModels` fails the whole chunk instead, which would turn one + * locked value into a stalled migration. + * + * Declared with method syntax deliberately: method parameters are checked + * bivariantly, so the SDK's narrower `Encrypted` payload type is accepted + * without a cast. + */ +export interface DecryptionClientLike { + bulkDecrypt( + payloads: Array<{ id?: string; data: Record | null }>, + ): PromiseLike +} + +/** + * Thrown when the first chunk of a run decrypts NOTHING. Its message is + * author-controlled and carries only counts and error codes — never a value — + * so the CLI prints it verbatim. + * + * A whole chunk failing is almost never "every one of these rows happens to + * be lock-context bound". It is credentials resolving to a different + * workspace or keyset, and carrying on would spend one ZeroKMS request per + * chunk across the entire table to decrypt nothing. The run stops before + * writing that chunk and records an `error` event carrying the last committed + * cursor, so fixing the credentials and re-running resumes from the last + * checkpoint rather than from the first primary key. + */ +export class UndecryptableBatchError extends Error { + constructor( + readonly rows: number, + readonly codes: Readonly>, + ) { + super( + `None of the ${rows} row(s) in the first batch could be decrypted (${formatCodes(codes)}). Nothing was written. This usually means the credentials in this shell resolve to a different workspace or keyset than the one that encrypted the data — export the CS_* values of the environment that wrote it and re-run. If these rows really are undecryptable with workspace credentials — values bound to an end-user lock context, or rows an earlier run already skipped — re-run with --continue-on-undecryptable to record them as skipped and carry on.`, + ) + this.name = 'UndecryptableBatchError' + } +} + +/** + * Thrown when the chunk's `UPDATE` fails — most often a decrypted value that + * does not cast to the plaintext column's declared type. + * + * Postgres puts the offending INPUT in the text of a cast error + * (`invalid input syntax for type integer: "…"`), and on this path the input + * is decrypted plaintext. The engine persists an error's message into the + * `cs_migrations` `error` event, so the database's message must never reach + * it. This error carries only the SQLSTATE, the constraint name when there is + * one, and the column and type names; the original error is deliberately not + * kept as `cause`. + */ +export class PlaintextWriteError extends Error { + constructor( + readonly sqlState: string | undefined, + readonly plaintextColumn: string, + readonly plaintextSqlType: string, + readonly constraint: string | undefined, + ) { + super( + `Could not write decrypted values into "${plaintextColumn}" (${plaintextSqlType})${sqlState ? `: SQLSTATE ${sqlState}` : ''}${constraint ? `, constraint "${constraint}"` : ''}. Nothing from this chunk was written. ${sqlState?.startsWith('22') ? `A decrypted value does not fit the column's type; widen "${plaintextColumn}" (for example to the type \`stash decrypt plan\` suggests) and re-run.` : 'Check the column and re-run; the database message is withheld because it can contain row data.'}`, + ) + this.name = 'PlaintextWriteError' + } +} + +/** + * The largest chunk one write can carry. Each decrypted row binds TWO + * parameters in the chunk's `UPDATE … FROM (VALUES ($1, $2), …)` (the primary + * key, and the value packed with the ciphertext it was decrypted from), and Postgres's wire protocol caps a statement at 65535 + * parameters, so 32767 rows is the most that can ever be written in one + * statement. A larger chunk would fail only after its ZeroKMS request. + */ +export const MAX_DECRYPT_CHUNK_SIZE = Math.floor(65535 / 2) + +/** Options for {@link runDecryptBackfill}. Mirrors {@link BackfillOptions}. */ +export interface DecryptBackfillOptions { + /** A pool client the runner owns for the call (`BEGIN`/`COMMIT` per chunk). */ + db: PoolClient + /** Initialised encryption client; only `bulkDecrypt` is used. */ + decryptionClient: DecryptionClientLike + /** Physical table name; `schema.table` supported. */ + tableName: string + /** The EQL v3 column to read ciphertext from. Events are keyed by it. */ + encryptedColumn: string + /** The plaintext column to fill. Must already exist (`stash decrypt plan`). */ + plaintextColumn: string + /** + * The plaintext column's declared type, as `format_type` renders it (see + * `columnSqlType`). Every decrypted value is cast to it on write, so the + * twin's real type is honoured, and JSON targets (`json`/`jsonb`) are + * serialised as JSON documents rather than as scalars. + */ + plaintextSqlType: string + /** Single-column primary key for keyset pagination. */ + pkColumn: string + /** + * Rows per chunk, which is also the size of each `bulkDecrypt` call — one + * ZeroKMS request per chunk. Default `500`. A whole number from 1 to + * {@link MAX_DECRYPT_CHUNK_SIZE}; anything else throws a `RangeError` + * before any event is recorded or any request is made. + */ + chunkSize?: number + /** Checked between chunks; the in-flight chunk always commits first. */ + signal?: AbortSignal + onProgress?: (progress: DecryptBackfillProgress) => void + /** + * Re-decrypt every row with ciphertext, including rows whose plaintext is + * already set, and overwrite that plaintext. Recovery path for drift (the + * application updated the ciphertext without writing the plaintext twin), + * and the way to replace plaintext that predates the reverse lifecycle. + * + * Safe only while dual-writes are live. It writes the decrypted ciphertext + * over whatever the plaintext column holds, so any row the application + * wrote to the plaintext column alone (a newer value, or a value cleared on + * purpose) gets the older ciphertext's value back. + */ + force?: boolean + /** + * Carry on when the first chunk of a run decrypts nothing, instead of + * throwing {@link UndecryptableBatchError}. For a table whose first rows by + * primary key genuinely are all lock-context bound, or a re-run where only + * previously skipped rows remain. Off by default because the far likelier + * cause is credentials for the wrong workspace or keyset. + */ + continueOnUndecryptable?: boolean +} + +/** Progress after each committed chunk. */ +export interface DecryptBackfillProgress { + /** Rows visited so far, decrypted or skipped (includes a resumed run). */ + rowsProcessed: number + rowsTotal: number + /** Cumulative rows that could not be decrypted and were left NULL. */ + rowsSkipped: number + lastPk: string | null + chunkSize: number + chunkIndex: number +} + +/** Result of {@link runDecryptBackfill}. */ +export interface DecryptBackfillResult { + resumed: boolean + /** Rows visited, decrypted or skipped. */ + rowsProcessed: number + rowsTotal: number + /** Rows written with plaintext. `rowsProcessed - rowsSkipped`. */ + rowsDecrypted: number + /** + * Rows whose ciphertext could not be decrypted with these credentials and + * whose plaintext was therefore left NULL — typically values bound to an + * end-user lock context. Cumulative across a resumed run. + */ + rowsSkipped: number + /** Skips in THIS run, by error code (`UNKNOWN` when the row carried none). */ + skippedByCode: Record + /** + * Primary keys (first {@link NULL_PLAINTEXT_SAMPLE_SIZE}, this run) of rows + * whose ciphertext decrypted to `null` for a non-JSON column — the + * `NULL_PLAINTEXT` skips. They are not a credentials problem, so they need + * a different remedy, and naming them is what makes it actionable. Keys + * only; never values. + */ + nullPlaintextPks: string[] + completed: boolean +} + +/** + * Whether this column migration owes a forced pass, and so every run must be + * one whatever the caller passes. It is owed from the moment a forced pass + * starts, or from the `decrypt_dual_writing` event when a twin that already + * held values was acknowledged with `force` (`details.acknowledgedWith`), and + * stops being owed when a forced pass completes. Reads at most two events: + * the latest `decrypt_dual_writing`, which starts the column migration, and + * the latest forced pass or completion event after it (after the first event + * when there is no `decrypt_dual_writing`). Unforced events neither open nor + * close the obligation, so they are not read. + */ +export async function forceObligationOpen( + db: ClientBase, + tableName: string, + encryptedColumn: string, +): Promise { + // `scope` is the latest `decrypt_dual_writing` (absent for a pass run + // without one, when every event counts). `latest` is the newest forced + // event after it that opens or closes the obligation. + const { rows } = await db.query<{ + acknowledged_with: string | null + latest_event: string | null + }>( + `WITH scope AS ( + SELECT id, details->>'acknowledgedWith' AS acknowledged_with + FROM cipherstash.cs_migrations + WHERE table_name = $1 AND column_name = $2 + AND event = 'decrypt_dual_writing' + ORDER BY id DESC + LIMIT 1 + ), latest AS ( + SELECT event + FROM cipherstash.cs_migrations + WHERE table_name = $1 AND column_name = $2 + AND id > COALESCE((SELECT id FROM scope), 0) + AND details->>'force' = 'true' + AND (event = $3 OR phase = $4) + ORDER BY id DESC + LIMIT 1 + ) + SELECT (SELECT acknowledged_with FROM scope) AS acknowledged_with, + (SELECT event FROM latest) AS latest_event`, + [ + tableName, + encryptedColumn, + REVERSE_EVENTS.completed, + REVERSE_EVENTS.runningPhase, + ], + ) + const row = rows[0] + if (row === undefined) return false + // The latest forced event decides: a completed forced pass closes the + // obligation, a started, checkpointed or failed one opens it. + if (row.latest_event !== null) { + return row.latest_event !== REVERSE_EVENTS.completed + } + // No forced event yet. A twin that already held values and was + // acknowledged with `force` owes a forced pass before any has run. + return row.acknowledged_with === 'force' +} + +/** How many `NULL_PLAINTEXT` primary keys a result carries. */ +export const NULL_PLAINTEXT_SAMPLE_SIZE = 20 + +/** The reverse lifecycle's `cs_migrations` vocabulary. */ +const REVERSE_EVENTS = { + started: 'decrypt_started', + checkpoint: 'decrypt_checkpoint', + completed: 'decrypted', + runningPhase: 'decrypting', + completedPhase: 'decrypted', +} as const satisfies ChunkedRunEvents + +/** + * Run a chunked, resumable, idempotent backfill of ciphertext → plaintext — + * the reverse of {@link runBackfill}, on the same engine. + * + * Per chunk: select the next keyset page where the encrypted column is set and + * the plaintext column is NULL; decrypt it with ONE `bulkDecrypt` call; then + * in a single transaction write the plaintext for every row that decrypted + * and append a `decrypt_checkpoint` event. + * + * **Rows that cannot be decrypted are skipped, not fatal.** Their plaintext + * stays NULL, the cursor moves past them, and they are counted + * (`rowsSkipped`, plus per-code tallies in the event details). The common + * cause is a value bound to an end-user lock context, which only that user + * can decrypt; there is deliberately no bypass. Because the skipped rows keep + * a NULL plaintext, {@link countUndecrypted} still counts them, so the + * coverage gate in `stash decrypt drop` refuses to drop the only copy of + * their data. A later run (after the cursor-based pass has completed) starts + * from the beginning and retries them. + * + * **Leak guard (reverse).** A decrypted value that is itself an EQL envelope + * means the client handed ciphertext back unchanged; writing it would put + * ciphertext in the plaintext column. The run fails before the chunk commits. + * + * Events are recorded under `(tableName, encryptedColumn)` with + * `details.plaintextColumn` on every event, so later commands recover the + * pairing from state alone. + * + * **Resume.** A run resumes from its last checkpoint, and also from an + * `error` event, which records the last committed cursor; a failed run does + * not throw away the progress before it. A forced pass, once started, opens + * an obligation ({@link forceObligationOpen}): until a forced pass completes, + * every run is forced whether or not `force` is passed, so an interrupted + * forced pass is always finished as one. A `force` run over an unforced pass + * starts from the first row. + * + * **Write errors.** A failed chunk write throws {@link PlaintextWriteError}, + * never the database's own message, which can contain the decrypted value. + */ +export async function runDecryptBackfill( + options: DecryptBackfillOptions, +): Promise { + const chunkSize = options.chunkSize ?? 500 + if ( + !Number.isSafeInteger(chunkSize) || + chunkSize < 1 || + chunkSize > MAX_DECRYPT_CHUNK_SIZE + ) { + throw new RangeError( + `chunkSize must be a whole number from 1 to ${MAX_DECRYPT_CHUNK_SIZE} (got ${String(chunkSize)}).`, + ) + } + const { + db, + tableName, + pkColumn, + encryptedColumn, + plaintextColumn, + plaintextSqlType, + } = options + const force = + (options.force ?? false) || + (await forceObligationOpen(db, tableName, encryptedColumn)) + const serialize = plaintextSerializer(plaintextSqlType) + const skippedByCode: Record = {} + const nullPlaintextPks: string[] = [] + + const result = await runChunkedMigration({ + db, + tableName, + stateColumn: encryptedColumn, + chunkSize, + signal: options.signal, + events: REVERSE_EVENTS, + trackSkipped: true, + resumeFromError: true, + // `force` on every event, so a resume knows which mode the pass ran in. + baseDetails: force ? { plaintextColumn, force: true } : { plaintextColumn }, + // A forced run never resumes an unforced pass: the rows before its + // cursor were never overwritten. It starts over from the first row. + canResume: (last) => (last.details?.force === true) === force, + countPending: (after) => + countUndecrypted( + db, + tableName, + encryptedColumn, + plaintextColumn, + force, + after === null ? null : { pkColumn, pk: after }, + ), + fetchPage: (after, limit) => + fetchUndecryptedPage(db, { + tableName, + pkColumn, + encryptedColumn, + plaintextColumn, + after, + limit, + force, + }), + prepareChunk: async (rows, { chunkIndex }) => { + const payloads = rows.map((row) => ({ + id: row.pk, + data: asPayload(row.ciphertext), + })) + const decrypted = await options.decryptionClient.bulkDecrypt(payloads) + if (decrypted.failure) { + throw new Error(`bulkDecrypt failed: ${decrypted.failure.message}`) + } + if (decrypted.data.length !== rows.length) { + throw new Error( + `bulkDecrypt returned ${decrypted.data.length} result(s) for ${rows.length} payload(s); refusing to correlate a short or long result positionally.`, + ) + } + + const writes: DecryptedWrite[] = [] + const chunkCodes: Record = {} + // Rows that decrypted, whether or not they produced a write. A row + // decrypting to `null` (NULL_PLAINTEXT) proves the credentials work. + let decryptedCount = 0 + for (const [i, item] of decrypted.data.entries()) { + const { pk } = rows[i] + if (item.error !== undefined) { + tally(chunkCodes, codeOf(item)) + continue + } + if (isEncryptedPayload(item.data)) { + // Only the pk goes into the message — the value is ciphertext, but + // nothing about this path should normalise putting row data in an + // exception. + throw new Error( + `Decryption client returned an EQL envelope instead of plaintext for pk=${pk}. Refusing to write ciphertext into the plaintext column "${plaintextColumn}".`, + ) + } + decryptedCount++ + const value = serialize(item.data) + if (value === undefined) { + tally(chunkCodes, 'NULL_PLAINTEXT') + if (nullPlaintextPks.length < NULL_PLAINTEXT_SAMPLE_SIZE) { + nullPlaintextPks.push(pk) + } + continue + } + // The ciphertext as read, so the write can refuse a row whose + // ciphertext changed while this chunk was being decrypted. + writes.push({ pk, value, ciphertext: rows[i].ciphertext }) + } + + // The first chunk of a run decrypting nothing is a credentials problem, + // not a column of locked values — see UndecryptableBatchError. + if ( + decryptedCount === 0 && + chunkIndex === 0 && + !options.continueOnUndecryptable + ) { + throw new UndecryptableBatchError(rows.length, chunkCodes) + } + + for (const [code, count] of Object.entries(chunkCodes)) { + skippedByCode[code] = (skippedByCode[code] ?? 0) + count + } + const skipped = rows.length - writes.length + + return { + skipped, + checkpointDetails: + skipped > 0 ? { chunkSkippedByCode: chunkCodes } : undefined, + write: (client) => + writeDecryptedChunk(client, { + tableName, + pkColumn, + encryptedColumn, + plaintextColumn, + plaintextSqlType, + writes, + force, + }), + } + }, + onProgress: options.onProgress + ? (progress: ChunkedRunProgress) => + options.onProgress?.({ + rowsProcessed: progress.rowsProcessed, + rowsTotal: progress.rowsTotal, + rowsSkipped: progress.rowsSkipped, + lastPk: progress.lastPk, + chunkSize: progress.chunkSize, + chunkIndex: progress.chunkIndex, + }) + : undefined, + }) + + return { + resumed: result.resumed, + rowsProcessed: result.rowsProcessed, + rowsTotal: result.rowsTotal, + rowsDecrypted: result.rowsProcessed - result.rowsSkipped, + rowsSkipped: result.rowsSkipped, + skippedByCode, + nullPlaintextPks, + completed: result.completed, + } +} + +/** + * The ciphertext as read from the row, narrowed to the object shape + * `bulkDecrypt` takes. The page query guarantees non-NULL and the column's + * EQL domain CHECK guarantees an object; anything else is a corrupt row, and + * handing it to the client as-is lets it be reported per row like any other + * undecryptable value rather than crashing the chunk here. + */ +function asPayload(value: unknown): Record | null { + return value !== null && typeof value === 'object' + ? (value as Record) + : null +} + +/** A per-row failure's error code, or `UNKNOWN` when it carried none. */ +function codeOf(item: { code?: unknown }): string { + return typeof item.code === 'string' && /^[A-Z0-9_]+$/.test(item.code) + ? item.code + : 'UNKNOWN' +} + +function tally(counts: Record, code: string): void { + counts[code] = (counts[code] ?? 0) + 1 +} + +/** `3 × UNKNOWN, 1 × INVALID_CIPHERTEXT` — stable order, for messages. */ +export function formatCodes(codes: Readonly>): string { + const entries = Object.entries(codes).sort( + ([a, x], [b, y]) => y - x || a.localeCompare(b), + ) + return entries.length === 0 + ? 'no error codes reported' + : entries.map(([code, count]) => `${count} × ${code}`).join(', ') +} + +const JSON_TYPES = new Set(['json', 'jsonb']) + +/** + * Build the serializer that turns a decrypted JS plaintext into the text + * parameter the write casts to the plaintext column's type. Returns + * `undefined` for a value that has no faithful text form for that type (a + * scalar column decrypting to `null`), which the caller counts as skipped + * rather than writing a NULL that would read as "not yet decrypted". + * + * - JSON targets: `JSON.stringify` of whatever came back, so a document that + * is itself a string, number or `null` keeps its JSON meaning. + * - `Date` (when a client reconstructs one): ISO-8601, which Postgres parses + * for `date`, `timestamp` and `timestamptz` alike. + * - `bigint`: its exact decimal string — never through `Number`. + * - Other scalars: their canonical string; Postgres does the type conversion, + * so `numeric`/`real`/`double precision` and integer widths all apply their + * own parsing and range checks. + */ +export function plaintextSerializer( + plaintextSqlType: string, +): (value: unknown) => string | undefined { + if (JSON_TYPES.has(plaintextSqlType.trim().toLowerCase())) { + return (value) => JSON.stringify(value === undefined ? null : value) + } + return (value) => { + if (value === null || value === undefined) return undefined + if (value instanceof Date) return value.toISOString() + switch (typeof value) { + case 'string': + return value + case 'number': + case 'bigint': + case 'boolean': + return String(value) + default: + return JSON.stringify(value) + } + } +} + +/** One row's decrypted plaintext, with the ciphertext it was decrypted from. */ +interface DecryptedWrite { + pk: string + /** Text form of the plaintext, cast to the column's type in SQL. */ + value: string + /** The ciphertext as the page query read it (parsed `jsonb`). */ + ciphertext: unknown +} + +interface WriteDecryptedChunkOptions { + tableName: string + pkColumn: string + encryptedColumn: string + plaintextColumn: string + plaintextSqlType: string + writes: DecryptedWrite[] + force: boolean +} + +/** + * One `UPDATE … FROM (VALUES …)` per chunk, the mirror of the forward + * `writeEncryptedChunk`. Values travel as text and are cast to the column's + * declared type in SQL, which is what lets one statement serve every type. + * + * Two guards keep a concurrent dual-write from being overwritten by an older + * decrypted value. The page is read, and decrypted, outside the chunk's + * transaction, so the application can write the row in between: + * + * - The ciphertext must still be the one that was decrypted. A row whose + * ciphertext changed since it was read is left alone: the application's + * dual-write already put the matching plaintext there. This is the only + * guard under `force`, where the plaintext is expected to be set. + * - Unforced, the plaintext must still be NULL. + * + * A row left alone by either guard is still counted as processed. That is + * safe for coverage: if it was left with a NULL plaintext, + * `countUndecrypted` still counts it and a later run retries it. + * + * Each row binds two parameters — the primary key, and one `jsonb` array + * `[plaintext, ciphertext]` — which is what {@link MAX_DECRYPT_CHUNK_SIZE} + * is computed from. + * + * Identifiers are interpolated, never bound, because SQL cannot bind them: + * the table and columns are quoted with `quoteIdent`/`qualifyTable`, and + * `plaintextSqlType` is `format_type()` output read from the catalog. Every + * value travels as a bound parameter. + */ +async function writeDecryptedChunk( + db: ClientBase, + opts: WriteDecryptedChunkOptions, +): Promise { + if (opts.writes.length === 0) return + + const table = qualifyTable(opts.tableName) + const pk = quoteIdent(opts.pkColumn) + const enc = quoteIdent(opts.encryptedColumn) + const plain = quoteIdent(opts.plaintextColumn) + + const params: unknown[] = [] + const valuesSql = opts.writes + .map(({ pk: pkValue, value, ciphertext }) => { + params.push(pkValue) + const pkParam = `$${params.length}` + params.push(JSON.stringify([value, ciphertext])) + return `(${pkParam}, $${params.length}::jsonb)` + }) + .join(', ') + + const unchanged = `t.${pk}::text = v.pk::text AND t.${enc}::jsonb = v.w->1` + const where = opts.force ? unchanged : `${unchanged} AND t.${plain} IS NULL` + + // `plaintextSqlType` is `format_type()` output from the catalog — valid, + // already-quoted type syntax — never user input. + try { + await db.query( + ` + UPDATE ${table} AS t + SET ${plain} = (v.w->>0)::${opts.plaintextSqlType} + FROM (VALUES ${valuesSql}) AS v(pk, w) + WHERE ${where} + `, + params, + ) + } catch (error) { + throw new PlaintextWriteError( + sqlStateOf(error), + opts.plaintextColumn, + opts.plaintextSqlType, + constraintOf(error), + ) + } +} + +/** A driver error's SQLSTATE, when it is one (five uppercase alphanumerics). */ +function sqlStateOf(error: unknown): string | undefined { + const code = fieldOf(error, 'code') + return code !== undefined && /^[0-9A-Z]{5}$/.test(code) ? code : undefined +} + +/** A driver error's constraint name — schema metadata, not row data. */ +function constraintOf(error: unknown): string | undefined { + return fieldOf(error, 'constraint') +} + +function fieldOf(error: unknown, key: string): string | undefined { + if (typeof error !== 'object' || error === null) return undefined + const value: unknown = Reflect.get(error, key) + return typeof value === 'string' ? value : undefined +} diff --git a/languages/typescript/packages/migrate/src/engine.ts b/languages/typescript/packages/migrate/src/engine.ts new file mode 100644 index 000000000..bf4434536 --- /dev/null +++ b/languages/typescript/packages/migrate/src/engine.ts @@ -0,0 +1,293 @@ +import type { ClientBase, PoolClient } from 'pg' +import type { KeysetPage } from './cursor.js' +import { + appendEvent, + type MigrationEvent, + type MigrationPhase, + type MigrationStateRow, + progress, +} from './state.js' + +/** + * The `cs_migrations` vocabulary one direction of the lifecycle writes. The + * forward backfill and the reverse (decrypt) backfill run the same engine and + * differ here, so their histories stay distinguishable and neither resumes + * from the other's cursor. + */ +export interface ChunkedRunEvents { + started: MigrationEvent + /** The event a resume looks for. Its `cursorValue` is the restart point. */ + checkpoint: MigrationEvent + completed: MigrationEvent + /** Phase recorded on `started`, every `checkpoint`, and any `error`. */ + runningPhase: MigrationPhase + completedPhase: MigrationPhase +} + +/** + * A chunk whose network work (encrypt or decrypt) is done and whose database + * write is ready to run inside the chunk's transaction. + */ +export interface PreparedChunk { + /** Issue the chunk's writes on `db`. Runs between `BEGIN` and `COMMIT`. */ + write(db: ClientBase): Promise + /** + * Rows in this chunk that were visited but deliberately not written — for + * the reverse direction, values that could not be decrypted. Counted only + * when {@link ChunkedRunOptions.trackSkipped} is set. + */ + skipped?: number + /** Extra fields merged into this chunk's checkpoint event `details`. */ + checkpointDetails?: Record +} + +/** Context handed to {@link ChunkedRunOptions.prepareChunk}. */ +export interface ChunkContext { + /** Zero-based index of this chunk within the current run. */ + chunkIndex: number + /** True when this run resumed from a recorded checkpoint. */ + resumed: boolean +} + +/** Progress snapshot, emitted after every committed chunk. */ +export interface ChunkedRunProgress { + rowsProcessed: number + rowsTotal: number + lastPk: string | null + chunkSize: number + chunkIndex: number + /** Cumulative skipped rows; always `0` unless skipping is tracked. */ + rowsSkipped: number +} + +export interface ChunkedRunOptions { + db: PoolClient + tableName: string + /** The `column_name` this run's events are recorded under. */ + stateColumn: string + chunkSize: number + signal?: AbortSignal + events: ChunkedRunEvents + /** + * Rows still to process, for `rowsTotal`. Evaluated once, at start, after + * the resume point is known: `after` is the cursor the run restarts from + * (`null` for a fresh pass). Count only rows past it — rows before it were + * already visited and are in the resumed `rowsProcessed`, including any the + * previous run skipped, which still match the pending predicate. + */ + countPending(after: string | null): Promise + /** The next keyset page after `after` (exclusive). */ + fetchPage(after: string | null, limit: number): Promise> + /** + * Do the chunk's network work BEFORE the transaction opens, so a slow + * ZeroKMS round trip never holds row locks. Throwing aborts the run (an + * `error` event is recorded and the error rethrown) with nothing written + * for this chunk and the cursor where it was. + */ + prepareChunk(rows: Row[], context: ChunkContext): Promise + /** Extra fields merged into the `started` event's `details`. */ + startedDetails?: Record + /** + * Fields merged into EVERY event's `details` (started, checkpoint, + * completed, error) — e.g. the reverse run records which plaintext column + * it fills, so later commands can recover the pairing from state alone. + */ + baseDetails?: Record + /** + * Maintain a cumulative `rowsSkipped` (restored from the resumed checkpoint) + * and record it on checkpoint/completed events. Off for the forward + * backfill, whose event details are pinned by existing history readers. + */ + trackSkipped?: boolean + /** + * Also resume from this direction's `error` event, which records the last + * COMMITTED cursor and counters. Without it an error hides the previous + * checkpoint and the next run starts a fresh pass from the first primary + * key. Off for the forward backfill, whose resume behaviour is pinned. + */ + resumeFromError?: boolean + /** + * Called with the event a run is about to resume from. Return `false` to + * start a fresh pass instead; throw to refuse the run before anything is + * recorded. Lets a direction refuse to resume a pass that ran in a + * different mode (the reverse backfill's `force`). + */ + canResume?: (last: MigrationStateRow) => boolean + onProgress?: (progress: ChunkedRunProgress) => void +} + +export interface ChunkedRunResult { + resumed: boolean + rowsProcessed: number + rowsTotal: number + completed: boolean + rowsSkipped: number +} + +/** + * The chunked, resumable, keyset-paginated loop both backfill directions run. + * + * Per chunk: fetch the next page after the cursor, prepare it (network work, + * outside any transaction), then in ONE transaction write it and append a + * checkpoint event, and commit. The checkpoint and the data land atomically, + * so a crash can never record progress for rows that were not written, and + * the `target IS NULL` guard in every page query makes a lost cursor + * harmless: a re-run simply re-selects whatever is still pending. + * + * Resume: when the column's latest event is this direction's checkpoint, the + * run restarts from its cursor and carries its counters forward. With + * `resumeFromError`, an `error` event recorded in this direction's running + * phase resumes the same way: it carries the last committed cursor. Any other + * latest event (a completed run, the other direction's history) starts a + * fresh pass from the beginning, which the idempotency guard keeps cheap. + * + * Abort: `signal` is checked between chunks; the in-flight chunk always + * finishes and checkpoints first. + */ +export async function runChunkedMigration( + options: ChunkedRunOptions, +): Promise { + const { db, tableName, stateColumn, chunkSize, events } = options + const base = options.baseDetails ?? {} + const trackSkipped = options.trackSkipped ?? false + + const last = await progress(db, tableName, stateColumn) + const resumable = + last !== null && + (last.event === events.checkpoint || + (options.resumeFromError === true && + last.event === 'error' && + last.phase === events.runningPhase && + last.cursorValue !== null)) + const resumedFrom = + resumable && (options.canResume?.(last) ?? true) ? last : null + const resumeCursor = resumedFrom ? resumedFrom.cursorValue : null + const resumed = resumeCursor !== null + const priorProcessed = resumedFrom ? (resumedFrom.rowsProcessed ?? 0) : 0 + const priorSkipped = + trackSkipped && resumedFrom ? readSkipped(resumedFrom.details) : 0 + + const pending = await options.countPending(resumeCursor) + const rowsTotal = priorProcessed + pending + + await appendEvent(db, { + tableName, + columnName: stateColumn, + event: events.started, + phase: events.runningPhase, + cursorValue: resumeCursor, + rowsProcessed: priorProcessed, + rowsTotal, + details: { + ...base, + chunkSize, + resumed, + ...options.startedDetails, + }, + }) + + let cursor = resumeCursor + let rowsProcessed = priorProcessed + let rowsSkipped = priorSkipped + let chunkIndex = 0 + let completed = false + const skippedDetails = () => (trackSkipped ? { rowsSkipped } : {}) + + try { + while (true) { + if (options.signal?.aborted) break + + const page = await options.fetchPage(cursor, chunkSize) + if (page.rows.length === 0) { + completed = true + break + } + + const prepared = await options.prepareChunk(page.rows, { + chunkIndex, + resumed, + }) + const chunkSkipped = trackSkipped ? (prepared.skipped ?? 0) : 0 + + // The counters and cursor advance only once the chunk has COMMITTED: + // an `error` event records them, and a resumed run trusts them. + const nextProcessed = rowsProcessed + page.rows.length + const nextSkipped = rowsSkipped + chunkSkipped + await db.query('BEGIN') + try { + await prepared.write(db) + await appendEvent(db, { + tableName, + columnName: stateColumn, + event: events.checkpoint, + phase: events.runningPhase, + cursorValue: page.lastPk, + rowsProcessed: nextProcessed, + rowsTotal, + details: { + ...base, + chunkIndex, + chunkRows: page.rows.length, + ...(trackSkipped ? { chunkSkipped, rowsSkipped: nextSkipped } : {}), + ...prepared.checkpointDetails, + }, + }) + await db.query('COMMIT') + } catch (err) { + await db.query('ROLLBACK').catch(() => {}) + throw err + } + rowsProcessed = nextProcessed + rowsSkipped = nextSkipped + cursor = page.lastPk + + options.onProgress?.({ + rowsProcessed, + rowsTotal, + lastPk: cursor, + chunkSize, + chunkIndex, + rowsSkipped, + }) + chunkIndex += 1 + } + + if (completed) { + await appendEvent(db, { + tableName, + columnName: stateColumn, + event: events.completed, + phase: events.completedPhase, + cursorValue: cursor, + rowsProcessed, + rowsTotal, + details: { ...base, chunkCount: chunkIndex, ...skippedDetails() }, + }) + } + } catch (err) { + await appendEvent(db, { + tableName, + columnName: stateColumn, + event: 'error', + phase: events.runningPhase, + cursorValue: cursor, + rowsProcessed, + rowsTotal, + details: { + ...base, + message: err instanceof Error ? err.message : String(err), + chunkIndex, + ...skippedDetails(), + }, + }) + throw err + } + + return { resumed, rowsProcessed, rowsTotal, completed, rowsSkipped } +} + +/** Cumulative skipped count from a checkpoint's details; `0` when absent. */ +function readSkipped(details: Record | null): number { + const value = details?.rowsSkipped + return typeof value === 'number' && Number.isFinite(value) ? value : 0 +} diff --git a/languages/typescript/packages/migrate/src/index.ts b/languages/typescript/packages/migrate/src/index.ts index cb34f42da..702d80f8b 100644 --- a/languages/typescript/packages/migrate/src/index.ts +++ b/languages/typescript/packages/migrate/src/index.ts @@ -13,6 +13,13 @@ * v3: schema-added → dual-writing → backfilling → backfilled → dropped * ``` * + * And the reverse lifecycle `stash decrypt` drives, back to plaintext, on the + * same chunked engine ({@link runDecryptBackfill}): + * + * ``` + * decrypt-dual-writing → decrypting → decrypted → encrypted-dropped + * ``` + * * State is split across three stores on purpose: * - `.cipherstash/migrations.json` — repo-side intent ({@link Manifest}) * - `cipherstash.cs_migrations` — append-only runtime state written here @@ -31,42 +38,89 @@ export { runBackfill, } from './backfill.js' export { + columnHasValues, countEncrypted, + countUndecrypted, countUnencrypted, + type DecryptKeysetPageOptions, + fetchUndecryptedPage, fetchUnencryptedPage, + type KeysetBound, type KeysetPage, type KeysetPageOptions, qualifyTable, } from './cursor.js' +export { + type BulkDecryptItem, + type BulkDecryptResult, + type DecryptBackfillOptions, + type DecryptBackfillProgress, + type DecryptBackfillResult, + type DecryptionClientLike, + forceObligationOpen, + formatCodes, + MAX_DECRYPT_CHUNK_SIZE, + NULL_PLAINTEXT_SAMPLE_SIZE, + PlaintextWriteError, + plaintextSerializer, + runDecryptBackfill, + UndecryptableBatchError, +} from './decrypt.js' +export { + type ChunkContext, + type ChunkedRunEvents, + type ChunkedRunOptions, + type ChunkedRunProgress, + type ChunkedRunResult, + type PreparedChunk, + runChunkedMigration, +} from './engine.js' export { installMigrationsSchema, MIGRATIONS_SCHEMA_SQL } from './install.js' export { type Manifest, type ManifestColumn, manifestPath, readManifest, + removeManifestColumn, setManifestTargetPhase, upsertManifestColumn, writeManifest, } from './manifest.js' +export { + type EqlDomainFamily, + type PlaintextSqlType, + type PlaintextTypeMapping, + plaintextTypeForEqlDomain, +} from './plaintext-type.js' export { quoteIdent } from './sql.js' export { appendEvent, type ColumnKey, + isReverseMigrationPhase, latestByColumn, type MigrationEvent, type MigrationPhase, type MigrationStateRow, progress, + REVERSE_MIGRATION_PHASES, + type ReverseMigrationEvent, + type ReverseMigrationPhase, } from './state.js' export { + type ColumnIndexInfo, classifyEqlDomain, columnExists, + columnSqlType, + countEqlColumnsInDatabase, detectColumnEqlVersion, type EncryptedColumnInfo, type EncryptedColumnResolution, + EQL_COLUMN_PREDICATE_SQL, type EqlVersion, + listColumnIndexes, listEncryptedColumns, pickEncryptedColumn, + primaryKeyColumns, type ResolvedEncryptedColumn, resolveEncryptedColumn, } from './version.js' diff --git a/languages/typescript/packages/migrate/src/manifest.ts b/languages/typescript/packages/migrate/src/manifest.ts index e67755597..7292ffdc0 100644 --- a/languages/typescript/packages/migrate/src/manifest.ts +++ b/languages/typescript/packages/migrate/src/manifest.ts @@ -185,3 +185,32 @@ export async function setManifestTargetPhase( ) await writeManifest(existing, cwd) } + +/** + * Remove one column's entry from the manifest, deleting the table key when it + * empties. No-op when there is no manifest or no such entry. + * + * For tooling that withdraws the intent to encrypt a column once its + * encrypted column is gone. `stash decrypt drop` deliberately does not call + * it: it only GENERATES the drop, whose apply-time check can still abort, so + * it leaves the entry for the user to remove after applying (meanwhile + * `stash encrypt status` shows it as reversed rather than as drift). + */ +export async function removeManifestColumn( + table: string, + columnName: string, + cwd: string = process.cwd(), +): Promise { + const existing = await readManifest(cwd) + const tableColumns = existing?.tables[table] + if (!existing || !tableColumns) return false + const remaining = tableColumns.filter((c) => c.column !== columnName) + if (remaining.length === tableColumns.length) return false + if (remaining.length === 0) { + delete existing.tables[table] + } else { + existing.tables[table] = remaining + } + await writeManifest(existing, cwd) + return true +} diff --git a/languages/typescript/packages/migrate/src/plaintext-type.ts b/languages/typescript/packages/migrate/src/plaintext-type.ts new file mode 100644 index 000000000..384326e34 --- /dev/null +++ b/languages/typescript/packages/migrate/src/plaintext-type.ts @@ -0,0 +1,110 @@ +/** + * The plaintext Postgres type an EQL v3 domain decrypts back to — the inverse + * of the forward mapping, where a plaintext column of some type is given an + * `eql_v3_[_]` domain twin. + * + * Keyed by the domain FAMILY, not the full domain name: every domain in a + * family (`eql_v3_text`, `_eq`, `_match`, `_ord`, `_ord_ore`, `_ord_ope`, + * `_search`, `_search_ore`) encrypts the same plaintext kind and differs only + * in which query terms it carries. The suffixes are dropped on the way back + * because a plaintext column needs no terms — ordinary Postgres operators and + * indexes answer every query the encrypted terms did. + * + * The families mirror `packages/eql/crates/eql-domains` (the catalog every EQL + * v3 domain is generated from). Two choices are deliberate: + * + * - `timestamp` → `timestamptz`. EQL's `timestamp` family encrypts an INSTANT + * (UTC-normalised); its catalog entry says so explicitly. A naive + * `timestamp` column would silently reinterpret those instants in the + * session time zone. + * - `json` → `jsonb`. Both the searchable document (`eql_v3_json_search`) and + * the storage-only `eql_v3_json` decrypt to a JSON value; `jsonb` is the type + * the forward path accepts for JSON and the one EQL's searchable JSON models. + * The `json_entry` / `json_query` domains are SteVec fragments and query + * needles, never a column's type, so they deliberately have no mapping. + */ +const FAMILY_PLAINTEXT_TYPES = { + text: 'text', + integer: 'integer', + smallint: 'smallint', + bigint: 'bigint', + numeric: 'numeric', + real: 'real', + double: 'double precision', + boolean: 'boolean', + date: 'date', + timestamp: 'timestamptz', + json: 'jsonb', +} as const + +/** An EQL v3 domain family this module can map back to plaintext. */ +export type EqlDomainFamily = keyof typeof FAMILY_PLAINTEXT_TYPES + +/** The plaintext Postgres type for an {@link EqlDomainFamily}. */ +export type PlaintextSqlType = (typeof FAMILY_PLAINTEXT_TYPES)[EqlDomainFamily] + +/** + * Suffixes each family's scalar domains may carry, per the EQL catalog. A + * domain name is accepted only as `eql_v3_` or + * `eql_v3__` with a suffix listed here, so a future domain + * this release does not know about is refused rather than guessed at. + */ +const ORDERED_SUFFIXES = ['eq', 'ord', 'ord_ore', 'ord_ope'] as const +const FAMILY_SUFFIXES: Record = { + text: [...ORDERED_SUFFIXES, 'match', 'search', 'search_ore'], + integer: ORDERED_SUFFIXES, + smallint: ORDERED_SUFFIXES, + bigint: ORDERED_SUFFIXES, + numeric: ORDERED_SUFFIXES, + real: ORDERED_SUFFIXES, + double: ORDERED_SUFFIXES, + date: ORDERED_SUFFIXES, + timestamp: ORDERED_SUFFIXES, + boolean: [], + json: ['search'], +} + +/** A domain resolved to its family and plaintext type. */ +export interface PlaintextTypeMapping { + /** The unqualified domain, e.g. `eql_v3_text_search`. */ + domain: string + family: EqlDomainFamily + /** The Postgres type to give the plaintext twin, e.g. `text`. */ + sqlType: PlaintextSqlType +} + +/** + * Map an EQL v3 domain name to the plaintext Postgres type its values decrypt + * to. Accepts the bare (`eql_v3_text_search`) or `public.`-qualified form. + * Returns `null` for anything that is not a known EQL v3 column domain — a + * legacy `eql_v2_encrypted`, a SteVec fragment, or a domain newer than this + * release — so callers refuse instead of inventing a type. + */ +export function plaintextTypeForEqlDomain( + domainName: string, +): PlaintextTypeMapping | null { + const domain = domainName.startsWith('public.') + ? domainName.slice('public.'.length) + : domainName + if (!domain.startsWith('eql_v3_')) return null + const rest = domain.slice('eql_v3_'.length) + + // Longest family first, so a hypothetical family that prefixes another can + // never be shadowed by the shorter one. + const families = ( + Object.keys(FAMILY_PLAINTEXT_TYPES) as EqlDomainFamily[] + ).sort((a, b) => b.length - a.length) + for (const family of families) { + if (rest === family) { + return { domain, family, sqlType: FAMILY_PLAINTEXT_TYPES[family] } + } + if (rest.startsWith(`${family}_`)) { + const suffix = rest.slice(family.length + 1) + if (FAMILY_SUFFIXES[family].includes(suffix)) { + return { domain, family, sqlType: FAMILY_PLAINTEXT_TYPES[family] } + } + return null + } + } + return null +} diff --git a/languages/typescript/packages/migrate/src/state.ts b/languages/typescript/packages/migrate/src/state.ts index a3f4ff2ca..08d84c981 100644 --- a/languages/typescript/packages/migrate/src/state.ts +++ b/languages/typescript/packages/migrate/src/state.ts @@ -20,6 +20,23 @@ export type MigrationEvent = | 'cut_over' | 'dropped' | 'error' + | ReverseMigrationEvent + +/** + * Events written by the REVERSE (encrypted → plaintext) lifecycle that + * `stash decrypt` drives. They mirror the forward events one for one, and are + * keyed by the ENCRYPTED column — the column being migrated away from — so a + * reverse run never shares a `(table, column)` timeline with the forward + * history recorded under the plaintext name. That separation is what keeps the + * forward resume logic (`backfill_checkpoint`) and the reverse resume logic + * (`decrypt_checkpoint`) from ever reading each other's cursor. + */ +export type ReverseMigrationEvent = + | 'decrypt_dual_writing' + | 'decrypt_started' + | 'decrypt_checkpoint' + | 'decrypted' + | 'encrypted_dropped' /** * The per-column lifecycle phase as surfaced in status/plan output and @@ -41,6 +58,63 @@ export type MigrationPhase = | 'backfilled' | 'cut-over' | 'dropped' + | ReverseMigrationPhase + +/** + * Phases of the reverse lifecycle `stash decrypt` drives, recorded against the + * encrypted column: + * + * ``` + * decrypt-dual-writing → app writes both the ciphertext and the plaintext twin + * decrypting → runDecryptBackfill is (or has been) decrypting rows + * decrypted → every decryptable row has its plaintext twin set + * encrypted-dropped → the encrypted column's drop migration was generated + * ``` + * + * There is no reverse "schema added" phase for the same reason the forward + * `schema-added` is never written by the CLI: adding the plaintext twin is a + * migration the user applies, not a transition the CLI observes. + */ +export type ReverseMigrationPhase = + | 'decrypt-dual-writing' + | 'decrypting' + | 'decrypted' + | 'encrypted-dropped' + +/** + * The reverse phases as a value, so readers can tell a reverse row from a + * forward one without restating the list. `satisfies` keeps it total: a phase + * added to {@link ReverseMigrationPhase} and not here fails to compile. + */ +export const REVERSE_MIGRATION_PHASES = [ + 'decrypt-dual-writing', + 'decrypting', + 'decrypted', + 'encrypted-dropped', +] as const satisfies readonly ReverseMigrationPhase[] + +type MissingReversePhase = Exclude< + ReverseMigrationPhase, + (typeof REVERSE_MIGRATION_PHASES)[number] +> +// Compile-time totality check for the list above (the `satisfies` clause only +// proves every member is a reverse phase, not that every reverse phase is a +// member). +const _reversePhasesAreTotal: [MissingReversePhase] extends [never] + ? true + : never = true +void _reversePhasesAreTotal + +/** True when `phase` belongs to the reverse (decrypt) lifecycle. */ +export function isReverseMigrationPhase( + phase: MigrationPhase | null | undefined, +): phase is ReverseMigrationPhase { + return ( + phase !== null && + phase !== undefined && + (REVERSE_MIGRATION_PHASES as readonly string[]).includes(phase) + ) +} /** * Composite key of `.`. Used as the map key by diff --git a/languages/typescript/packages/migrate/src/version.ts b/languages/typescript/packages/migrate/src/version.ts index 36b268b79..2c2bbca0f 100644 --- a/languages/typescript/packages/migrate/src/version.ts +++ b/languages/typescript/packages/migrate/src/version.ts @@ -134,6 +134,33 @@ export async function columnExists( return rows[0]?.exists === true } +/** + * The columns of `tableName`'s primary key, in attribute order. Empty when + * the table has no primary key or does not exist. + * + * Resolves through {@link REGCLASS_SQL}, so an unqualified name follows + * `search_path` and stays case-exact. Matching `relname` in every schema + * instead would merge same-named tables (Supabase's `public.users` and + * `auth.users`) into one apparent composite key. + */ +export async function primaryKeyColumns( + client: ClientBase, + tableName: string, +): Promise { + const { schema, table } = splitTableName(tableName) + const { rows } = await client.query<{ column_name: string }>( + `SELECT a.attname AS column_name + FROM pg_index i + JOIN pg_attribute a + ON a.attrelid = i.indrelid AND a.attnum = ANY(i.indkey) + WHERE i.indisprimary + AND i.indrelid = ${REGCLASS_SQL} + ORDER BY a.attnum ASC`, + [table, schema], + ) + return rows.map((row) => row.column_name) +} + /** * Every EQL-domain column on a table, classified. The EQL types are * self-describing, so this is the ground truth for "which columns on this @@ -234,3 +261,120 @@ export async function resolveEncryptedColumn( const candidates = await listEncryptedColumns(client, tableName) return pickEncryptedColumn(candidates, plaintextColumn, hint) } + +/** + * The column's declared type as Postgres itself renders it + * (`format_type(atttypid, atttypmod)`, e.g. `character varying(255)`, + * `timestamp with time zone`, `jsonb`), or `null` when the column does not + * exist. + * + * The reverse backfill casts every decrypted value to this exact string, so it + * honours whatever type the user actually gave the plaintext twin — a + * `varchar(255)` instead of the suggested `text`, say — rather than the type + * the CLI would have proposed. `format_type` output is designed to be valid + * type syntax, quoting included, which is why it is safe to splice. + */ +export async function columnSqlType( + client: ClientBase, + tableName: string, + columnName: string, +): Promise { + const { schema, table } = splitTableName(tableName) + const { rows } = await client.query<{ sql_type: string }>( + `SELECT format_type(a.atttypid, a.atttypmod) AS sql_type + FROM pg_attribute a + WHERE a.attrelid = ${REGCLASS_SQL} + AND a.attname = $3 + AND a.attnum > 0 + AND NOT a.attisdropped`, + [table, schema, columnName], + ) + return rows[0]?.sql_type ?? null +} + +/** An index that depends on a column, as {@link listColumnIndexes} reports it. */ +export interface ColumnIndexInfo { + /** Schema the index lives in. */ + schema: string + /** The index's own name. */ + name: string + /** `pg_get_indexdef` output, for display. */ + definition: string +} + +/** + * Every index that depends on `columnName` — plain column indexes AND + * expression indexes over it (an EQL functional index such as + * `eql_v3.ord_term(email_encrypted)` references the column only through its + * expression, so `indkey` alone would miss it; `pg_depend` records both). + * + * Used by `stash decrypt drop` to drop the EQL-term indexes explicitly and + * name them, rather than leaving them to `DROP COLUMN`'s implicit cascade. + */ +export async function listColumnIndexes( + client: ClientBase, + tableName: string, + columnName: string, +): Promise { + const { schema, table } = splitTableName(tableName) + const { rows } = await client.query<{ + schema: string + name: string + definition: string + }>( + `SELECT DISTINCT n.nspname AS schema, + ic.relname AS name, + pg_get_indexdef(i.indexrelid) AS definition + FROM pg_index i + JOIN pg_class ic ON ic.oid = i.indexrelid + JOIN pg_namespace n ON n.oid = ic.relnamespace + JOIN pg_attribute a + ON a.attrelid = i.indrelid + AND a.attname = $3 + AND a.attnum > 0 + AND NOT a.attisdropped + JOIN pg_depend d + ON d.classid = 'pg_class'::regclass + AND d.objid = i.indexrelid + AND d.refclassid = 'pg_class'::regclass + AND d.refobjid = i.indrelid + AND d.refobjsubid = a.attnum + WHERE i.indrelid = ${REGCLASS_SQL} + ORDER BY ic.relname`, + [table, schema, columnName], + ) + return rows +} + +/** + * SQL predicate (over `pg_attribute a`, `pg_type t`, `pg_class c`) matching a + * user-visible relation column whose type is an EQL v3 domain, or an array of + * one. Shared by {@link countEqlColumnsInDatabase} and the apply-time recheck + * `stash decrypt drop --uninstall-eql` embeds in its migration, so the CLI's + * check and the migration's check cannot disagree. + */ +export const EQL_COLUMN_PREDICATE_SQL = `a.attnum > 0 + AND NOT a.attisdropped + AND c.relkind IN ('r', 'p', 'v', 'm', 'f') + AND (t.typname LIKE 'eql\\_v3\\_%' OR t.typname LIKE '\\_eql\\_v3\\_%')` + +/** + * How many columns in the WHOLE database (every schema) are typed with an EQL + * v3 domain. Uninstalling EQL v3 drops its schemas with `CASCADE`, which + * removes every such column along with the domains — so `stash decrypt drop + * --uninstall-eql` refuses unless the column it is dropping is the last one. + * Legacy `eql_v2_encrypted` columns are not counted: the v3 uninstall does not + * touch the v2 schema they depend on. + */ +export async function countEqlColumnsInDatabase( + client: ClientBase, +): Promise { + const { rows } = await client.query<{ count: string }>( + `SELECT count(*)::text AS count + FROM pg_attribute a + JOIN pg_type t ON t.oid = a.atttypid + JOIN pg_class c ON c.oid = a.attrelid + WHERE ${EQL_COLUMN_PREDICATE_SQL}`, + ) + return Number(rows[0]?.count ?? 0) +} diff --git a/skills/stash-auth/SKILL.md b/skills/stash-auth/SKILL.md index d6f1a112f..0fb7f2edf 100644 --- a/skills/stash-auth/SKILL.md +++ b/skills/stash-auth/SKILL.md @@ -357,6 +357,15 @@ const result = await client - A lock-context denial is per-value and per-caller (the "gate 2" failure in `stash-zerokms`): ZeroKMS refused to release that key to that caller. It does not mean the credentials are wrong. +- **Bulk tooling cannot decrypt lock-bound values.** `stash decrypt backfill` + (moving a column back to plaintext) runs as the workspace's client, never + as an end user, so a value bound to a user's claim fails on its own row: + the backfill skips it, counts it, and leaves its plaintext NULL, and + `stash decrypt drop` refuses to drop the encrypted column while any remain. + There is deliberately no bypass. Decrypt those rows in the application, as + each user (an `OidcFederationStrategy` client for that user and the same + `identityClaim`), write the plaintext column, then re-run the backfill. The + same applies to any job of your own that decrypts with an access key. ### Deprecated, and why diff --git a/skills/stash-cli/SKILL.md b/skills/stash-cli/SKILL.md index d5dd7cf73..97b25a01b 100644 --- a/skills/stash-cli/SKILL.md +++ b/skills/stash-cli/SKILL.md @@ -1,6 +1,6 @@ --- name: stash-cli -description: Drive CipherStash setup and encryption migrations through the `stash` CLI — `init`, `plan`, `impl`, `status`, `auth login`, `eql preflight/install/verify/migration/repair/upgrade/status/validate`, `encrypt backfill/drop`, `schema build`, and `manifest --json`. Covers the agent / non-interactive interface, credential rules, and the staged EQL v3 rollout lifecycle. +description: Drive CipherStash setup and encryption migrations through the `stash` CLI — `init`, `plan`, `impl`, `status`, `auth login`, `eql preflight/install/verify/migration/repair/upgrade/status/validate`, `encrypt backfill/drop`, `decrypt status/plan/backfill/drop`, `schema build`, and `manifest --json`. Covers the agent / non-interactive interface, credential rules, the staged EQL v3 rollout lifecycle, and its reverse (moving encrypted columns back to plaintext). --- # CipherStash CLI (`stash`) @@ -133,7 +133,7 @@ There is **no global `--non-interactive` or `--json` flag** (and no global `--ye | Database URL (all `db` / `eql` / `schema` commands) | `--database-url ` or `DATABASE_URL` | | Agent target (`plan`, `impl`) | `--target ` | | Skills destination (`init`) | `--target ` — selects the destination only; `init` performs no handoff | -| Dual-write confirmation (`encrypt backfill`) | `--confirm-dual-writes-deployed` | +| Dual-write confirmation (`encrypt backfill`, `decrypt backfill`) | `--confirm-dual-writes-deployed` | | Machine-readable output | `--json` on `status`, `manifest`, `auth login`, `auth regions` | When a required value is missing in a non-TTY context, the command exits non-zero with an actionable message naming the flag and env var — it never hangs. @@ -213,7 +213,7 @@ export default defineConfig({ | Option | Required | Default | Purpose | |---|---|---|---| | `databaseUrl` | yes | — | PostgreSQL connection string | -| `client` | no | `./src/encryption/index.ts` | Encryption client, loaded by `eql validate` and `encrypt backfill` (`schema build` only writes here; `encrypt drop` resolves against the database). **Not required in Prisma Next projects** — see below. | +| `client` | no | `./src/encryption/index.ts` | Encryption client, loaded by `eql validate`, `encrypt backfill` and `decrypt backfill` (`schema build` only writes here; `encrypt drop` and the other `decrypt` commands resolve against the database). **Not required in Prisma Next projects** — see below. | In a **Prisma Next** project there is deliberately no client file: encrypted columns are declared in the PSL contract. When the configured `client` path is missing, `encrypt backfill` — the only command that loads it — detects Prisma Next, reads the emitted `contract.json` (searched at `src/prisma/`, `prisma/`, then the project root), and derives the schemas with the adapter's own `deriveStackSchemasV3`, so the CLI's schema view matches the application's. (`encrypt status` and `encrypt drop` never read the client file; they resolve against the database.) Both `@cipherstash/stack-prisma` and `@cipherstash/stack` are resolved from *your* project, not the CLI's dependency tree. Run `prisma-next contract emit` first; if the contract declares no `cipherstash.*()` column, the command says so rather than reporting a missing client file. @@ -643,6 +643,83 @@ Runs from the `backfilled` phase and drops the original plaintext ``. It ve Flags: `--table`, `--column`, `--migrations-dir `. +### Decrypt + +The reverse lifecycle: moving encrypted columns back to plaintext, for leaving CipherStash or un-encrypting a column. It mirrors `encrypt` step for step, on the same `@cipherstash/migrate` engine, and records its own events in `cipherstash.cs_migrations` — keyed by the **encrypted** column, so they never mix with the forward history kept under the plaintext name. Decryption is never charged. + +`decrypt-dual-writing → decrypting → decrypted → encrypted-dropped`. As with v3 encryption there is no cut-over command: the application switches reads to the plaintext column by name. + +**Keep dual-writing until the drop migration is applied.** From the deploy before `decrypt backfill` until the `decrypt drop` migration has run, the application writes BOTH the encrypted column and its plaintext twin on every insert and update, including after reads move to the plaintext column. This is the opposite of the forward rollout, where dual-write code is removed once reads move to the encrypted column. The backfill, `--force` and the drop's coverage check all assume both columns stay in step. If writes to the encrypted column stop early, a value cleared in the plaintext column (an erasure request, say) leaves its old ciphertext behind: `decrypt drop` refuses, and re-running the backfill, or `--force`, writes the old value back. Stop writing the encrypted column only after the drop is applied. + +**`--column` names the column being migrated away from — the encrypted one.** The plaintext twin is resolved in this order: `--plaintext-column`; the pairing recorded by the reverse migration in progress (a flag that contradicts it is refused; a pairing from an earlier reverse migration that finished, before the column was encrypted again, is not binding); the original name in `.cipherstash/migrations.json`; the name without its `_encrypted` suffix; `_plaintext`. `decrypt status`, `decrypt plan` and `decrypt backfill` apply the same rules, so they always show the same name. + +#### `decrypt status` / `decrypt plan` + +`status` lists every EQL v3 column in the current schema plus any reverse lifecycle on record, with the plaintext twin, phase, progress, and the count of values the backfill could not decrypt (`UNDECRYPTABLE`). A twin name already taken by an EQL column is flagged `plaintext-col-is-eql`. Read-only. + +`plan` with no flags is the read-only overview: each encrypted column, its proposed twin and type, and the next command. With `--table` (optionally `--column`, `--plaintext-column`) it generates the change that adds the twins, **nullable**: + +- **Drizzle:** prints the schema lines (`email: text('email'),`, `bigint(…, { mode: 'bigint' })`, `timestamp(…, { withTimezone: true })`, …) for `drizzle-kit generate`. It writes no file on purpose — a `--custom` ADD COLUMN never reaches drizzle-kit's snapshot, so the next `generate` would emit it again and fail with "column already exists". +- **Everything else:** writes `_add_plaintext_
_.sql` (`ADD COLUMN IF NOT EXISTS`) into `--migrations-dir`. Prisma Next projects also get the contract change to make. `--dry-run` prints it instead. + +When the twin already exists, `plan` adds nothing for it. If it already holds values, `plan` warns with the same text `decrypt backfill` refuses with (see below) and shows the backfill command with the flag still to choose. If it is the source column of a forward encryption that has not reached `dropped`, `plan` warns that the backfill will refuse it outright (see below). + +When the twin does NOT exist but a forward encryption under its name is still open, the original plaintext column was removed outside stash (an ORM migration rather than `encrypt drop`). `plan` is the only step that can see this, because once the twin is added back the column looks like a forward source that was never dropped. So `plan --table` records that forward migration as `dropped`, marked as observed rather than generated, and plans the twin as usual; `--dry-run` only reports it. If you already re-added the column by hand, that record is missing and `decrypt backfill` refuses: drop the re-added empty column, run `decrypt plan`, and add it through the plan. + +Type mapping (the inverse of the forward ladder; suffixes like `_eq`/`_ord`/`_search` only add query terms and are dropped): `text*` → `text`; `integer*`, `smallint*`, `bigint*`, `numeric*`, `real*` → the same; `double*` → `double precision`; `boolean` → `boolean`; `date*` → `date`; `timestamp*` → `timestamptz` (EQL timestamps are UTC instants); `json`, `json_search` → `jsonb`. A domain the CLI does not know is refused rather than guessed — add that twin by hand. + +#### `decrypt backfill` + +```bash +stash decrypt backfill --table users --column email_encrypted +stash decrypt backfill --table users --column email_encrypted --plaintext-column email --chunk-size 200 +``` + +Same engine as `encrypt backfill`: keyset pagination, one transaction per chunk with its checkpoint, SIGINT/SIGTERM finishes the chunk, and the ` IS NOT NULL AND IS NULL` guard makes re-runs converge. Re-running resumes from the last checkpoint, and also after a failed run: the failure is recorded with the last committed position, so a fix-and-rerun does not start again from the first row. Each chunk is **one `bulkDecrypt` call — one ZeroKMS request** (default 500 rows, at most 32767: each row binds two parameters in the chunk's write, and Postgres allows 65535 per statement). Values are cast to the twin's *declared* type, so a `varchar(255)` twin works as well as the suggested `text`; a value that does not fit stops the run with its SQLSTATE, column and type, never the value. The twin must already exist; the command points at `decrypt plan` if not. + +**A forward encryption still in progress cannot be reversed.** If the twin is the source column of a forward encryption that has not reached `dropped`, `decrypt backfill` refuses, whatever flags are passed: only one migration may be open on a pair of columns at a time. Finish the forward one with `encrypt drop` first (if the original column is already gone, see `decrypt plan` above). To abandon encryption instead, stop writing the encrypted column, switch reads back to the plaintext column and drop the encrypted column. While dual-writes are live the plaintext column is still current, so there is nothing to decrypt. + +**A twin that already holds values is refused on the first run.** The backfill only fills rows where the twin is NULL, so existing values would be kept as they are, the column marked `decrypted`, and `decrypt drop` would then remove the only current copy. So before recording anything, the first run refuses a twin that holds values until you choose: + +- `--force` overwrites every row of the twin with the decrypted ciphertext. Use this when in doubt. +- `--confirm-plaintext-current` keeps the existing values and fills only the NULLs. Use it only when the application has written the plaintext column on every insert and update for the whole time the encrypted column existed. + +The choice is recorded in `cs_migrations`. Later runs of the same reverse migration do not ask again: the twin's values are then its own. Choosing `--force` makes a forced pass owed from that moment, and so does starting any `--force` run: every later run is forced, with or without `--force`, until one completes, so an interrupted `--force` run is always finished as one, even if it failed before writing anything. A `--force` run after an interrupted plain one starts again from the first row. + +**Lock contexts — known limitation, no bypass.** A value encrypted with an end-user lock context can only be decrypted as that user (see `stash-auth`). With workspace credentials it fails *on its own row* (`bulkDecrypt` reports per row), so the backfill **skips it, counts it, leaves the twin NULL**, and reports the count and error codes; the count is also recorded in `cs_migrations` and shown by `decrypt status`. Decrypt those rows in your application on each user's behalf (writing the plaintext column), then re-run. `decrypt drop` refuses while any remain. + +**Ciphertext that decrypts to null.** A non-JSON twin can only store an empty value as NULL, which is also what "not decrypted yet" looks like, so these rows are skipped as `NULL_PLAINTEXT` and reported separately, with their primary keys. They are not a credentials or lock-context problem and a re-run will not change them. `decrypt drop` refuses while they hold ciphertext: once you have confirmed the values really are empty, set their encrypted column to NULL, then generate the drop. + +**Whole first batch undecryptable → stop.** If the first chunk of a run decrypts nothing, the command stops before writing anything (`UndecryptableBatchError`): that is almost always credentials resolving to a different workspace or keyset. Re-run with `--continue-on-undecryptable` only when those rows really are lock-bound (or are the leftovers of an earlier run). + +**Credentials.** The backfill loads your encryption client exactly as `encrypt backfill` does — the client file, or the Prisma Next contract — so the **keyset precondition** above applies unchanged: export the `CS_*` values of the environment that wrote the data. Proxy-only deployments with no SDK client in the project cannot run it, exactly as they cannot run `encrypt backfill`. + +| Flag | Description | +|---|---| +| `--table` / `--column` | Required. `--column` is the encrypted column. | +| `--plaintext-column <name>` | The twin (see resolution order above) | +| `--chunk-size <n>` | Default 500 — rows per `bulkDecrypt` / ZeroKMS request. A whole number from 1 to 32767 | +| `--pk-column <name>` | Override primary-key detection | +| `--confirm-dual-writes-deployed` | Non-interactive equivalent of the prompt: the app writes the encrypted column **and** its plaintext twin on every insert and update | +| `--force` | Re-decrypt rows whose plaintext is already set and overwrite it: drift recovery, or replacing a twin that held values before the first run. Safe only while dual-writes are live: any row the application wrote to the plaintext column alone (a newer value, or one cleared on purpose) gets the older ciphertext's value back. | +| `--confirm-plaintext-current` | First run only: keep the values a twin already holds, asserting they are current (see above) | +| `--continue-on-undecryptable` | Do not stop when the first batch decrypts nothing | + +#### `decrypt drop` + +```bash +stash decrypt drop --table users --column email_encrypted +stash decrypt drop --table users --column email_encrypted --uninstall-eql +``` + +Runs from `decrypted`, against the plaintext twin the backfill **recorded** (a fully-populated but different column would make the coverage check pass vacuously). It verifies live coverage — no row with ciphertext set and plaintext NULL — then writes a migration that re-checks coverage at apply time under `ACCESS EXCLUSIVE`, **drops every index built on the encrypted column** (the EQL-term functional indexes; listed in the output), and drops the column, all in one `DO` block. Not applied for you; Drizzle projects get a journaled `drizzle-kit generate --custom` file. + +Generating records `encrypted-dropped`, which stops a second drop being generated for the same column. It does not edit `.cipherstash/migrations.json`, because the migration has not run and its apply-time check can still abort it. After applying it, remove the column's entry from that file (the command's output names it); until then `encrypt status` shows the entry as `reversed` rather than as drift. If the apply-time check aborts, re-run `decrypt backfill`, then `decrypt drop` again. Keep dual-writing until the migration is applied. + +`--uninstall-eql` appends the EQL v3 uninstall (`@cipherstash/eql`'s script: `DROP SCHEMA eql_v3, eql_v3_internal … CASCADE`) to the same block, plus the `public.eql_v3_*` column domains that script deliberately leaves behind (they live outside the disposable schemas so upgrades never touch encrypted columns). Only when this is the **last** EQL v3 column in the whole database — checked now and again at apply time, because the cascade would delete any other encrypted column. `cipherstash.cs_migrations` is kept. + +Flags: `--table`, `--column`, `--plaintext-column`, `--migrations-dir <path>`, `--uninstall-eql`. + ### Deployment #### `env` diff --git a/skills/stash-encryption/SKILL.md b/skills/stash-encryption/SKILL.md index 46cb8dc99..cf4ae37a5 100644 --- a/skills/stash-encryption/SKILL.md +++ b/skills/stash-encryption/SKILL.md @@ -968,6 +968,8 @@ Once dual-writes are recorded as live in `cs_migrations`: **Create functional indexes between backfill and the read switch.** Build the `eql_v3.*` extractor indexes for every queried capability and run `ANALYZE`. +**Reversing, back to plaintext.** `stash decrypt plan` / `backfill` / `drop` run this lifecycle backwards; the Decrypt section of `stash-cli` is the reference. Two differences to plan for. In reverse, the application keeps writing BOTH columns until the encrypted column's drop migration is applied, including after reads move to the plaintext column. And while the forward migration is still open (before `stash encrypt drop` has generated its drop migration), `stash decrypt backfill` refuses the original plaintext column outright, whatever flags are passed: only one migration may be open on a pair of columns, so finish the forward one with `encrypt drop` first. If the original column was already removed outside stash, `stash decrypt plan` notices it is missing and closes the forward migration itself (see `stash-cli`). Once `encrypt drop` has generated the drop but it has not been applied, the original column still exists but stopped being updated when the dual-write code above was removed, so the first `decrypt backfill` run refuses to reuse it as-is: pass `--force` to overwrite it from the ciphertext, or `--confirm-plaintext-current` only if it was written on every change. Values encrypted under an end-user lock context cannot be decrypted by the CLI (see `stash-auth`). + ### State storage Two current sources of truth, kept separate on purpose: