Skip to content

feat(export): add chunked, resumable R2 multipart database dumps with DO alarm breathing intervals (#59) - #359

Open
yumc-afk wants to merge 1 commit into
outerbase:mainfrom
yumc-afk:feat/chunked-r2-multipart-dump-59
Open

yumc-afk wants to merge 1 commit into
outerbase:mainfrom
yumc-afk:feat/chunked-r2-multipart-dump-59

Conversation

@yumc-afk

@yumc-afk yumc-afk commented Nov 21, 2026 •

Copy link
Copy Markdown

Fixes #59
/claim #59

Summary

Replaces the unbounded in-memory SELECT * database dump with a paginated, memory-bounded exporter supporting large (1 GB – 10 GB) Durable Object SQLite databases without exhausting Worker isolate memory, timing out client requests, or starving concurrent DO queries:

  • Keyset-paginated shared producer & ReadableStream fallback (GET /export/dump): Uses 64-bit string CAST(_rowid_ AS TEXT) keyset pagination (with collision-free cursor aliasing, automatic WITHOUT ROWID / _rowid_ OFFSET fallback, adaptive page-size halving on wide-row RPC payload limits, and RLS-parser-compatible integer LIMIT/OFFSET) to avoid $O(N^2)$ scans and deletion-induced offset skips on standard rowid tables. When DUMP_BUCKET is unbound (or for non-admin callers, preserving RLS/allowlist enforcement), streams chunks via a pull-based ReadableStream with backpressure and cancellation.
  • Automatic sync-to-async R2 promotion & explicit ?async=true (admin-only): Admin requests with DUMP_BUCKET bound enjoy a fast synchronous download (200 OK, database_dump.sql) under the configurable sync_timeout_ms (default 20s latency policy addressing issue Database dumps do not work on large databases #59's 30s repro) and 16 MiB memory cap, automatically promoting to an R2 Multipart Upload (dump_YYYYMMDD-HHMMSS.sql, atomically deduplicated via UNIQUE(r2_key)) and returning 202 Accepted with statusUrl and downloadUrl.
  • Crash-safe R2 multipart & DO alarm breathing intervals: StarbaseDBDurableObject.alarm() is the sole executor for async dump jobs—slicing exact 5 MiB non-final parts (per R2's equal-part-size rule), checkpointing sub-part tails under copy-on-write versioned staging keys (${key}.pending.${bytesWritten}: write new → commit SQL → delete old, and terminal complete() → commit SQL → cleanup staging), and yielding for breathing_ms (default 5s) after each work_budget_ms (default 5s) slice.
  • Starvation-free CronPlugin alarm coexistence: A 2-slot durable alarm multiplexer (tmp_alarm_slots) caches slots to skip no-op CronPlugin middleware writes, preserves final-second cron deadlines without clamping past physical alarms, never postpones earlier/due physical alarms under traffic, uses strict cron <= now checks so dump ticks never fire cron early, and automatically re-arms waiting cron alarms once dumps complete.
  • Status, streaming download & isolated webhooks: Adds admin-gated GET /export/dump/:jobId (omitting callback_url secrets), GET /export/dump/:jobId/download (streaming object.body directly from R2), and callback_url / X-Callback-Url completion/failure notifications with 10s timeouts on both sync and async paths isolated from terminal job state.
  • SQL serialization & internal table hygiene: Properly formats NULL, BLOB (X'...'), booleans, BigInt, quotes SQLite reserved keywords (order, group, etc.), emits explicit column lists for numeric column names, sanitizes newlines in -- Table: comments, trims trailing DDL ;, and excludes sqlite_*, _cf_*, and StarbaseDB internal tables (tmp_query_log, tmp_cache, etc.) while keeping user tmp_* tables.

Notes

  • Multi-slice dumps across breathing intervals intentionally yield to live traffic and are therefore non-snapshot exports (64-bit rowid keyset pagination prevents deletion-induced offset skips on standard rowid tables).
  • Pre-existing src/import/dump.ts line-splitting behavior on multiline strings containing -- / ;\n and SQLite generated columns remain unchanged.

Configuration (wrangler.toml)

[[r2_buckets]]
binding = "DUMP_BUCKET"
bucket_name = "starbasedb-dumps"

Verification

  • pnpm exec vitest run src/export/ src/import/ src/do.test.ts src/handler.test.ts — 75/75 passed (coverage on src/export/dump.ts: 90.40% stmts / 82.04% branches / 88.70% funcs / 91.17% lines; src/do.ts: 72.00% stmts / 63.15% branches / 71.83% lines).
  • Full repo pnpm exec vitest run — 18/19 files, 174/178 passed (only the 4 pre-existing src/rls/index.test.ts failures on main).
  • Live local Cloudflare workerd/Miniflare verification (useSQLite: true, R2 DUMP_BUCKET, ENABLE_RLS=0 & 1) confirmed: clean empty dump (0 internal tables / 0 recursive tmp_query_log reads), "order" + numeric column + >2^53 64-bit rowid + shadowed rowid export/import round-trip (200 OK), 18×100 KB wide-row adaptive RPC halving, 3 concurrent ?async=true unique keys, and client-role ENABLE_RLS=1 sync 200 OK + async/status 403.

@yumc-afk
yumc-afk force-pushed the feat/chunked-r2-multipart-dump-59 branch from b98613e to 2efe443 Compare November 21, 2026 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database dumps do not work on large databases

1 participant