Repository navigation
Conversation
… DO alarm breathing intervals (outerbase#59)
yumc-afk
force-pushed
the
feat/chunked-r2-multipart-dump-59
branch
from
November 21, 2026 13:06
b98613e to
2efe443
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #59
/claim #59
Summary
Replaces the unbounded in-memory
SELECT *database dump with a paginated, memory-bounded exporter supporting large (1 GB – 10 GB) Durable Object SQLite databases without exhausting Worker isolate memory, timing out client requests, or starving concurrent DO queries:ReadableStreamfallback (GET /export/dump): Uses 64-bit stringCAST(_rowid_ AS TEXT)keyset pagination (with collision-free cursor aliasing, automaticWITHOUT ROWID/_rowid_OFFSETfallback, adaptive page-size halving on wide-row RPC payload limits, and RLS-parser-compatible integerLIMIT/OFFSET) to avoidDUMP_BUCKETis unbound (or for non-admin callers, preserving RLS/allowlist enforcement), streams chunks via a pull-basedReadableStreamwith backpressure and cancellation.?async=true(admin-only): Admin requests withDUMP_BUCKETbound enjoy a fast synchronous download (200 OK,database_dump.sql) under the configurablesync_timeout_ms(default 20s latency policy addressing issue Database dumps do not work on large databases #59's 30s repro) and16 MiBmemory cap, automatically promoting to an R2 Multipart Upload (dump_YYYYMMDD-HHMMSS.sql, atomically deduplicated viaUNIQUE(r2_key)) and returning202 AcceptedwithstatusUrlanddownloadUrl.StarbaseDBDurableObject.alarm()is the sole executor for async dump jobs—slicing exact5 MiBnon-final parts (per R2's equal-part-size rule), checkpointing sub-part tails under copy-on-write versioned staging keys (${key}.pending.${bytesWritten}: write new → commit SQL → delete old, and terminalcomplete()→ commit SQL → cleanup staging), and yielding forbreathing_ms(default 5s) after eachwork_budget_ms(default 5s) slice.CronPluginalarm coexistence: A 2-slot durable alarm multiplexer (tmp_alarm_slots) caches slots to skip no-opCronPluginmiddleware writes, preserves final-second cron deadlines without clamping past physical alarms, never postpones earlier/due physical alarms under traffic, uses strictcron <= nowchecks so dump ticks never fire cron early, and automatically re-arms waiting cron alarms once dumps complete.GET /export/dump/:jobId(omittingcallback_urlsecrets),GET /export/dump/:jobId/download(streamingobject.bodydirectly from R2), andcallback_url/X-Callback-Urlcompletion/failure notifications with 10s timeouts on both sync and async paths isolated from terminal job state.NULL,BLOB(X'...'), booleans,BigInt, quotes SQLite reserved keywords (order,group, etc.), emits explicit column lists for numeric column names, sanitizes newlines in-- Table:comments, trims trailing DDL;, and excludessqlite_*,_cf_*, and StarbaseDB internal tables (tmp_query_log,tmp_cache, etc.) while keeping usertmp_*tables.Notes
rowidkeyset pagination prevents deletion-induced offset skips on standard rowid tables).src/import/dump.tsline-splitting behavior on multiline strings containing--/;\nand SQLite generated columns remain unchanged.Configuration (
wrangler.toml)Verification
pnpm exec vitest run src/export/ src/import/ src/do.test.ts src/handler.test.ts— 75/75 passed (coverage onsrc/export/dump.ts: 90.40% stmts / 82.04% branches / 88.70% funcs / 91.17% lines;src/do.ts: 72.00% stmts / 63.15% branches / 71.83% lines).pnpm exec vitest run— 18/19 files, 174/178 passed (only the 4 pre-existingsrc/rls/index.test.tsfailures onmain).workerd/Miniflare verification (useSQLite: true, R2DUMP_BUCKET,ENABLE_RLS=0&1) confirmed: clean empty dump (0 internal tables / 0 recursivetmp_query_logreads),"order"+ numeric column +>2^5364-bit rowid + shadowedrowidexport/import round-trip (200 OK), 18×100 KB wide-row adaptive RPC halving, 3 concurrent?async=trueunique keys, and client-roleENABLE_RLS=1sync200 OK+ async/status403.