Skip to content

Add incremental BLOB streams - #361

Open
quinnj wants to merge 2 commits into
fix/statement-handle-lifetimefrom
feature/incremental-blob-io
Open

quinnj wants to merge 2 commits into
fix/statement-handle-lifetimefrom
feature/incremental-blob-io

Conversation

@quinnj

@quinnj quinnj commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

Large SQLite BLOBs currently require whole-value query materialization. Add qualified SQLite.Blob <: IO so callers can read or overwrite an existing fixed-size value using normal Julia IO and a reusable buffer. It defaults to read-only access, exposes raw BLOB/TEXT bytes, and leaves ordinary query conversion unchanged. No new export or dependency is added.

This PR is stacked on #362. BLOB and statement handles share one database lifecycle lock. The DB retains handle references without retaining stream objects; explicit DB close consumes all tracked BLOBs, including those with pending finalizers, so they cannot commit later. GC finalizers defer while the lock is busy. Close errors still consume handles and are reported after cleanup; a do-block preserves both callback and close failures. Ordinary stream/connection use still requires caller synchronization.

Validation on the actual stacked source:

  • Full suites pass 655 assertions each on Julia 1.9.4 / SQLite 3.51.2 and Julia 1.13 / SQLite 3.53.4, including 171 BLOB and 100 statement lifetime assertions. The original 89 independent IO/ownership assertions also pass on each runtime. Single-thread BLOB tests pass 165 assertions with one intentional cross-thread-only skip.
  • Deterministic controls on the unsynchronized stack reproduce premature cleanup under a held lifecycle lock and blocked BLOB finalization during a query callback on both runtimes. The corrected tests cover shared statement/BLOB callback ordering and concurrent GC.
  • Strict documentation build and executable examples pass. Reused 4 KiB reads and writes still allocate zero Julia heap bytes for 1 MiB and 64 MiB values on both runtimes, excluding buffer creation and native SQLite allocation.
  • Extended the existing isolated native workload with public Blob operations. It compiles with JuliaC safe trimming on Julia 1.13 and passes five runtime-input runs with code generation disabled. Prepared-query result checks use raw C column reads; this does not claim native DBInterface/Tables materialization.

The docs cover fixed-size storage, row expiry, implicit commits, explicit rollback, bounded-buffer IO, and close-error behavior. Incremental IO copies bytes between SQLite and the caller's buffer; whole-value operations remain available. This update makes no new transfer-throughput claim.

Closes #345.

All seven hosted runtime, documentation, and native-core checks pass at e1b1eae, including public BLOB IO in the Linux x86_64 native executable.

Co-authored by Codex

@quinnj
quinnj force-pushed the feature/incremental-blob-io branch from 113eb3b to e1b1eae Compare September 26, 2026 11:05
@quinnj
quinnj changed the base branch from master to fix/statement-handle-lifetime September 26, 2026 11:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: BLOB API

1 participant