Skip to content

Retain aggregate state between SQLite callbacks - #363

Open
quinnj wants to merge 5 commits into
masterfrom
investigate/aggregate-state
Open

quinnj wants to merge 5 commits into
masterfrom
investigate/aggregate-state

Conversation

@quinnj

@quinnj quinnj commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

Aggregate callbacks currently serialize, copy, and deserialize the entire accumulator after every row. A callback that appends to a vector therefore does quadratic copying even though its own update is small. This change keeps one owned first-result snapshot per group, then roots later Julia values through the existing function registration. It removes the manually allocated native state buffer and releases roots before final/error callbacks can enter other SQL.

Compatibility: the first callback still receives the original init, and its result is serialized/deserialized at the existing first boundary. Later intermediate objects are now kept by reference; their Serialization hooks no longer run per row, and references saved by a callback observe later mutations. SQL argument/result conversion and empty-input behavior are unchanged. The existing shared mutable-seed limitation remains documented: use an immutable seed and create fresh mutable state in the first step when groups need independent state.

This branch builds on #362's shared database lifecycle lock. It adds no exports, dependencies, public mode, or separate lock. CI now runs for pull requests targeting any branch so this stack receives the same runtime, documentation, and native checks. Push triggers remain unchanged.

Validation:

  • Julia 1.9.4 and 1.13.0: 637/637 full-suite assertions each, including 153 new aggregate lifetime/compatibility assertions; all 58 existing callback-error assertions remain unchanged. New aggregate checks also pass with one Julia thread.
  • Parent composed this patch with the published Add incremental BLOB streams #361 BLOB stack: all 808 combined assertions pass on both runtimes with unchanged dependency versions. Production patches apply unchanged; only test-runner includes were composed.
  • Nested SQL using the same registration from step/final/error callbacks, multiple active contexts/groups, forced GC, exact error formatting, zero-argument and changing-type states, interruption, query reset, statement/DB close, and weak-reference cleanup.
  • A custom serializer retaining its writer buffer reproduced a first-snapshot aliasing error in the initial prototype; the owned-copy control and unchanged baseline both pass. The regression is in the permanent tests.
  • Strict docs pass. The existing native statement core compiles and runs three times with runtime code generation disabled.

Matched local in-memory measurements used three alternating-order pairs on each runtime, seven warmed samples per process, and independent result oracles. At 4,096 rows:

Workload Julia 1.13 wall / heap Julia 1.9 wall / heap
Scalar sum 5.751→1.006 ms; 8.323→0.657 MB 6.840→0.866 ms; 12.876→0.658 MB
Growing vector, then sum 40.437→1.011 ms; 459.817→0.706 MB 56.526→0.877 ms; 330.745→0.653 MB

The fixed cost is about 44–45 extra Julia bytes per one-row group (holder plus periodic dictionary rehashing), and 336/352 extra bytes per registration. For 4,096 one-row groups, allocation rises 1.98%/1.37%; current timings ranged from 0.97× to 1.13× baseline speed. Four-row groups were 2.37×/2.64× faster with substantially fewer allocations. Empty queries retain their allocation counts; current empty scalar timing changed 0.822→0.874 µs, minimum 0.891→0.878 µs. First-query compilation allocation did not materially increase in the retained controls. An unrelated Julia process remained active, so these are bounded local measurements, not general throughput claims.

Native limitation: registered aggregate callbacks still do not pass safe trimming. The identical fixed-aggregate probe reports 54 verifier diagnostics on the base and 57 on this branch, involving dynamic callback/Serialization state. This is not an improvement to callback trimming. SQLite also still owns/allocates the native aggregate context; only the manually managed state buffer is removed.

Hosted validation: all seven runtime, documentation, and native checks are green at b352ce6dc2325fec916bbf66557c86564e82eb5c.

Co-authored by Codex

Base automatically changed from fix/statement-handle-lifetime to master September 30, 2026 15:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant