Skip to content

perf: batch database reads and bound data loading - #715

Merged
MateoLostanlen merged 5 commits into
mainfrom
codex/bounded-query-loading
Oct 22, 2026
Merged

MateoLostanlen merged 5 commits into
mainfrom
codex/bounded-query-loading

Conversation

@frgfm

@frgfm frgfm commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

Reduce latency and memory by batching reads, selecting recent frames in SQL, and streaming bounded CSV batches. Remove the obsolete lookup and reuse existing test fixtures.

Workload Median latency, ms (base → PR) Peak RSS, MiB (base → PR)
Camera list: 100 cameras 99.3 → 11.9 164.6 → 163.8
Bbox-selection step: 20 sequences 19.4 → 7.0 162.9 → 162.9
Frame-selection step: 100,000 frames 4,660.5 → 297.5 368.2 → 162.8
CSV download: 10,000 rows 4,641.2 → 2,853.0 183.1 → 164.1

Local PostgreSQL; three fresh workers/revision, one warmup then three serial requests each. Latency is request p50; RSS is median worker VmHWM including startup, excluding DB/client. CSV timing covers the full streamed download; selection steps exclude model inference. Read responses match the base.

List endpoints enforce page limits; the client supports limit/offset (defaults in changelog). Exports can reflect concurrent edits. Regression tests, Ruff and Ty pass.

@codecov

codecov Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.00990% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 94.68%. Comparing base (9f4c75f) to head (2fc28f4).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
src/app/crud/crud_detection.py 96.96% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #715      +/-   ##
==========================================
+ Coverage   94.48%   94.68%   +0.19%     
==========================================
  Files          59       59              
  Lines        3246     3253       +7     
==========================================
+ Hits         3067     3080      +13     
+ Misses        179      173       -6     
Flag Coverage Δ
backend 94.83% <99.00%> (+0.20%) ⬆️
client 91.30% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@frgfm
frgfm requested a review from MateoLostanlen October 15, 2026 12:15

@MateoLostanlen MateoLostanlen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this optimization 😄

@MateoLostanlen
MateoLostanlen merged commit 219cfac into main Oct 22, 2026
23 checks passed
@MateoLostanlen
MateoLostanlen deleted the codex/bounded-query-loading branch October 22, 2026 12:52
@frgfm frgfm self-assigned this Oct 22, 2026
@frgfm frgfm added the type: improvement New feature or request label Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants