[muhammad-ali-e]

Requested by ali · project thread

What

• Release the request's Django DB connection(s) before each sleep in the synchronous wait loops of WorkflowHelper.execute_workflow_async and WorkflowHelper.wait_for_execution.
• New helper WorkflowHelper._release_db_connection().

Why

• An API deployment call with a timeout sleep-polls for up to that timeout (max 300 s). Django holds the connection for the whole request (no CONN_MAX_AGE). Behind pgbouncer's session pool, that pins one Postgres server connection per waiting request while it does nothing.
• In the integration load tests this is what let the session pool reach ~1,500 server connections, 95–98% Cloud SQL memory, and an instance restart (UN-4177).
• This is step 1 of finding Add support for configuring and using OCR adapters #5. Step 2 caps the session pool (Zipstack/unstract-cloud#1892) and must land after this: while the connection is still pinned, a cap would starve the backend, because up to ~520 waiting requests each hold a slot.

How

• For every initialized connection alias, the helper calls Django's close_if_unusable_or_obsolete() (the check close_old_connections uses). It does not hard-close. With today's CONN_MAX_AGE=0 it closes the connection. If persistent connections are ever enabled, it only closes expired ones.
• It never touches a connection with work in flight: an atomic block (ATOMIC_REQUESTS) or autocommit turned off. Closing those would lose the work. When it has to skip, it logs a WARNING on the first poll of a wait and DEBUG after that, so the fix can't switch itself off silently.
• It never raises. A failure here would reach the post-dispatch handler and return EXECUTING without waiting (the UN-3602 failure mode). Such a failure is logged once per wait.
• It runs before every sleep, not only the first, because a Redis cache miss in _get_execution_status reads the row and reopens a connection.
• The reads after the wait (row refetch, results, acknowledge) reconnect lazily. backend.custom_db re-applies search_path on connect.

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

• No. Only the sync-wait path changes. The async path (timeout=-1) and an execution that is already complete never enter the loop.
• With ATOMIC_REQUESTS on, the helper is a logged no-op. It is off in every chart: DJANGO_ATOMIC_REQUESTS: False.
• Cost: one reconnect after the wait, plus one per poll while the status cache misses. Each is a pgbouncer login, far cheaper than holding a server connection for up to 300 s.
• Known trade-off: if pgbouncer is saturated when the execution finishes, the post-wait reconnect can queue or fail. The response is then EXECUTING with no result, and the client can still fetch status/results. Before this change, the held connection is what kept the pool saturated in the first place.

Database Migrations

• None

Env Config

• None

Relevant Docs

• PG Queue & DB Load findings, Add support for configuring and using OCR adapters #5

Related Issues or PRs

• UN-4177
• Zipstack/unstract-cloud#1892 (step 2, pool cap + server_idle_timeout; held until this is deployed and load-tested)

Dependencies Versions

• None

Notes on Testing

• workflow_manager/workflow_v2/tests/test_sync_wait_releases_db_connection.py, 21 tests:
  • Unit (DB-free):
    • The release precedes every sleep in both loops.
    • The release goes through the expiry check and never close().
    • Nothing is released on the async path, for an already-completed execution, inside an atomic block, with autocommit off, or on an unopened connection.
    • Every alias is covered.
    • A failing release does not abort the wait.
    • Logging is loud only on the first poll.
  • Integration (real Postgres):
    • The ORM reconnects after a release, with the right current_schema().
    • An unexpired persistent connection is kept.
    • A closed connection is not reopened.
    • Inside transaction.atomic() the connection stays usable.
• A mutation check: reverting the helper to a hard close() fails the unit tier.
• Ran locally against Postgres 16 + Redis:
  • Backend unit tier: 1665 passed.
  • workflow_manager + api_v2 + pg_queue integration: 107 passed.
  • The Postgres destination-connector tests were deselected (they need connector creds and are untouched here).
• Three local review rounds before review; findings fixed in follow-up commits. ruff (pinned v0.3.4) is clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RAwdv4B9xPDFEg9c18f5Az

[greptile-apps]

via Greptile

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking issues were found.

Summary

The PR releases idle database connections before each sleep in both synchronous wait loops.

• Workflow waits release idle database connections before sleeping.

No actionable new issues were found. No previous review threads were supplied.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Execution still running] --> B[Check initialized connections]
    B --> C{Open transaction?}
    C -->|Yes| D[Keep connection]
    C -->|No| E[Close if expired or unusable]
    D --> F[Sleep]
    E --> F
    F --> G[Read execution status]
    G --> H{Finished or timed out?}
    H -->|No| A
    H -->|Yes| I[Return response]

Loading

_{Reviews (2) · Last reviewed commit: "UN-4177 [FIX] Pin the release to the exp..." · Reviewed by Greptile}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

Unstract test results

Per-group results

Status	Group	Tier	Passed	Failed	Errors	Skipped	Duration (s)
⚪	e2e-agent-kv	e2e	0	0	0	1	0.0
✅	e2e-api-deployment	e2e	3	0	0	0	11.7
✅	e2e-coowners	e2e	1	0	0	0	1.0
✅	e2e-etl	e2e	1	0	0	0	5.6
✅	e2e-login	e2e	2	0	0	0	0.9
✅	e2e-prompt-studio	e2e	1	0	0	0	14.4
✅	e2e-smoke	e2e	2	0	0	0	0.8
✅	e2e-workflow	e2e	1	0	0	0	17.7
✅	frontend	unit	629	0	0	0	18.3
✅	integration-backend	integration	1424	0	0	26	82.1
✅	integration-connectors	integration	1	0	0	7	9.2
✅	integration-workers	integration	181	0	0	1	55.3
❌	ui	e2e	0	1	0	0	0.0
✅	unit-agent-kv-schema	unit	143	0	0	0	1.7
✅	unit-backend	unit	2050	0	0	1	57.6
✅	unit-connectors	unit	82	0	0	0	11.3
✅	unit-core	unit	281	0	0	0	3.2
✅	unit-filesystem	unit	1	0	0	0	4.3
✅	unit-platform-service	unit	18	0	0	0	2.7
✅	unit-rig	unit	120	0	0	0	4.4
✅	unit-runner	unit	12	0	0	0	2.9
✅	unit-sdk1	unit	878	0	0	0	41.8
✅	unit-tool-sidecar	unit	8	0	0	0	1.7
✅	unit-workers	unit	1534	0	0	2	127.3
	TOTAL		7373	1	0	38	475.8

Critical paths

⚠️ Critical paths not yet covered

• workflow-execution-fan-out — Multi-file workflow execution fans out to file-processing workers and rejoins. (declared coverage: no groups declared)

✅ Covered critical paths

• auth-login — covered by e2e-login
• adapter-register-llm — covered by integration-backend
• workflow-author — covered by integration-backend
• co-owner-manage — covered by integration-backend, e2e-coowners
• workflow-create-execute — covered by e2e-workflow
• api-deployment-provision — covered by integration-backend
• api-deployment-auth — covered by integration-backend
• api-deployment-run — covered by e2e-api-deployment
• mcp-server-auth — covered by integration-backend
• mcp-platform-auth — covered by integration-backend
• platform-key-whoami — covered by integration-backend
• prompt-studio-author — covered by integration-backend
• prompt-studio-fetch-response — covered by e2e-prompt-studio
• connector-register-test — covered by integration-backend
• pipeline-etl-execute — covered by e2e-etl
• usage-aggregate-read — covered by integration-backend
• usage-token-tracking — covered by e2e-api-deployment
• callback-result-delivery — covered by e2e-api-deployment