[muhammad-ali-e]

Requested by ali · project thread

What

• get_workflow_execution (worker client) now always sends file_execution explicitly, defaults it to false, and builds its query through params.
• The callback asks for the file-execution rows once (billing needs them) and hands that context to the status step instead of fetching the execution twice more.
• Each file fetches its execution context once, and its source/destination config lookups reuse that fetch instead of making their own.

Why

Load tests showed ~29 internal API calls per document, 8 of them GET workflow-execution/<id>/, at 0.5–1.5 s each under load. Every one of them also ran list(execution.file_executions.values()):

• the backend defaults file_execution to true;
• the client only ever appended ?file_execution=true and never sent false, so every caller passing file_execution=False still got all rows.

For an ETL run with N files, the per-file fetches made that O(N²) rows. Separately, include_cost=True with the old default produced ?include_cost=true?file_execution=true, which the backend parsed as include_cost="true?file_execution=true", so no cost was returned. Current callers happen to avoid it.

How

Per document (API deployment, one file), GETs before → after:

Step	Before	After
API orchestrator tool-validation check	1 (with rows)	1 (no rows)
File batch setup	1 (with rows)	1 (no rows)
Per file: execution context	1 (with rows)	1 (no rows)
Per file: source config	1 (with rows)	0, reuses the file's fetch
Callback context	1 (with rows)	1 (with rows, needed for billing)
Callback wall-clock time	1 (with rows)	0, reuses the callback's
Callback expected files	1 (with rows)	0, reuses the callback's
Callback cost UI log (include_cost)	1 (with rows)	1 (no rows)
Total	8, all with rows	5, one with rows

ETL saves 1 GET per file, and none of the per-file or per-batch fetches scale with the file count any more.

• shared/clients/execution_client.py: params={"file_execution": "true"|"false", ["include_cost": "true"]}; default False (also in the InternalAPIClient facade).
• callback/tasks.py: _extract_callback_parameters and process_batch_callback_api request file_execution=True, which matches what they actually received before. The context is stored as CallbackContext.execution_context and passed to _determine_execution_status_unified(execution_context=...).
• shared/processing/files/time_utils.py: calculate_execution_time(..., execution_context=None) reuses the context when given (created_at is immutable; now is still taken at call time) and refetches if it is empty.
• shared/workflow/execution/service.py: the file's context is kept, keyed by execution_id, for _get_source_config / _get_destination_config.

The context is deliberately not shared from the batch to each file. The file's source connector settings (including auth tokens) come from this fetch, so sharing it would freeze them for the whole batch, which can run for an hour.

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

• Subscription billing: it reads context.file_executions. Both callback fetches now ask for the rows explicitly, so billing gets the same data as before. A test covers this.
• Rolling deploys: the backend is unchanged and still defaults to true, so older workers behave exactly as before. New workers simply say what they want.
• Status determination: unchanged. expected_files is still read from the top level of the response, as before. That key doesn't exist there (it lives under execution), so expected_files is always 0 today, on main too. I left it alone because fixing it would change the timeout-failure behaviour; it's noted on UN-4514 for a separate change.
• Connector settings freshness: unchanged. Each file still reads them at its own start.
• No caller outside callback/ reads file_executions from this endpoint (checked in OSS and unstract-cloud).

Database Migrations

• None

Env Config

• None

Relevant Docs

• PG Queue & DB Load findings, item Corrected alignment of the OnBoard page to accommodate various screen sizes #6

Related Issues or PRs

• UN-4514
• UN-4513 [FIX] Stop execution status updates from blocking file-execution inserts #2376 (UN-4513, finding Basic workflow and adapter exception messages #4) changes nearby execution-status code in the backend. The two PRs touch no files in common.

Dependencies Versions

• None

Notes on Testing

• New workers/tests/test_workflow_execution_fetch_reuse.py (17 tests): query params, callback fetch/reuse, wall-clock reuse and fallbacks, per-file fetch-once, and cache keying.
• Full workers suite: 1464 passed, 184 skipped.
• Not yet run on a live stack. Worth an ETL plus an API deployment run on a dev namespace, checking that billing commits still happen and that the backend access log shows fewer workflow-execution GETs.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WpxxVeu41XtPNkvP2Lm8H4

[greptile-apps]

via Greptile

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable new issue was found.

Summary

The PR reduces repeated execution fetches and avoids loading file-execution rows unless requested.

• Workflow execution requests skip file rows unless a caller asks.
• Callbacks keep billing rows and reuse their execution details.
• Each file reuses its execution details for configuration lookups.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Callback fetches execution with file rows] --> B[Billing uses file rows]
  A --> C[Status and timing reuse context]
  D[Each file fetches fresh execution without file rows] --> E[Source and destination lookups reuse context]

Loading

_{Reviews (2) · Last reviewed commit: "UN-4514 [FIX] Address review: keep per-f..." · Reviewed by Greptile}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[muhammad-ali-e]

Self-review before asking for a human review. I ran three review rounds; this is what changed in 5cdd74a and what I deliberately left out.

Fixed

• Each file no longer reuses the batch's execution context. The file's source connector settings, including auth tokens, come from that fetch, so sharing it would have frozen them for a batch that can run an hour. Each file now fetches once with no rows, and its config lookups reuse that fetch.
• The per-file context cache is keyed by execution_id.
• The status step tolerates a missing context, and wall-clock time refetches on an empty one.
• The full response is now named execution_context everywhere.

Not changed, with reasons

• expected_files reads total_files from the wrong level of the response, so it is always 0, on main too. Fixing it changes when an execution is marked ERROR (for example, when every file is skipped by file history). That needs its own change; it's noted on UN-4514.
• The backend still defaults file_execution to true. Flipping it would let older workers bill zero files during a rolling deploy. That can follow once every worker sends the flag.
• The callback still loads all rows for billing. That happens once per execution, not per document, and it is the same data billing uses today. Loading them lazily inside the billing step changes the billing data source, so it belongs in a separate change.
• The separate include_cost fetch, the per-file tool-instance and workflow fetches, and a shared fetch helper are out of scope (finding Set server_tokens to off in the nginx configuration #8 covers the per-file lookups).

---

Generated by Claude Code

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

Unstract test results

Per-group results

Status	Group	Tier	Passed	Failed	Errors	Skipped	Duration (s)
⚪	e2e-agent-kv	e2e	0	0	0	1	0.0
✅	e2e-api-deployment	e2e	3	0	0	0	15.8
✅	e2e-coowners	e2e	1	0	0	0	0.9
✅	e2e-etl	e2e	1	0	0	0	15.7
✅	e2e-login	e2e	2	0	0	0	0.8
✅	e2e-prompt-studio	e2e	1	0	0	0	5.9
✅	e2e-smoke	e2e	2	0	0	0	0.7
✅	e2e-workflow	e2e	1	0	0	0	13.6
✅	frontend	unit	629	0	0	0	17.4
✅	integration-backend	integration	1420	0	0	26	84.9
✅	integration-connectors	integration	1	0	0	7	9.7
✅	integration-workers	integration	181	0	0	1	55.7
❌	ui	e2e	0	1	0	0	0.0
✅	unit-agent-kv-schema	unit	143	0	0	0	1.7
✅	unit-backend	unit	2033	0	0	1	58.2
✅	unit-connectors	unit	82	0	0	0	11.3
✅	unit-core	unit	281	0	0	0	3.1
✅	unit-filesystem	unit	1	0	0	0	4.3
✅	unit-platform-service	unit	18	0	0	0	2.7
✅	unit-rig	unit	120	0	0	0	4.5
✅	unit-runner	unit	12	0	0	0	2.9
✅	unit-sdk1	unit	878	0	0	0	42.4
✅	unit-tool-sidecar	unit	8	0	0	0	1.8
✅	unit-workers	unit	1551	0	0	2	130.8
	TOTAL		7369	1	0	38	484.7

Critical paths

⚠️ Critical paths not yet covered

• workflow-execution-fan-out — Multi-file workflow execution fans out to file-processing workers and rejoins. (declared coverage: no groups declared)

✅ Covered critical paths

• auth-login — covered by e2e-login
• adapter-register-llm — covered by integration-backend
• workflow-author — covered by integration-backend
• co-owner-manage — covered by integration-backend, e2e-coowners
• workflow-create-execute — covered by e2e-workflow
• api-deployment-provision — covered by integration-backend
• api-deployment-auth — covered by integration-backend
• api-deployment-run — covered by e2e-api-deployment
• mcp-server-auth — covered by integration-backend
• mcp-platform-auth — covered by integration-backend
• platform-key-whoami — covered by integration-backend
• prompt-studio-author — covered by integration-backend
• prompt-studio-fetch-response — covered by e2e-prompt-studio
• connector-register-test — covered by integration-backend
• pipeline-etl-execute — covered by e2e-etl
• usage-aggregate-read — covered by integration-backend
• usage-token-tracking — covered by e2e-api-deployment
• callback-result-delivery — covered by e2e-api-deployment