[lewisjared]

Streams execution archives as a hand-built tar.gz, so /executions/{group_id}/archive holds at most one chunk in memory instead of writing the whole archive to a temp file before the first byte. It also fixes a few smaller API issues found in an audit.

• tar_gz_iterator writes each tar header with TarInfo.tobuf, then the file in 64 KB chunks through one zlib compressor. Streaming 512 MB of random data peaks at 0.7 MB under tracemalloc, against 35 MB for a 32 MB file with tarfile on a buffer. test_memory_does_not_grow_with_file_size guards this.
• Symlinks and other special files are now left out of archives. A file that shrinks while being read is zero-filled so the header stays true.
• Log downloads were named execution_result_None.log when no execution_id was passed. They now use the resolved execution's id.
• GET /executions/ caps limit at 1000, the page size the failures view uses.
• Diagnostic summaries no longer carry execution_groups: list[int], which loaded every group id per request. The frontend never read it, but this is a public response change, so external scripts reading that field will break.

Summary by CodeRabbit

• Improvements
  • Execution archive downloads now start sooner and use less server memory, including for large outputs. Symbolic links are excluded from downloaded archives.
• Bug Fixes
  • Log downloads now use the correct execution ID in the filename.
• Breaking Changes
  • Diagnostic summaries no longer include execution group IDs.
  • Execution group lists are now limited to 1,000 results per page.

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 52 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: e52b8d0c-1e49-4925-b697-1ddbe656ab37

Commits

Reviewing files that changed from the base of the PR and between f4934d3 and 54e55bb.

Files selected for processing: 1

• changelog/172.breaking.md

Walkthrough

The PR removes execution-group IDs from diagnostic summaries, updates diagnostic summary construction, and caps recent execution-group results at 1,000. It adds chunked tar.gz streaming for execution archives and uses the resolved execution ID in log attachment names.

Changes: Diagnostic summaries

Layer / File(s)	Summary
Diagnostic summary contract and routes backend/src/ref_backend/models/diagnostics.py, backend/src/ref_backend/api/routes/diagnostics.py, frontend/src/client/types.gen.ts, frontend/src/lib/resourceUsageCsv.test.ts, changelog/172.breaking.md	Diagnostic summary models, API routes, generated frontend types, and a test fixture no longer include execution-group IDs. Grouping now receives an optional concrete diagnostic, and missing AFT diagnostics are logged at debug level.

Execution archives and endpoints

Layer / File(s)	Summary
Stream execution archives backend/src/ref_backend/core/file_handling.py, backend/src/ref_backend/api/routes/executions.py, backend/tests/test_core/test_file_handling.py, backend/tests/test_api/test_routes/test_executions.py, changelog/172.improvement.md	A tar.gz iterator streams regular files from the output directory. The archive endpoint uses this iterator. Tests cover archive contents, symlink exclusion, files that shrink during reading, and memory use.
Execution limits and log filenames backend/src/ref_backend/api/routes/executions.py, backend/tests/test_api/test_routes/test_executions.py, changelog/172.fix.md	The recent execution-group endpoint rejects limits above 1,000. Log attachment filenames use the resolved execution ID.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant ExecutionArchive
  participant TarGzIterator
  participant OutputDirectory
  Client->>ExecutionArchive: request execution archive
  ExecutionArchive->>TarGzIterator: stream the output directory as a tar.gz archive
  TarGzIterator->>OutputDirectory: read regular files in chunks
  OutputDirectory-->>TarGzIterator: return file data
  TarGzIterator-->>ExecutionArchive: yield compressed chunks
  ExecutionArchive-->>Client: stream archive response

Loading

Possibly related issues

• Publish a Python client for the REF API #171: The issue identifies the diagnostic execution_groups field, archive endpoint behaviour, and unbounded execution-group pagination, which this PR changes.

Possibly related PRs

• Climate-REF/ref-app#145: Both PRs change how _build_group_by_summary handles diagnostics that are not registered.

Priority: Normal

Change: Bug fix

Merge Risk: Low · previous review f4934

Archive downloads now stream in bounded memory, and the log filename, limit cap and diagnostic summary changes work as intended. API clients sending a limit above 1000 will now get HTTP 422, and the changelog should say so. The remaining archive hardening concern depends on who can write to the results directory and is not a blocker.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly identifies the main change: streaming archives in constant memory. It also accurately refers to related execution route updates.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 14 functions across 6 files. (3 skipped: 3 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1

📝 Generate docstrings: 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🔇 Additional comments: 1

backend/src/ref_backend/models/diagnostics.py-67-67: 1

67-67: Minor · Data Integrity & Integration · Quick win

⚠️ Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Keep a way to list historical groups by diagnostic.

The removed execution_groups field included IDs from earlier diagnostic versions. Both diagnostic and recent-execution listings exclude superseded and retracted groups, so clients cannot use those routes to discover the old IDs. The dataset route requires a matching dataset ID. (github.com)

Add options to a listing route to include superseded and retracted groups.

---

Review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 55f3c7c3-2803-4b3d-bb85-8b764dc33d7d

Commits

Reviewing files that changed from the base of the PR and between a0f0ae9 and f4934d3.

Files selected for processing: 11

• backend/src/ref_backend/api/routes/diagnostics.py
• backend/src/ref_backend/api/routes/executions.py
• backend/src/ref_backend/core/file_handling.py
• backend/src/ref_backend/models/diagnostics.py
• backend/tests/test_api/test_routes/test_executions.py
• backend/tests/test_core/test_file_handling.py
• changelog/172.breaking.md
• changelog/172.fix.md
• changelog/172.improvement.md
• frontend/src/client/types.gen.ts
• frontend/src/lib/resourceUsageCsv.test.ts

Files with no reviewable changes: 2

• frontend/src/client/types.gen.ts
• frontend/src/lib/resourceUsageCsv.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.