Skip to content

Fix 5xx/429 retries in api_client so they actually fire, with backoff - #228

Open
cx-happy-yang wants to merge 2 commits into
checkmarx-ts:masterfrom
cx-happy-yang:fix/backoff-on-5xx-retries
Open

cx-happy-yang wants to merge 2 commits into
checkmarx-ts:masterfrom
cx-happy-yang:fix/backoff-on-5xx-retries

Conversation

@cx-happy-yang

Copy link
Copy Markdown
Contributor

check_response() was called inside call_api() before the @Retry decorator could inspect the response, so the 500/502/503/504 and 429 retry branches were dead code — every server error raised ValueError on the first attempt with zero retries (verified against customer logs on 1.9.1: 504 gateway timeout on SCA report export).

Move check_response() into the retry wrapper, after the retry decision:

  • 500/502/503/504: honor Retry-After header when present, otherwise exponential backoff (5s, 10s, 20s, ... capped at 30s), retry up to max_retries, then raise the usual ValueError format on exhaustion.
  • 429: same treatment — Retry-After first, then exponential backoff (60s base, capped at 300s).
  • Non-retryable statuses (e.g. 404) still raise immediately, and success responses pass through unchanged.

Adds tests covering 504 retry-with-backoff, growing backoff, Retry-After, exhausted-retries raising, 429 Retry-After, and 4xx non-retry.

Problem

Customers on 1.9.1 still get 504 gateway timeouts (SCA report export) with zero retries. Traceback verified against tag b2e0095:

api_client.py:271 in call_api -> check_response(response)
api_client.py:204 in check_response -> raise ValueError
ValueError: ('HttpStatusCode: 504', 'ErrorMessage: {"message": "Endpoint request timed out"}')

Root cause: check_response() was called inside call_api() before the @retry decorator could inspect the response, so the 500/502/503/504 and 429 retry branches in the decorator were dead code — every server error raised on the first attempt.

Fix

Move check_response() into the retry wrapper, after the retry decision:

  • 500/502/503/504: honor Retry-After header when present, otherwise exponential backoff (5s, 10s, 20s, ... capped at 30s), retry up to max_retries, then raise the usual ValueError format on exhaustion.
  • 429: same treatment — Retry-After first, otherwise exponential backoff (60s base, capped at 300s).
  • Non-retryable statuses (e.g. 404) still raise immediately; success responses pass through unchanged.

Tests

Six new tests in TestServerErrorRetry covering: 504 retry-with-backoff then success, growing backoff (5s then 10s), Retry-After honored, exhausted retries raising the expected ValueError, 429 with Retry-After, and 4xx raising without retry. tests/test_api_client.py: 13/13 pass.

check_response() was called inside call_api() before the @Retry
decorator could inspect the response, so the 500/502/503/504 and 429
retry branches were dead code — every server error raised ValueError on
the first attempt with zero retries (verified against customer logs on
1.9.1: 504 gateway timeout on SCA report export).

Move check_response() into the retry wrapper, after the retry decision:

- 500/502/503/504: honor Retry-After header when present, otherwise
  exponential backoff (5s, 10s, 20s, ... capped at 30s), retry up to
  max_retries, then raise the usual ValueError format on exhaustion.
- 429: same treatment — Retry-After first, then exponential backoff
  (60s base, capped at 300s).
- Non-retryable statuses (e.g. 404) still raise immediately, and
  success responses pass through unchanged.

Adds tests covering 504 retry-with-backoff, growing backoff,
Retry-After, exhausted-retries raising, 429 Retry-After, and 4xx
non-retry.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@cx-happy-yang
cx-happy-yang requested a review from a team October 27, 2026 07:46
Moves the Unreleased entries (CxOne export renames and the api_client
5xx/429 retry fix) under a 1.9.2 release section.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants