feat: Retry indefinitely after a data source failure instead of stopping permanently

[jsonbailey]

Note

This requires ld-eventsource 3.0.1. 3.0.0 (launchdarkly/ruby-eventsource#91) adds SSE::Errors::StreamClosedByServerError, and 3.0.1 (launchdarkly/ruby-eventsource#95) hardens the reconnect loop. The gemspec pins it exactly.

BEGIN_COMMIT_OVERRIDE
feat: Retry indefinitely after a data source failure instead of stopping permanently
fix: Warn and use the documented default for an invalid poll interval or initial reconnect delay
fix: Measure a repeating task's interval from when the task returns
fix: Log normal HTTP errors from a data source at warn instead of error
fix: Update ld-eventsource to 3.0.1
END_COMMIT_OVERRIDE

Requirements

• I have added test coverage for new or changed functionality
• I have followed the repository's pull request submission guidelines
• I have validated my changes against all supported platform versions (local runs on Ruby 4.0 only; CI covers the rest once the gem is released)

Related issues

• Jira: SDK-2793 (gem: SDK-3204)
• feat!: Report a server-initiated stream close to the error handler ruby-eventsource#91
• fix: Close the connection before reporting a stream error, and keep the reconnect loop alive ruby-eventsource#95 (follow-up hardening, released in 3.0.1)

Describe the solution you've provided

The FDv1 streaming and polling data sources now follow the RETRY spec. No HTTP or transport failure stops a data source.

Each failure is normal or unexpected:

Failure	Class
400, 408, 429, any 5xx	normal
Any other 4xx, including 401, 403	unexpected
Network, TLS, timeout, bad content type, bad JSON, server closes the stream	normal

	Streaming	Polling
Normal delay	initial_reconnect_delay (1s), doubling to 30s	poll_interval (flat)
Extended delay (after an unexpected failure)	5 min, doubling to 1 h	5 min, doubling to 1 h, never below poll_interval
Jitter	subtract up to half	subtract up to half
Reset	60s of healthy operation	two successful polls in a row

The SDK owns the reconnect delay. The SSE client gets reconnect_time: 0, and the SDK waits inside its error handler on an event that stop sets, so stop ends even an hour-long wait at once. A server close arrives in the error handler as SSE::Errors::StreamClosedByServerError.

Behavior changes (for release notes)

1. A bad or revoked SDK key no longer fails fast. LDClient.new and postfork wait the full wait_for_sec and return uninitialized, and the SDK keeps retrying.
2. FDv1 reports OFF only after shutdown, and no status follows it. HTTP errors give INTERRUPTED, or keep INITIALIZING before the first success. A health check that treats OFF as "SDK key rejected" no longer fires: use initialized? and the status's last_error instead.
3. A rejected SDK key logs at error about once an hour, not once.
4. A server stream close logs a warning and backs off. A normal stream rotation by the server now shows as an INTERRUPTED then VALID status change.
5. Failure logs state the real delay, e.g. HTTP error 401 (invalid SDK key) for streaming connection - will retry in 300.0s. Unexpected HTTP errors (401, 403, other 4xx) log at error. Normal ones (5xx, 429, 408, 400) now log at warn, as in the Go and Python SDKs.
6. An invalid poll_interval or initial_reconnect_delay logs a warning and uses the default.
7. Every repeating task (polling, big-segment status, store availability check, FDv2 timer) now waits its full interval after the run returns, not the interval minus the run time.

What changed

• impl/retry_state.rb (new): the delay calculation, both reset policies, and input validation. It has no HTTP knowledge.
• impl/data_source/stream.rb: one failure path for every error, including bad JSON (one failure, one INVALID_DATA status); the wait on a stop event, which also runs if logging or reporting raises; stop wakes a waiting handler first and always reports OFF; only data events count as healthy operation; reconnect_time: 0 as an Integer.
• impl/data_source/polling.rb: the retry state drives the task interval; the stop-on-unexpected-status branch is gone.
• impl/repeating_task.rb: the interval can be a callable, and every wait starts when the run returns. A callable that raises or returns something other than a number of zero or more is logged and waits 1s.
• impl/util.rb: HTTP status classification, the failure log helpers, and TIMEOUT_MAX. http_error_recoverable? uses the shared classification, so FDv2 and events do not change.
• impl/data_source.rb: the FDv1 update sink ignores every status after OFF.
• impl/data_system/streaming.rb (FDv2, one line): StreamClosedByServerError reports as NETWORK_ERROR, not UNKNOWN.
• interfaces/data_source.rb, ldclient.rb: INITIALIZING, INTERRUPTED, OFF, update_status and initialized? docs.
• contract-tests/service.rb: declares retry-conformance-fdv1-streaming and retry-conformance-fdv1-polling.

Known gaps / out of scope

• FDv2 still stops on an unexpected status, and the event processor still disables itself on one. Both are SDK-2776.
• A server retry: field makes the gem add its own wait on top of the SDK's (RETRY §1.11 is not implemented). LaunchDarkly does not send that field.

Testing

• bundle exec rspec spec: 1237 examples, 0 failures. RuboCop: 192 files, no offenses. Run against ld-eventsource 3.0.1, after merging main.
• Contract tests with -enable-long-running-tests, run out of band:
  • Red, on unchanged main with the two capabilities declared: 13 failures, exactly the 9 streaming and 4 polling conformance tests.
  • Green, on this branch at 3a91fee with ld-eventsource 3.0.1: 4697 ran, 0 failed. (1386bda after it is a refactor with no behavior change.)
• CI does not pass -enable-long-running-tests, so the conformance tests do not run in CI. Declaring the capabilities also turns off the legacy "do not retry" tests.

Describe alternatives you've considered

Retry profiles inside the gem (the Go and Java approach). Rejected: the algorithm would then exist twice, in the gem for streaming and in the SDK for polling.

---

Note

Overview
FDv1 streaming and polling now follow the RETRY spec: failures no longer permanently stop the data source. A new RetryState drives exponential backoff with jitter (normal vs extended regimes for unexpected 4xx like 401/403), and both paths keep reporting INTERRUPTED/INITIALIZING while retrying instead of OFF for bad SDK keys.

Streaming owns reconnect timing (reconnect_time: 0 on the SSE client, wait in handle_failure on a stop-wakable event). Polling uses a callable interval on RepeatingTask so backoff applies between polls. OFF is final in the FDv1 update sink; docs and initialized? text reflect that invalid keys mean “wait full timeout, stay uninitialized, keep retrying.”

Also bumps ld-eventsource to 3.0.1 (server stream close as StreamClosedByServerError), tightens HTTP error logging (unexpected at error, normal at warn with explicit retry delay), validates invalid poll/reconnect delays, and declares FDv1 retry contract-test capabilities.

^{Reviewed by Cursor Bugbot for commit 1386bda. Bugbot is set up for automated code reviews on this repo. Configure here.}