[jkwok-atg]

Summary

Even since this incident https://status.amplitude.com/incidents/0h7rsg8tsv5t, we've noticed a significant increase in our error logs from the SDK when trying to initialize the client. The error messages returned were [Experiment] Failed to fetch variants: Request-sent or [Experiment] Retry failed: Request-sent. These suggested the http.client connection was stuck in the _CS_REQ_SENT ("Request-sent") internal state after a non-200 response status. Searching online suggested that the most common reason for this is a failure to drain or read the previous response body completely from the socket. Seeing as our code had set the config to fetch_retries=1, those errors and that reasoning lined up. Hopefully the below is a permanent fix for this bug.

Checklist

• [x ] Does your PR title have the correct title format?
• Does your PR have a breaking change?: no

---

Note

Low Risk
Small behavioral change in error-path handling for remote variant fetch; improves connection reuse with no auth or data-model changes.

Overview
Fixes pooled HTTP connections getting stuck after failed variant fetches by always draining the response body in __do_fetch before handling status codes or parsing JSON.

Previously the body was only read on HTTP 200, so error responses could leave http.client in a Request-sent state and break the next request on the same keep-alive connection (matching Request-sent errors on init/retry). The bytes are now read once up front and reused for JSON parsing on success.

^{Reviewed by Cursor Bugbot for commit b871da9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit bc94924. Configure here.}

[jkwok-atg]

@aliaksandr-kazarez @sandeep-madugula @cmyui Is it possible to get a review for this?

[cmyui]

Hi @jkwok-atg. I am not a maintainer of this library nor employee of Amplitude, just an occasional contributor.

I am consistently able to reproduce the problem you are seeing, I get ResponseNotReady('Request-sent') each time, and I can confirm that your code fixes it.

This seems to be the correct, ideal and clearest fix.

If this is an urgent issue for you, I would recommend you create a support ticket with Amplitude in their support portal to have this issue prioritized. This is what I did for my recently merged changes.

---

On the code side, one recommendation might be to create a small regression test. Both for regressions, and to help maintainers confirm the problem.

Additionally, here is the way I was able to reproduce and confirm the fix of the problem:

# Repro: an unread non-200 body breaks the next request on the pooled connection.
# pip install amplitude-experiment && python repro_min.py
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

from amplitude_experiment import RemoteEvaluationClient, RemoteEvaluationConfig, User

class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"  # keep-alive, like api.lab.amplitude.com
    count = 0

    def do_POST(self):
        self.rfile.read(int(self.headers["Content-Length"]))
        Handler.count += 1
        status, body = (503, b'{"error":"unavailable"}') if Handler.count == 1 else (200, b"{}")
        self.send_response(status)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass

server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()

client = RemoteEvaluationClient("server-key", RemoteEvaluationConfig(
    server_url=f"http://127.0.0.1:{server.server_address[1]}",
    fetch_retries=1,
    fetch_retry_backoff_min_millis=0,
))
try:
    client.fetch_v2(User(user_id="user"))
    print("OK: retry succeeded after the 503")
except Exception as e:
    print(f"BUG: retry failed with {type(e).__name__}: {e}")

[kyeh-amp]

Thanks for tracking this down and sending a fix. We appreciate it.

I reproduced the issue. An unread error body leaves the pooled connection in the Request-sent state, so the retry fails, and your change fixes that common case.

During review I found one related path. If the error body is slow and the read times out, the connection still goes back to the pool and the retry fails the same way. I'll bring your fix in on a new branch, close that gap, and add regression tests. Your commit and authorship will stay as they are. I'll link the new PR here once it's open.

[kyeh-amp]

The fix is continued in #84. It includes your original commit with authorship preserved, plus the pooled-connection follow-ups and local HTTP regression tests. Thank you for the original fix.

[kyeh-amp]

Fixes were published in today's 1.13.1 release. Please check them out! I tried to preserve the authorship

Your engagement is super valued. Thank you so much for your contribution 🙏

[jkwok-atg]

@kyeh-amp thanks for the follow-up and release!