[po-et]

An unexpected failure of one synchronous BatchStream currently closes and recreates the entire shared client connection. While that reconnect is in progress, queries on the same client fail with WeaviateClosedClientError and another active batch RPC terminates, even when the server is otherwise healthy.

This change retries the failed RPC on the existing connection. It stops the old request iterator before handing over queued work, replays only sent requests without final results, and preserves the terminal stop when the caller has already left the batch context. Consecutive retry attempts are bounded, and explicit client.close() interrupts recovery without reopening the client. Reference serialization uses a copy so replay does not mutate the cached reference. Recovery preserves skip_init_checks and the configured stream/init timeouts; it does not add a total recovery deadline.

Related to #2139. The server's explicit shutting_down reconnect policy remains unchanged. Startup and context-exit error propagation remain outside this change and are addressed separately in #2141; this PR does not claim to resolve the whole issue.

Validation uses public SDK calls against real local HTTP/gRPC fixtures, including one or two failed streams; object and reference failures before ACK, after ACK, and after a final result; successive replacement failures; queued work ownership; context exit; retry exhaustion; and explicit close. The service records actual request counts, metadata, deadlines, and RPC termination. No Weaviate node crash, Docker cluster, or multi-node failover test has been run.

Validation on the final source: 540 unit tests passed, 1 skipped; the full local mock group had 86 passed and 1 failure, including all 17 new recovery cases passing. The failure was mock_tests/test_auth.py::test_refresh: a rotated refresh token no longer matches the fixture's original-token handler. The same failure reproduced on pristine base 671f0a5c when running the auth module (22 passed, 1 failed); that base test passed in isolation. No auth code or assertions were changed. Full Pyright and changed-file Ruff/format/flake8 checks passed. Independent real gRPC checks also verified that unavailable/denied Health services are not called when initialization checks are skipped.

Developed with OpenAI Codex and independently reviewed and tested by a separate Codex agent.

[orca-security-eu]

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0   0   0   0	View in Orca
Passed	SAST	0   0   0   0	View in Orca
Passed	Secrets	0   0   0   0	View in Orca
Passed	Vulnerabilities	0   0   0   0	View in Orca

[weaviate-git-bot]

To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Forum?