Skip to content

test(proxy): cover mapping-disabled recovery after failed queries - #469

Open
tobyhede wants to merge 1 commit into
cj/cip-4167-proxy-connection-pooler-compatibility-test-matrix-pgbouncerfrom
claude/project-thread-y0sclv
Open

tobyhede wants to merge 1 commit into
cj/cip-4167-proxy-connection-pooler-compatibility-test-matrix-pgbouncerfrom
claude/project-thread-y0sclv

Conversation

@tobyhede

@tobyhede tobyhede commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Requested by Toby · project thread

Before: with mapping disabled, nothing tested that a connection keeps working after a failed simple query in autocommit (Idle) or inside a failed transaction.

After: two regression tests in settings_rollback_tests.rs cover the mapping-disabled recovery paths.

This PR started as the fix for the mapping-disabled simple-query hang. The base branch now fixes it itself in 189296c ("end discard state at ReadyForQuery"), including the pipelined Sync case from review. The merge takes that implementation unchanged, so this PR adds tests only.

How:

  • failed_simple_query_with_mapping_disabled_does_not_suppress_next_query fails SELECT 1 / 0 with ReadyForQuery Idle and then FailedTransaction, and asserts the next Query is forwarded and the connection is not discarding.
  • failed_pipelined_execute_with_mapping_disabled_does_not_suppress_next_query sends Parse/Bind/Execute/Sync before any backend reply, fails the Execute, and asserts the same. It overlaps with the base's mapping_disabled_extended_error_after_sync_does_not_suppress_pipelined_query, but goes through the full frontend and backend middleware pair.
  • fail_query now takes the ReadyForQuery status.

Both tests failed before the fix. All 315 proxy unit tests pass against the base's implementation.

Acknowledgment

By submitting this pull request, I confirm that CipherStash can use, modify, copy, and redistribute this contribution, under the terms of CipherStash's choice.

https://claude.ai/code/session_01UeSgBAqVkHyKtZh6vWgtLR

@tobyhede tobyhede self-assigned this Oct 14, 2026
@tobyhede
tobyhede marked this pull request as ready for review October 14, 2026 22:35
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-14T22:38:02.920458Z 1146cd4 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1146cd4fd0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/cipherstash-proxy/src/postgresql/middleware/frontend.rs Outdated
@tobyhede
tobyhede force-pushed the cj/cip-4167-proxy-connection-pooler-compatibility-test-matrix-pgbouncer branch from 9c14803 to 010e7f3 Compare October 14, 2026 22:57
@tobyhede
tobyhede force-pushed the claude/project-thread-y0sclv branch from 1146cd4 to e0173cd Compare October 14, 2026 22:57

@freshtonic freshtonic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix for simple queries is correct. It does the same thing as the mapped error arm, and the regression test fails without it. But the mapping-disabled arm has the same defect for Sync, so extended-protocol clients can still hang.

Blocking: Sync in the mapping-disabled arm does not record a boundary.

In frontend.rs, the if self.context.mapping_disabled() arm sends FrontendMessage::Sync to _ => {}. The mapped arm calls self.context.sync(operation)? (lifecycle.rs:344), which records the schema boundary and the settings boundary. The passthrough arm records neither.

Scenario (development.disable_mapping = true):

  1. A driver sends Parse/Bind/Execute/Sync in one packet. The Execute fails (for example, a unique violation).
  2. The proxy reads the Sync before the ErrorResponse comes back. There is no boundary at or after the failed operation.
  3. backend.rs calls report_outcome → postgres_failed → SettingsProtocol::failed, which sets discarding_until_sync = true.
  4. ready() does not clear that flag. Only sync() clears it.
  5. The next client message is not a Sync, so the gate at the top of intercept suppresses it. A simple Query hangs. A new P/B/E/S batch forwards only its Sync, so the client gets ReadyForQuery without ParseComplete.

Most drivers use the extended protocol, so this case is more common than the simple-query case.

Suggested change: in the same arm, add

FrontendMessage::Sync => self.context.sync(operation)?,

For Query, call self.context.sync(operation)? too, not only settings_protocol_boundary, so that the schema boundary is also recorded, as in the mapped error arm. Add a test with the real config value: mapping disabled, P/B/E/S, an ErrorResponse on E, then the next Query is forwarded.

Suggestion: record the boundary in one place.

There are now three sites that must remember to record a boundary: the passthrough Query, the mapped Query error arm, and simple_query. The defect class comes from this. One check near the top of intercept, for Query and Sync in all mapping modes, would make it impossible to forget the boundary in a new arm.

Minor: the test.

  • fail_query sends ReadyForQuery(FailedTransaction). A failed autocommit SELECT 1 / 0 returns Idle. Thus the test shows the explicit-transaction case only. Please add the Idle case too, or make the status a parameter.
  • Also assert !client.frontend.context.discarding_extended_messages().unwrap() (or a similar check), so that the test shows why the query is forwarded.

I agree with your reply about the CHANGELOG. This bug was never released.

tobyhede commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor Author

@freshtonic thanks, the Sync case was real. Update: the base branch now fixes the hang itself in 189296c, and the merge in 5ecc8e3 takes that implementation unchanged. This PR now adds regression tests only.

  • Sync boundary (blocking): fixed on the base by unmapped_sync (frontend.rs:143), which records the settings boundary for a mapping-disabled Sync. SettingsProtocol::ready now also clears discarding_until_sync. The new test failed_pipelined_execute_with_mapping_disabled_does_not_suppress_next_query sends P/B/E/S before any backend reply, fails the Execute, and asserts the next Query is forwarded. It passes against the base's code.
  • Schema boundary on Query: unmapped_query records only the settings boundary (readiness_boundary), not context.sync. As far as I can tell that is sufficient: the mapping-disabled paths never queue schema executions, so there is nothing for a schema ReadyBoundary to bound. Please say if you see a path that does.
  • Test minors: fail_query takes the ReadyForQuery status, and the simple-query test now covers both Idle and FailedTransaction. Both tests assert !discarding_extended_messages() before checking the forward.
  • Single boundary site (suggestion): not in this PR. Boundaries are still recorded in unmapped_query, unmapped_sync and register_query. Happy to ticket the consolidation if you want it.

All 315 proxy unit tests and CI pass on 5ecc8e3.


Generated by Claude Code

@tobyhede
tobyhede requested a review from freshtonic October 15, 2026 00:27
@tobyhede tobyhede changed the title fix(proxy): stop simple-query hang after errors with mapping disabled test(proxy): cover mapping-disabled recovery after failed queries Oct 15, 2026

@cipherstash-bot cipherstash-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommendation: 🟡 merge after changes

One change before merging: make the simple-query test detect a missing unmapped_query boundary. Today it passes when that boundary is removed, and no other unit test detects that regression. The pipelined-test reorder and the query_is_forwarded output change are optional.

Both new tests send ReadyForQuery before they check the next Query. SettingsProtocol::ready always clears the discard state, so the tests check ready and not the boundaries that the base branch records in unmapped_query and unmapped_sync. I confirmed this with mutation runs in an isolated copy of HEAD. I also ran the suggested fixes: each one fails on the mutated code and passes on HEAD. I made no changes to this checkout.

How this review was made
Agent Model Review type Result
claude claude-opus-5-5 test-gap 2 found, 2 posted
claude claude-opus-5-5 infracode 0 found, 0 posted
claude claude-opus-5-5 rust 3 found, 2 posted
codex gpt-5.6-terra test-gap 0 found, 0 posted
codex gpt-5.6-terra infracode 0 found, 0 posted
codex gpt-5.6-terra rust 0 found, 0 posted

Synthesis: claude-opus-5-5 merged the findings, removed duplicates and dropped findings it could not confirm in the code. 0 posted finding(s) were raised by two or more models.

Plain language: claude-opus-5-5 read every comment as a new reader would. 2 comment(s) had a problem that stopped the reader acting; it rewrote 2.

Stack: 1 earlier and 0 later pull request(s) (#467, *️⃣ #469). *️⃣ marks this pull request.

Context loaded: the description, 0 linked issue(s) and 6 discussion entries.

Comment thread packages/cipherstash-proxy/src/postgresql/middleware/settings_rollback_tests.rs Outdated
Comment thread packages/cipherstash-proxy/src/postgresql/middleware/settings_rollback_tests.rs Outdated
@tobyhede
tobyhede force-pushed the cj/cip-4167-proxy-connection-pooler-compatibility-test-matrix-pgbouncer branch from 1ecf3a5 to f83b959 Compare October 15, 2026 04:37
Check that a failed simple query (autocommit and failed transaction) and
a failed pipelined Execute leave the next Query forwarded with mapping
disabled. Each test asserts the discard state before ReadyForQuery, so it
fails if unmapped_query or unmapped_sync stops recording its boundary.

Claude-Session: https://claude.ai/code/session_01UeSgBAqVkHyKtZh6vWgtLR
@tobyhede
tobyhede force-pushed the claude/project-thread-y0sclv branch from 532bd5b to b04a123 Compare October 15, 2026 04:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants