Skip to content

Bound authentication work during reconnect storms - #9

Draft
rayshobby wants to merge 5 commits into
fix/connection-lifecycle-hardeningfrom
fix/reconnect-storm-hardening
Draft

rayshobby wants to merge 5 commits into
fix/connection-lifecycle-hardeningfrom
fix/reconnect-storm-hardening

Conversation

@rayshobby

@rayshobby rayshobby commented Oct 30, 2026 •

Copy link
Copy Markdown

Reconnect storms can leave authentication work queued after controllers disconnect or hit their deadline. This change bounds running lookups and waiting jobs, removes abandoned waiters before SQL, and rejects duplicates before querying while preserving the existing owner and final ownership check.

It also caps incoming WebSocket messages at 4 MiB before authentication, enforces MySQL pool/deadline limits, and adds bounded aggregate telemetry independent of sampled logs. Existing firmware remains supported; heartbeat changes, caching, owner probing and pre-upgrade admission are deferred.

This PR is stacked on #7 (fix/connection-lifecycle-hardening) and contains four implementation commits plus a regression-test commit. The MySQL changes depend on the authentication context interface; coordinated reverts may be necessary.

Validation:

  • Hosted CI passed all 70 tests with zero skips and TypeScript checks on Node 22.18 and 24.19, including six real-driver MariaDB cases; the Docker build passed too: https://github.com/OpenThingsIO/OpenThings-Framework-Request-Proxy/actions/runs/40998855235
  • Isolated nginx/Node/MariaDB tests reached 4,000 devices with repeated reconnects, slow SQL and legacy/mixed retry policies. Authentication remained within 10 running and 500 waiting jobs. The 50/50 mixed retry run passed 8,000 device/endpoint content checks and demonstrated a longer recovery tail. Detailed evidence remains private.
  • Effective Docker log rotation was verified before debug staging. The verified image is now on debug for a 48-hour observation; production and Traefik were not restarted or updated. Soak completion and production promotion remain separate gates.

No production rollout is included. The missing hard socket-admission/output-buffer bounds remain explicit follow-up scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant