fix(sdk): webhooks-django × codex-baseline — The agent hand-rolled HMAC signature verification (parsing a… - #736
Draft
workos-sdk-automation[bot] wants to merge 1 commit into
Conversation
…AC signature verification (parsing a… The agent hand-rolled HMAC signature verification (parsing a comma-separated 't=...,v1=...' header) instead of using the official WorkOS SDK/webhook-verification helper; if the grader's test vectors are generated via the real WorkOS SDK's signing scheme, a subtly different parsing/tolerance implementation would fail invalid_signature_rejected and webhook_delivery_e2e even though it passes self-authored tests. Evidence: https://evals.workos.tools/runs/01M3BD1BXZVYHG0F10DR65TQG7/attempts/01M3BD3FYTJ2V837E7G6RAGV6C?artifact=transcript&lines=20-26#L20 Finding: https://evals.workos.tools/runs/01M3BD1BXZVYHG0F10DR65TQG7?tab=diagnosis Workflow: https://github.com/workos/evals/actions/runs/36141608655
workos-sdk-automation
Bot
added
the
eval-diagnosis
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Added a "Webhook Signature Verification" section to
README.mddocumenting the SDK'sexisting
client.webhooks.verify_event()/verify_header()helpers, with a Django-styleexample. The helpers were undocumented outside docstrings, so agents were hand-rolling the
HMAC check and getting the details subtly wrong; the section spells out the exact scheme —
WorkOS-Signatureformattedt=<ms>, v1=<hex>with a,separator, HMAC-SHA256 over"{timestamp}.{raw body}", raw bytes rather than a re-serialized dict, and the 180s defaulttolerance. Docs-only change; no Python source touched. I could not run
uv run pytest— thecommand required an approval that was not granted in this session.
Finding
webhooks-django × codex-baseline · target
sdk· confidence medium · seen in 1 sweep · still failing in the newest sweepThe agent hand-rolled HMAC signature verification (parsing a comma-separated 't=...,v1=...' header) instead of using the official WorkOS SDK/webhook-verification helper; if the grader's test vectors are generated via the real WorkOS SDK's signing scheme, a subtly different parsing/tolerance implementation would fail invalid_signature_rejected and webhook_delivery_e2e even though it passes self-authored tests.
Suggested change: Provide or document a reference WorkOS webhook-verification utility (e.g., via workos-python SDK) that agents can call, so the signing/verification algorithm matches what the eval's test fixtures expect.
Evidence
Checks
uv run pytest→ passedThe workflow ran these after the agent finished;
checks=passedin the marker above is their result.Provenance
Opened from run https://evals.workos.tools/runs/01M3BD1BXZVYHG0F10DR65TQG7?tab=diagnosis by garen.torikian@workos.com, dispatched 2026-09-25T13:32:29.116Z · Actions run · branch
eval-diagnosis/27fc772371c2d215de9905f716065fa56bc8091e08da5cbbc10e9021775340cf· fingerprint27fc772371c2d215de9905f716065fa56bc8091e08da5cbbc10e9021775340cfThese findings are model-drafted hypotheses, not verdicts — verify the cited lines before merging. This PR was opened as a draft by the propose-fixes workflow in workos/evals; the agent that drafted the change edited only under the route's path prefix and did not push or open the PR itself.