Skip to content

fix: keep retry tasks rescheduled during execution - #1151

Merged
alexluong merged 2 commits into
mainfrom
fix/scheduler-receipt-delete
Oct 21, 2026
Merged

alexluong merged 2 commits into
mainfrom
fix/scheduler-receipt-delete

Conversation

@alexluong

@alexluong alexluong commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes a race condition in the retry scheduler between deleting a finished task and rescheduling it under the same ID.

The race

A retry task's ID is fixed per event and destination (RetryID), so scheduling the next retry replaces the queued task. When the delivery fails before the retry scheduler deletes the task it just ran, the delete removes the next retry:

retry scheduler                            delivery handler
(scheduler/scheduler.go)                   (deliverymq/messagehandler.go)
------------------------                   ------------------------------
receive task RetryID
exec: deliverymq.Publish(task)  -------->  Handle(task)
                                           delivery fails
                                           retryScheduler.Schedule(task,
                                             WithTaskID(RetryID))
                                           -> replaces the queued task
DeleteMessage(RetryID)
-> deletes the task just scheduled

Fix

Each receive gets a receipt. Delete and backoff act only while the receipt is current; rescheduling the task clears it.

receive task       -> receipt 41
schedule retry 2   -> receipt cleared
delete(ID, 41)     -> mismatch, retry 2 kept
  1. rsmq: ReceiveMessage returns QueueMessage.Receipt; SendMessage with a custom ID clears it.
  2. rsmq: DeleteMessageWithReceipt and ChangeMessageVisibilityWithReceipt, one Lua script each.
  3. scheduler: uses them after exec, in the exec-error backoff and when moving a task to the dead-letter queue.

🤖 Generated with Claude Code

@alexluong
alexluong force-pushed the fix/scheduler-receipt-delete branch from cd7fce8 to 0c87e9b Compare October 21, 2026 15:31
@alexluong
alexluong merged commit 26f2b26 into main Oct 21, 2026
1 check passed
@alexluong
alexluong deleted the fix/scheduler-receipt-delete branch October 21, 2026 15:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants