Provide environment information
n/a
Describe the bug
Renaming or consolidating scheduled tasks can permanently block deploys, even when the project ends up well under its schedule limit.
syncDeclarativeSchedules does its work in this order:
- loop over every declared scheduled task: quota-check it, then create or update its schedule
- after the loop: delete schedules that are no longer declared
So a deploy that removes some scheduled tasks and adds one still sees the removed schedules in the count when it quota-checks the new one. If that pushes it to the limit, the deploy throws in step 1 and step 2 never runs. The schedules that would free up the quota are only deleted after the quota is checked, so the next deploy fails identically. There is no way out from the dashboard or the SDK, because declarative schedules can't be deleted through either
Reproduction repo
n/a
To reproduce
On a plan with a limit of 10 schedules:
- Deploy 10 declarative scheduled tasks (
cron on the task). Succeeds, 10/10.
- Rename one of them: change its task id, keeping the
cron. The project still declares 10 scheduled tasks, so it is not over the limit.
- Deploy.
The deploy fails with:
You have created 10/10 schedules so you'll need to increase your limits or delete some schedules.
The 9 unchanged tasks match their existing schedules and skip the limit check. The renamed one is new, and it is checked while the old schedule it replaces is still registered so the count is 10 and it throws. The deletion of the now-undeclared old schedule only happens after the loop, which never completes. Every subsequent deploy fails identically.
Deleting tasks instead of renaming does the same thing: delete 6 of the 10 and add 1 new one, so the project declares 5, and the deploy still fails at 10/10.
Additional information
Suggested fix: reconcile deletions before the create loop, or check the quota against the post-sync set rather than the current row count. Wrapping the sync in a transaction would also stop a failed deploy leaving schedules behind.
Workaround
Split it into two deploys so the cleanup gets a chance to run:
- Remove the
cron from the new scheduled task (the one with no schedule registered yet) and deploy. Every remaining declared task already has a schedule, so none is re-checked against the limit, the loop completes, and the stale schedules are deleted, dropping the count.
- Add the
cron back and deploy again. There's now headroom, so it's created normally.
It has to be the new task whose cron you remove. Removing one that's already registered doesn't help, since it isn't deleted until the end of the deploy and still occupies quota while the new one is checked.
Related: #3209 same deadlock, reached via imperative schedules instead.
Provide environment information
n/a
Describe the bug
Renaming or consolidating scheduled tasks can permanently block deploys, even when the project ends up well under its schedule limit.
syncDeclarativeSchedulesdoes its work in this order:So a deploy that removes some scheduled tasks and adds one still sees the removed schedules in the count when it quota-checks the new one. If that pushes it to the limit, the deploy throws in step 1 and step 2 never runs. The schedules that would free up the quota are only deleted after the quota is checked, so the next deploy fails identically. There is no way out from the dashboard or the SDK, because declarative schedules can't be deleted through either
Reproduction repo
n/a
To reproduce
On a plan with a limit of 10 schedules:
cronon the task). Succeeds, 10/10.cron. The project still declares 10 scheduled tasks, so it is not over the limit.The deploy fails with:
The 9 unchanged tasks match their existing schedules and skip the limit check. The renamed one is new, and it is checked while the old schedule it replaces is still registered so the count is 10 and it throws. The deletion of the now-undeclared old schedule only happens after the loop, which never completes. Every subsequent deploy fails identically.
Deleting tasks instead of renaming does the same thing: delete 6 of the 10 and add 1 new one, so the project declares 5, and the deploy still fails at 10/10.
Additional information
Suggested fix: reconcile deletions before the create loop, or check the quota against the post-sync set rather than the current row count. Wrapping the sync in a transaction would also stop a failed deploy leaving schedules behind.
Workaround
Split it into two deploys so the cleanup gets a chance to run:
cronfrom the new scheduled task (the one with no schedule registered yet) and deploy. Every remaining declared task already has a schedule, so none is re-checked against the limit, the loop completes, and the stale schedules are deleted, dropping the count.cronback and deploy again. There's now headroom, so it's created normally.It has to be the new task whose
cronyou remove. Removing one that's already registered doesn't help, since it isn't deleted until the end of the deploy and still occupies quota while the new one is checked.Related: #3209 same deadlock, reached via imperative schedules instead.