fix(webhooks): add scheduled task to unlock stale webhook request locks#3546
Open
cymulatereouven wants to merge 1 commit intopostalserver:mainfrom
Open
fix(webhooks): add scheduled task to unlock stale webhook request locks#3546cymulatereouven wants to merge 1 commit intopostalserver:mainfrom
cymulatereouven wants to merge 1 commit intopostalserver:mainfrom
Conversation
When a worker process crashes or is killed (common in Kubernetes with rolling updates, OOM kills, etc.), locks held on webhook_requests are never released. Unlike QueuedMessage which has TidyQueuedMessagesTask to clean up stale locks, WebhookRequest had no equivalent mechanism, causing webhook delivery to be permanently blocked. This adds: - A `with_stale_lock` scope on WebhookRequest (locks older than 1 hour) - A TidyWebhookRequestsTask scheduled task that runs hourly to unlock stale webhook requests so they can be retried Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
webhook_requestsrows are never released. UnlikeQueuedMessagewhich hasTidyQueuedMessagesTaskto clean up stale locks,WebhookRequesthad no equivalent cleanup mechanism — causing webhook delivery to be permanently blocked.TidyWebhookRequestsTaskscheduled task (runs hourly) that finds webhook requests locked for more than 1 hour and unlocks them so they can be retried.with_stale_lockscope onWebhookRequest(mirrors the existing pattern onQueuedMessage)Changes
app/models/webhook_request.rbwith_stale_lockscope (locks older than 1 hour)app/scheduled_tasks/tidy_webhook_requests_task.rbapp/lib/worker/process.rbTidyWebhookRequestsTaskin the worker TASKS arrayWhy unlock instead of destroy?
TidyQueuedMessagesTaskdestroys stale queued messages because they represent outbound email delivery attempts that are likely no longer relevant. Webhook requests, however, carry event notifications that downstream systems may still need — so we unlock them to allow retry rather than silently dropping them.Test plan
WebhookRequest.with_stale_lockreturns only records withlocked_atolder than 1 hourTidyWebhookRequestsTaskunlocks stale records (setslocked_byandlocked_atto nil)ProcessWebhookRequestsJobafter unlock🤖 Generated with Claude Code