-
Notifications
You must be signed in to change notification settings - Fork 291
Closed
Labels
DEV: backendP0 - criticalPriority: Release blocker or regressionPriority: Release blocker or regression
Description
Observed behavior
A chef script produced thousands of enqueued calculate_user_storage_tasks for the same user, which overwhelmed the task workers
Expected behavior
For most tasks, any attempts to enqueue the task should only enqueue it if there isn't already a queued or currently executing task for the same arguments
User-facing consequences
Other users actively editing channels experience long delays for operations like copying and publishing, where the UI clearly shows the progress of the operation
Errors and logs
Seems as though the fetch_or_enqueue behavior isn't working properly
INFO:root:Enqueuing task:id calculate_user_storage_task:3a505d01cb9e447f9360158b0dad61e1 for user:channel 2858:None | {'user_id': 2858}
INFO:root:Didn't fetch matching task calculate_user_storage_task for user 2858 | {'user_id': 2858}
Additional information
https://learningequality.slack.com/archives/C0LK8QS9J/p1668640815323059
celery/django-celery-results#334
Despite enabling extended results, the pending result does not have the 'extended' information
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
DEV: backendP0 - criticalPriority: Release blocker or regressionPriority: Release blocker or regression