-
Notifications
You must be signed in to change notification settings - Fork 387
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: Job data not passed to worker, all queue jobs removed and excessive amount of requests sent to redis server #2763
Comments
The number of requests are probably just normal. |
@manast So far, I don't have any explicit code to remove items from the queue, and the only queue removal setting is the default one, which is done after completion or failure by Bull. The job data being removed might have been resolved after I refactored my code from the NestJS process decorator to an explicit worker class, but the issue of queue data getting removed remains frequent. We can't add any items to the queue, as they are removed immediately. I thought the excessive number of requests could also be the cause, as it was sending many DEL commands to the Redis server when I used throttle/limiter. // Request sent to Redis when throttled // Queue settings
|
I am quite confident the issue with missing data is not a bug in BullMQ. |
Hi @samundrak, you are using quite old version, could you pls try to use the latest one and let us know |
Thank you for the response @manast I was thinking same for most of the the time, but I couldn't find place in the implementation that could be the reason for all queue data being removed. One thing I did recently was to remove the throttle in worker settings and since then it hasn't occurred but am still giving some time to confirm. @roggervalf I will update you on the status once I update the versions. |
Version
v3.10.3
Platform
NodeJS
What happened?
I have been facing this issue for a week now. A few changes I made included increasing the delay and adding a limiter. After these changes, I started encountering issues (Not sure if this is the reason), such as the job data not being passed to the worker—it was basically empty. To fix this temporarily, I removed a few queues, but another problem arose: all the queue data was suddenly being removed, which is causing serious issues in production.
I couldn't pinpoint the problem, so I switched from AWS ElastiCache to a self-hosted Redis to ensure the settings were configured correctly according to the documentation. It worked well for a few days, but then the issue of the queue being automatically removed started again. I did some debugging, checked the logs using RedisInsight, and discovered that an excessive number of requests were being sent to the Redis server.
Framework: NestJS ^9.0.0
Screen.Recording.2024-09-07.at.19.03.10.mov
How to reproduce.
Not able to reproduce it on dev environment or in local environment.
Relevant log output
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: