Add blocking mode with auto-reconnection and some other features. #632
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm glad to introduce a lot of features that were made in the driver working on other projects.
Here is the short list:
blocking
mode option [Consumer]ttl
less than a secong [Queue]Some of the features were added to improve a perfomance of hanling messages using Consumer, and the others were added to improve durability.
There options are production ready (I migrated them from the v13, so there could be some bugs because of that).
These changes don't break anything and disabled by default (except the point 5 about new messages)
Now let me explain why here a lof ot options added.
Improving the Consumer
Increasing perfomance
In the default mode (non-blocking) when you don't have any messages in a queue the consumer tries to get a job, and if gets nothing it sleeps, but while it sleeps we can get a job in a queue and that job will wait, so we lose a lot of perfomance and the consumer consumes resources like CPU. To fix the problem
the blocking
mode was added.When you use
blocking mode
the worker just waits for some time (or forever) for new messages, and while it waits it consumes no resources and when a job comes the worker gets it ASAP and starts working on it.And the last one about perfomance: in the production we don't need any simple logs about successfuly processed jobs, but about errors. To fix it the
verbose-messages option
was added, you can use it to write messages ONLY if you start worker in the verbose mode (-v
)Increasing durability
We don't live in the perfect world, shit happens sometime and we need to try to handle as much as we can.
To increase the Consumber durability we need to enable the
auto-reconnection option
.When something is wrong with the connection the consumer will automatically reconnect. Also we can use
initializing queue option
, because if the problem happens because of the RabbitMQ and the RabbitMQ restarted (you might don't have anything in it) you will get an error after reconnecting becasue the queus doesn't exist anymore.When you use the
blocking mode
the connection can stuck in some strange state (the consumer works, the RabbitMQ sees the consumer, but messages don't reach the consumer at all), it could happen when you use a complex architecture with vurtial machines and/or proxies. To check that the connection is stuck we must call some action and wait for the response, for this thealive-check option
was added. Just set N seconds and the consumer will check that all is OK with the connection every N seconds of working (it happens when NO messages comes in N seconds, so this doesn't affect the perfomance a lot).Also the
keepalive
option must be enabled for theblocking mode
, that's why it's added to the options.And the last thing for the consumer: I added more logs in it to simplify an investigation about what happened, because sometimes the consumer dies and you don't get anything about what was going on (maybe it was stopped because of some options, maybe it was killed by the system)
Improving the Queue
First of all I want to speak about the
option to cache declared
. The origial code doesn't have it and it caches all declared exchanges and queues in the memory, but it's completely WRONG (see the section aboutauto-reconnection option
above), because the original code doesn't handle any problems with the connection and don't clear them in the memory. Now it's fixed and the option was added just in case to simplify any investigations of some problems of caching (useful with the Laravel Octane).The
retry options
was added to automatically reconnect and try again if we get some problems while we send messages, it helps to avoid problems during short temporary network problems. (Maybe it's better to make another class for it, but now it's included in the Queue class)The
another expiration strategy
was added to impove latering messages. The original strategy creates a lot of queues (one queue per TTL), but the new one just uses the expiration option of messages, and the latering messages will become more effective with this option enabled.The
option to declare full route
was added to improve durability and stability. If you face some problems with connection and the RabbitMQ lost everything you might want to restore the full route of message, right? So, you can enable this option and before first sending the Queue class will declare exchange + queue + bind them, and you will get all things work.What else can be improved
Affected PRs
blocking mode
addedgetDelayQueueArguments
#627 must be fixed accorging to the new expiration strategy addedAffected issues
auto-retries
added in the coreauto-retries
added in the core