Add support for worker pools that exclude specific queues#1148
Open
njakobsen wants to merge 10 commits into
Open
Add support for worker pools that exclude specific queues#1148njakobsen wants to merge 10 commits into
njakobsen wants to merge 10 commits into
Conversation
Co-authored-by: Nicholas Jakobsen <nicholas.jakobsen@gmail.com>
Co-authored-by: Nicholas Jakobsen <nicholas.jakobsen@gmail.com>
The addition of `exclude_specified_queues` allowed all specified queues to be treated as exclusions, implying that all jobs in those queues not be handled by the workers. However, this only allows application of this behaviour globally. What we want is the ability to create two or more pools, where some pools handle jobs from the given queues and other pools handle jobs that are not from the given queues. To implement this, we check if the `exclude_specified_queues` is not set and treat this as a request for automatic detection of exclusions. By adding a "!" to the beginning of a queue list, only the workers that handle that list of queues will behaves as if the `exclude_specified_queues` flag had been set.
|
Any chance this one could be reviewed and merged? |
|
@albus522 could you please review this PR? |
Align this repo with the Combinaut org standard. Config mirrors the other repos: weekly Monday schedule, patch auto-merge grouped into a single PR, manual approval for minor/major, dependency dashboard enabled.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Delayed Job already supports worker pools that draw from specific queues. This is useful to ensure those tasks are processed in a timely manner and do not starve. Coupling that with a pool that processes all jobs is a common practice. However, it is possible to create so many jobs for a specific that other jobs are delayed to an unacceptable degree. In this case we want to have a worker pool that only processes jobs that are not from that queue (or queues). There is currently no mechanism for this, hence this PR.
The changes allow worker pools to be named with a
!prefix which would indicate that we want this pool to process any jobs except those in the given queues. It is intended to be coupled with a change at the adapter level which would read theexclude_specified_queuesflag and return the appropriate jobs based on whether the list of queues is intended as an inclusion or exclusion filter.See this PR for changes made to DJ active_record adapter to support this new feature collectiveidea/delayed_job_active_record#194.