Optimize PG reservation query for PG >=9.5 #215
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
What does this optimization do? It handles an issue where two DJ servers "workers" would try to pull the top `delayed_jobs` record at the same time. One would succeed and the second would block. "SKIP LOCKED", introduced in PostgreSQL 9.5, allows the second query to skip records that are already locked by another transaction. This results in significant improvement in performance when there are multiple worker servers all trying to get the next available job row at the same time. What about older PostgreSQL versions? ActiveRecord requires PG>=9.3, so we cannot rely on the built-in checks. But we can use the same mechanisms that AR provides to check the current version and enable an optimization in that case. Meaning newer versions of AR (at least 5.0+) will inspect the current PG version and make the optimization if they can. We're not going to worry about older ARs as everything that old is EOL'd anyhow. And if they're running a Rails that old, the PG is likely older too.
|
That should work for MySQL too: https://dev.mysql.com/blog-archive/mysql-8-0-1-using-skip-locked-and-nowait-to-handle-hot-rows/ |
stevecrozz
approved these changes
May 28, 2024
gektin
approved these changes
May 28, 2024
albus522
approved these changes
Aug 14, 2024
|
Thanks for pulling this in, @albus522. Do you have a plan for an eventual release? Given the number of changes being pulled in, are you thinking a new major version, or just a minor ( |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this optimization do?
It handles an issue where two DelayedJob servers (i.e., "workers") would try to pull the top
delayed_jobsrecord at the same time. One would succeed and the second would block.SKIP LOCKED, introduced in PostgreSQL 9.5, allows the second query to skip records that are already locked by another transaction. This results in significant improvement in performance when there are multiple worker servers all trying to get the next availabledelayed_jobsrow at the same time.What about older PostgreSQL versions?
ActiveRecordrequires Postgres >= 9.3, so we cannot rely on the built-in version checks. But we can use the same mechanisms thatActiveRecordprovides to check the current version and enable an optimization in that case. This means newer versions ofActiveRecord(at least 5.0+) will inspect the current Postgres version and make the optimization if they can. We're not going to worry about olderActiveRecordversions as everything that old is EOL'd anyhow. And if they're running a Rails that old, the Postgres is likely older too.I understand there's been a hesitance to introduce Postgres version-specific functionality in the past, but this optimization drops our time to dequeue a job from several seconds to milliseconds. We've been running with this patch for years in production and I'm hoping this is a more favorable time to upstream it.
If so, I can adjust the code to be more intention revealing - extracting some methods so we need fewer comments, and to make it easier to pull this out when Postgres 9.5+ is required in some future version of this gem. We might also memoize the result of the check so we only do it once per connection object.
Thank you for your consideration.