Use smgrexists() instead of access() to enforce uniqueness of generated relfilenumber #439
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
knizhnik
force-pushed
the
get_relfilenum_fix_v16
branch
from
cf28299 to
9f6daf0
Compare
added 2 commits
July 3, 2024 17:12
knizhnik
force-pushed
the
get_relfilenum_fix_v16
branch
from
9f6daf0 to
b39f316
Compare
poojanilangekar
pushed a commit
to umd-dslam/sunstorm-postgres
that referenced
this pull request
Aug 26, 2024
…epers (neondatabase#439). It is intended to solve the following problems: a) Chicken-or-the-egg one: compute postgres needs data directory with non-rel files that are downloaded from pageserver by calling basebackup@LSN. This LSN is not arbitrary, it must include all previously committed transactions and defined through consensus voting, which happens... in walproposer, a part of compute node. b) Just warranting such LSN is not enough, we must also actually commit it and make sure there is a safekeeper who knows this LSN is committed so WAL before it can be streamed to pageserver -- otherwise basebackup will hang waiting for WAL. Advancing commit_lsn without playing consensus game is impossible, so speculative 'let's just poll safekeepers, learn start LSN of future epoch and run basebackup' won't work. Currently --sync-safekeepers is considered completed when 1) at least majority of safekeepers and 2) *all* safekeepers with live connection to walproposer switch to new epoch and advance commit_lsn allowing basebackup to proceed. 2) limits availablity, but that's because currently we don't have a mechanism defining which safekeeper should stream WAL into pageserver.
lubennikovaav
pushed a commit
that referenced
this pull request
May 13, 2025
…epers (#439). It is intended to solve the following problems: a) Chicken-or-the-egg one: compute postgres needs data directory with non-rel files that are downloaded from pageserver by calling basebackup@LSN. This LSN is not arbitrary, it must include all previously committed transactions and defined through consensus voting, which happens... in walproposer, a part of compute node. b) Just warranting such LSN is not enough, we must also actually commit it and make sure there is a safekeeper who knows this LSN is committed so WAL before it can be streamed to pageserver -- otherwise basebackup will hang waiting for WAL. Advancing commit_lsn without playing consensus game is impossible, so speculative 'let's just poll safekeepers, learn start LSN of future epoch and run basebackup' won't work. Currently --sync-safekeepers is considered completed when 1) at least majority of safekeepers and 2) *all* safekeepers with live connection to walproposer switch to new epoch and advance commit_lsn allowing basebackup to proceed. 2) limits availablity, but that's because currently we don't have a mechanism defining which safekeeper should stream WAL into pageserver.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Postgres is using
access()function inGetNewRelFileNumberto check if assigned relfilenumber is not used for any other relation. This check will not work in Neon, because we do not have all files in local storage.Use smgrexists() instead which will check at page server if such relfilenode is used.