Region snapshot replacement for read-only regions #7435
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reuse the region snapshot replacement machinery to replace read-only regions. This is done by storing a replacement type in the region snapshot replacement record such that either a region snapshot _or_ a read-only region can be the subject of this type of replacement. The procedure for both types is the same so all the code can be reused. A future commit will rename region snapshot replacement (and all references) to "read-only target replacement" to reflect that the machinery now applies to both region snapshots and read-only regions. This will be a mostly mechanical set of changes that can be reviewed separately with much less scrutiny. Right now manually requesting a region replacement with omdb is done through the region replacement manual request, not the region snapshot replacement manual request. This will change in that future commit to be part of a read-only target replacement request.
leftwo
reviewed
Feb 5, 2025
Contributor
leftwo
left a comment
leftwo
left a comment
There was a problem hiding this comment.
It looks pretty good, just some general questions.
I've been replacing read only regions in snapshots for about a week now, no issues found.
nexus/src/app/background/tasks/region_snapshot_replacement_garbage_collect.rs
Outdated
Show resolved
Hide resolved
addresses test flake due to slow systems being at the start of execution, then bailing out of the wait for condition, then querying for the state and seeing that it's in the middle of execution
I set every poll interval related to the background task and crucible polling to 500 milliseconds, and this is way too long for computers. Cut that by 10 to 50 milliseconds. Also, more `wait_for_all_replacements` calls to make sure replacement tasks are finished. This should address timeout test flakes in CI.
leftwo
reviewed
Feb 12, 2025
| request.volume_id, | ||
| ) | ||
| } else { | ||
| String::from("UNKNOWN") |
Contributor
There was a problem hiding this comment.
This would happen if someone changed how step_invalidated was assigned and did not add a place for it here?
leftwo
approved these changes
Feb 12, 2025
Contributor
leftwo
left a comment
leftwo
left a comment
There was a problem hiding this comment.
I don't see anything in the code, and I've done 26 test runs and not seen the test flake.
I also did more than 7 sled expunges and as long as I don't have running instances on the sled I expunged, things have been working correclty.
…ents also, run all replacement tasks _before_ checking if there are any replacements left, because the replacement tasks add replacements!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reuse the region snapshot replacement machinery to replace read-only regions. This is done by storing a replacement type in the region snapshot replacement record such that either a region snapshot or a read-only region can be the subject of this type of replacement. The procedure for both types is the same so all the code can be reused.
A future commit will rename region snapshot replacement (and all references) to "read-only target replacement" to reflect that the machinery now applies to both region snapshots and read-only regions. This will be a mostly mechanical set of changes that can be reviewed separately with much less scrutiny. Right now manually requesting a region replacement with omdb is done through the region replacement manual request, not the region snapshot replacement manual request. This will change in that future commit to be part of a read-only target replacement request.
Fixes #6172