Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: removal of a store in a multi-store setup can break wal failover #131468

Open
nicktrav opened this issue Sep 26, 2024 · 1 comment
Open
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-storage Storage Team

Comments

@nicktrav
Copy link
Collaborator

nicktrav commented Sep 26, 2024

With WAL Failover, we currently assume the mapping from primary to secondary is fixed.

If we omit a store flag, we'll exit because an existing store can no longer find its secondary WAL.

Jira issue: CRDB-42551

Epic CRDB-41111

@nicktrav nicktrav added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-storage Storage Team labels Sep 26, 2024
@blathers-crl blathers-crl bot added the A-storage Relating to our storage engine (Pebble) on-disk storage. label Sep 26, 2024
@github-project-automation github-project-automation bot moved this to Incoming in Storage Sep 26, 2024
@nicktrav nicktrav moved this from Incoming to Backlog in Storage Oct 1, 2024
@jbowens
Copy link
Collaborator

jbowens commented Nov 4, 2024

Additionally, setting the flag --wal-failover=disabled on a multi-store node that has never had WAL failover enabled may fail, because WAL failover expects the directory containing the secondary WALs to exist.

jbowens added a commit to jbowens/cockroach that referenced this issue Nov 4, 2024
Fix the multi-store-remove roachtest. Setting --wal-failover=disabled on a
multi-store node that has never had WAL failover enabled is not yet supported.

Fixes cockroachdb#133939.
Informs cockroachdb#131468.
Epic: none
Release note: none
craig bot pushed a commit that referenced this issue Nov 7, 2024
134209: roachtest: move utilities to roachtestutil r=srosenberg a=andrewbaptist

Previously many of the test utilities were inside the tests package. This forced any tests to also be inside that package. By moving them into roachtestutil (and making public) it becomes easier to split tests out.

Epic: None

Release note: None

134213: roachtest: fix multi-store-remove roachtest r=itsbilal a=jbowens

Fix the multi-store-remove roachtest. Setting --wal-failover=disabled on a multi-store node that has never had WAL failover enabled is not yet supported.

Fixes #133939.
Informs #131468.
Epic: none
Release note: none

134335: roachtest: disable 23.1 -> 23.2 testing for follower reads r=miraradeva a=andrewbaptist

In #133092 the min version was set at 23.2 for the test, however it was only applied to the "locality=global" variation. This also applies to the single-region variation.

Fixes: #133433

Release note: None

134422: crosscluster: require 24.3 for LDR r=stevendanna a=stevendanna

We now require a number of options that are only available on 24.3.

Epic: none
Release note: None

134514: sqlsmith: disable stats forecasting in sqlsmith-based tests r=mgartner,rytaft a=michae2

In #125267 we added CREATE STATISTICS and ANALYZE statements to all of the sqlsmith-based randomized tests. But because CREATE STATISTICS and ANALYZE produce table statistics with nondeterministic `createdAt` times, this makes the forecasts based on these stats nondeterministic as well. I think the proper fix here would be to add an option to CREATE STATISTICS to set the `createdAt` time, but for now simply disable stats forecasting.

Epic: None

Release note: None

Co-authored-by: Andrew Baptist <baptist@cockroachlabs.com>
Co-authored-by: Jackson Owens <jackson@cockroachlabs.com>
Co-authored-by: Steven Danna <danna@cockroachlabs.com>
Co-authored-by: Michael Erickson <michae2@cockroachlabs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-storage Storage Team
Projects
Status: Backlog
Development

No branches or pull requests

2 participants