Prevent direct upgrade of indices from 6.8 to 8.0 #82689

grcevski · 2022-01-17T17:20:46Z

This PR adds a check that relies on 7.17 calculating the overall index age
metadata from the existing indices we have. Once we have the oldest index
age computed, we add a check to prevent upgrades to 8.0 with any indices
that are older than 7.0.

Closes #81326

This PR introduces a check to prevent upgrading to 8.0 without first upgrading to 7.last. Closes elastic#81865

…rom_7.nonlast' into enhancement/prevent_upgrades_from_7.nonlast

…ecurity/SecurityImplicitBehaviorBootstrapCheckTests.java Co-authored-by: David Turner <david.turner@elastic.co>

Upgrading to 8.0 with 6.8 indices causes data corruption, since we modify the indices directories in irreversible manner. This change introduces the following: - code to compute the oldest seen index version - code to check early enough for the incompatible indices and bail before any irreversible changes are made.

elasticmachine · 2022-01-17T17:20:48Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

elasticsearchmachine · 2022-01-17T17:21:10Z

Hi @grcevski, I've created a changelog YAML for you.

grcevski · 2022-01-17T17:23:38Z

Still a draft because I need to figure out how to test this, other than the manual testing I did.

DaveCTurner · 2022-01-18T10:08:43Z

server/src/main/java/org/elasticsearch/env/NodeEnvironment.java

+            () -> TimeValue.nsecToMSec(System.nanoTime())
+        );
+
+        PersistedClusterStateService.OnDiskState onDiskState = persistedClusterStateService.loadBestOnDiskState();


I think this is too early in the node's lifecycle for loading the cluster state (that's why you had to create your own NamedXContentRegistry). Instead we should record the oldest index version in NodeMetadata, which means adding an entry to the userData attached to the Lucene commit for the cluster state.

OK, great. I'll try that approach instead.

DaveCTurner

Looks good except for one tiny thing...

DaveCTurner · 2022-01-20T12:55:52Z

server/src/main/java/org/elasticsearch/env/NodeEnvironment.java

+        try {
+            checkForIndexCompatibility(logger, legacyNodeLock.getNodePaths());
+        } finally {
+            legacyNodeLock.close();


I think we should only release the legacyNodeLock on failure - on success we need to keep hold of it until the end of this method.

We could move the checkForIndexCompatibility call within the next try block right?

Ah good point, I was fixing a bug and you are right, I release it too soon.

Add oldest index version calculation and record it in NodeMetadata. This check will be used by future releases to prevent irreversible upgrades of incompatible indices, e.g. 6.x indices to version 8.

DaveCTurner

LGTM

elasticsearchmachine · 2022-01-20T16:05:51Z

💔 Backport failed

Status	Branch	Result
❌	8.0	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 82689

Backport of #82689

Relates to elastic#82689

Relates to #82689

The mutation function would rarely fail to mutate the input object. This commit fixes that. Relates elastic#82689

The mutation function would rarely fail to mutate the input object. This commit fixes that. Relates #82689

The mutation function would rarely fail to mutate the input object. This commit fixes that. Relates elastic#82689

The mutation function would rarely fail to mutate the input object. This commit fixes that. Relates #82689

…#83194) * Fix NodeMetadataTests#testEqualsHashcodeSerialization (#83170) The mutation function would rarely fail to mutate the input object. This commit fixes that. Relates #82689 * Old-skool switch

Nikola Grcevski and others added 15 commits January 6, 2022 11:01

Prevent upgrades to 8.0 from 7.non-last

c4b7415

This PR introduces a check to prevent upgrading to 8.0 without first upgrading to 7.last. Closes elastic#81865

Update docs/changelog/82321.yaml

938c28f

Fix tests and improve message.

543595e

Merge remote-tracking branch 'grcevski/enhancement/prevent_upgrades_f…

dd07b96

…rom_7.nonlast' into enhancement/prevent_upgrades_from_7.nonlast

Fix test message comparison.

36b917a

Revert wrong test fix.

f1ad482

Merge branch 'master' into enhancement/prevent_upgrades_from_7.nonlast

a0922d5

Update x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/s…

eca4b62

…ecurity/SecurityImplicitBehaviorBootstrapCheckTests.java Co-authored-by: David Turner <david.turner@elastic.co>

Fixes as per the review comments.

6f71d66

Merge branch 'master' into enhancement/prevent_upgrades_from_7.nonlast

ca13dca

Merge branch 'master' into enhancement/prevent_upgrades_from_7.nonlast

18b65f4

Fix tests

fda84e2

Spotless.

8710cef

Merge branch 'master' into enhancement/prevent_6.8_indices_upgrade

812d1b6

grcevski added >enhancement :Core/Infra/Core Core issues without another label v8.0.0 Team:Core/Infra Meta label for core/infra team auto-backport Automatically create backport pull requests when merged labels Jan 17, 2022

grcevski marked this pull request as draft January 17, 2022 17:20

elasticsearchmachine added the v8.1.0 label Jan 17, 2022

Update docs/changelog/82689.yaml

8427934

grcevski changed the title ~~Enhancement/prevent 6.8 indices upgrade~~ Prevent 6.8 indices upgrade to 8.0 Jan 17, 2022

grcevski added the WIP label Jan 17, 2022

Better way to compute the version.

dc1ce06

DaveCTurner reviewed Jan 18, 2022

View reviewed changes

albertzaharovits added v7.17.1 and removed v7.17.0 labels Jan 20, 2022

DaveCTurner reviewed Jan 20, 2022

View reviewed changes

Move code to release the lock at the right place.

543f1e6

DaveCTurner approved these changes Jan 20, 2022

View reviewed changes

grcevski merged commit be29e29 into elastic:master Jan 20, 2022

grcevski pushed a commit to grcevski/elasticsearch that referenced this pull request Jan 20, 2022

Backport of elastic#82689

afd3245

grcevski mentioned this pull request Jan 20, 2022

[8.0] Prevent 6.8 indices upgrade to 8.0 (#82689) #82876

Merged

grcevski added a commit that referenced this pull request Jan 20, 2022

[8.0] Prevent 6.8 indices upgrade to 8.0 (#82689) (#82876)

0db7ca3

Backport of #82689

grcevski pushed a commit to grcevski/elasticsearch that referenced this pull request Jan 20, 2022

Re-enable BWC

215ec5b

Relates to elastic#82689

grcevski mentioned this pull request Jan 20, 2022

Re-enable BWC #82888

Merged

grcevski added a commit that referenced this pull request Jan 20, 2022

Re-enable BWC (#82888)

3aa9716

Relates to #82689

albertzaharovits added v7.17.0 and removed v7.17.1 labels Jan 25, 2022

albertzaharovits changed the title ~~Prevent 6.8 indices upgrade to 8.0~~ Prevent direct upgrade of indices from 6.8 to 8.0 Jan 25, 2022

grcevski deleted the enhancement/prevent_6.8_indices_upgrade branch January 25, 2022 22:54

DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this pull request Jan 26, 2022

Fix NodeMetadataTests#testEqualsHashcodeSerialization

d0a33bb

The mutation function would rarely fail to mutate the input object. This commit fixes that. Relates elastic#82689

DaveCTurner mentioned this pull request Jan 26, 2022

Fix NodeMetadataTests#testEqualsHashcodeSerialization #83170

Merged

DaveCTurner added a commit that referenced this pull request Jan 27, 2022

Fix NodeMetadataTests#testEqualsHashcodeSerialization (#83170)

847dd43

The mutation function would rarely fail to mutate the input object. This commit fixes that. Relates #82689

elasticsearchmachine pushed a commit that referenced this pull request Jan 27, 2022

Fix NodeMetadataTests#testEqualsHashcodeSerialization (#83170) (#83193)

aa7a749

The mutation function would rarely fail to mutate the input object. This commit fixes that. Relates #82689

DaveCTurner mentioned this pull request May 17, 2022

Downgrading from 8.x to 7.x to fix 6.x indices is sometimes impossible #86870

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Prevent direct upgrade of indices from 6.8 to 8.0 #82689

Prevent direct upgrade of indices from 6.8 to 8.0 #82689

Uh oh!

grcevski commented Jan 17, 2022

Uh oh!

elasticmachine commented Jan 17, 2022

Uh oh!

elasticsearchmachine commented Jan 17, 2022

Uh oh!

grcevski commented Jan 17, 2022

Uh oh!

DaveCTurner Jan 18, 2022

Uh oh!

grcevski Jan 18, 2022

Uh oh!

DaveCTurner left a comment

Uh oh!

DaveCTurner Jan 20, 2022

Uh oh!

grcevski Jan 20, 2022

Uh oh!

DaveCTurner left a comment

Uh oh!

elasticsearchmachine commented Jan 20, 2022

Uh oh!

Uh oh!

Prevent direct upgrade of indices from 6.8 to 8.0 #82689

Prevent direct upgrade of indices from 6.8 to 8.0 #82689

Uh oh!

Conversation

grcevski commented Jan 17, 2022

Uh oh!

elasticmachine commented Jan 17, 2022

Uh oh!

elasticsearchmachine commented Jan 17, 2022

Uh oh!

grcevski commented Jan 17, 2022

Uh oh!

DaveCTurner Jan 18, 2022

Choose a reason for hiding this comment

Uh oh!

grcevski Jan 18, 2022

Choose a reason for hiding this comment

Uh oh!

DaveCTurner left a comment

Choose a reason for hiding this comment

Uh oh!

DaveCTurner Jan 20, 2022

Choose a reason for hiding this comment

Uh oh!

grcevski Jan 20, 2022

Choose a reason for hiding this comment

Uh oh!

DaveCTurner left a comment

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Jan 20, 2022

💔 Backport failed

Uh oh!

Uh oh!