Closed
Description
Running the 7.x -> 8.x -> 9.x legacy upgrade test results in failure to upgrade to 9.1.0-SNAPSHOT.
make smoketest/run-version TEST_DIR=./testing/smoke/legacy-standalone-major-managed SMOKETEST_VERSION=latest
The error logs in ECH:
[instance-0000000000] fatal exception while booting Elasticsearch java.lang.IllegalStateException: The index [.ds-.logs-deprecation.elasticsearch-default-2025.03.13-000001/mZkPod7hSCWCrQg8ktu3Bg] created in version [7.17.25-8.0.0] with current compatibility version [7.17.25-8.0.0] must be marked as read-only using the setting [index.blocks.write] set to [true] before upgrading to 9.1.0. at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.isReadOnlySupportedVersion(IndexMetadataVerifier.java:180) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?] at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkSupportedVersion(IndexMetadataVerifier.java:126) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?] at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:98) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?] at org.elasticsearch.gateway.GatewayMetaState.upgradeProjectMetadata(GatewayMetaState.java:318) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?] at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:299) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?] at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:286) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?] at org.elasticsearch.gateway.GatewayMetaState.createOnDiskPersistedState(GatewayMetaState.java:194) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?] at org.elasticsearch.gateway.GatewayMetaState.createPersistedState(GatewayMetaState.java:148) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?] at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:106) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?] at org.elasticsearch.node.Node.start(Node.java:315) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?] at org.elasticsearch.bootstrap.Elasticsearch.start(Elasticsearch.java:568) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?] at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:365) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:95) ~[elasticsearch-9.1.0-SNAPSHOT.jar:?]
There are critical deprecation warnings from GET /_migration/deprecations:
{
"cluster_settings": [],
"node_settings": [],
"index_settings": {
"apm-7.17.29-transaction-000001": [
{
"level": "critical",
"message": "Old index with a compatibility version < 8.0",
"url": "https://www.elastic.co/guide/en/elastic-stack/9.0/upgrading-elastic-stack.html",
"details": "This index has version: 7.17.29-8.0.0",
"resolve_during_rolling_upgrade": false,
"_meta": {
"reindex_required": true
}
}
],
".kibana-event-log-7.17.29-snapshot-000001": [
{
"level": "critical",
"message": "Old index with a compatibility version < 8.0",
"url": "https://www.elastic.co/guide/en/elastic-stack/9.0/upgrading-elastic-stack.html",
"details": "This index has version: 7.17.29-8.0.0",
"resolve_during_rolling_upgrade": false,
"_meta": {
"reindex_required": true
}
}
],
"apm-7.17.29-metric-000001": [
{
"level": "critical",
"message": "Old index with a compatibility version < 8.0",
"url": "https://www.elastic.co/guide/en/elastic-stack/9.0/upgrading-elastic-stack.html",
"details": "This index has version: 7.17.29-8.0.0",
"resolve_during_rolling_upgrade": false,
"_meta": {
"reindex_required": true
}
}
],
"apm-7.17.29-span-000001": [
{
"level": "critical",
"message": "Old index with a compatibility version < 8.0",
"url": "https://www.elastic.co/guide/en/elastic-stack/9.0/upgrading-elastic-stack.html",
"details": "This index has version: 7.17.29-8.0.0",
"resolve_during_rolling_upgrade": false,
"_meta": {
"reindex_required": true
}
}
],
"apm-7.17.29-profile-000001": [
{
"level": "critical",
"message": "Old index with a compatibility version < 8.0",
"url": "https://www.elastic.co/guide/en/elastic-stack/9.0/upgrading-elastic-stack.html",
"details": "This index has version: 7.17.29-8.0.0",
"resolve_during_rolling_upgrade": false,
"_meta": {
"reindex_required": true
}
}
],
"apm-7.17.29-onboarding-2025.03.14": [
{
"level": "critical",
"message": "Old index with a compatibility version < 8.0",
"url": "https://www.elastic.co/guide/en/elastic-stack/9.0/upgrading-elastic-stack.html",
"details": "This index has version: 7.17.29-8.0.0",
"resolve_during_rolling_upgrade": false,
"_meta": {
"reindex_required": true
}
}
],
"apm-7.17.29-error-000001": [
{
"level": "critical",
"message": "Old index with a compatibility version < 8.0",
"url": "https://www.elastic.co/guide/en/elastic-stack/9.0/upgrading-elastic-stack.html",
"details": "This index has version: 7.17.29-8.0.0",
"resolve_during_rolling_upgrade": false,
"_meta": {
"reindex_required": true
}
}
]
},
"data_streams": {
"ilm-history-5": [
{
"level": "critical",
"message": "Old data stream with a compatibility version < 8.0",
"url": "https://www.elastic.co/guide/en/elasticsearch/reference/current/migrating-8.0.html#breaking-changes-8.0",
"details": "This data stream has backing indices that were created before Elasticsearch 8.0.0",
"resolve_during_rolling_upgrade": false,
"_meta": {
"indices_requiring_upgrade": [
".ds-ilm-history-5-2025.03.14-000001"
],
"indices_requiring_upgrade_count": 1,
"total_backing_indices": 1,
"reindex_required": true
}
}
],
".logs-deprecation.elasticsearch-default": [
{
"level": "critical",
"message": "Old data stream with a compatibility version < 8.0",
"url": "https://www.elastic.co/guide/en/elasticsearch/reference/current/migrating-8.0.html#breaking-changes-8.0",
"details": "This data stream has backing indices that were created before Elasticsearch 8.0.0",
"resolve_during_rolling_upgrade": false,
"_meta": {
"indices_requiring_upgrade": [
".ds-.logs-deprecation.elasticsearch-default-2025.03.14-000001"
],
"indices_requiring_upgrade_count": 1,
"total_backing_indices": 1,
"reindex_required": true
}
}
],
".slm-history-5": [
{
"level": "critical",
"message": "Old data stream with a compatibility version < 8.0",
"url": "https://www.elastic.co/guide/en/elasticsearch/reference/current/migrating-8.0.html#breaking-changes-8.0",
"details": "This data stream has backing indices that were created before Elasticsearch 8.0.0",
"resolve_during_rolling_upgrade": false,
"_meta": {
"indices_requiring_upgrade": [
".ds-.slm-history-5-2025.03.14-000001"
],
"indices_requiring_upgrade_count": 1,
"total_backing_indices": 1,
"reindex_required": true
}
}
]
},
"ml_settings": [],
"templates": {},
"ilm_policies": {}
}
We will need to reindex the data streams and indices before upgrading, might be a job for functional tests, as it is quite troublesome to do in current smoke tests.