[8.0 Upgrade Assistant] Reindex failed when the index was ever frozen

[hubbleview]

Kibana version: 7.2.0

Elasticsearch version: 7.2.0

Describe the bug:
When Elasticsearch and Kibana at 6.x, some indices were ever frozen/unfrozen, so the index settings is added following private settings:

    "settings" : {
      "index" : {
        "search" : {
          "throttled" : "false"
        },
        "frozen" : "false",

After upgrading Elasticsearch and Kibana from 6.x to 7.2, running 8.0 Upgrade Assistant, it prompts to convert the index created before 7.0 should be reindexed. However, the step creating new index failed with following error:

Creating new index
There was an error

[index_creation_exception] failed to create index [reindexed-v7-kibana_sample_data_logs] :: {"path":"/reindexed-v7-kibana_sample_data_logs","query":{},"body":"{\"settings\":{\"index.auto_expand_replicas\":\"0-1\",\"index.frozen\":\"false\",\"index.number_of_replicas\":\"0\",\"index.number_of_shards\":\"1\",\"index.search.throttled\":\"false\"},\"mappings\":{...}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"index_creation_exception\",\"reason\":\"failed to create index [reindexed-v7-kibana_sample_data_logs]\"}],\"type\":\"validation_exception\",\"reason\":\"Validation Failed: 1: private index setting [index.frozen] can not be set explicitly;2: private index setting [index.search.throttled] can not be set explicitly;\"},\"status\":400}"}

Looks like, when creating index, the index private settings are also used, which is not allowed.

Steps to reproduce:

1. Freeze the Kibana sample index in 6.x (I was using 6.8.1). Observing the index settings was added following:

    "settings" : {
      "index" : {
        "search" : {
          "throttled" : "false"
        },
        "frozen" : "false",

After that, even unfreeze, the settings are still there.
2. Upgrade ES/Kibana from 6.x to 7.2.0.
3. Run Kibana > 8.0 Upgrade Assistant > Indices > click Reindex for Index created before 7.0, then it would fail with errors.

Creating new index
There was an error

[index_creation_exception] failed to create index [reindexed-v7-kibana_sample_data_logs] :: {"path":"/reindexed-v7-kibana_sample_data_logs","query":{},"body":"{\"settings\":{\"index.auto_expand_replicas\":\"0-1\",\"index.frozen\":\"false\",\"index.number_of_replicas\":\"0\",\"index.number_of_shards\":\"1\",\"index.search.throttled\":\"false\"},\"mappings\":{...}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"index_creation_exception\",\"reason\":\"failed to create index [reindexed-v7-kibana_sample_data_logs]\"}],\"type\":\"validation_exception\",\"reason\":\"Validation Failed: 1: private index setting [index.frozen] can not be set explicitly;2: private index setting [index.search.throttled] can not be set explicitly;\"},\"status\":400}"}

Expected behavior:
Private settings of index should be excluded when creating new index with format 7.0.

[dakrone]

I opened an issue for this on the ES side to be able to determine whether a setting is private: elastic/elasticsearch#44440

[joshFive]

@dakrone do you know if there is manual way to get the customer past this point? to be able to upgrade his indices?

[dakrone]

@joshFive they can manually re-index, copying only the non-private settings (in this case, the index.frozen and index.search.throttled settings should be elided). So manually perform the actions the upgrade assistant is doing, but without these two settings.

[elasticmachine]

Pinging @elastic/es-ui

[alisonelizabeth]

This should be addressed via #91193. The Upgrade Assistant was updated to handle these private ES settings.

[alisonelizabeth]

I wanted to add that while the underlying code has been fixed, starting in 7.12, we have decided to disable any Upgrade Assistant actions until the last minor before the 8.0 release. This was implemented via #92834.