[BUG] Cannot update cluster settings after upgrading to 3.0.0

[taaniel]

Describe the bug

After upgrading our clusters from 2.19.2 to version 3.0.0, we cannot do any PUT requests to _cluster/settings as it results with an error message:

{
  "error": {
    "root_cause": [
      {
        "type": "settings_exception",
        "reason": "unknown setting [archived.plugins.index_state_management.metadata_migration.status] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
        "suppressed": [
          {
            "type": "settings_exception",
            "reason": "unknown setting [archived.plugins.index_state_management.template_migration.control] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
          }
        ]
      }
    ],
    "type": "settings_exception",
    "reason": "unknown setting [archived.plugins.index_state_management.metadata_migration.status] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
    "suppressed": [
      {
        "type": "settings_exception",
        "reason": "unknown setting [archived.plugins.index_state_management.template_migration.control] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
      }
    ]
  },
  "status": 400
}

When using GET _cluster/settings, I can see these archived settings but I cannot remove these settings in any way:

    "archived": {
      "plugins": {
        "index_state_management": {
          "metadata_migration": {
            "status": "1"
          },
          "template_migration": {
            "control": "-1"
          }
        }
      }
    }

Trying to set them to null:

PUT _cluster/settings
{
   "persistent":{
      "archived.plugins.index_state_management.metadata_migration.status": null,
      "archived.plugins.index_state_management.template_migration.status": null
   }
}
{
  "error": {
    "root_cause": [
      {
        "type": "settings_exception",
        "reason": "unknown setting [archived.plugins.index_state_management.template_migration.control] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
      }
    ],
    "type": "settings_exception",
    "reason": "unknown setting [archived.plugins.index_state_management.template_migration.control] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
  },
  "status": 400
}

And now we are stuck as we cannot reinstall opensearch-index-management temporarily to get rid of them. How can we resolve this issue? Would it be possible to make the archived settings validation more lax or offer a way to get rid of these settings?

Related component

Plugins

To Reproduce

Not sure how to reproduce exactly - I just know that we had indexes with ISM enabled in 2.19.2, then we upgraded to 3.0.0 (also, migrated from indexes to data streams during the upgrade) and now we cannot change cluster settings due to this bug.

Expected behavior

Any cluster setting updates/changes are successful.

Additional Details

Plugins
[opensearch@opensearch-cluster-master-0 ~]$ opensearch-plugin list
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-custom-codecs
opensearch-flow-framework
opensearch-geospatial
opensearch-job-scheduler
opensearch-knn
opensearch-ltr
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-skills
opensearch-sql
opensearch-system-templates
prometheus-exporter
query-insights
repository-azure

Screenshots

Host/Environment (please complete the following information):

• OS: Azure AKS cluster with Ubuntu 22.04.5 LTS nodes
• Version: 3.0.0

Additional context
Can provide additional information if needed.

[andrross]

Thanks @taaniel. This looks similar to the known bug mentioned in #8714

The work-around for now is to delete all the settings with archived prefix.

[taaniel]

@andrross how can I delete them? I already tried this, as I also described. Can you give the appropriate request to delete these settings please?

[andrross]

@taaniel Apologies, I missed the detail that you attempted to delete them per the workaround.

It looks like your delete command had a small mistake. It should be "control" on the second setting instead of "status". Can you try this command?

PUT _cluster/settings
{
   "persistent":{
      "archived.plugins.index_state_management.metadata_migration.status": null,
      "archived.plugins.index_state_management.template_migration.control": null
   }
}

I'm curious about how exactly you got to this situation. Did you remove the index management plugin on the 3.0 version when you did the upgrade?

[andrross]

For the record, @taaniel you're right that the validation is broken with regard to archived settings and it should be fixed to not block updates.

[taaniel]

Hey, @andrross - I think that this was a typo from my side. The updated PUT request worked and removed all of the archived settings, thus allowing us to make changes to cluster settings again. Thank you very much! 🙂 As this resolves the problem described in the scope of this issue (general validation aside, but I assume that this is addressed in some separate ticket already), I will close this ticket. Thanks again!

[andrross]

@taaniel Just want to bump my previous question. Did you intentionally uninstall the index management plugin on the upgrade? I want to make sure there isn't some other issue here as well.

[taaniel]

@andrross sadly, I cannot reopen the issue again... But to answer your question - no. The only thing I did was update the helm chart version and then deployed it using Terraform. We are using the official chart. I tried to install the plugin manually but that resulted in an error.

PS: I updated from 2.19.1 (helm charts were 2.34.0 for OS and 2.28.0 for OSD if I remember correctly).

[andrross]

@taaniel Is it possible that something in your infrastructure is uninstalling the opensearch-index-management plugin? We have testing of all the distribution flavors that should fail if any plugin is not present. I've confirmed that the docker distributions have the plugin installed.

[taaniel]

I'm not sure to be honest. The only plugins that we have added via helm's plugins are repository-azure and prometheus-exporter-plugin-for-opensearch (no official version released so had to update and re-package it myself for it to work with 3.0.0) but those both installed successfully and there were no error messages present regarding plugins in the logs. What changed is that together with the upgrade we transitioned from indexes to data streams and all of the related ISM configurations were removed and new ones for data streams were added. Could that have caused this?

[andrross]

What changed is that together with the upgrade we transitioned from indexes to data streams and all of the related ISM configurations were removed and new ones for data streams were added. Could that have caused this?

I don't think so. Plugins have the ability to register settings, and the settings in question here are registered by the index-management plugin. The fact that they got renamed to "archived.*" means that the cluster started without that plugin installed, which caused the cluster to view these as unknown settings and move them to the archived prefix. The output of the opensearch-plugin list command also shows that index-management is not installed. So the question is really how you got into the state where that plugin is not installed. The only theories I have are:

1. The command opensearch-plugin remove opensearch-index-management was run
2. The directory plugins/opensearch-index-management was deleted after the software was installed
3. One of the distributions is not including opensearch-index-management

I've verified the docker and tarball distros do have the plugin installed as expected. Helm charts is community-supported, but I'm assuming they are using the docker distro.