Plugin upgrade path from versions prior to OpenSearch 1.0.0

[saratvemulapalli]

As part of getting the plugins ready for backwards compatibility the last of the items on #12 are:

• Run in a backwards compatible way on top of OpenSearch 1.0 that has joined an ODFE 1.13.x cluster
• Run in a backwards compatible way on top of OpenSearch 1.0 that has joined an ES 7.10.x cluster

As 1 and 2 are very similar, to break down the problem we'll focus on 1.
We see 2 different upgrade paths:

1. In place upgrade: A node running ODFE/ES with a plugin is shutdown and OpenSearch with the plugin is started.
2. Rolling upgrade, Blue/Green: A cluster running ODFE/ES will be joined by OpenSearch node, replicate context and data.
3. Automate these tests via opensearch gradle plugin.

In order to perform and verify this backwards compatibility during upgrades plugins need to have a way to test these scenarios.
Ref: OpenSearch does this today with Bwc tests. Eg: https://github.com/opensearch-project/OpenSearch/blob/main/qa/mixed-cluster/build.gradle

[saratvemulapalli]

We did mixed cluster test Elasticsearch 7.x and OpenSearch 1.x without any plugins. The OpenSearch node doesn’t join the cluster due to an open issue: opensearch-project/OpenSearch#669

[saratvemulapalli]

Related issue after this is taken care of is: opensearch-project/opensearch-build#42

[saratvemulapalli]

I have tested manually using these steps:

Restart Upgrade:

• Install ODFE in single node cluster
• Make sure this ODFE cluster works (along with plugins)
• Index some data and a sample plugin feature
• Disable the shards.
• Bring down the ODFE cluster
• Upgrade to OpenSearch
• Make sure this OpenSearch cluster works (along with plugins)
• Verify the index data on ODFE is searchable in OpenSearch.

Rolling Upgrade:

• Install ODFE in 3 nodes (at the least)
• Make sure this ODFE cluster works (along with plugins)
  • Index random test data
  • Random search test data w/ assertThat validation
  • Add cluster setting from one of the plugin (AD)
• Disable shard allocation of replicas (set cluster.routing.allocation.enable to primaries)
• Ensure nothing is being indexed (stop any test indexing threads)
• Flush all indexes / data streams (e.g., POST /_flush)
• Bring down one node and Upgrade this node to OpenSearch
• Restart the node and wait for it to join the quorum
• Make sure the mixed cluster works (along with plugins). Cluster is GREEN
• Repeat the activity for other ODFE nodes
• Final testing ( after all the nodes are upgraded to OpenSearch ) verify that the search results, cluster settings from the plugin as expected.

[dblock]

I think we should close this, the feature is implemented across the board, @saratvemulapalli ?

[saratvemulapalli]

Agreed. I'll open a new one for automated tests.

[saratvemulapalli]

Reopening this as indexing during rolling upgrades is not supported and potentially will impact Alerting, AD etc

[skkosuri-amzn]

I have tested manually using these steps:

Restart Upgrade:

• Install ODFE in single node cluster
• Make sure this ODFE cluster works (along with plugins)
• Index some data and a sample plugin feature
• Disable the shards.
• Bring down the ODFE cluster
• Upgrade to OpenSearch
• Make sure this OpenSearch cluster works (along with plugins)
• Verify the index data on ODFE is searchable in OpenSearch.

Rolling Upgrade:

• Install ODFE in 3 nodes (at the least)

• Make sure this ODFE cluster works (along with plugins)

  • Index random test data
  • Random search test data w/ assertThat validation
  • Add cluster setting from one of the plugin (AD)

• Disable shard allocation of replicas (set cluster.routing.allocation.enable to primaries)

• Ensure nothing is being indexed (stop any test indexing threads)

• Flush all indexes / data streams (e.g., POST /_flush)

• Bring down one node and Upgrade this node to OpenSearch

• Restart the node and wait for it to join the quorum

• Make sure the mixed cluster works (along with plugins). Cluster is GREEN

• Repeat the activity for other ODFE nodes

• Final testing ( after all the nodes are upgraded to OpenSearch ) verify that the search results, cluster settings from the plugin as expected.

Did the above test is with security enabled?

[saratvemulapalli]

I have tested manually using these steps:
Restart Upgrade:

• Install ODFE in single node cluster
• Make sure this ODFE cluster works (along with plugins)
• Index some data and a sample plugin feature
• Disable the shards.
• Bring down the ODFE cluster
• Upgrade to OpenSearch
• Make sure this OpenSearch cluster works (along with plugins)
• Verify the index data on ODFE is searchable in OpenSearch.

Rolling Upgrade:

• Install ODFE in 3 nodes (at the least)

• Make sure this ODFE cluster works (along with plugins)

  • Index random test data
  • Random search test data w/ assertThat validation
  • Add cluster setting from one of the plugin (AD)

• Disable shard allocation of replicas (set cluster.routing.allocation.enable to primaries)

• Ensure nothing is being indexed (stop any test indexing threads)

• Flush all indexes / data streams (e.g., POST /_flush)

• Bring down one node and Upgrade this node to OpenSearch

• Restart the node and wait for it to join the quorum

• Make sure the mixed cluster works (along with plugins). Cluster is GREEN

• Repeat the activity for other ODFE nodes

• Final testing ( after all the nodes are upgraded to OpenSearch ) verify that the search results, cluster settings from the plugin as expected.

Did the above test is with security enabled?

No this was without security plugin. Infact all the plugins were purged and installed Job-Scheduler, AD.

[saratvemulapalli]

Closing this as the fix in security is merged.