Ensure k-NN can be a drop-in replacement for k-NN working in ODFE 1.13 and ES 7.10.x for v1.0.0

[jmazanec15]

Coming from opensearch-project/OpenSearch#671 and opensearch-project/opensearch-plugins#12.

• Renaming Namespaces (e.g. com.amazon.opendistroforelasticsearch to org.opensearch)
• Renaming Classes, Methods, Variables
• Renaming Remaining Identifiers (e.g. Opendistro to OpenSearch).
• Renaming, and Backwards Compatibility for Rest APIs
• Renaming, and Backwards Compatibility for Settings
• Renaming, and Backwards Compatibility for Indices
• Run in a backwards compatible way on top of OpenSearch 1.0 that has joined an ES 7.10.x cluster
• Run in a backwards compatible way on top of OpenSearch 1.0 that has joined an ODFE 1.13.x cluster
• Drop in replacement for the Opendistro version of the plugin

[dblock]

What's our status for this?

[VijayanB]

@dblock We will work on remaining in-complete steps in next week.

[jmazanec15]

Marking upgrade tests as complete. Will outline my test process in this comment:

Test script

I created a test script, main.py, to run different workloads during upgrade and monitor the cluster state. The import commands

// HOSTNAME is either localhost or the ip address (without the port)

// Setup the initial cluster
python3 main.py ${HOST_NAME} setup

// Run tests when there are odfe and opensearch nodes in same cluster
python3 main.py ${HOST_NAME} mixed

// Exclude ODFE nodes from shard allocation
python3 main.py ${HOST_NAME} exclude

// Run tests when upgrade is complete
python3 main.py ${HOST_NAME} post

Docker Commands

// create os-net so separate dockercompose files can use same network
docker network create os-net --driver=bridge

// Stop an odfe node
docker container stop odfe-node3

// Remove volumes
docker volume rm $(docker volume ls -q)

Rolling Upgrade

1. Bring up ODFE cluster
2. Run python setup to initialize cluster state
3. Bring up rolling OS cluster so nodes join
4. Run python exclude to exclude ODFE nodes from shard allocation
5. Run python mixed tests
6. Get IP address of opensearch node
7. Remove ODFE nodes from cluster
8. Run python post tests with IP address gotten from step 6

After each steps, I validated that stats, cluster health, etc. checked out

Restart Upgrade

1. Delete all existing volumes
2. Bring up ODFE cluster
3. Run python setup to initialize cluster state
4. Bring down ODFE cluster
5. Bring up restart OS cluster
6. Run python post tests

After each steps, I validated that stats, cluster health, etc. checked out

Other notes

I manually checked warmup API and settings update to make sure it works in each cluster state.

Artifacts

1. docker-compose.yml for odfe 1.13 3 node cluster
2. docker-compose.ymlfor opensearch rolling upgrade
3. docker-compose.yml for opensearch restart upgrade
4. main.py for running tests

[jmazanec15]

@dblock What is meant by "Drop in replacement for the Opendistro version of the plugin"?

[dblock]

@dblock What is meant by "Drop in replacement for the Opendistro version of the plugin"?

This was the original nomenclature. Meaning that you don't need to do any manual work as you perform a rolling or a restart upgrade with the k-NN plugin enabled before and after to see everything working as before. I would say that if k-NN is working in all the scenarios in https://opensearch.org/faq/#c3, you can close this.