Support rolling upgrades for OpenSearch plugins, and restart upgrades for OpenSearch Dashboards Plugins 1.0

[saratvemulapalli]

Coming from opensearch-project/OpenSearch#671. Plugins are upgraded to OpenSearch, while supporting backward compatibility. We need design and documentation for the following.

• Renaming Namespaces com.amazon.opendistroforelasticsearch to org.opensearch
• Renaming Classes, Methods, Variables
• Renaming Remaining Identifiers (e.g. Opendistro to OpenSearch).
• Renaming, and Backwards Compatibility for Rest APIs
• Renaming, and Backwards Compatibility for Settings
• Renaming, and Backwards Compatibility for Indices
• Run in a backwards compatible way on top of OpenSearch 1.0 that has joined an ES 7.10.x cluster
• Run in a backwards compatible way on top of OpenSearch 1.0 that has joined an ODFE 1.13.x cluster
• Rolling upgrades for the Opendistro version 1.13 of the plugin

[dblock]

Settings: #15 (example: opensearch-project/job-scheduler#20).

[dblock]

API endpoints discussion in #13

[cliu123]

Settings: #15 (example: opensearch-project/job-scheduler#20).

@dblock This looks great! Looking forward to a POC for migrating indices.

[hsiang9431-amzn]

Any idea on indices?

[dblock]

I opened #17 to track indices, @saratvemulapalli is looking into that.

[saratvemulapalli]

APIs: Take a look at example opensearch-project/anomaly-detection#35
Let me know if you have any concerns.

[cliu123]

@dblock @saratvemulapalli
I see that Renaming Namespaces has been checked. Have we already had a decision/POC for it?
How does renaming all class names and namespaces support backward compatibility, e.g Java serialization, Transport Actions? Especially during B/G deployment on the managed service where nodes are upgraded/migrated from opendistro cluster to opensearch cluster, how would opensearch cluster recognize the transport requests that have opendistro namespaces and classes serialized in the header and know how to deserialize them?

[dblock]

Renaming was focused on just the naming part. For the items you describe we already have some bugs open like opensearch-project/OpenSearch#669 that @shwetathareja is looking into. If there's a scenario that you know of that is not covered, let's at least open an issue in OpenSearch?

[shwetathareja]

@dblock @saratvemulapalli
I see that Renaming Namespaces has been checked. Have we already had a decision/POC for it?
How does renaming all class names and namespaces support backward compatibility, e.g Java serialization, Transport Actions? Especially during B/G deployment on the managed service where nodes are upgraded/migrated from opendistro cluster to opensearch cluster, how would opensearch cluster recognize the transport requests that have opendistro namespaces and classes serialized in the header and know how to deserialize them?

@cliu123 In general the Java Serialization and Transport action shouldn't fail during upgrade from Elasticsearch to OpenSearch as actual serialization and actions are not dependent on the class/ package names. As long the class attributes data types and their ordering in not changing while reading and write to stream, It should work as is. And as @dblock mentioned the version compatibility logic is already taken as part of opensearch-project/OpenSearch#775

[dblock]

Are we ready to close this issue? What remains?

[saratvemulapalli]

Are we ready to close this issue? What remains?

We should be able to close this as the follow up is: #39