[BUG] 1.0.x have issues BWC/rolling upgrade to 2.x

[peterzhuamazon]

I have talked to @qreshi @ohltyler on this extensively.

When using opensearch 1.0.x artifacts doing bwc test, as odfe is deprecated right now, we see this:

Caused by: java.lang.IllegalStateException: Received message from unsupported version: [2.1.0] minimal compatible version is: [6.8.0]
»       at org.opensearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:168) ~[opensearch-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]

This exception getting thrown due to min compat version for OS 1.0.0 is ES 6.8.0 and it thinks 2.0.0 is ES 2.0.0 which is not on or after 6.8.0 as a number.

This issue was initially raised by @cliu123 here: #3001

It later get fixed by @nknize in this commit: 7f524c7

---

The issue is, when we release bundle 1.0.1 we did not update the opensearch-min code but rather using min 1.0.0 + 1.0.0.0/1.0.1.0 plugins to produce the bundle 1.0.1 release:
https://opensearch.org/versions/opensearch-1-0-1.html
https://opensearch.org/docs/1.0/opensearch/install/plugins/
https://opensearch.org/docs/1.0/dashboards/install/plugins/

This means, no 1.0.1 core standalone tarball, so @nknize fix did not get released until 1.1.0.
This also means the possible concern of no 1.0.x version available for users to rolling upgrade from 1.0.x to 2.x.

---

I welcome more discussion on this so we can get more information on this issue.
As of now, the bwc tests we are switching are using 1.1.0 instead of 1.0.0.

cc: @CEHENKLE @bbarani @saratvemulapalli @dblock

Thanks.

[saratvemulapalli]

Thanks @peterzhuamazon for the context.

Trying to understand how to learn from this, couple of questions:

• did we miss including 1.0.1 min in the bundle or was it intentional?
• Was this caught by automation ?
• Why did this surface now? (plugins should have already been testing 1.0.0 to 2.x prior to 2.2 release and opendistro archival).

That said, I agree moving forward with 1.1.0 is the probably the right call for plugins to test against.
But we should document a path forward for customers who are on OpenSearch 1.0 and call it out explicitly the upgrade path doesnt exist.

[peterzhuamazon]

Thanks @peterzhuamazon for the context.

Trying to understand how to learn from this, couple of questions:

* did we miss including `1.0.1` min in the bundle or was it intentional?

* Was this caught by automation ?

* Why did this surface now? (plugins should have already been testing 1.0.0 to 2.x prior to 2.2 release and opendistro archival).

That said, I agree moving forward with 1.1.0 is the probably the right call for plugins to test against. But we should document a path forward for customers who are on OpenSearch 1.0 and call it out explicitly the upgrade path doesnt exist.

Hi @saratvemulapalli

1. We did not miss it was purely a decision to only update plugins as a patch version release, therefore no core changes and we did not realize this was a problem back then due to BWC test was running on ODFE.
2. It was caught due to ODFE deprecated and plugin owners tried to use OpenSearch 1.0.0 for BWC and failed.
3. See 2.

Thanks.

[saratvemulapalli]

Thanks @peterzhuamazon
Ideally with/without ODFE, our plugins should have tested OpenSearch 1.0 to 2.x, probably a learning.

Anyway I am ok with plugins testing with 1.1.0, but we need to document it in our docs explaining why OpenSearch 1.0 is not directly upgradable to 2.x. Let me know if you guys have any other suggestions.

[dblock]

I think you got this right @saratvemulapalli. Care to open an issue in docs or just PR the caveat?

[saratvemulapalli]

Opened up an issue[1], if its not getting attention I'll PR it.

[1] opensearch-project/documentation-website#904