Support for bwc tests for plugins #1051

VachaShah · 2021-08-05T22:17:12Z

Signed-off-by: Vacha vachshah@amazon.com

Description

This PR adds supporting logic for implementing bwc tests in plugins. This are being used in PR anomaly-detection#158.

Issues Resolved

As part of #1002

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Vacha <vachshah@amazon.com>

opensearch-ci-bot · 2021-08-05T22:19:13Z

✅ DCO Check Passed e616879

opensearch-ci-bot · 2021-08-05T22:19:49Z

✅ Gradle Wrapper Validation success e616879

opensearch-ci-bot · 2021-08-05T22:28:17Z

✅ Gradle Precommit success e616879

saratvemulapalli

Dropped a few comments.
How do we test this new functionality? Can you add unit/integ tests ?

saratvemulapalli · 2021-08-06T16:37:48Z

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchCluster.java

@@ -404,6 +409,19 @@ public void nextNodeToNextVersion() {
        node.start();
    }

+    public void upgradeNodeAndPluginToNextVersion(List<Provider<RegularFile>> plugins) {


This is nice. It works for rolling upgrade/mixed cluster cases where we upgrade node by node.
For full cluster restarts/restart upgrades we need another function to handle it for plugins.
Take a look at:

public void goToNextVersion()

Minor nit: I wish we didn't duplicate it but I dont see another way too :/

I missed that! I will add that method as well since we are adding bwc support. For the duplication, yeah I had to create a new method since the plugins have to be added before starting the node and this would otherwise affect OpenSearch bwc tests :/

Added support for full cluster restarts/restart upgrades. I have also tried to reduce duplication where possible.

Awesome, we should be able to leverage restart cluster infra for snapshot upgrades. Essentially its taking snapshot one old version, store it in a repo, restart the cluster to new version and restore the snapshot.
Lets open an issue to take care fo snapshot upgrades.

Sure, opened an issue for tracking snapshot upgrade tests #1088.

saratvemulapalli · 2021-08-06T16:54:24Z

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchCluster.java

@@ -186,6 +186,11 @@ public void plugin(String pluginProjectPath) {
        nodes.all(each -> each.plugin(pluginProjectPath));
    }

+    @Override
+    public void upgradePlugin(List<Provider<RegularFile>> plugins) {
+        nodes.all(each -> each.upgradePlugin(plugins));


I assume the only valid case for this function is during restart upgrades.

Yeah I had to override it since it is added in interface TestClusterConfiguration.

VachaShah · 2021-08-06T20:53:09Z

Dropped a few comments.
How do we test this new functionality? Can you add unit/integ tests ?

Currently, there looks to be no tests specifically for this class, they are directly used in the bwc testing in OpenSearch. I will see if I can find a place for them or start adding tests for it.

Signed-off-by: Vacha <vachshah@amazon.com>

opensearch-ci-bot · 2021-08-10T21:12:17Z

✅ Gradle Wrapper Validation success 7216b78

opensearch-ci-bot · 2021-08-10T21:12:46Z

✅ DCO Check Passed 7216b78

opensearch-ci-bot · 2021-08-10T21:19:43Z

✅ Gradle Precommit success 7216b78

VachaShah · 2021-08-12T17:47:36Z

Dropped a few comments.
How do we test this new functionality? Can you add unit/integ tests ?

Currently, there looks to be no tests specifically for this class, they are directly used in the bwc testing in OpenSearch. I will see if I can find a place for them or start adding tests for it.

For the tests for OpenSearchCluster, it looks like it might take some time to setup, we could create an issue for it and revisit it next week, the methods in OpenSearchCluster are all used in bwc tests. WDYT?

Dropped a few comments.
How do we test this new functionality? Can you add unit/integ tests ?

Currently, there looks to be no tests specifically for this class, they are directly used in the bwc testing in OpenSearch. I will see if I can find a place for them or start adding tests for it.

I have created an issue #1086 for adding the tests.

saratvemulapalli

This is awesome, plugins would love it and great work!
Thanks a lot @VachaShah !!

saratvemulapalli · 2021-08-12T19:50:39Z

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchCluster.java

@@ -404,6 +409,19 @@ public void nextNodeToNextVersion() {
        node.start();
    }

+    public void upgradeNodeAndPluginToNextVersion(List<Provider<RegularFile>> plugins) {


Awesome, we should be able to leverage restart cluster infra for snapshot upgrades. Essentially its taking snapshot one old version, store it in a repo, restart the cluster to new version and restore the snapshot.
Lets open an issue to take care fo snapshot upgrades.

saratvemulapalli · 2021-08-12T19:56:49Z

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchCluster.java

+        nodes.all(OpenSearchNode::goToNextVersion);
+        upgradePlugin(plugins);
+        start();
+        writeUnicastHostsFiles();


Curious: Do we need to writeUnicastHostFiles?
Unicast is used for discovery of cluster nodes, if the cluster has the same ports already published and written to unicast_host.txt and upgrading the nodes the new version, wouldnt the framework start the new version of OpenSearch with the same ports?

saratvemulapalli · 2021-08-12T19:58:25Z

start gradle check

opensearch-ci-bot · 2021-08-12T20:25:37Z

✅ Gradle Check success 7216b78
Log 392

Reports 392

VachaShah · 2021-08-12T20:26:31Z

This is awesome, plugins would love it and great work!
Thanks a lot @VachaShah !!

Thank you @saratvemulapalli !

saratvemulapalli · 2021-08-13T22:41:50Z

@VachaShah we should backport this PR to 1.x.

VachaShah · 2021-08-13T22:48:54Z

@VachaShah we should backport this PR to 1.x.

Sure, I will raise a PR for backporting it.

* Support for bwc tests for plugins Signed-off-by: Vacha <vachshah@amazon.com> * Adding support for restart upgrades for plugins bwc Signed-off-by: Vacha <vachshah@amazon.com>

Support for bwc tests for plugins

e616879

Signed-off-by: Vacha <vachshah@amazon.com>

VachaShah requested a review from saratvemulapalli August 5, 2021 22:19

saratvemulapalli reviewed Aug 6, 2021

View reviewed changes

Adding support for restart upgrades for plugins bwc

7216b78

Signed-off-by: Vacha <vachshah@amazon.com>

VachaShah requested a review from saratvemulapalli August 12, 2021 16:36

VachaShah mentioned this pull request Aug 12, 2021

Introduce tests for OpenSearchCluster in bwc test framework #1086

Open

saratvemulapalli approved these changes Aug 12, 2021

View reviewed changes

saratvemulapalli added enhancement Enhancement or improvement to existing feature or request v1.1.0 Issues, PRs, related to the 1.1.0 release labels Aug 12, 2021

saratvemulapalli merged commit 072ccda into opensearch-project:main Aug 12, 2021

saratvemulapalli added the pending backport Identifies an issue or PR that still needs to be backported label Aug 12, 2021

VachaShah mentioned this pull request Aug 14, 2021

[Backport-1.x] Support for bwc tests for plugins (#1051) #1090

Merged

5 tasks

saratvemulapalli added the backwards-compatibility label Aug 15, 2021

saratvemulapalli removed the pending backport Identifies an issue or PR that still needs to be backported label Aug 15, 2021

VachaShah deleted the bwc_support_for_plugins branch August 17, 2021 16:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for bwc tests for plugins #1051

Support for bwc tests for plugins #1051

VachaShah commented Aug 5, 2021

opensearch-ci-bot commented Aug 5, 2021

opensearch-ci-bot commented Aug 5, 2021

opensearch-ci-bot commented Aug 5, 2021

saratvemulapalli left a comment

saratvemulapalli Aug 6, 2021

VachaShah Aug 6, 2021 •

edited

Loading

VachaShah Aug 10, 2021

saratvemulapalli Aug 12, 2021

VachaShah Aug 12, 2021

saratvemulapalli Aug 6, 2021

VachaShah Aug 6, 2021

VachaShah commented Aug 6, 2021

opensearch-ci-bot commented Aug 10, 2021

opensearch-ci-bot commented Aug 10, 2021

opensearch-ci-bot commented Aug 10, 2021

VachaShah commented Aug 12, 2021

saratvemulapalli left a comment

saratvemulapalli Aug 12, 2021

saratvemulapalli Aug 12, 2021

saratvemulapalli commented Aug 12, 2021

opensearch-ci-bot commented Aug 12, 2021

VachaShah commented Aug 12, 2021

saratvemulapalli commented Aug 13, 2021

VachaShah commented Aug 13, 2021

Support for bwc tests for plugins #1051

Support for bwc tests for plugins #1051

Conversation

VachaShah commented Aug 5, 2021

Description

Issues Resolved

Check List

opensearch-ci-bot commented Aug 5, 2021

opensearch-ci-bot commented Aug 5, 2021

opensearch-ci-bot commented Aug 5, 2021

saratvemulapalli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VachaShah Aug 6, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VachaShah commented Aug 6, 2021

opensearch-ci-bot commented Aug 10, 2021

opensearch-ci-bot commented Aug 10, 2021

opensearch-ci-bot commented Aug 10, 2021

VachaShah commented Aug 12, 2021

saratvemulapalli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saratvemulapalli commented Aug 12, 2021

opensearch-ci-bot commented Aug 12, 2021

VachaShah commented Aug 12, 2021

saratvemulapalli commented Aug 13, 2021

VachaShah commented Aug 13, 2021

VachaShah Aug 6, 2021 •

edited

Loading