Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add bwc tests to run with CI #181

Merged
merged 2 commits into from
Aug 17, 2021
Merged

Add bwc tests to run with CI #181

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
892d4cc
Adding support for BWC tests to run in CI
VachaShah Aug 16, 2021
8bf54ca
Adding bwc test commands in developer guide
VachaShah Aug 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/workflows/CI.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,24 @@ jobs:
cp ./build/distributions/*.zip ../src/test/resources/job-scheduler
echo "Copied ./build/distributions/*.zip to ../src/test/resources/job-scheduler ..."
ls ../src/test/resources/job-scheduler
echo "Creating ../src/test/resources/org/opensearch/ad/bwc/job-scheduler/1.1.0.0-SNAPSHOT ..."
mkdir -p ../src/test/resources/org/opensearch/ad/bwc/job-scheduler/1.1.0.0-SNAPSHOT
echo "Copying ./build/distributions/*.zip to ../src/test/resources/org/opensearch/ad/bwc/job-scheduler/1.1.0.0-SNAPSHOT ..."
ls ./build/distributions/
cp ./build/distributions/*.zip ../src/test/resources/org/opensearch/ad/bwc/job-scheduler/1.1.0.0-SNAPSHOT
echo "Copied ./build/distributions/*.zip to ../src/test/resources/org/opensearch/ad/bwc/job-scheduler/1.1.0.0-SNAPSHOT ..."
ls ../src/test/resources/org/opensearch/ad/bwc/job-scheduler/1.1.0.0-SNAPSHOT

- name: Assemble anomaly-detection
run: |
./gradlew assemble -Dopensearch.version=1.1.0-SNAPSHOT
echo "Creating ./src/test/resources/org/opensearch/ad/bwc/anomaly-detection/1.1.0.0-SNAPSHOT ..."
mkdir -p ./src/test/resources/org/opensearch/ad/bwc/anomaly-detection/1.1.0.0-SNAPSHOT
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this?
Gradle plugin automatically can pick up the plugin from the build directory.
Take a look at: https://github.com/opensearch-project/anomaly-detection/blob/main/build.gradle#L235

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is needed since the upgradeNodeAndPluginToNextVersion method is designed to take in a list of Provider<RegularFile> for installing plugins on the node during the upgrade. The plugin method is separate which has two implementations, one for taking in a file and one for taking in the path which is used for the node.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. That makes sense. I think it worth overloading that function to have another method to take care of this.
We should avoid as much as possible manually copying artifacts for tests.

We can goahead with this change, but can you also make a change in OpenSearch to add this functionality?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this method has the file list as input since the job-scheduler artifact would be a zip file and since AD has a dependency on that, we have to supply it at the same time so that the newer version plugins can be added when the node is upgraded. The new functionality can be useful for plugins that don't have any other plugins as dependency.

echo "Copying ./build/distributions/*.zip to ./src/test/resources/org/opensearch/ad/bwc/anomaly-detection/1.1.0.0-SNAPSHOT ..."
ls ./build/distributions/
cp ./build/distributions/*.zip ./src/test/resources/org/opensearch/ad/bwc/anomaly-detection/1.1.0.0-SNAPSHOT
echo "Copied ./build/distributions/*.zip to ./src/test/resources/org/opensearch/ad/bwc/anomaly-detection/1.1.0.0-SNAPSHOT ..."
ls ./src/test/resources/org/opensearch/ad/bwc/anomaly-detection/1.1.0.0-SNAPSHOT

- name: Build and Run Tests
run: |
Expand Down Expand Up @@ -124,6 +142,21 @@ jobs:
./gradlew integTest -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="docker-cluster"
fi

- name: Mixed cluster backwards compatibility tests
run: |
echo "Running mixed cluster backwards compatibility tests ..."
./gradlew adBwcCluster#mixedClusterTask -DmixedCluster=true -Dtests.security.manager=false

- name: Rolling upgrade backwards compatibility tests
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nice to separate out dedicated tests but is there a way to run all BWC tests with a single command?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is currently no task to run that, I can add one that runs all three of them together.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah definitely, I think its worth adding that task and will help us when we would want to publish nightly builds and run these tests in one go.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely in agreement, I will work on adding a single task which can be invoked to run the bwc tests after this PR.

run: |
echo "Running rolling upgrade backwards compatibility tests ..."
./gradlew adBwcCluster#rollingUpgradeClusterTask -DrollingUpgradeCluster=true -Dtests.security.manager=false

- name: Full cluster restart backwards compatibility tests
run: |
echo "Running full restart cluster backwards compatibility tests ..."
./gradlew adBwcCluster#fullRestartClusterTask -DfullRestartCluster=true -Dtests.security.manager=false

- name: Upload Coverage Report
uses: codecov/codecov-action@v1
with:
Expand Down
3 changes: 3 additions & 0 deletions DEVELOPER_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ Currently we just put RCF jar in lib as dependency. Plan to publish to Maven and
4. ` ./gradlew :integTest --tests="**.test execute foo"` runs a single integration test class or method
5. `./gradlew integTest -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="docker-cluster" -Dhttps=true -Duser=admin -Dpassword=admin` launches integration tests against a local cluster and run tests with security
6. `./gradlew spotlessApply` formats code. And/or import formatting rules in `.eclipseformat.xml` with IDE.
7. `./gradlew adBwcCluster#mixedClusterTask -DmixedCluster=true -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by upgrading one of the nodes with the current version of OpenSearch with anomaly-detection and job-scheduler creating a mixed cluster.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the three nodes run three different OpenSearch versions? Or 1 node runs old version, other 2 nodes run new version?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old version cluster will have 3 nodes of the old (same) version of OpenSearch and then one node will be upgraded to current version (for example 1.1) along with anomaly-detection and job-scheduler upgrade to 1.1 on that node. 2 other nodes will still be on the old version, this creates a mixed cluster.

8. `./gradlew adBwcCluster#rollingUpgradeClusterTask -DrollingUpgradeCluster=true -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by performing rolling upgrade of each node with the current version of OpenSearch with anomaly-detection and job-scheduler.
9. `./gradlew adBwcCluster#fullRestartClusterTask -DfullRestartCluster=true -Dtests.security.manager=false` launches a cluster with three nodes of bwc version of OpenSearch with anomaly-detection and job-scheduler and tests backwards compatibility by performing a full restart on the cluster upgrading all the nodes with the current version of OpenSearch with anomaly-detection and job-scheduler.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have some historical data of old version ready before full restart ? For AD, that will be create some detectors and run it to generate some AD result; then fully restart and test.

Same question for other 2 BWC tests: mixed cluster and rolling upgrade.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AD can definitely add more assertions to the tests according to the various scenarios.


When launching a cluster using one of the above commands logs are placed in `/build/cluster/run node0/opensearch-<version>/logs`. Though the logs are teed to the console, in practices it's best to check the actual log file.

Expand Down