add name and update file location for test manifest (opensearch-proje…

…ct#1163)

* add name and update file location for test manifest

Signed-off-by: Tianle Huang <tianleh@amazon.com>

* modify

Signed-off-by: Tianle Huang <tianleh@amazon.com>

* remove

Signed-off-by: Tianle Huang <tianleh@amazon.com>

* update docs

Signed-off-by: Tianle Huang <tianleh@amazon.com>

ONBOARDING.md

@@ -42,12 +42,12 @@ Add the new plugin to the [opensearch-plugins meta](https://github.com/opensearc
 
 ### Onboard to `test-workflow`
 
-1. Update the test configuration file, [test_manifest.yml](src/test_workflow/config/test_manifest.yml), for a particular release, to include your plugin. This test configuration defines full suite of tests - `integ`, `bwc`, that can be run on the plugin.
+1. Update the test configuration file, [test_manifest.yml](src/test_workflow/config/opensearch/test_manifest.yml), for a particular release, to include your plugin. This test configuration defines full suite of tests - `integ`, `bwc`, that can be run on the plugin.
 
-2. For integration testing, the `test-workflow` runs integration tests available in the plugin repository. You will need to add `integ-test` config for your plugin in test_manifest.yml, [example](src/test_workflow/config/test_manifest.yml#L3).
+2. For integration testing, the `test-workflow` runs integration tests available in the plugin repository. You will need to add `integ-test` config for your plugin in test_manifest.yml, [example](src/test_workflow/config/opensearch/test_manifest.yml#L3).
 
     1. It supports two test configs - `with-security` and `without-security`, which runs test with security plugin enabled and disabled respectively. Choose one or both depending on what your plugin integration tests support.
 
     2. If your plugin is dependent on `job-scheduler` zip, you can define that in `build-dependencies` in the config. Currently, the test workflow only supports `job-scheduler` as build dependency. Please create an issue if your plugin needs more support.
 
-3. For backward compatibility testing, the `test-workflow` runs backward compatibility tests available in the plugin repository, (see [reference]((https://github.com/opensearch-project/anomaly-detection/blob/d9a122d05282f7efc1e24c61d64f18dec0fd47af/build.gradle#L428))). Like integration test, it has a set of configurable options defined in test_manifest.yml, [example](src/test_workflow/config/test_manifest.yml#L8).
+3. For backward compatibility testing, the `test-workflow` runs backward compatibility tests available in the plugin repository, (see [reference]((https://github.com/opensearch-project/anomaly-detection/blob/d9a122d05282f7efc1e24c61d64f18dec0fd47af/build.gradle#L428))). Like integration test, it has a set of configurable options defined in test_manifest.yml, [example](src/test_workflow/config/opensearch/test_manifest.yml#L8).

src/manifests/test_manifest.py

@@ -13,6 +13,7 @@ class TestManifest(ComponentManifest):
 
     The format for schema version 1.0 is:
         schema-version: '1.0'
+        name: 'OpenSearch'
         components:
           - name: index-management
             working-directory: optional relative directory to run commands in
@@ -31,6 +32,7 @@ class TestManifest(ComponentManifest):
 
     SCHEMA = {
         "schema-version": {"required": True, "type": "string", "allowed": ["1.0"]},
+        "name": {"required": True, "type": "string", "allowed": ["OpenSearch", "OpenSearch Dashboards"]},
         "components": {
             "type": "list",
             "schema": {
@@ -60,10 +62,12 @@ class TestManifest(ComponentManifest):
 
     def __init__(self, data):
         super().__init__(data)
+        self.name = str(data["name"])
 
     def __to_dict__(self):
         return {
             "schema-version": "1.0",
+            "name": self.name,
             "components": self.components.__to_dict__()
         }
 

src/run_integ_test.py

@@ -24,7 +24,7 @@
 def main():
     args = TestArgs()
     console.configure(level=args.logging_level)
-    test_manifest_path = os.path.join(os.path.dirname(__file__), "test_workflow", "config", "test_manifest.yml")
+    test_manifest_path = os.path.join(os.path.dirname(__file__), "test_workflow", "config", "opensearch", "test_manifest.yml")
     test_manifest = TestManifest.from_path(test_manifest_path)
     bundle_manifest = BundleManifest.from_urlpath("/".join([args.path.rstrip("/"), "dist/manifest.yml"]))
     build_manifest = BuildManifest.from_urlpath("/".join([args.path.rstrip("/"), "builds/manifest.yml"]))

src/test_workflow/README.md

@@ -10,7 +10,7 @@ The overall CICD infrastructure is divided into two main workflows - `build` and
 
 Once a new bundle is ready, the `build-job` kicks off the [test-orchestrator-pipeline](../../jenkins_workflow/test/orchestrator/Jenkinsfile) with input parameters `(build_id, architecture, opensearch_version)` that uniquely identify the bundle. The test orchestrator-pipeline generate a unique `test_run_id`, that uniquely identifies the test execution and invokes all three test jobs - `integ-test, bwc-test, perf-test` in parallel.
 
-The [integ-test job](../../jenkins_workflow/test/testsuite/Jenkinsfile) starts by pulling the manifest files and installing the required dependencies for running plugin integration tests. It then kicks off the integration test for each plugin based on the `test-configs` defined in [test-manifest.yml](config/test_manifest.yml). It executes each configuration separately from others by spinning up a dedicated local test cluster. It uses `integtest.sh` script to run the integration test. There is a [default](../../scripts/default/integtest.sh) version of this script present in opensearch-build repo and also allows plugins to override the default by having a custom integtest.sh in plugin repo.
+The [integ-test job](../../jenkins_workflow/test/testsuite/Jenkinsfile) starts by pulling the manifest files and installing the required dependencies for running plugin integration tests. It then kicks off the integration test for each plugin based on the `test-configs` defined in [test-manifest.yml](config/opensearch/test_manifest.yml). It executes each configuration separately from others by spinning up a dedicated local test cluster. It uses `integtest.sh` script to run the integration test. There is a [default](../../scripts/default/integtest.sh) version of this script present in opensearch-build repo and also allows plugins to override the default by having a custom integtest.sh in plugin repo.
 
 Once all tests complete, the notifications job can send out the notifications to the subscribed channels. Below figure illustrates how different components of the test workflow would interact with each other. 
 
@@ -58,7 +58,7 @@ The development is tracked by [meta issue #126](https://github.com/opensearch-pr
 
 Manifest files are configurations for a particular bundle. `test-workflow` uses three types of manifest files to run test suites. 
 
-1. `test-manifest.yml` provides a list of test configurations to run against a given component in the bundle. An example of a configuration would be, integration test `index-management` plugin `with-security` and `without-security`. This manifest file serves as a support matrix config for the testing and should be updated by plugins if new components or test suites are to be added as part of the release workflow. See [here](config/test_manifest.yml)
+1. `test-manifest.yml` provides a list of test configurations to run against a given component in the bundle. An example of a configuration would be, integration test `index-management` plugin `with-security` and `without-security`. This manifest file serves as a support matrix config for the testing and should be updated by plugins if new components or test suites are to be added as part of the release workflow. See [here](config/opensearch/test_manifest.yml)
 2. `build-manifest.yml` created by the build-workflow and provides a list of artifacts built as part of the build-workflow. It assists `test-workflow` pull the maven and build dependencies to run the test suites.
 3. `bundle-manfest.yml` created by the build-workflow and provides a list of components packaged in a given bundle.  It assists `test-workflow` to identify what components should be tested for a given bundle.
 

src/test_workflow/config/test_manifest.yml → src/test_workflow/config/opensearch/test_manifest.yml

@@ -1,3 +1,4 @@
+name: OpenSearch
 components:
   - name: index-management
     integ-test:

tests/tests_manifests/data/opensearch-test-1.1.0.yml

@@ -1,3 +1,4 @@
+name: OpenSearch
 components:
   - name: index-management
     integ-test:

tests/tests_manifests/data/test/opensearch-test-schema-version-1.0.yml

@@ -1,3 +1,4 @@
+name: OpenSearch
 components:
   - name: index-management
     integ-test:

tests/tests_test_workflow/data/test_manifest.yml

@@ -1,3 +1,4 @@
+name: OpenSearch
 components:
   - name: job-scheduler
     integ-test:

tests/tests_test_workflow/test_integ_workflow/integ_test/data/test_manifest.yml

@@ -1,3 +1,4 @@
+name: OpenSearch
 components:
   - name: job-scheduler
     integ-test: