[Testing] Enhance sanity testing framework

[DarshitChanpura]

As part of initial effort, sanity tests were added to verify whether security plugin was installed correctly. With that in place, this issue proposes addition of a series of tests that run with :integTestRemote and the intent is to automate tests performed manually after every release.
These tests will run in every CI run as well as when distribution builds are being invoked. These tests will also help keep healthy check on whether incoming changes change security plugin behavior.

Here is the runbook for the tests:
sanity-testing-runbook.txt

Exit criteria:

• Ensure all tests are written in security plugin (Translate these scenarios into individual tests)
• Analyze all steps and implement tests required in security-dashboards-plugin. (enhance tests covered in functional test repo)

[stephen-crawford]

[Triage] Hi @DarshitChanpura, thank you for filing this issue. More testing is always helpful and this has clear steps. Marking as triaged.

[samuelcostae]

Gonna start looking into this

[samuelcostae]

@DarshitChanpura the existing tests make use of a number mock API Calls. I would like to confirm that for the scope this issue we don't want to use these mocks(For the main calls at least) so we can sanity check both OSS Plugin and OSS Dashboards Plugin. Is that correct?

[samuelcostae]

@DarshitChanpura Another question, in the text file with the steps. the section "Dashboards Basics", it mentions opening the browser in incognito mode and check if the data is correct. The incognito part made me not sure if that sections another set of tests, or its a manual verifcation that i should do now to check if the tests are working as expected.

[DarshitChanpura]

the existing tests make use of a number mock API Calls. I would like to confirm that for the scope this issue we don't want to use these mocks(For the main calls at least) so we can sanity check both OSS Plugin and OSS Dashboards Plugin. Is that correct?

We would like to replicate the steps as test. They are all mostly for front-end (jest_integration). The API calls should be translated to tests on the backend (we can utilize the test framework to do so)

Another question, in the text file with the steps. the section "Dashboards Basics", it mentions opening the browser in incognito mode and check if the data is correct. The incognito part made me not sure if that sections another set of tests, or its a manual verifcation that i should do now to check if the tests are working as expected.

your observation is correct. It's manual verification by logging in as another user to verify that the action was successful.

[peternied]

@DarshitChanpura Its been a while since there have been updates on this issue, can you check where this is at?

[DarshitChanpura]

Here is the PR from @samuelcostae : opensearch-project/opensearch-dashboards-functional-test#958 to add tests from the runbook

[DarshitChanpura]

This issue is currently blocked awaiting review from maintainers of the functional tests repo.

[DarshitChanpura]

Closed as completed via: opensearch-project/opensearch-dashboards-functional-test#958