EPIC: AQA Change-based testing for PRs (various repos and applications)

[smlambert]

This is an Epic for the CanOSP Winter 2021 term. Each of the bulleted items can be expanded and/or spun off into their own issue within this repo.

Part 0) Optimize the current PR testing

• openjdk-test PR test build gets triggered without any code change #2072 Limit the automatic triggered PR testing. Only auto trigger when PR is ready for review PushOrPRtestbuild.yml_L15
• Change default SDK to nightly in PR build #2187 Change release SDK to nightly PushOrPRtestbuild.yml_L36
• Comment triggered PR builds #2199 Proposal for comment triggered PR builds (related to Proposal: TKG PR builds TKG#109)
  -- Trigger PR testing manually using comment in the git PR
  -- Allow users to specify SDK (default is nightly). Users can specify releases/nightly
  -- Allow users to specify test targets (sanity.system, extended.openjdk, etc)
  -- Allow other values to be specified. For example:
  action nightly sanity.functional xlinuxlargeheap jdk11
action releases testA,testB xlinux,osx jdk8,jdk11 hotspot,openj9

Part 1) Change-based for openjdk-tests repo changes

• Certain test directories/files change trigger for PR testing #2200 Start with a Static map of which openjdk-tests dir was changed -> what action to take. In openjdk-tests, identify what changed, if test changes: example, if perf dir changed, run sanity.perf test target.
• Workflow job triggered by a change to the buildenv directory #2204 If buildenv dir changed, how to trigger a Jenkins job in a github workflow, to trigger a Grinder. If both test dirs and buildenv, dir changes, can a Grinder be triggered with targets from the playlists within the test dirs.
• The prototype would output a result, which is a change-based test target, or a comma-separated list of test targets to be ingested by run-aqa github action to run a build with specific test target to run against changes in test category or trigger specific jenkins build to test jenkins changes.

Part 2) Change-based for OpenJDK changes

• Re-enable code coverage testing #2205 Re-enable code coverage testing (from jcov work). This involves an instrumented build of the jdk repo (openjdk/jdk using jcov, see related issues list).

• Note other coverage tools can be used for certain types of tests, but we will start with OpenJDK test material (and jcov for the coverage info). This item can be broken down and approached in several ways:

  • try to revive the actual openjdk8_jcov job
  • See if we can we leverage build-jdk (and later build-openj9) github action? (adding the build args to instrument and create jcov output).
    In either approach above (running jcov in Jenkins or in github workflow, create an artifact that can be archived and pulled into in a 'next step'...

• Next step is to look at what information jcov provides you, and utilize code coverage information to predict which test(s) will cover a PR made into the jdk repo (against which the codecov was run).

• Using code coverage information, determine which tests should be run against a particular PR based on what existing tests historically cover the code. Based on PR code changes (type of added code, changed code, deleted code), suggest a set of test targets to run and recommend if new test targets are required in order to cover the changes (highlighting sections of the PR that are not covered by testing).

• There may be an existing prototype that did something similar. Assess the prototype and see if it can be rewritten to work with Github PRs (was previously written to work with Eclipse Changesets). It previously handled 7-8 scenarios for variations of code changes, including addition of if-else blocks, cases to switch statements, etc.

#2205 issue can be repeated for jdk11+ (which may have different instructions for running jcov).
Make notes on how to extend for other test suites (like measuring code coverage information for functional tests).

Part 3) CTD exploration
For the recommendation to add new tests, can this change-based testing approach generate tests or an outline of a test to help jumpstart the developer to add the test? Could we leverage Combinatorial Test Design (CTD) to generate test data input?

FYI @llxia @sophia-guo

[smlambert]

Closing this as completed.