Skip to content

Commit

Permalink
Update disable element
Browse files Browse the repository at this point in the history 
- use new format of disable
- update readme file

Signed-off-by: renfeiw <renfeiw@ca.ibm.com>
  • Loading branch information
@renfeiw
renfeiw committed Jul 8, 2021
1 parent 8702ab5 commit b674367
Show file tree
Hide file tree
Showing 20 changed files with 302 additions and 212 deletions.
8 changes: 5 additions & 3 deletions test/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,11 @@ automated test builds at the AdoptOpenJDK project.
- In playlist.xml, to disable a test target, add

```
<disabled>
<comment>issue url or issue comment url</comment>
</disabled>
<disables>
<disable>
<comment>issue url or issue comment url</comment>
<disable>
</disables>
```

inside the `<test>` element that you want to disable.
Expand Down
131 changes: 75 additions & 56 deletions test/docs/OpenJ9TestUserGuide.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -310,33 +310,37 @@ To exclude the 2nd variation listed which is assigned suffix_1 ```-Xmx1024m``` a
After the comment is left, there will be a auto PR created with the exclude change in the playlist.xml. The PR will be linked to issue. If the testName can not be found in the repo, no PR will be created and there will be a comment left in the issue linking to the failed workflow run for more details. In the case where the parameter contains space separated values, use single quotes to group the parameter.

#### Manually exclude a test target
Search the test name to find its playlist.xml file. Add a ```<disabled>``` element after ```<testCaseName>``` element. The ```<disabled>``` element should always contain a ```<comment>``` element to specify the related issue url (or issue comment url).
Search the test name to find its playlist.xml file. Add a ```<disables>``` element after ```<testCaseName>``` element. The ```<disables>``` element is used to capsulate all ```<disable>``` elements. ```<disable>``` should always contain a ```<comment>``` element to specify the related issue url (or issue comment url).

For example:

```
<test>
<testCaseName>jdk_test</testCaseName>
<disabled>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
</disabled>
<disables>
<disable>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<disable>
</disables>
...
```

This will disable the entire test suite. The following section describes how to disable the specific test cases.

##### Exclude a specific test variation:
Add a ```<variation>``` element in the ```<disabled>``` element to specify the variation. The ```<variation>``` element must match an element defined in the ```<variations>``` element.
Add a ```<variation>``` element in the ```<disable>``` element to specify the variation. The ```<variation>``` element must match an element defined in the ```<variations>``` element.

For example, to exclude the test case with variation ```-Xmx1024m```:

```
<test>
<testCaseName>jdk_test</testCaseName>
<disabled>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<variation>-Xmx1024m</variation>
</disabled>
<disables>
<disable>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<variation>-Xmx1024m</variation>
</disable>
</disables>
...
<variations>
<variation>NoOptions</variation>
Expand All @@ -346,103 +350,116 @@ For example, to exclude the test case with variation ```-Xmx1024m```:
```

##### Exclude a test against specific java implementation:
Add a ```<impl>``` element in the ```<disabled>``` element to specify the implementation.
Add a ```<impl>``` element in the ```<disable>``` element to specify the implementation.

For example, to exclude the test for openj9 only:

```
<test>
<testCaseName>jdk_test</testCaseName>
<disabled>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<impl>openj9</impl>
</disabled>
<disables>
<disable>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<impl>openj9</impl>
</disable>
</disables>
...
```

##### Exclude a test against specific java vendor:
Add a ```<vendor>``` element in the ```<disabled>``` element to specify the vendor information.
Add a ```<vendor>``` element in the ```<disable>``` element to specify the vendor information.

For example, to exclude the test for AdoptOpenJDK only:

```
<test>
<testCaseName>jdk_test</testCaseName>
<disabled>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<vendor>adoptopenjdk</vendor>
</disabled>
<disables>
<disable>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<vendor>adoptopenjdk</vendor>
</disable>
</disables>
...
```

##### Exclude a test against specific java version:
Add a ```<version>``` element in the ```<disabled>``` element to specify the version.
Add a ```<version>``` element in the ```<disable>``` element to specify the version.

For example, to exclude the test for java 11 and up:

```
<test>
<testCaseName>jdk_test</testCaseName>
<disabled>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<version>11+</version>
</disabled>
<disables>
<disable>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<version>11+</version>
</disable>
</disables>
...
```


##### Exclude a test against specific platform:
Add a ```<platform>``` element in the ```<disabled>``` element to specify the platform in regular expression. All platforms can be found here: https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/openjdk_tests
Add a ```<platform>``` element in the ```<disable>``` element to specify the platform in regular expression. All platforms can be found here: https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/openjdk_tests

For example, to exclude the test for all linux platforms:

```
<test>
<testCaseName>jdk_test</testCaseName>
<disabled>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<platform>.*linux.*</plat>
</disabled>
<disables>
<disable>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<platform>.*linux.*</plat>
</disable>
</disables>
...
```


##### Exclude test against multiple criteria:
Defined a combination of ```<variation>```, ```<impl>```, ```<version>```, and ```<platform>``` in the ```<disabled>``` element.
Defined a combination of ```<variation>```, ```<impl>```, ```<version>```, and ```<platform>``` in the ```<disable>``` element.

For example, to exclude the test with variation ```-Xmx1024m``` against adoptopenjdk openj9 java 8 on windows only:

```
<test>
<testCaseName>jdk_test</testCaseName>
<disabled>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<variation>-Xmx1024m</variation>
<version>8</version>
<impl>openj9</impl>
<vendor>adoptopenjdk</vendor>
<platform>.*windows.*</platform>
</disabled>
<disables>
<disable>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<variation>-Xmx1024m</variation>
<version>8</version>
<impl>openj9</impl>
<vendor>adoptopenjdk</vendor>
<platform>.*windows.*</platform>
</disable>
</disables>
...
```

Note: Same element cannot be defined multiple times inside one ```<disabled>``` element. It is because the elements inside the disable element are in AND relationship.
Note: Same element cannot be defined multiple times inside one ```<disable>``` element. It is because the elements inside the disable element are in AND relationship.

For example, to exclude test on against hotspot and openj9. It is required to define multiple ```<disabled>``` elements, each with a single ```<impl>``` element inside:
For example, to exclude test on against hotspot and openj9. It is required to define multiple ```<disable>``` elements, each with a single ```<impl>``` element inside:

```
<test>
<testCaseName>jdk_test</testCaseName>
<disabled>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<version>8</version>
<impl>openj9</impl>
</disabled>
<disabled>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<version>8</version>
<impl>hotspot</impl>
</disabled>
<disables>
<disable>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<version>8</version>
<impl>openj9</impl>
</disable>
<disable>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<version>8</version>
<impl>hotspot</impl>
</disable>
</disables>
...
```

Expand All @@ -451,15 +468,17 @@ Or remove ```<impl>``` element to exclude test against all implementations:
```
<test>
<testCaseName>jdk_test</testCaseName>
<disabled>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<version>8</version>
</disabled>
<disables>
<disable>
<comment>https://github.com/adoptium/aqa-tests/issues/123456</comment>
<version>8</version>
</disable>
</disables>
...
```
#### Execute excluded test target

If a test is disabled using `<disabled>` tag in playlist.xml, it can be executed by specifying the test target or adding `disabled` in front of its top-level test target.
If a test is disabled using `<disable>` tag in playlist.xml, it can be executed by specifying the test target or adding `disabled` in front of its top-level test target.

```
make _disabled.testA // testA has <disabled> tag in playlist.xml
Expand Down Expand Up @@ -548,7 +567,7 @@ capabilities.

- `DISABLED` tests

If a test is disabled, it means that this test is disabled using `<disabled>` tag in playlist.xml.
If a test is disabled, it means that this test is disabled using `<disable>` tag in playlist.xml.

#### Results in html files

Expand Down
50 changes: 31 additions & 19 deletions test/functional/JLM_Tests/playlist.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -432,9 +432,11 @@
</test>
<test>
<testCaseName>testSoftMxRemote</testCaseName>
<disabled>
<comment>https://github.com/eclipse-openj9/openj9/issues/480</comment>
</disabled>
<disables>
<disable>
<comment>https://github.com/eclipse-openj9/openj9/issues/480</comment>
</disable>
</disables>
<variations>
<variation>Mode110</variation>
<variation>Mode119</variation>
Expand Down Expand Up @@ -465,9 +467,11 @@
</test>
<test>
<testCaseName>testSoftMxRemote_zlinux_64</testCaseName>
<disabled>
<comment>https://github.com/eclipse-openj9/openj9/issues/480</comment>
</disabled>
<disables>
<disable>
<comment>https://github.com/eclipse-openj9/openj9/issues/480</comment>
</disable>
</disables>
<variations>
<variation>Mode110</variation>
<variation>Mode119</variation>
Expand Down Expand Up @@ -498,9 +502,11 @@
</test>
<test>
<testCaseName>testSoftMxRemote_zlinux_31</testCaseName>
<disabled>
<comment>https://github.com/eclipse-openj9/openj9/issues/480</comment>
</disabled>
<disables>
<disable>
<comment>https://github.com/eclipse-openj9/openj9/issues/480</comment>
</disable>
</disables>
<variations>
<variation>Mode110</variation>
<variation>Mode119</variation>
Expand Down Expand Up @@ -531,9 +537,11 @@
</test>
<test>
<testCaseName>testSoftMxRemote_LP4k</testCaseName>
<disabled>
<comment>https://github.com/eclipse-openj9/openj9/issues/480</comment>
</disabled>
<disables>
<disable>
<comment>https://github.com/eclipse-openj9/openj9/issues/480</comment>
</disable>
</disables>
<variations>
<variation>Mode110</variation>
<variation>Mode119</variation>
Expand Down Expand Up @@ -564,9 +572,11 @@
</test>
<test>
<testCaseName>testSoftMxRemote_LP64k</testCaseName>
<disabled>
<comment>https://github.com/eclipse-openj9/openj9/issues/480</comment>
</disabled>
<disables>
<disable>
<comment>https://github.com/eclipse-openj9/openj9/issues/480</comment>
</disable>
</disables>
<variations>
<variation>Mode110</variation>
<variation>Mode119</variation>
Expand Down Expand Up @@ -1256,10 +1266,12 @@
</test>
<test>
<testCaseName>testSoftMxUserScenario</testCaseName>
<disabled>
<comment>github.com/eclipse-openj9/openj9/issues/8967</comment>
<platform>aarch64.*</platform>
</disabled>
<disables>
<disable>
<comment>github.com/eclipse-openj9/openj9/issues/8967</comment>
<platform>aarch64.*</platform>
</disable>
</disables>
<variations>
<variation>Mode110</variation>
<variation>Mode119</variation>
Expand Down
10 changes: 6 additions & 4 deletions test/functional/Java11andUp/playlist.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -73,10 +73,12 @@
-->
<test>
<testCaseName>Nestmate_virtual_private</testCaseName>
<disabled>
<comment>https://github.com/eclipse-openj9/openj9/issues/9167</comment>
<platform>.*zos.*</platform>
</disabled>
<disables>
<disable>
<comment>https://github.com/eclipse-openj9/openj9/issues/9167</comment>
<platform>.*zos.*</platform>
</disable>
</disables>
<variations>
<variation>$(SQ)-Xjit:rtResolve,disableAsyncCompilation,limit={org/openj9/test/nestmates/NestmatesTest.testVirtualUnresolvedInterpreted*},{org/openj9/test/nestmates/NestmatesTest.testVirtualUnresolvedInterpreted*}(count=0)$(SQ)</variation>
<variation>$(SQ)-Xjit:rtResolve,disableAsyncCompilation,limit={org/openj9/test/nestmates/NestmatesTest.testVirtualUnresolvedJitted*,org/openj9/test/nestmates/NestmatesTest$InnerClass.innerPrivateMethod*},{org/openj9/test/nestmates/NestmatesTest.testVirtualUnresolvedInterpreted*}(count=0),{org/openj9/test/nestmates/NestmatesTest$InnerClass.innerPrivateMethod*}(count=100)$(SQ)</variation>
Expand Down
20 changes: 12 additions & 8 deletions test/functional/Java8andUp/playlist.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -306,10 +306,12 @@
</test>
<test>
<testCaseName>TestAttachErrorHandling_SE80</testCaseName>
<disabled>
<comment>https://github.com/eclipse-openj9/openj9/issues/3081</comment>
<platform>.*aix.*</platform>
</disabled>
<disables>
<disable>
<comment>https://github.com/eclipse-openj9/openj9/issues/3081</comment>
<platform>.*aix.*</platform>
</disable>
</disables>
<command>$(JAVA_COMMAND) $(JVM_OPTIONS) \
-cp $(Q)$(RESOURCES_DIR)$(P)$(TEST_JDK_HOME)$(D)lib$(D)tools.jar$(P)$(TESTNG)$(P)$(TEST_RESROOT)$(D)GeneralTest.jar$(Q) \
-Dcom.ibm.tools.attach.enable=yes \
Expand All @@ -334,10 +336,12 @@
</test>
<test>
<testCaseName>TestAttachErrorHandling</testCaseName>
<disabled>
<comment>https://github.com/eclipse-openj9/openj9/issues/3081</comment>
<platform>.*aix.*</platform>
</disabled>
<disables>
<disable>
<comment>https://github.com/eclipse-openj9/openj9/issues/3081</comment>
<platform>.*aix.*</platform>
</disable>
</disables>
<command>$(JAVA_COMMAND) $(JVM_OPTIONS) \
-cp $(Q)$(RESOURCES_DIR)$(P)$(TESTNG)$(P)$(TEST_RESROOT)$(D)GeneralTest.jar$(Q) \
-Dcom.ibm.tools.attach.enable=yes \
Expand Down
Loading

0 comments on commit b674367

Please sign in to comment.