Parse problem list and generate json file #2698

llxia · 2021-06-30T16:24:15Z

As part of Create a Disabled tests pipeline, we need to parse openjdk problemlist and get json file:

input: problem list
https://github.com/adoptium/aqa-tests/tree/master/openjdk/excludes
expected output:

[{
"JDK_VERSION" : 17,
"JDK_IMPL" : "hotspot",
"TARGET" : "jdk_custom",
"CUSTOM_TARGET" : "java/lang/ClassLoader/nativeLibrary/NativeLibraryTest.java",
"GIT_ISSUE" : "https://github.com/adoptium/temurin-build/issues/248",
"PLATFORM" : "all"
},{
"JDK_VERSION" : 16,
"JDK_IMPL" : "openj9",
"TARGET" : "jdk_custom",
"CUSTOM_TARGET" : "java/lang/ClassLoader/Assert.java",
"GIT_ISSUE" : "https://github.com/eclipse-openj9/openj9/issues/6668",
"PLATFORM" : "x86-64_mac"
}]

The text was updated successfully, but these errors were encountered:

deepak-gittyup · 2021-09-01T17:10:56Z

Can I pick up this task ?

smlambert · 2021-09-01T17:22:19Z

Yes, certainly @deepak-gittyup ! Please ask questions in this issue if you have any.

deepak-gittyup · 2021-09-01T18:26:58Z

Thanks @smlambert

Here is my understanding of the requirement:
I need to write a parser (in JAVA) that takes the excludes directory as its input and creates a json file adhering to the format shown in the expected output shown above

What should be JDK_IMPL field ?
I see that the problem list files look like:
ProblemList_openjdk<JDK_VERSION>-<JDK_IMPL>.txt // For JDK_IMPL = openj9 and sap
or
ProblemList_openjdk<JDK_VERSION>.txt // For JDK_IMPL = hotspot ?

Where should this parser be placed ?
Should it be in the same dir that contains excludes dir something like this:
aqa-tests/openjdk/tools/CreateProblemListJson.java

I picturing the usage something like this

${JAVA_HOME}/bin/java CreateProblemListJson <path to the `excludes dir`>

smlambert · 2021-09-07T20:08:32Z

hi @deepak-gittyup - @llxia will give you some direction on this issue.

llxia · 2021-09-08T01:09:56Z

I need to write a parser (in JAVA) that takes the excludes directory as its input and creates a json file adhering to the format shown in the expected output shown above

Personally, I think python will be much simpler. I know this makes python one of the prerequisites, but we should only run this script on xlinux for parsing. Plus, we have some code for Query git issue url to get issue status in python already.

What should be JDK_IMPL field ?
I see that the problem list files look like:
ProblemList_openjdk<JDK_VERSION>-<JDK_IMPL>.txt // For JDK_IMPL = openj9 and sap
or
ProblemList_openjdk<JDK_VERSION>.txt // For JDK_IMPL = hotspot ?

Yes, that is correct. If JDK_IMPL is provided in the file name, use it. Otherwise, defaults to the hotspot.

Where should this parser be placed ?

Maybe create a folder called disabledTestParser under https://github.com/adoptium/aqa-tests/tree/master/. We should put all related scripts into this folder.

deepak-gittyup · 2021-09-10T02:19:03Z

Thanks for the clarification @llxia
Couple of other questions

What should be the value of the TARGET?
In the examples above, I see that both of them are set to jdk_custom
"TARGET" : "jdk_custom"

What should be the value of PLATFORM ?
The first example entry above seems to be picked up from ProblemList_openjdk17.txt
Here is the line

java/lang/ClassLoader/nativeLibrary/NativeLibraryTest.java	https://github.com/adoptium/temurin-build/issues/248	generic-all

So generic-all here is mapped to "PLATFORM" : "all"

Similarly, for the 2nd example entry is from ProblemList_openjdk16-openj9.txt
Here is the line

java/lang/ClassLoader/Assert.java	https://github.com/eclipse-openj9/openj9/issues/6668	macosx-x64

macosx-x64 is mapped to "PLATFORM" : "x86-64_mac"

Is there a general rule to this name mapping ?

I see other types too:

linux-s390x
macosx-all
windows-all
aix-all
linux-all,aix-all,macosx-all
.
.
.

llxia · 2021-09-10T13:23:24Z

@deepak-gittyup Thanks for looking into this.

What should be the value of the TARGET?

"TARGET" : "jdk_custom"
This is the same for all tests. CUSTOM_TARGET should set to the *.java file path.

Is there a general rule to this name mapping ?

The PLATFORM that we are using is https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/openjdk_tests#L3. You can ignore SPEC and LABEL.
The value in problemlist has to match with PLATFORM. We can have an array that contains PLATFORM. For example, macosx-all should match all PLATFORMS that contain _mac. Therefore, in json, we have

"PLATFORM" :  "x86-64_mac,aarch64_mac"

deepak-gittyup · 2021-09-10T15:43:34Z

I extracted all the unique Platform values from all the ProblemList*.txt files (deliberately omitting *-all for now):
LIST1

arm-linux
linux-aarch64
linux-arm
linux-ppc64le
linux-s390x
linux-x64
macosx-x64
windows-x64
windows-x86
x86-64_windows
z/OS-s390x

Here is all the supported Platform from PLATFORM_MAP (extracted from https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/openjdk_tests#L3):
LIST2

arm_linux
aarch64_mac
aarch64_linux
ppc32_aix
ppc32_linux
ppc64_aix
ppc64_linux
ppc64le_linux
riscv64_linux
riscv64_linux_xl
s390_linux
s390x_linux
s390x_zos
s390x_zos_xl
s390_zos
sparcv9_solaris
x86-64_solaris
x86-64_alpine-linux
x86-32_linux
x86-32_windows
x86-64_linux
x86-64_mac
x86-64_windows

My understanding is that now I have to map items in LIST1 to items in LIST2 before adding them to the final json
Here is my attempt at mapping them:

Platforms from ProblemList (LIST1)	Platform from PLATFORM_MAP (equivalent from LIST2)	Does these LIST2 items match the corresponding LIST1 item ?
arm-linux	arm_linux
linux-aarch64	aarch64_linux
linux-arm	arm_linux
linux-ppc64le	ppc64le_linux
linux-s390x	s390x_linux	s390x_zos_xl
linux-x64	x86-64_linux
macosx-x64	x86-64_mac	aarch64_mac
windows-x64	x86-64_windows
windows-x86		x86-32_windows
x86-64_windows	x86-64_windows
z/OS-s390x	s390x_zos	s390x_linux, s390x_zos_xl

Hi @llxia

Does the table mapping look correct to you ?
From LIST1, what is the difference between the below 2 items. For now, I am assuming both are same
** arm-linux vs linux-arm
** windows-x64 vs x86-64_windows
Comments on the last column in the table above ? (Does these LIST2 items match the corresponding LIST1 item ?)
None of the entires in LIST1 have an equivalent mapping for the below entires in LIST2. Any suggestions for them ?
ppc32_linux
ppc64_linux
riscv64_linux
riscv64_linux_xl
s390_linux
s390_zos
sparcv9_solaris
x86-64_solaris
x86-64_alpine-linux
x86-32_linux
Should the 32bit arch type be included in the -all equivalent. For example is the below expectation correct:
- linux-all: arm_linux,aarch64_linux,ppc32_linux,ppc64_linux,ppc64le_linux,riscv64_linux,riscv64_linux_xl,s390_linux,s390x_linux,x86-64_alpine-linux,x86-32_linux,x86-64_linux
- windows-all: x86-32_windows,x86-64_windows
- aix-all: ppc32_aix, ppc64_aix
Last type: generic-all should literally match everything ? Or simply use all ("PLATFORM" : "all")

llxia · 2021-09-10T19:53:18Z

You can ignore *_xl. We will remove them in the future. I think windows-x86 matches with x86-64_windows and x86-32_windows

None of the entires in LIST1 have an equivalent mapping for the below entires in LIST2. Any suggestions for them ?

We need to map problemlist value to PLATFORM. Not the other way around.

Should the 32bit arch type be included in the -all equivalent.

yes

Last type: generic-all should literally match everything ? Or simply use all ("PLATFORM" : "all")

Use "PLATFORM" : "all"

deepak-gittyup · 2021-09-11T08:51:45Z

Thanks for quick reply @llxia

We need to map problemlist value to PLATFORM. Not the other way around.

Yes
What I was trying to say is, currently list of all supported PLATFORM types is known
But say tomorrow one of the entires in problem list line appears like this:

java/lang/Class/forName/NonJavaNames.sh		https://github.com/eclipse-openj9/openj9/issues/xxxx	linux-apline-x64

Then the mapping will fail since I currently don't know what the problemlist value linux-alpine-x64 should map to
Visually I can see that it should be mapped to x86-64_alpine-linux

Just like we have https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/openjdk_tests#L3, is there a corresponding list of problemlist value that may appear in the ProblemList*.txt files ?

deepak-gittyup · 2021-09-11T09:18:01Z

Here is my first attempt at writing this parser:
#2858

Please review it

llxia · 2021-09-16T14:45:04Z

@sophia-guo we seem to have two formats of Linux arm in problem list. Is it correct?

arm-linux:

aqa-tests/openjdk/excludes/ProblemList_openjdk8.txt

Line 51 in 7f68564

compiler/tiered/Level2RecompilationTest.java https://github.com/adoptium/aqa-tests/issues/2671 linux-ppc64le,aix-all,arm-linux,windows-x86
linux-arm:

aqa-tests/openjdk/excludes/ProblemList_openjdk17.txt

Line 190 in df3cf62

sun/tools/jstat/jstatLineCounts1.sh https://bugs.openjdk.java.net/browse/JDK-8248691 linux-ppc64le,aix-all,linux-arm

Similar question for x86-64_windows in problem list.

aqa-tests/openjdk/excludes/ProblemList_openjdk11-openj9.txt

Line 30 in 034e5c2

    
           java/lang/Class/GetPackageBootLoaderChildLayer.java	https://github.com/eclipse-openj9/openj9/issues/5274	x86-64_windows

llxia · 2021-09-16T15:01:40Z

is there a corresponding list of problemlist value that may appear in the ProblemList*.txt files ?

@deepak-gittyup thanks for PR. I am ok with hardcoded the map for the initial version. For enhancement, I think we can split the value in problem list by -. The matching PLATFORM should contain the tokens with a few exceptions.

sophia-guo · 2021-09-17T16:41:08Z

@llxia that must be added by mistake. The correct format for exclude lable should be OSNAME-ARCH. That is saying linux-arm, windows-x64 are the correct one.

llxia · 2021-09-28T14:39:02Z

@sophia-guo could you please open issues for them? I think they are good first issues.

sophia-guo · 2021-09-28T15:53:00Z

Yes, Will do. @llxia #2908

gervaisj · 2022-01-29T20:28:44Z

@llxia @smlambert Should this still be open? #2858 was merged in and the parser is functional (#2910 and #2905 track further development already)

smlambert · 2022-01-30T04:56:56Z

Closed via #2858

llxia added the good first issue label Jun 30, 2021

llxia mentioned this issue Jun 30, 2021

EPIC: Enable a Disabled tests pipeline or workflow #1853

Open

4 tasks

smlambert assigned deepak-gittyup Sep 1, 2021

llxia mentioned this issue Sep 16, 2021

Fix:2698 Parse problem list and generate json file #2858

Merged

sophia-guo mentioned this issue Sep 28, 2021

Support other bug tracker when parse problem list and query corresponding bug tracker status #2905

Open

llxia mentioned this issue Sep 28, 2021

Dynamically map platforms from ProblemList to PLATFORM #2910

Closed

smlambert added the canosp label Jan 22, 2022

smlambert unassigned deepak-gittyup Jan 22, 2022

smlambert closed this as completed Jan 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parse problem list and generate json file #2698

Parse problem list and generate json file #2698

llxia commented Jun 30, 2021

deepak-gittyup commented Sep 1, 2021

smlambert commented Sep 1, 2021

deepak-gittyup commented Sep 1, 2021

smlambert commented Sep 7, 2021

llxia commented Sep 8, 2021

deepak-gittyup commented Sep 10, 2021

llxia commented Sep 10, 2021 •

edited

Loading

deepak-gittyup commented Sep 10, 2021

llxia commented Sep 10, 2021

deepak-gittyup commented Sep 11, 2021

deepak-gittyup commented Sep 11, 2021 •

edited

Loading

llxia commented Sep 16, 2021 •

edited

Loading

llxia commented Sep 16, 2021

sophia-guo commented Sep 17, 2021

llxia commented Sep 28, 2021

sophia-guo commented Sep 28, 2021 •

edited

Loading

gervaisj commented Jan 29, 2022

smlambert commented Jan 30, 2022

Parse problem list and generate json file #2698

Parse problem list and generate json file #2698

Comments

llxia commented Jun 30, 2021

deepak-gittyup commented Sep 1, 2021

smlambert commented Sep 1, 2021

deepak-gittyup commented Sep 1, 2021

smlambert commented Sep 7, 2021

llxia commented Sep 8, 2021

deepak-gittyup commented Sep 10, 2021

llxia commented Sep 10, 2021 • edited Loading

deepak-gittyup commented Sep 10, 2021

llxia commented Sep 10, 2021

deepak-gittyup commented Sep 11, 2021

deepak-gittyup commented Sep 11, 2021 • edited Loading

llxia commented Sep 16, 2021 • edited Loading

llxia commented Sep 16, 2021

sophia-guo commented Sep 17, 2021

llxia commented Sep 28, 2021

sophia-guo commented Sep 28, 2021 • edited Loading

gervaisj commented Jan 29, 2022

smlambert commented Jan 30, 2022

llxia commented Sep 10, 2021 •

edited

Loading

deepak-gittyup commented Sep 11, 2021 •

edited

Loading

llxia commented Sep 16, 2021 •

edited

Loading

sophia-guo commented Sep 28, 2021 •

edited

Loading