MAPREDUCE-7497. mapreduce tests have stopped running. #7343
Conversation
|
🎊 +1 overall
This message was automatically generated. |
hadoop-project/pom.xml
Outdated
| @@ -185,8 +185,8 @@ | |||
| --add-opens=java.base/sun.security.x509=ALL-UNNAMED | |||
| </extraJavaTestArgs> | |||
| <!-- Plugin versions and config --> | |||
| <maven-surefire-plugin.argLine>-Xmx2048m -XX:+HeapDumpOnOutOfMemoryError ${extraJavaTestArgs}</maven-surefire-plugin.argLine> | |||
| <maven-surefire-plugin.version>3.0.0-M1</maven-surefire-plugin.version> | |||
| <maven-surefire-plugin.argLine>-Xmx2048m -Xss2m -XX:+HeapDumpOnOutOfMemoryError ${extraJavaTestArgs}</maven-surefire-plugin.argLine> | |||
There was a problem hiding this comment.
Does the default stack size not work?
There was a problem hiding this comment.
This change was made because the ResourceManager module in #7335 reported an unable to create new native thread exception. Additionally, the third build result of #7335 appeared abnormal, and I suspected that the issue might be caused by the large number of compiled modules. I increased the -Xss size, which seemed to have some effect. The default value for this parameter is 1m.
The build report is as follows:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7335/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
If either #7343 or #7335 is merged, the other PR can remove this change.
| @@ -43,7 +43,12 @@ public ApplicationEntity(TimelineEntity entity) { | |||
| } | |||
|
|
|||
| public String getQueue() { | |||
| return getInfo().get(QUEUE_INFO_KEY).toString(); | |||
| if (getInfo() != null) { | |||
There was a problem hiding this comment.
Is this an unrelated change?
There was a problem hiding this comment.
This is a necessary change, and I have provided a detailed explanation of the modification in the comment below.
There was a problem hiding this comment.
Thank you for the explanation.
Is this potentially a backward-incompatible change? For example, does it mean clients may now be getting back a queue element with an empty string, where previously it was omitted entirely from the response?
There was a problem hiding this comment.
Thank you very much for your comment! I have further refined this part of the code and added the @JsonInclude(JsonInclude.Include.NON_EMPTY) annotation to ensure compatibility of this logic.
There was a problem hiding this comment.
From my perspective, we need to continue iterating and improving the YARN Timeline-related code, implementing parsing for both JSON and XML formats in the way Jersey2 does. I hope that after completing the JDK17 support, we can continue advancing these improvements.
|
@cnauroth Thank you very much for reviewing this PR! I apologize for not providing a detailed explanation of why we made these changes. Let me explain in detail why we made these modifications. After upgrading Jersey to version 2.46, we found that Jersey 2 handles requests differently compared to Jersey 1. Overall, it has become stricter.
This unit test is for the following interface: AMWebServices#get, with the code as follows: This interface only accepts Therefore, we need to improve this unit test.
TestHsWebServices#testInvalidAccept also requires improvement for reasons similar to Change 1.
ApplicationEntity#getQueue is a necessary change. We made this modification because the unit test was failing previously. We can see this error at the following link: patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt This is a subtle error, and it took me a long time to identify the issue. The problem arises from the difference in how Jersey 1 and Jersey 2 handle complex type conversions to JSON. In Jersey 1, we could customize the conversion process, but in Jersey 2, we need to define custom JSON readers and writers. In my case, I defined a Therefore, I have implemented handling for the conversion of this field. Since it is the timeclient-v2 version, it will eventually reach the Inside, there is a complex data type, This will throw a |
There was a problem hiding this comment.
+1, pending completion of the pre-submit run on the latest changes.
Thank you, @slfan1989 !
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@cnauroth Thank you very much for reviewing the code! I will merge this PR into the trunk branch to begin the upgrade of the MapReduce module from |
Co-authored-by: Chris Nauroth <cnauroth@apache.org> Reviewed-by: Chris Nauroth <cnauroth@apache.org> Signed-off-by: Shilun Fan <slfan1989@apache.org>
Description of PR
JIRA: MAPREDUCE-7497. mapreduce tests have stopped running.
After completing HADOOP-15984, we added JUnit5 dependencies to some modules. These dependencies caused maven-surefire to fail to recognize JUnit4 tests, resulting in the MapReduce module's unit tests stopping. We will track and fix this issue in this JIRA.
How was this patch tested?
Unit Test.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?