Skip to content

HADOOP-17745. Wrap IOException with InterruptedException cause properly #3076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: trunk
Choose a base branch
from
Open

HADOOP-17745. Wrap IOException with InterruptedException cause properly #3076

wants to merge 3 commits into from
+68 −0

Conversation

eric-maynard
Copy link

The Azure client sometimes throws an IOException with an InterruptedException cause which can be converted to an InterruptedIOException. This is important for downstream consumers that rely on an InterruptedIOException to gracefully close.

@eric-maynard
Copy link
Author

Hi @steveloughran, what tests should I run in order to make this change?

@steveloughran
Copy link
Contributor

aah, you need to look at the Testing Azure doc. I can see the code here doesn't directly go near the store, but as its called from abfs you are going to have to do a test run. Sorry.

steveloughran
steveloughran requested changes Jun 8, 2021
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Production code looks good, one minor change. Test-wise, I'd prefer tests in assertJ as they'll report on exception class mismatches better.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 52s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 33m 19s trunk passed
+1 💚 compile 23m 43s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 20m 51s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 1m 1s trunk passed
+1 💚 mvnsite 1m 42s trunk passed
+1 💚 javadoc 1m 16s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 52s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 3m 2s trunk passed
+1 💚 shadedclient 21m 44s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 16s the patch passed
+1 💚 compile 28m 31s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 28m 31s the patch passed
+1 💚 compile 24m 56s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 24m 56s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 1m 7s /results-checkstyle-hadoop-common-project_hadoop-common.txt hadoop-common-project/hadoop-common: The patch generated 31 new + 16 unchanged - 0 fixed = 47 total (was 16)
+1 💚 mvnsite 1m 52s the patch passed
+1 💚 javadoc 1m 2s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 39s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 59s the patch passed
+1 💚 shadedclient 21m 1s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 19m 52s hadoop-common in the patch passed.
+1 💚 asflicense 0m 56s The patch does not generate ASF License warnings.
213m 32s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3076/2/artifact/out/Dockerfile
GITHUB PR #3076
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux a2fe8be534d4 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1ab84de
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3076/2/testReport/
Max. process+thread count 2137 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3076/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran
steveloughran requested changes Jun 23, 2021
View reviewed changes
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Some minor changes in the tests, but nothing much

@steveloughran
Copy link
Contributor

Hi @steveloughran, what tests should I run in order to make this change?

all the abfs tests in hadoop-azure. Ideally with as many auth options as you can.
hadoop-tools/hadoop-azure/dev-support/testrun-scripts/runtests.sh helps there.

At the very least:

mvn clean verify  -Dparallel-tests=abfs -DtestsThreadCount=5 -Dscale

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 8s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 33m 4s trunk passed
+1 💚 compile 22m 33s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 19m 10s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 1m 1s trunk passed
+1 💚 mvnsite 1m 31s trunk passed
+1 💚 javadoc 1m 2s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 37s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 25s trunk passed
+1 💚 shadedclient 18m 3s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 55s the patch passed
+1 💚 compile 21m 44s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 21m 44s the patch passed
+1 💚 compile 19m 3s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 19m 3s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 1m 1s /results-checkstyle-hadoop-common-project_hadoop-common.txt hadoop-common-project/hadoop-common: The patch generated 29 new + 16 unchanged - 0 fixed = 45 total (was 16)
+1 💚 mvnsite 1m 29s the patch passed
+1 💚 javadoc 1m 0s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 35s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 36s the patch passed
+1 💚 shadedclient 18m 33s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 16m 57s hadoop-common in the patch passed.
+1 💚 asflicense 0m 48s The patch does not generate ASF License warnings.
187m 47s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3076/3/artifact/out/Dockerfile
GITHUB PR #3076
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 89c443ac23e8 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / efdfacb
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3076/3/testReport/
Max. process+thread count 2235 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3076/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@apache apache deleted a comment from hadoop-yetus Jul 2, 2021
@steveloughran
Copy link
Contributor

changes look good. This leaves checkstyle complaints to deal with. Indentation and some lines need to be split.

+1 pending these changes

./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:32:    @Test: 'method def modifier' has incorrect indentation level 4, expected level should be 2. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:34:        InterruptedIOException inputException = new InterruptedIOException("message");: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:34:        InterruptedIOException inputException = new InterruptedIOException("message");: Line is longer than 80 characters (found 86). [LineLength]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:35:        NullPointerException causeException = new NullPointerException("cause");: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:36:        inputException.initCause(causeException);: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:37:        Exception outputException = IOUtils.wrapException("path", "methodName", inputException);: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:37:        Exception outputException = IOUtils.wrapException("path", "methodName", inputException);: Line is longer than 80 characters (found 96). [LineLength]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:40:        Assertions.assertThat(outputException).isInstanceOf(InterruptedIOException.class);: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:40:        Assertions.assertThat(outputException).isInstanceOf(InterruptedIOException.class);: Line is longer than 80 characters (found 90). [LineLength]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:41:        Assertions.assertThat(outputException.getCause()).isInstanceOf(NullPointerException.class): 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:41:        Assertions.assertThat(outputException.getCause()).isInstanceOf(NullPointerException.class): Line is longer than 80 characters (found 98). [LineLength]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:43:        assertEquals(outputException.getMessage(), inputException.getMessage());: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:44:        assertEquals(outputException.getCause(), inputException.getCause());: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:45:    }: 'method def rcurly' has incorrect indentation level 4, expected level should be 2. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:47:    @Test: 'method def modifier' has incorrect indentation level 4, expected level should be 2. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:48:    public void testWrapExceptionWithInterruptedCauseException() throws Exception {: Line is longer than 80 characters (found 83). [LineLength]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:49:        IOException inputException = new IOException("message");: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:50:        InterruptedException causeException = new InterruptedException("cause");: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:51:        inputException.initCause(causeException);: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:52:        Exception outputException = IOUtils.wrapException("path", "methodName", inputException);: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:52:        Exception outputException = IOUtils.wrapException("path", "methodName", inputException);: Line is longer than 80 characters (found 96). [LineLength]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:55:        // but be an InterruptedIOException because the cause was an InterruptedException: Line is longer than 80 characters (found 89). [LineLength]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:56:        Assertions.assertThat(outputException).isInstanceOf(InterruptedIOException.class);: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:56:        Assertions.assertThat(outputException).isInstanceOf(InterruptedIOException.class);: Line is longer than 80 characters (found 90). [LineLength]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:57:        Assertions.assertThat(outputException.getCause()).isInstanceOf(InterruptedException.class);: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:57:        Assertions.assertThat(outputException.getCause()).isInstanceOf(InterruptedException.class);: Line is longer than 80 characters (found 99). [LineLength]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:58:        assertEquals("getMessage()",: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:60:        assertEquals("getCause()",: 'method def' child has incorrect indentation level 8, expected level should be 4. [Indentation]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtilsWrapExceptionSuite.java:62:    }: 'method def rcurly' has incorrect indentation level 4, expected level should be 2. [Indentation]

@steveloughran
Copy link
Contributor

@eric-maynard any chance to deal with the checkstyle complaints yet?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@steveloughran steveloughran Awaiting requested review from steveloughran

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@eric-maynard @steveloughran @hadoop-yetus