HADOOP-16138. hadoop fs mkdir / of nonexistent abfs container raises NPE

[bgaborg]

Change-Id: I2f637865c871e400b95fe7ddaa24bf99fa192023

[mackrorysd]

I only see test changes. Is there another change to actually fix the error and fail with a more graceful error message?

[bgaborg]

Yeah, I need to add a message which can be understood easily.

The hard thing is that this message comes from org.apache.hadoop.fs.shell.Mkdir#processPath:

  @Override
  protected void processPath(PathData item) throws IOException {
    if (item.stat.isDirectory()) {
      if (!createParents) {
        throw new PathExistsException(item.toString());
      }
    } else {
      throw new PathIsNotDirectoryException(item.toString());
    }
  }

--- this throws PathExistsException.
so not from the org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem, so it's a little more tricky than I expected (at least for me) to solve this nicely.

[bgaborg]

@mackrorysd I found out what's happening. We auto-create all containers during the test run, so I should turn that feature off for this test. I forgot to do that, and so I ended up a lot of random containers in the testing account.

[steveloughran]

:)

[steveloughran]

OK, so this is the test. What about the underlying NPE?

[bgaborg]

I haven't figured out yet where to handle that. We can talk about that offline - it will be just a few minutes.

…

On Fri, 6 Sep 2019 at 22:34, Steve Loughran ***@***.***> wrote: OK, so this is the test. What about the underlying NPE? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1302?email_source=notifications&email_token=ABHN7L2K2IXM3AGYEKZ5ZULQIK5D7A5CNFSM4IL6CAC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6D7PKA#issuecomment-529004456>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABHN7L6ML4GIZ5MLSGJ6363QIK5D7ANCNFSM4IL6CACQ> .

[steveloughran]

LGTM, some minor typos and style

+1 pending those changes

looking at the mkdir() code, i can see that it raises a PathIOE on failure without any text. Do you think we should add a cause like "mkdir failed"?

[bgaborg]

looking at the mkdir() code, i can see that it raises a PathIOE on failure without any text. Do you think we should add a cause like "mkdir failed"?

yeah, I can add since I'm already there.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	42	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	29	Maven dependency ordering for branch
+1	mvninstall	1131	trunk passed
+1	compile	983	trunk passed
+1	checkstyle	141	trunk passed
+1	mvnsite	132	trunk passed
+1	shadedclient	1024	branch has no errors when building and testing our client artifacts.
+1	javadoc	111	trunk passed
0	spotbugs	64	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	181	trunk passed
		_ Patch Compile Tests _
0	mvndep	24	Maven dependency ordering for patch
+1	mvninstall	74	the patch passed
+1	compile	932	the patch passed
+1	javac	932	the patch passed
+1	checkstyle	146	the patch passed
+1	mvnsite	129	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	705	patch has no errors when building and testing our client artifacts.
+1	javadoc	113	the patch passed
+1	findbugs	197	the patch passed
		_ Other Tests _
+1	unit	541	hadoop-common in the patch passed.
+1	unit	99	hadoop-azure in the patch passed.
+1	asflicense	54	The patch does not generate ASF License warnings.
		6832

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1302/10/artifact/out/Dockerfile
GITHUB PR	#1302
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 558f9f8445c4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / f6d884c
Default Java	1.8.0_222
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1302/10/testReport/
Max. process+thread count	1373 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1302/10/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	81	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	29	Maven dependency ordering for branch
+1	mvninstall	1116	trunk passed
+1	compile	1053	trunk passed
+1	checkstyle	159	trunk passed
+1	mvnsite	135	trunk passed
-1	shadedclient	401	branch has errors when building and testing our client artifacts.
+1	javadoc	133	trunk passed
0	spotbugs	64	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	190	trunk passed
		_ Patch Compile Tests _
0	mvndep	25	Maven dependency ordering for patch
+1	mvninstall	75	the patch passed
+1	compile	1036	the patch passed
+1	javac	1036	the patch passed
+1	checkstyle	140	the patch passed
+1	mvnsite	137	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
-1	shadedclient	58	patch has errors when building and testing our client artifacts.
+1	javadoc	133	the patch passed
+1	findbugs	212	the patch passed
		_ Other Tests _
-1	unit	535	hadoop-common in the patch failed.
+1	unit	95	hadoop-azure in the patch passed.
+1	asflicense	51	The patch does not generate ASF License warnings.
		5818

Reason	Tests
Failed junit tests	hadoop.util.TestReadWriteDiskValidator

Subsystem	Report/Notes
Docker	Client=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1302/11/artifact/out/Dockerfile
GITHUB PR	#1302
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 292da6e63620 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 1654497
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1302/11/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1302/11/testReport/
Max. process+thread count	1367 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1302/11/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.