Skip to content

HDFS-17843. FileSystem.createFile() fails with relative path.#8279

Open
balodesecurity wants to merge 1 commit intoapache:trunkfrom
balodesecurity:HDFS-17843
Open

HDFS-17843. FileSystem.createFile() fails with relative path.#8279
balodesecurity wants to merge 1 commit intoapache:trunkfrom
balodesecurity:HDFS-17843

Conversation

@balodesecurity
Copy link

Description of PR

JIRA: https://issues.apache.org/jira/browse/HDFS-17843

DistributedFileSystem.createFile() (builder pattern) and create() with favored nodes threw IllegalArgumentException when passed a relative path, while fs.create() worked correctly.

Root Cause

In three create() overloads, the doCall(Path p) method passed the original outer variable f (potentially relative) to getPathName() instead of the resolved absolute path p computed by fixRelativePart() + symlink resolution:

// Before (buggy)
public HdfsDataOutputStream doCall(final Path p) throws IOException {
    final DFSOutputStream out = dfs.create(getPathName(f), ...);

// After (fixed)
public HdfsDataOutputStream doCall(final Path p) throws IOException {
    final DFSOutputStream out = dfs.create(getPathName(p), ...);

The three affected methods are:

  • create(Path, FsPermission, boolean, int, short, long, Progressable, InetSocketAddress[]) — create with favored nodes
  • create(Path, FsPermission, EnumSet, int, short, long, Progressable, ChecksumOpt, InetSocketAddress[], String, String) — create with EC/storage policy (used by createFile() builder)
  • createNonRecursive(Path, FsPermission, EnumSet, int, short, long, Progressable, ChecksumOpt, InetSocketAddress[], String, String)

Testing

Added TestDistributedFileSystem#testCreateFileWithRelativePath covering both createFile() builder and create() with favored nodes using relative paths. Also verified manually against a running single-node HDFS cluster.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

In three create() overloads within DistributedFileSystem, the doCall()
method was passing the original (possibly relative) path `f` to
getPathName() instead of the resolved absolute path `p`. This caused
an IllegalArgumentException when using relative paths with createFile()
builder and create() with favored nodes, while fs.create() worked fine.

Fix: pass `p` (the symlink-resolved, absolute path) to getPathName()
in all three affected doCall() implementations.

Added TestDistributedFileSystem#testCreateFileWithRelativePath to
verify both createFile() builder and create() with favored nodes work
correctly with relative paths.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 36s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 3m 12s Maven dependency ordering for branch
+1 💚 mvninstall 43m 33s trunk passed
+1 💚 compile 5m 27s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 compile 5m 49s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 checkstyle 2m 14s trunk passed
+1 💚 mvnsite 3m 18s trunk passed
+1 💚 javadoc 2m 42s trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 2m 40s trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 spotbugs 7m 50s trunk passed
+1 💚 shadedclient 30m 40s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 29s Maven dependency ordering for patch
+1 💚 mvninstall 2m 15s the patch passed
+1 💚 compile 5m 2s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javac 5m 2s the patch passed
+1 💚 compile 5m 16s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 javac 5m 16s the patch passed
+1 💚 blanks 0m 1s The patch has no blanks issues.
+1 💚 checkstyle 1m 37s the patch passed
+1 💚 mvnsite 2m 21s the patch passed
+1 💚 javadoc 1m 38s the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚 javadoc 1m 38s the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚 spotbugs 7m 8s the patch passed
+1 💚 shadedclient 30m 14s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 2m 36s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 214m 39s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 47s The patch does not generate ASF License warnings.
382m 42s
Reason Tests
Failed junit tests hadoop.hdfs.TestBlockRecoveryCauseStandbyNameNodeCrash
Subsystem Report/Notes
Docker ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8279/1/artifact/out/Dockerfile
GITHUB PR #8279
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux ee85d77d20ab 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 44899f6
Default Java Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8279/1/testReport/
Max. process+thread count 3467 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8279/1/console
versions git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

@balodesecurity
Copy link
Author

The TestBlockRecoveryCauseStandbyNameNodeCrash failure is a pre-existing flaky test unrelated to this patch — it also failed in PR #8268 (an unrelated change) around the same time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants