Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDFS-16495: RBF should prepend the client ip rather than append it. #4054

Closed
wants to merge 2 commits into from
Closed

HDFS-16495: RBF should prepend the client ip rather than append it. #4054

wants to merge 2 commits into from

Conversation

omalley
Copy link
Contributor

@omalley omalley commented Mar 8, 2022

Description of PR

Makes RBF prepends the client ip & port to the caller context and removes previous values. This avoids a couple problems:

  • User can't fake their network address to the NN.
  • It is less likely to have false positives (accidental conflicts), since the critical information that is under system control comes first.

How was this patch tested?

The relevant unit tests were fixed and run.

@omalley omalley requested review from ferhui and ayushtkn March 8, 2022 23:21
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 50s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 36m 18s trunk passed
+1 💚 compile 0m 40s trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 0m 35s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 checkstyle 0m 23s trunk passed
+1 💚 mvnsite 0m 41s trunk passed
+1 💚 javadoc 0m 41s trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 0m 53s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 1m 19s trunk passed
+1 💚 shadedclient 24m 8s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 32s the patch passed
+1 💚 compile 0m 36s the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 0m 36s the patch passed
+1 💚 compile 0m 30s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 javac 0m 30s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 15s /results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚 mvnsite 0m 32s the patch passed
+1 💚 javadoc 0m 31s the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 0m 50s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
-1 ❌ spotbugs 1m 26s /new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 shadedclient 23m 27s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 50m 4s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 0m 43s The patch does not generate ASF License warnings.
147m 4s
Reason Tests
SpotBugs module:hadoop-hdfs-project/hadoop-hdfs-rbf
Possible null pointer dereference of null in org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.addClientIpToCallerContext() Dereferenced at RouterRpcClient.java:null in org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.addClientIpToCallerContext() Dereferenced at RouterRpcClient.java:[line 604]
Failed junit tests hadoop.fs.contract.router.TestRouterHDFSContractCreate
hadoop.fs.contract.router.web.TestRouterWebHDFSContractSeek
hadoop.hdfs.server.federation.metrics.TestNameserviceRPCMetrics
hadoop.fs.contract.router.TestRouterHDFSContractSetTimes
hadoop.hdfs.server.federation.router.TestRouterFederationRenamePermission
hadoop.hdfs.server.federation.router.TestRouterMountTable
hadoop.hdfs.server.federation.router.TestRouterFederationRenameInKerberosEnv
hadoop.hdfs.server.federation.router.TestRouterClientRejectOverload
hadoop.fs.contract.router.TestRouterHDFSContractConcatSecure
hadoop.hdfs.server.federation.router.TestRouterWebHdfsMethods
hadoop.hdfs.server.federation.router.TestRouterMultiRack
hadoop.hdfs.server.federation.router.TestRouterQuota
hadoop.fs.contract.router.TestRouterHDFSContractRootDirectorySecure
hadoop.hdfs.server.federation.router.TestRouterNamenodeMonitoring
hadoop.hdfs.server.federation.router.TestRouterRpcStoragePolicySatisfier
hadoop.hdfs.server.federation.router.TestRouterFederationRename
hadoop.fs.contract.router.TestRouterHDFSContractAppend
hadoop.fs.contract.router.TestRouterHDFSContractAppendSecure
hadoop.fs.contract.router.TestRouterHDFSContractCreateSecure
hadoop.hdfs.server.federation.fairness.TestRouterHandlersFairness
hadoop.fs.contract.router.TestRouterHDFSContractOpen
hadoop.hdfs.server.federation.router.TestRouterRPCClientRetries
hadoop.fs.contract.router.TestRouterHDFSContractOpenSecure
hadoop.fs.contract.router.TestRouterHDFSContractRootDirectory
hadoop.hdfs.server.federation.router.TestRouterMissingFolderMulti
hadoop.hdfs.server.federation.router.TestRouterAllResolver
hadoop.fs.contract.router.web.TestRouterWebHDFSContractMkdir
hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate
hadoop.hdfs.server.federation.router.TestRouterFaultTolerant
hadoop.fs.contract.router.TestRouterHDFSContractDeleteSecure
hadoop.fs.contract.router.TestRouterHDFSContractGetFileStatusSecure
hadoop.fs.contract.router.TestRouterHDFSContractGetFileStatus
hadoop.hdfs.rbfbalance.TestMountTableProcedure
hadoop.fs.contract.router.web.TestRouterWebHDFSContractDelete
hadoop.fs.contract.router.TestRouterHDFSContractRename
hadoop.hdfs.server.federation.router.TestRouterTrash
hadoop.fs.contract.router.TestRouterHDFSContractDelegationToken
hadoop.hdfs.server.federation.router.TestRouterFsck
hadoop.fs.contract.router.TestRouterHDFSContractRenameSecure
hadoop.hdfs.server.federation.router.TestDisableNameservices
hadoop.hdfs.server.federation.router.TestRouterAdmin
hadoop.fs.contract.router.TestRouterHDFSContractSeek
hadoop.fs.contract.router.TestRouterHDFSContractSeekSecure
hadoop.fs.contract.router.TestRouterHDFSContractMkdir
hadoop.fs.contract.router.web.TestRouterWebHDFSContractRename
hadoop.fs.contract.router.TestRouterHDFSContractConcat
hadoop.fs.contract.router.TestRouterHDFSContractSetTimesSecure
hadoop.fs.contract.router.web.TestRouterWebHDFSContractRootDirectory
hadoop.fs.contract.router.web.TestRouterWebHDFSContractAppend
hadoop.hdfs.server.federation.router.TestRouterNetworkTopologyServlet
hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure
hadoop.fs.contract.router.web.TestRouterWebHDFSContractConcat
hadoop.hdfs.server.federation.router.TestSafeMode
hadoop.fs.contract.router.TestRouterHDFSContractDelete
hadoop.hdfs.server.federation.router.TestRouterRpcSingleNS
hadoop.hdfs.server.federation.metrics.TestRouterClientMetrics
hadoop.fs.contract.router.web.TestRouterWebHDFSContractOpen
hadoop.fs.contract.router.TestRouterHDFSContractMkdirSecure
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4054/1/artifact/out/Dockerfile
GITHUB PR #4054
JIRA Issue HDFS-16495
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux a250790c75be 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e7c0c118a0fe28f9fe78e493f9b4d9e01454a975
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4054/1/testReport/
Max. process+thread count 3412 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4054/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 49s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 36m 20s trunk passed
+1 💚 compile 0m 41s trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 0m 36s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 checkstyle 0m 24s trunk passed
+1 💚 mvnsite 0m 40s trunk passed
+1 💚 javadoc 0m 41s trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 0m 56s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 1m 18s trunk passed
+1 💚 shadedclient 23m 51s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 32s the patch passed
+1 💚 compile 0m 36s the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 0m 36s the patch passed
+1 💚 compile 0m 30s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 javac 0m 30s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 15s the patch passed
+1 💚 mvnsite 0m 33s the patch passed
+1 💚 javadoc 0m 31s the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 0m 47s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 1m 21s the patch passed
+1 💚 shadedclient 23m 49s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 31m 43s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 0m 31s The patch does not generate ASF License warnings.
128m 32s
Reason Tests
Failed junit tests hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4054/2/artifact/out/Dockerfile
GITHUB PR #4054
JIRA Issue HDFS-16495
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 96bedbe2a1f0 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 31edfde
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4054/2/testReport/
Max. process+thread count 2292 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4054/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 52s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 13m 2s Maven dependency ordering for branch
+1 💚 mvninstall 28m 45s trunk passed
+1 💚 compile 24m 23s trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 20m 52s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 checkstyle 3m 50s trunk passed
+1 💚 mvnsite 2m 28s trunk passed
+1 💚 javadoc 1m 55s trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 2m 43s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 3m 55s trunk passed
+1 💚 shadedclient 24m 19s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 22s Maven dependency ordering for patch
+1 💚 mvninstall 1m 33s the patch passed
+1 💚 compile 23m 47s the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 23m 47s the patch passed
+1 💚 compile 20m 56s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 javac 20m 56s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 3m 46s the patch passed
+1 💚 mvnsite 2m 28s the patch passed
+1 💚 javadoc 1m 52s the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 2m 42s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 4m 17s the patch passed
+1 💚 shadedclient 24m 48s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 17m 47s hadoop-common in the patch passed.
-1 ❌ unit 32m 7s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 0m 52s The patch does not generate ASF License warnings.
266m 40s
Reason Tests
Failed junit tests hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4054/3/artifact/out/Dockerfile
GITHUB PR #4054
JIRA Issue HDFS-16495
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux b70e3394632d 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / f414315
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4054/3/testReport/
Max. process+thread count 3143 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4054/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

ayushtkn
ayushtkn approved these changes Mar 12, 2022
View reviewed changes
Copy link
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM.

@omalley omalley closed this in 7b5eac2 Mar 14, 2022
omalley added a commit that referenced this pull request Mar 14, 2022
@omalley
HDFS-16495: RBF should prepend the client ip rather than append it.
c52b97d 
Fixes #4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
@omalley
HDFS-16495: RBF should prepend the client ip rather than append it.
aa02845 
Fixes apache#4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
LiuGuH pushed a commit to LiuGuH/hadoop that referenced this pull request Mar 26, 2024
@omalley
HDFS-16495: RBF should prepend the client ip rather than append it.
b16fadf 
Fixes apache#4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
NyteKnight pushed a commit to NyteKnight/hadoop that referenced this pull request Jun 25, 2024
@omalley
HDFS-16495: RBF should prepend the client ip rather than append it.
1f3c5d4 
Fixes apache#4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>

RB=3230199
BUG=LIHADOOP-64054
G=storage-reviewers
R=hchaverr,sdzinama
A=sdzinama
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simbadzina simbadzina simbadzina left review comments

@hchaverri hchaverri hchaverri left review comments

@ayushtkn ayushtkn ayushtkn approved these changes

@ferhui ferhui Awaiting requested review from ferhui

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@omalley @hadoop-yetus @simbadzina @ayushtkn @hchaverri