Skip to content

HADOOP-16922. ABFS: Change User-Agent header #1938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Apr 21, 2020

Conversation

bilaharith
Copy link
Contributor

@bilaharith bilaharith commented Apr 3, 2020

HADOOP-16922. Changing User-Agent header.

  • Add more inforrmation to the User-Agent header like cluster name, cluster type, java vendor etc.
  • Add APN/1.0 in the begining

At present the User-Agent header would look like:
Azure Blob FS/3.4.0-SNAPSHOT (JavaJRE 1.8.0_241; Linux 5.3.0-46-generic; SunJSSE-1.8) MSFT

With this PR it would be:
APN/1.0 Azure Blob FS/3.4.0-SNAPSHOT (OracleCorporation JavaJRE 1.8.0_241; Linux 5.3.0-46-generic/amd64; SunJSSE-1.8; UNKNOWN/UNKNOWN) MSFT

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 6s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 21m 19s trunk passed
+1 💚 compile 0m 26s trunk passed
+1 💚 checkstyle 0m 20s trunk passed
+1 💚 mvnsite 0m 30s trunk passed
+1 💚 shadedclient 16m 59s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 22s trunk passed
+0 🆗 spotbugs 0m 50s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 0m 49s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 26s the patch passed
+1 💚 compile 0m 22s the patch passed
+1 💚 javac 0m 22s the patch passed
-0 ⚠️ checkstyle 0m 14s hadoop-tools/hadoop-azure: The patch generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5)
+1 💚 mvnsite 0m 25s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 6s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 21s the patch passed
+1 💚 findbugs 0m 52s the patch passed
_ Other Tests _
-1 ❌ unit 1m 9s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 28s The patch does not generate ASF License warnings.
62m 45s
Reason Tests
Failed junit tests hadoop.fs.azurebfs.services.TestAbfsClient
Subsystem Report/Notes
Docker Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/1/artifact/out/Dockerfile
GITHUB PR #1938
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux f70d7455ace7 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / ff50ecd
Default Java 1.8.0_242
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/1/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/1/testReport/
Max. process+thread count 308 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 45s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 19m 41s trunk passed
+1 💚 compile 0m 31s trunk passed
+1 💚 checkstyle 0m 25s trunk passed
+1 💚 mvnsite 0m 37s trunk passed
+1 💚 shadedclient 15m 20s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 27s trunk passed
+0 🆗 spotbugs 0m 55s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 0m 53s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 28s the patch passed
+1 💚 compile 0m 23s the patch passed
+1 💚 javac 0m 23s the patch passed
-0 ⚠️ checkstyle 0m 16s hadoop-tools/hadoop-azure: The patch generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5)
+1 💚 mvnsite 0m 28s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 13m 51s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 23s the patch passed
+1 💚 findbugs 0m 54s the patch passed
_ Other Tests _
+1 💚 unit 1m 24s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 32s The patch does not generate ASF License warnings.
58m 55s
Subsystem Report/Notes
Docker Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/2/artifact/out/Dockerfile
GITHUB PR #1938
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux bc07c17c1887 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 1695d8d
Default Java 1.8.0_242
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/2/testReport/
Max. process+thread count 482 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@bilaharith
Copy link
Contributor Author

Driver test results using accounts in Central India
mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Account with HNS Support
[INFO] Tests run: 54, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 416, Failures: 0, Errors: 0, Skipped: 66
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24

Account without HNS support
[INFO] Tests run: 54, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 416, Failures: 0, Errors: 0, Skipped: 240
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24

Copy link
Contributor

@DadanielZ DadanielZ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall it looks good to me, but plz fix the checkstyle issues reported in the Yetus.
Could you also provide some background why this change is needed and will it affect the existing customers?

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 2m 15s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 19m 59s trunk passed
+1 💚 compile 0m 31s trunk passed
+1 💚 checkstyle 0m 25s trunk passed
+1 💚 mvnsite 0m 35s trunk passed
+1 💚 shadedclient 15m 7s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 28s trunk passed
+0 🆗 spotbugs 0m 53s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 0m 49s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 29s the patch passed
+1 💚 compile 0m 24s the patch passed
+1 💚 javac 0m 24s the patch passed
-0 ⚠️ checkstyle 0m 17s hadoop-tools/hadoop-azure: The patch generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5)
+1 💚 mvnsite 0m 28s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 13m 56s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 23s the patch passed
+1 💚 findbugs 0m 53s the patch passed
_ Other Tests _
+1 💚 unit 1m 23s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 33s The patch does not generate ASF License warnings.
60m 31s
Subsystem Report/Notes
Docker Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/3/artifact/out/Dockerfile
GITHUB PR #1938
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 722df17dc024 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / e6455cc
Default Java 1.8.0_242
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/3/testReport/
Max. process+thread count 413 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/3/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 35s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 22m 40s trunk passed
+1 💚 compile 0m 32s trunk passed
+1 💚 checkstyle 0m 24s trunk passed
+1 💚 mvnsite 0m 34s trunk passed
+1 💚 shadedclient 15m 16s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 24s trunk passed
+0 🆗 spotbugs 0m 52s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 0m 50s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 28s the patch passed
+1 💚 compile 0m 24s the patch passed
+1 💚 javac 0m 24s the patch passed
-0 ⚠️ checkstyle 0m 16s hadoop-tools/hadoop-azure: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5)
+1 💚 mvnsite 0m 27s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 13m 57s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 24s the patch passed
+1 💚 findbugs 1m 7s the patch passed
_ Other Tests _
+1 💚 unit 1m 28s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 35s The patch does not generate ASF License warnings.
61m 55s
Subsystem Report/Notes
Docker Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/4/artifact/out/Dockerfile
GITHUB PR #1938
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux b70f37ac97ce 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / e6455cc
Default Java 1.8.0_242
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/4/testReport/
Max. process+thread count 420 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/4/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@steveloughran steveloughran changed the title HADOOP-16922. Changing User-Agent header HADOOP-16922. abfs: Changing User-Agent header Apr 6, 2020
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • This could be pulled out of AbfsClient or made package-protected static and so be easier to test
  • Is there a regexp somewhere of a well-formed HTTP UA header must look like? If so, test should verify it.

@@ -679,32 +679,60 @@ public AuthType getAuthType() {

@VisibleForTesting
String initializeUserAgent(final AbfsConfiguration abfsConfiguration,
final String sslProviderName) {
final String sslProviderName) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you pulled this out somewhere or made static you could have a unit test which would validate some of this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are test cases written to validate this as this method is package protected and with the @VisibleForTesting.

StringBuilder sb = new StringBuilder();
sb.append("(JavaJRE ");

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have org.apache.hadoop.util.VersionInfo,getVersion() in there too? Lets people pin the blame on the logs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CLIENT_VERSION includes org.apache.hadoop.util.VersionInfo,getVersion()

Copy link
Contributor Author

@bilaharith bilaharith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review comments addressed

@bilaharith bilaharith changed the title HADOOP-16922. abfs: Changing User-Agent header HADOOP-16922. ABFS: Changing User-Agent header Apr 8, 2020
@bilaharith
Copy link
Contributor Author

Driver test results using accounts in Central India
mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Account with HNS Support
[INFO] Tests run: 54, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 416, Failures: 0, Errors: 0, Skipped: 66
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24

Account without HNS support
[INFO] Tests run: 54, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 416, Failures: 0, Errors: 0, Skipped: 240
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 10s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 21m 59s trunk passed
+1 💚 compile 0m 29s trunk passed
+1 💚 checkstyle 0m 21s trunk passed
+1 💚 mvnsite 0m 32s trunk passed
+1 💚 shadedclient 16m 16s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 22s trunk passed
+0 🆗 spotbugs 0m 50s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 0m 47s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 25s the patch passed
+1 💚 compile 0m 22s the patch passed
+1 💚 javac 0m 22s the patch passed
+1 💚 checkstyle 0m 15s the patch passed
+1 💚 mvnsite 0m 25s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 15s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 20s the patch passed
+1 💚 findbugs 0m 54s the patch passed
_ Other Tests _
+1 💚 unit 1m 21s hadoop-azure in the patch passed.
+1 💚 asflicense 0m 27s The patch does not generate ASF License warnings.
63m 10s
Subsystem Report/Notes
Docker Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/5/artifact/out/Dockerfile
GITHUB PR #1938
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux cddd5a112ae7 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 11d1741
Default Java 1.8.0_242
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/5/testReport/
Max. process+thread count 307 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1938/5/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@DadanielZ DadanielZ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, +1
@steveloughran do you have pending request for changes?

@steveloughran
Copy link
Contributor

@DadanielZ I'm happy if you are.

@bilaharith
Copy link
Contributor Author

At present the User-Agent header would look like:
Azure Blob FS/3.4.0-SNAPSHOT (JavaJRE 1.8.0_241; Linux 5.3.0-46-generic; SunJSSE-1.8) MSFT

With this PR it would be:
APN/1.0 Azure Blob FS/3.4.0-SNAPSHOT (OracleCorporation JavaJRE 1.8.0_241; Linux 5.3.0-46-generic/amd64; SunJSSE-1.8; UNKNOWN/UNKNOWN) MSFT

@steveloughran steveloughran changed the title HADOOP-16922. ABFS: Changing User-Agent header HADOOP-16922. ABFS: Change User-Agent header Apr 21, 2020
@steveloughran steveloughran merged commit 264e49c into apache:trunk Apr 21, 2020
asfgit pushed a commit that referenced this pull request Apr 22, 2020
arjun4084346 pushed a commit to arjun4084346/hadoop that referenced this pull request Apr 11, 2022
Contributed by Bilahari T H.

(cherry picked from commit 264e49c)
zhangxiping1 pushed a commit to zhangxiping1/hadoop that referenced this pull request Dec 13, 2022
jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023
Contributed by Bilahari T H.
 Conflicts:
	hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java

Change-Id: Icc968be7234dcf93f352280c056d645a9439c459
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants