Skip to content

[WIP][SPARK-29250][BUILD][test-hadoop3.2][test-maven] Upgrade to Hadoop 3.2.1 #25932

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

[WIP][SPARK-29250][BUILD][test-hadoop3.2][test-maven] Upgrade to Hadoop 3.2.1 #25932

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 18 additions & 16 deletions dev/deps/spark-deps-hadoop-3.2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ automaton-1.11-8.jar
avro-1.8.2.jar
avro-ipc-1.8.2.jar
avro-mapred-1.8.2-hadoop2.jar
bcpkix-jdk15on-1.60.jar
bcprov-jdk15on-1.60.jar
Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Sep 25, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, we already have the following in NOTICE-binary. We may need to remove the word optionally, or to exclude these two like before.

This product optionally depends on 'Bouncy Castle Crypto APIs' to generate
a temporary self-signed X.509 certificate when the JVM does not provide the
equivalent functionality.  It can be obtained at:

  * LICENSE:
    * license/LICENSE.bouncycastle.txt (MIT License)
  * HOMEPAGE:
    * http://www.bouncycastle.org/

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@srowen . Could you give me some advice?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few parts here. First, that NOTICE statement is from Hadoop's NOTICE, so I'd copy whatever it says now, to update.

Second, if it's a first-class dependency now, it needs to have a line in LICENSE-binary and a copy of the license in licenses-binary/. It's MIT-licensed so should be OK.

Finally, BC is a special case because it's subject to crypto export laws. We will have to update http://www.apache.org/licenses/exports/ to say that it's again a dependency in 3.0. I can go figure that out again as and when this is merged.

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Sep 26, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops. It sounds too much for me. Please help me after I merge this. 😄

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, the process for the last part is https://www.apache.org/dev/crypto.html#sources
I can do it afterwards, it's not hard.

However, hm, I wonder if Hadoop needs a similar disclosure at http://www.apache.org/licenses/exports/ ? It's possible that somehow it isn't distributed directly by Hadoop, but would be surprised if it's a first-class dep and it makes binary releases.

Maybe, eh, CC @steveloughran in case he knows anything about this angle.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ooh, it is in the binary isn't it

hadoop-3.2.1 find . -print | grep bcp
./share/hadoop/yarn/lib/bcprov-jdk15on-1.60.jar
./share/hadoop/yarn/lib/bcpkix-jdk15on-1.60.jar

let me chase this up

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @srowen and @steveloughran .
I will use this chance to learn this legal process. :)

bonecp-0.8.0.RELEASE.jar
breeze-macros_2.12-1.0.jar
breeze_2.12-1.0.jar
Expand Down Expand Up @@ -65,20 +67,20 @@ gson-2.2.4.jar
guava-14.0.1.jar
guice-4.0.jar
guice-servlet-4.0.jar
hadoop-annotations-3.2.0.jar
hadoop-auth-3.2.0.jar
hadoop-client-3.2.0.jar
hadoop-common-3.2.0.jar
hadoop-hdfs-client-3.2.0.jar
hadoop-mapreduce-client-common-3.2.0.jar
hadoop-mapreduce-client-core-3.2.0.jar
hadoop-mapreduce-client-jobclient-3.2.0.jar
hadoop-yarn-api-3.2.0.jar
hadoop-yarn-client-3.2.0.jar
hadoop-yarn-common-3.2.0.jar
hadoop-yarn-registry-3.2.0.jar
hadoop-yarn-server-common-3.2.0.jar
hadoop-yarn-server-web-proxy-3.2.0.jar
hadoop-annotations-3.2.1.jar
hadoop-auth-3.2.1.jar
hadoop-client-3.2.1.jar
hadoop-common-3.2.1.jar
hadoop-hdfs-client-3.2.1.jar
hadoop-mapreduce-client-common-3.2.1.jar
hadoop-mapreduce-client-core-3.2.1.jar
hadoop-mapreduce-client-jobclient-3.2.1.jar
hadoop-yarn-api-3.2.1.jar
hadoop-yarn-client-3.2.1.jar
hadoop-yarn-common-3.2.1.jar
hadoop-yarn-registry-3.2.1.jar
hadoop-yarn-server-common-3.2.1.jar
hadoop-yarn-server-web-proxy-3.2.1.jar
hive-storage-api-2.6.0.jar
hk2-api-2.5.0.jar
hk2-locator-2.5.0.jar
Expand All @@ -94,8 +96,8 @@ jackson-core-2.9.10.jar
jackson-core-asl-1.9.13.jar
jackson-databind-2.9.10.jar
jackson-dataformat-yaml-2.9.10.jar
jackson-jaxrs-base-2.9.5.jar
jackson-jaxrs-json-provider-2.9.5.jar
jackson-jaxrs-base-2.9.8.jar
jackson-jaxrs-json-provider-2.9.8.jar
jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations-2.9.10.jar
jackson-module-paranamer-2.9.10.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2977,7 +2977,7 @@
<profile>
<id>hadoop-3.2</id>
<properties>
<hadoop.version>3.2.0</hadoop.version>
<hadoop.version>3.2.1</hadoop.version>
<curator.version>2.13.0</curator.version>
<zookeeper.version>3.4.13</zookeeper.version>
<hive.group>org.apache.hive</hive.group>
Expand Down