Skip to content

[SPARK-45376][CORE] Add netty-tcnative-boringssl-static dependency #43164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hasnain-db wants to merge 1 commit into from
Closed

[SPARK-45376][CORE] Add netty-tcnative-boringssl-static dependency #43164

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 33 additions & 1 deletion common/network-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,26 @@
<artifactId>netty-transport-native-kqueue</artifactId>
<classifier>osx-x86_64</classifier>
</dependency>
<dependency>
Copy link
Contributor

@LuciferYang LuciferYang Sep 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the network-yarn module have a requirement to use these native codes?

Copy link
Contributor Author

@hasnain-db hasnain-db Sep 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe it needs a direct dependency here if you look at the code changes to resource-managers/yarn in #42685 - we do add tests to the yarn code that do pass.

We do change ShuffleTransportContext which is in network-common, and that's what is used by yarn here. network-yarn depends on network-shuffle which depends on network-common so I think that's how the dependency gets pulled in.

Copy link
Member

@srowen srowen Sep 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These need to be test scope right?

Copy link
Contributor Author

@hasnain-db hasnain-db Sep 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@srowen the bouncycastle and network-common-helper dependencies are only needed in the test scope, but the netty-tcnative-boringssl-static dependencies need to be available in the production build. We need this dependency in order to provide SSL support.

<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<classifier>linux-x86_64</classifier>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<classifier>linux-aarch_64</classifier>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<classifier>osx-aarch_64</classifier>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<classifier>osx-x86_64</classifier>
</dependency>
<!-- Netty End -->

<dependency>
Expand Down Expand Up @@ -141,12 +161,24 @@
<artifactId>log4j-slf4j2-impl</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk15on</artifactId>
<version>${bouncycastle.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcpkix-jdk15on</artifactId>
<version>${bouncycastle.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-common-utils_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependency>

<!--
This spark-tags test-dep is needed even though it isn't used in this module, otherwise testing-cmds that exclude
Expand Down
7 changes: 7 additions & 0 deletions core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -446,6 +446,13 @@
<classifier>tests</classifier>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-network-common_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-network-shuffle_${scala.binary.version}</artifactId>
Expand Down
6 changes: 6 additions & 0 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,12 @@ netty-common/4.1.99.Final//netty-common-4.1.99.Final.jar
netty-handler-proxy/4.1.99.Final//netty-handler-proxy-4.1.99.Final.jar
netty-handler/4.1.99.Final//netty-handler-4.1.99.Final.jar
netty-resolver/4.1.99.Final//netty-resolver-4.1.99.Final.jar
netty-tcnative-boringssl-static/2.0.61.Final/linux-aarch_64/netty-tcnative-boringssl-static-2.0.61.Final-linux-aarch_64.jar
netty-tcnative-boringssl-static/2.0.61.Final/linux-x86_64/netty-tcnative-boringssl-static-2.0.61.Final-linux-x86_64.jar
netty-tcnative-boringssl-static/2.0.61.Final/osx-aarch_64/netty-tcnative-boringssl-static-2.0.61.Final-osx-aarch_64.jar
netty-tcnative-boringssl-static/2.0.61.Final/osx-x86_64/netty-tcnative-boringssl-static-2.0.61.Final-osx-x86_64.jar
netty-tcnative-boringssl-static/2.0.61.Final/windows-x86_64/netty-tcnative-boringssl-static-2.0.61.Final-windows-x86_64.jar
netty-tcnative-classes/2.0.61.Final//netty-tcnative-classes-2.0.61.Final.jar
netty-transport-classes-epoll/4.1.99.Final//netty-transport-classes-epoll-4.1.99.Final.jar
netty-transport-classes-kqueue/4.1.99.Final//netty-transport-classes-kqueue-4.1.99.Final.jar
netty-transport-native-epoll/4.1.99.Final/linux-aarch_64/netty-transport-native-epoll-4.1.99.Final-linux-aarch_64.jar
Expand Down
25 changes: 25 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,7 @@
<bouncycastle.version>1.70</bouncycastle.version>
<tink.version>1.9.0</tink.version>
<netty.version>4.1.99.Final</netty.version>
<netty-tcnative.version>2.0.61.Final</netty-tcnative.version>
<!--
If you are changing Arrow version specification, please check
./python/pyspark/sql/pandas/utils.py, and ./python/setup.py too.
Expand Down Expand Up @@ -928,6 +929,30 @@
<version>${netty.version}</version>
<classifier>osx-x86_64</classifier>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
Copy link
Member

@srowen srowen Sep 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the license of this code? if it's the same as netty, I think we're all good. Otherwise need to check it and possibly update LICENSE and licenses/

Copy link
Contributor Author

@hasnain-db hasnain-db Sep 28, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@srowen netty-tcnative-boringssl-static seems to be apache2 licensed: https://github.com/netty/netty-tcnative/blob/main/LICENSE.txt

which is the same as netty as far as I can tell: https://github.com/netty/netty/blob/4.1/LICENSE.txt

<version>${netty-tcnative.version}</version>
<classifier>linux-x86_64</classifier>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>${netty-tcnative.version}</version>
<classifier>linux-aarch_64</classifier>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>${netty-tcnative.version}</version>
<classifier>osx-aarch_64</classifier>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>${netty-tcnative.version}</version>
<classifier>osx-x86_64</classifier>
</dependency>
<!-- Netty End -->

<dependency>
Expand Down
7 changes: 7 additions & 0 deletions resource-managers/yarn/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,13 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-network-common_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-api</artifactId>
Expand Down