Skip to content

[SPARK-45464][CORE] Fix network-yarn distribution build #43289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hasnain-db wants to merge 3 commits into from
Closed

[SPARK-45464][CORE] Fix network-yarn distribution build #43289

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
31b0e9b
working
hasnain-db Oct 8, 2023
f47370b
rework
hasnain-db Oct 9, 2023
c47bd23
keep license folder not file
hasnain-db Oct 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions common/network-yarn/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>META-INF/LICENSE</exclude>
Copy link
Contributor

@LuciferYang LuciferYang Oct 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

[WARNING] jackson-annotations-2.15.2.jar, jackson-core-2.15.2.jar, jackson-databind-2.15.2.jar, jackson-module-scala_2.13-2.15.2.jar, log4j-1.2-api-2.20.0.jar, log4j-api-2.20.0.jar, log4j-core-2.20.0.jar, log4j-slf4j2-impl-2.20.0.jar, spark-common-utils_2.13-4.0.0-SNAPSHOT.jar, spark-network-common_2.13-4.0.0-SNAPSHOT.jar, spark-network-shuffle_2.13-4.0.0-SNAPSHOT.jar, spark-network-yarn_2.13-4.0.0-SNAPSHOT.jar define 1 overlapping resource: 
[WARNING]   - META-INF/LICENSE

From the compile log, META-INF/LICENSE might come from multiple jars.

Since this exclude is indiscriminate, in this case, all META-INF/LICENSE would be excluded. If it includes a non-Apache LICENSE, it would also be excluded. Would this be a problem @srowen ?

Of course, even without the exclude, there would still be issues currently because multiple files with the same name would overwrite each other, and it seems difficult to determine which META-INF/LICENSE would be retained in the end.

Copy link
Member

@srowen srowen Oct 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we include the license if all of those already in the LICENSE file (or no action is required) so it seems OK here. While it's nice to retain the license buried in the artifact, we should be addressing all licenses in the top level LICENSE and license/ dir, rather than rely on this anyway

Copy link
Contributor

@LuciferYang LuciferYang Oct 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood, thank you for your explanation @srowen

</excludes>
</filter>
</filters>
Expand Down Expand Up @@ -183,6 +184,14 @@
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_aarch_64.so" />
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_aarch_64.jnilib" />
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_tcnative_linux_x86_64.so"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_tcnative_linux_x86_64.so" />
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_tcnative_linux_aarch_64.so"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_tcnative_linux_aarch_64.so" />
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_tcnative_osx_x86_64.jnilib"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_tcnative_osx_x86_64.jnilib" />
<move file="${project.build.directory}/exploded/META-INF/native/libnetty_tcnative_osx_aarch_64.jnilib"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_tcnative_osx_aarch_64.jnilib" />
<jar destfile="${shuffle.jar}" basedir="${project.build.directory}/exploded" />
</target>
</configuration>
Expand Down