Skip to content

Commit

Permalink
[KYUUBI #6297] Package Spark SQL engine both Scala 2.12 and 2.13
Browse files Browse the repository at this point in the history
# 🔍 Description
## Issue References 🔗

This pull request fixes #6297

## Describe Your Solution 🔧

Here are my main modifications:
1. In the `build/dist` file, I added an additional packaging command that detects the version of `SCALA_VERSION`. If it is version 2.12, then it will additionally package version 2.13, otherwise it will package version 2.12
2. In the section of `# Copy spark engines`, the Jar packages of versions 2.12 and 2.13 will be moved to `$DISTDIR/externals/engines/spark/`

This is my first time submitting a PR, so there may be many unfamiliar areas, and I don't know if my changes meet the needs of the community. More guidance may be needed. When testing locally, it was able to be packaged successfully

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

![image](https://github.com/apache/kyuubi/assets/96274454/fb1981e0-dc3f-47a3-a497-2ae0d3161e6b)

![image](https://github.com/apache/kyuubi/assets/96274454/b7307738-7699-4c2c-b6ad-08b323539d90)

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6305 from PorterZhang2021/issue-6297.

Closes #6297

efc8643 [PorterZhang2021] Improved task "Package Spark SQL engine both Scala 2.12 and 2.13" #6297
cb9310b [PorterZhang2021] Improved task "Package Spark SQL engine both Scala 2.12 and 2.13" #6297
b6c9075 [PorterZhang2021] Finished task "Package Spark SQL engine both Scala 2.12 and 2.13" #6297

Authored-by: PorterZhang2021 <porterzhang@aliyun.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
  • Loading branch information
PorterZhang2021 authored and pan3793 committed Apr 15, 2024
1 parent 968fa09 commit 9086cfe
Showing 1 changed file with 14 additions and 1 deletion.
15 changes: 14 additions & 1 deletion build/dist
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,18 @@ echo -e "\nBuilding with..."
# shellcheck disable=SC2145
echo -e "\$ ${BUILD_COMMAND[@]}\n"

# shellcheck disable=SC2050
if [ "$SCALA_VERSION" = "2.12" ]; then
EXTRA_SPARK_ENGINE_BUILD_COMMAND=("$MVN" install $MVN_DIST_OPT $@ -Pscala-2.13 -pl :kyuubi-spark-sql-engine_2.13 -am)
else
EXTRA_SPARK_ENGINE_BUILD_COMMAND=("$MVN" install $MVN_DIST_OPT $@ -Pscala-2.12 -pl :kyuubi-spark-sql-engine_2.12 -am)
fi

# shellcheck disable=SC2145
echo -e "\$ ${EXTRA_SPARK_ENGINE_BUILD_COMMAND[@]}\n"

"${BUILD_COMMAND[@]}"
"${EXTRA_SPARK_ENGINE_BUILD_COMMAND[@]}"

# Make directories
rm -rf "$DISTDIR"
Expand Down Expand Up @@ -296,7 +307,9 @@ done
cp "$KYUUBI_HOME/externals/kyuubi-flink-sql-engine/target/kyuubi-flink-sql-engine_${SCALA_VERSION}-${VERSION}.jar" "$DISTDIR/externals/engines/flink/"

# Copy spark engines
cp "$KYUUBI_HOME/externals/kyuubi-spark-sql-engine/target/kyuubi-spark-sql-engine_${SCALA_VERSION}-${VERSION}.jar" "$DISTDIR/externals/engines/spark/"
for scala_version in 2.12 2.13; do
cp "$KYUUBI_HOME/externals/kyuubi-spark-sql-engine/target/kyuubi-spark-sql-engine_${scala_version}-${VERSION}.jar" "$DISTDIR/externals/engines/spark/"
done

# Copy trino engines
cp "$KYUUBI_HOME/externals/kyuubi-trino-engine/target/kyuubi-trino-engine_${SCALA_VERSION}-${VERSION}.jar" "$DISTDIR/externals/engines/trino/"
Expand Down

0 comments on commit 9086cfe

Please sign in to comment.