Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TASK][EASY] Package Spark SQL engine both Scala 2.12 and 2.13 #6297

Closed
2 of 3 tasks
pan3793 opened this issue Apr 11, 2024 · 2 comments
Closed
2 of 3 tasks

[TASK][EASY] Package Spark SQL engine both Scala 2.12 and 2.13 #6297

pan3793 opened this issue Apr 11, 2024 · 2 comments
Assignees
@PorterZhang2021

Comments

@pan3793
Copy link
Member

pan3793 commented Apr 11, 2024 •
edited
Loading

What's the level of this task?

EASY

Code of Conduct

Search before creating

  • I have searched in the task list and found no similar tasks.

Mentor

  • I have sufficient expertise on this task, and I volunteer to be a mentor of this task to guide contributors through the task.

Skill requirements

  • Shell scripts, Maven packaging
  • knowledge of how Kyuubi launches Spark applications

Background and Goals

The Kyuubi Spark SQL engine is a version-agnostic jar, which means the kyuubi-spark-sql-engine_2.12-1.9.0.jar built with Spark 3.5 should also work on Spark 3.2 and all supported versions.
Since 1.8, the whole Kyuubi project has supported building with Scala 2.13, so users could create a custom distribution with Scala 2.13. Another idea is, that one Kyuubi tgz ships both kyuubi-spark-sql-engine_2.12-1.9.0.jar and kyuubi-spark-sql-engine_2.13-1.9.0.jar, then it can support Scala 2.12 and 2.13 in same time.

Implementation steps

Kyuubi Server already supports extracting Spark's scala version from $SPARK_HOME/jars/spark-core_*.jar and choosing the right kyuubi-spark-sql-engine_*.jar, so we need to modify dev/dist to make the final binary tarball always(no matter the Kyuubi server compiled with Scala 2.12 or 2.13) ships

externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.10.0-SNAPSHOT.jar
externals/engines/spark/kyuubi-spark-sql-engine_2.13-1.10.0-SNAPSHOT.jar

Additional context

Introduction of 2024H1 Kyuubi Code Contribution Program
@PorterZhang2021
Copy link
Contributor

Please assign this task to me, I will try to solve it.

@pan3793
Copy link
Member Author

pan3793 commented Apr 12, 2024

@PorterZhang2021 thanks, assigned

PorterZhang2021 pushed a commit to PorterZhang2021/kyuubi that referenced this issue Apr 13, 2024
Finished task "Package Spark SQL engine both Scala 2.12 and 2.13" apa…
b6c9075 
…che#6297
@PorterZhang2021 PorterZhang2021 mentioned this issue Apr 13, 2024
Closed
4 tasks
PorterZhang2021 pushed a commit to PorterZhang2021/kyuubi that referenced this issue Apr 14, 2024
Improved task "Package Spark SQL engine both Scala 2.12 and 2.13" apa…
cb9310b 
…che#6297
PorterZhang2021 pushed a commit to PorterZhang2021/kyuubi that referenced this issue Apr 14, 2024
Improved task "Package Spark SQL engine both Scala 2.12 and 2.13" apa…
efc8643 
…che#6297
pan3793 pushed a commit that referenced this issue May 20, 2024
@pan3793
[KYUUBI #6297] Package Spark SQL engine both Scala 2.12 and 2.13
43ec939 
# 🔍 Description
## Issue References 🔗

This pull request fixes #6297

## Describe Your Solution 🔧

Here are my main modifications:
1. In the `build/dist` file, I added an additional packaging command that detects the version of `SCALA_VERSION`. If it is version 2.12, then it will additionally package version 2.13, otherwise it will package version 2.12
2. In the section of `# Copy spark engines`, the Jar packages of versions 2.12 and 2.13 will be moved to `$DISTDIR/externals/engines/spark/`

This is my first time submitting a PR, so there may be many unfamiliar areas, and I don't know if my changes meet the needs of the community. More guidance may be needed. When testing locally, it was able to be packaged successfully

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

![image](https://github.com/apache/kyuubi/assets/96274454/fb1981e0-dc3f-47a3-a497-2ae0d3161e6b)

![image](https://github.com/apache/kyuubi/assets/96274454/b7307738-7699-4c2c-b6ad-08b323539d90)

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6305 from PorterZhang2021/issue-6297.

Closes #6297

efc8643 [PorterZhang2021] Improved task "Package Spark SQL engine both Scala 2.12 and 2.13" #6297
cb9310b [PorterZhang2021] Improved task "Package Spark SQL engine both Scala 2.12 and 2.13" #6297
b6c9075 [PorterZhang2021] Finished task "Package Spark SQL engine both Scala 2.12 and 2.13" #6297

Authored-by: PorterZhang2021 <porterzhang@aliyun.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@PorterZhang2021 PorterZhang2021

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[KYUUBI #6297] Package Spark SQL engine both Scala 2.12 and 2.13 PorterZhang2021/kyuubi
2 participants
@pan3793 @PorterZhang2021