[KYUUBI #6411] Grpc common support #6412

davidyuan1223 · 2024-05-23T03:24:42Z

🔍 Description

Issue References 🔗

This pull request fixes #6411

Describe Your Solution 🔧

Add a common module common-grpc to support grpc type fronendservice/backendservice, which can compitable with another grpc engine, not only spark connect

Types of changes 🔖

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

Checklist 📝

This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

1. "$@" is a array, we want use string to compare. so update "$@" => "$*" 2. `tty` mean execute the command, we can use $(tty) replace it 3. param $# is a number, compare number should use -gt/-lt,not >/<

1. "$@" is a array, we want use string to compare. so update "$@" => "$*" 2. `tty` mean execute the command, we can use $(tty) replace it 3. param $# is a number, compare number should use -gt/-lt,not >/< 4. not sure the /bin/kyuubi line 63 'exit -1' need modify? so the directory bin only have a shellcheck note in /bin/kyuubi

# Conflicts: # bin/docker-image-tool.sh

Fix apache#4057

fix_4186

Revert "fix_4186"

fix_4186

# Conflicts: # externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala # externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkConsoleProgressBar.scala

…ommon-support # Conflicts: # kyuubi-grpc/pom.xml

davidyuan1223 · 2024-05-23T11:32:05Z

@wForget @yaooqinn @ulysses-you This pr is to add common-grpc module based on #6365, i add a simple test case with two simple grpc request/response, what do you think this.

codecov-commenter · 2024-05-23T12:38:21Z

Codecov Report

Attention: Patch coverage is 51.93199% with 311 lines in your changes are missing coverage. Please review.

Project coverage is 58.34%. Comparing base (47a1091) to head (c6af8ed).
Report is 207 commits behind head on master.

Files	Patch %	Lines
...rg/apache/spark/ui/JakartaHttpServletRequest.scala	0.00%	71 Missing ⚠️
.../org/apache/spark/ui/JavaxHttpServletRequest.scala	7.04%	66 Missing ⚠️
...ubi/plugin/spark/authz/serde/tableExtractors.scala	76.28%	20 Missing and 3 partials ⚠️
...g/apache/kyuubi/engine/spark/KyuubiSparkUtil.scala	0.00%	21 Missing ⚠️
.../main/scala/org/apache/spark/ui/SparkUIUtils.scala	62.74%	17 Missing and 2 partials ⚠️
...g/apache/spark/sql/kyuubi/SparkDatasetHelper.scala	56.09%	17 Missing and 1 partial ⚠️
...src/main/scala/org/apache/spark/ui/EngineTab.scala	69.38%	14 Missing and 1 partial ⚠️
...rg/apache/kyuubi/engine/spark/SparkSQLEngine.scala	55.55%	7 Missing and 5 partials ⚠️
...uubi/engine/spark/operation/ExecuteStatement.scala	16.66%	10 Missing ⚠️
.../engine/spark/session/SparkSQLSessionManager.scala	23.07%	9 Missing and 1 partial ⚠️
... and 11 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #6412      +/-   ##
============================================
- Coverage     61.19%   58.34%   -2.86%     
- Complexity       23       24       +1     
============================================
  Files           623      656      +33     
  Lines         37060    40283    +3223     
  Branches       5024     5498     +474     
============================================
+ Hits          22680    23504     +824     
- Misses        11948    14271    +2323     
- Partials       2432     2508      +76

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ulysses-you · 2024-05-24T09:10:40Z

Thank you @davidyuan1223 , I think we should make the first commit goal clear. Custom grpc things do not help us support Spark connect (the key make ExecutePlanRequest to be compatible with Kyuubi ?). I think we can add a test: use Spark connect client to query/cancel on a Kyuubi grpc server.

davidyuan1223 · 2024-05-24T09:15:44Z

Thank you @davidyuan1223 , I think we should make the first commit goal clear. Custom grpc things do not help us support Spark connect (the key make ExecutePlanRequest to be compatible with Kyuubi ?). I think we can add a test: use Spark connect client to query/cancel on a Kyuubi grpc server.

You are right, I will add a test case to test spark connect client with kyuubi server

davidyuan1223 · 2024-07-23T16:17:39Z

i'm trying to build a simple case to test spark grpc, but it always report Error

*** RUN ABORTED ***
  java.lang.NoClassDefFoundError: org/sparkproject/guava/util/concurrent/internal/InternalFutureFailureAccess
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
  at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
  at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
  ...
  Cause: java.lang.ClassNotFoundException: org.sparkproject.guava.util.concurrent.internal.InternalFutureFailureAccess
  at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
  at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
  at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:369)

I have tried to solve this issue many times by searching for a solution on https://issues.apache.org/jira/browse/SPARK-45201, but it seems futile. Can we focus on discussing the structure instead? We can then use the new module kyuubi-grpc-spark to test this. This bug is quite challenging for me.
@yaooqinn @ulysses-you

ulysses-you · 2024-07-24T01:53:31Z

@davidyuan1223 thank you for the work. Do you require this patch apache/spark#45775 ?

This issue is due to connect required a higher guava version which is different with Spark core itself, but connect depends on the Spark core's shaded guava.

I think this issue has been fixed in Spark3.5.2 which is on RCing, we can use the 3.5.2 rc version to verify it and change to use 3.5.2 after released.

davidyuan1223 · 2024-07-24T02:12:57Z

@davidyuan1223 thank you for the work. Do you require this patch apache/spark#45775 ?

This issue is due to connect required a higher guava version which is different with Spark core itself, but connect depends on the Spark core's shaded guava.

I think this issue has been fixed in Spark3.5.2 which is on RCing, we can use the 3.5.2 rc version to verify it and change to use 3.5.2 after released.

thanks, i will try! The spark-connect package has troubled me for a long time. Its proto classes conflict with connect-common, forcing me to shade it. Then I encountered a problem with Guava, and despite trying many methods, it still didn't work.

ulysses-you · 2024-07-24T02:22:05Z

thank you @davidyuan1223 , if you find any other issues related to Spark, just craete PR to Spark community. We can help review and push it.

davidyuan1223 and others added 23 commits January 13, 2023 16:40

fix-apache#4057 info:

dd39efd

1. "$@" is a array, we want use string to compare. so update "$@" => "$*" 2. `tty` mean execute the command, we can use $(tty) replace it 3. param $# is a number, compare number should use -gt/-lt,not >/<

Merge remote-tracking branch 'origin/fix-#4057' into fix-apache#4057

cb11935

# Conflicts: # bin/docker-image-tool.sh

Merge pull request #10 from xiaoyuandajian/fix-#4057

55a0a43

Fix apache#4057

remove the used blank lines

c48ad38

Merge branch 'apache:master' into master

16237c2

Merge branch 'apache:master' into master

bfa6cbf

Merge branch 'apache:master' into master

e244029

fix_4186

b616044

Merge branch 'master' into fix_4186

360d183

Merge pull request #11 from davidyuan1223/fix_4186

c83836b

fix_4186

Revert "fix_4186"

40e80d9

Merge pull request #12 from davidyuan1223/revert-11-fix_4186

0925a4b

Revert "fix_4186"

Merge branch 'apache:master' into master

2beccb6

Merge branch 'apache:master' into master

c8eb9a2

fix_4186

56b91a3

Merge pull request #13 from davidyuan1223/fix

57ec746

fix_4186

Merge branch 'apache:master' into master

72e7aea

Merge remote-tracking branch 'origin/master'

bcb0cf3

# Conflicts: # externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala # externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkConsoleProgressBar.scala

add common method to get session level config

8b51840

Merge branch 'apache:master' into master

6376466

Merge branch 'apache:master' into master

46001d4

add new module common-grpc

3d6c53b

github-actions bot added module:common kind:build labels May 23, 2024

davidyuan1223 and others added 5 commits May 23, 2024 11:27

fix_4186

e16fd7f

add common method to get session level config

a3a4ebd

add new module common-grpc

631ac9a

fork master new version

0a52c2a

Merge remote-tracking branch 'origin/grpc-common-support' into grpc-c…

d92fdb6

…ommon-support # Conflicts: # kyuubi-grpc/pom.xml

davidyuan1223 added 3 commits May 23, 2024 11:33

fix format error

4c5a4ae

fix format error

3e302aa

update settings.md

c6af8ed

github-actions bot added the kind:documentation Documentation is a feature! label May 23, 2024

davidyuan1223 and others added 6 commits May 28, 2024 16:32

update

a1e0447

update

5bf1814

update

e031c8c

update

55cbae3

update

0c17121

add new module kyuubi-grpc

2b959f9

tigrulya-exe mentioned this pull request Aug 15, 2024

[TASK][CHALLENGE] Support Spark Connect Frontend/Backend #5383

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[KYUUBI #6411] Grpc common support #6412

[KYUUBI #6411] Grpc common support #6412

davidyuan1223 commented May 23, 2024 •

edited

Loading

davidyuan1223 commented May 23, 2024

codecov-commenter commented May 23, 2024

ulysses-you commented May 24, 2024

davidyuan1223 commented May 24, 2024

davidyuan1223 commented Jul 23, 2024 •

edited

Loading

ulysses-you commented Jul 24, 2024

davidyuan1223 commented Jul 24, 2024

ulysses-you commented Jul 24, 2024

[KYUUBI #6411] Grpc common support #6412

Are you sure you want to change the base?

[KYUUBI #6411] Grpc common support #6412

Conversation

davidyuan1223 commented May 23, 2024 • edited Loading

🔍 Description

Issue References 🔗

Describe Your Solution 🔧

Types of changes 🔖

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

Checklist 📝

davidyuan1223 commented May 23, 2024

codecov-commenter commented May 23, 2024

Codecov Report

ulysses-you commented May 24, 2024

davidyuan1223 commented May 24, 2024

davidyuan1223 commented Jul 23, 2024 • edited Loading

ulysses-you commented Jul 24, 2024

davidyuan1223 commented Jul 24, 2024

ulysses-you commented Jul 24, 2024

davidyuan1223 commented May 23, 2024 •

edited

Loading

davidyuan1223 commented Jul 23, 2024 •

edited

Loading