[KYUUBI #502][SPARK-35002][INFRA] Fix the java.net.BindException when testing with Github Action #503

turboFei · 2021-04-09T12:36:03Z

Why are the changes needed?

Refer apache/spark#32096, apache/spark#32102, this PR tries to fix the java.net.BindException when testing with Github Action.

SparkOperationSuite:
*** RUN ABORTED ***
  java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address.
  at sun.nio.ch.Net.bind0(Native Method)
  at sun.nio.ch.Net.bind(Net.java:461)
  at sun.nio.ch.Net.bind(Net.java:453)
  at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
  at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134)
  at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550)
  at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
  at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)
  at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)
  at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)

Also transfer FRONTEND_BIND_HOST by connection string to fix similar issue.

Cause: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:org.apache.kyuubi.KyuubiException: Failed to initialize frontend service on fv-az207-19/10.1.1.0:0.
	at org.apache.kyuubi.service.FrontendService.initialize(FrontendService.scala:102)
	at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
	at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
	at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:44)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.initialize(SparkSQLEngine.scala:49)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:105)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:118)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before make a pull request

codecov · 2021-04-09T12:52:43Z

Codecov Report

Merging #503 (1488405) into master (9b3c22d) will decrease coverage by 1.03%.
The diff coverage is n/a.

❗ Current head 1488405 differs from pull request most recent head 1b10253. Consider uploading reports for the commit 1b10253 to get more accurate results

@@             Coverage Diff              @@
##             master     #503      +/-   ##
============================================
- Coverage     81.70%   80.66%   -1.04%     
- Complexity        0      648     +648     
============================================
  Files           116      116              
  Lines          4247     4247              
  Branches        514      514              
============================================
- Hits           3470     3426      -44     
- Misses          498      543      +45     
+ Partials        279      278       -1

Impacted Files	Coverage Δ	Complexity Δ
...scala/org/apache/kyuubi/operation/GetColumns.scala	`0.00% <0.00%> (-94.45%)`	`0.00% <0.00%> (ø%)`
...he/kyuubi/engine/spark/shim/CatalogShim_v3_0.scala	`59.21% <0.00%> (-28.95%)`	`21.00% <0.00%> (+21.00%)`	⬇️
...ache/kyuubi/operation/KyuubiOperationManager.scala	`55.40% <0.00%> (-6.76%)`	`0.00% <0.00%> (ø%)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9b3c22d...1b10253. Read the comment docs.

turboFei · 2021-04-09T13:07:41Z

  Cause: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:org.apache.kyuubi.KyuubiException: Failed to initialize frontend service on fv-az207-19/10.1.1.0:0.
	at org.apache.kyuubi.service.FrontendService.initialize(FrontendService.scala:102)
	at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
	at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
	at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:44)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.initialize(SparkSQLEngine.scala:49)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:105)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:118)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.BindException: Cannot assign requested address (Bind failed)
	at java.net.PlainSocketImpl.socketBind(Native Method)
	at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
	at java.net.ServerSocket.bind(ServerSocket.java:392)
	at java.net.ServerSocket.<init>(ServerSocket.java:254)
	at org.apache.kyuubi.service.FrontendService.initialize(FrontendService.scala:76)
 See more: /home/runner/work/kyuubi/kyuubi/kyuubi-main/target/work/runner/kyuubi-spark-sql-engine.log.11
  at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:58)
  at org.apache.kyuubi.engine.ProcBuilder.$anonfun$start$1(ProcBuilder.scala:125)
  at java.lang.Thread.run(Thread.java:748)

turboFei · 2021-04-09T13:08:50Z

Also set KYUUBI_FRONTEND_BIND_HOST env to 127.0.0.1 and have a try.

yaooqinn · 2021-04-09T14:37:52Z

.github/workflows/master.yml

@@ -37,6 +37,9 @@ jobs:
          - '-Pspark-3.0 -Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.1.1 -Dspark.archive.name=spark-3.1.1-bin-hadoop2.7.tgz -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.DataLakeTest'
          - '-Pspark-3.1'
          - '-Pspark-3.1 -Dhadoop.binary.version=3.2'
+    env:
+      SPARK_LOCAL_IP: 127.0.0.1
+      KYUUBI_FRONTEND_BIND_HOST: 127.0.0.1


How about just add envs and properties in scalatest-maven-plugin, add a new env is overkill for me to only fix a CI issue

sounds good

turboFei · 2021-04-09T15:44:57Z

https://github.com/yaooqinn/kyuubi/pull/503/checks?check_run_id=2306833395#step:9:24197

It seems that we have to use System env solution to fix it.

@yaooqinn
Because the suite is triggered by kyuubi server side, and the property value could not be transferred to engine side.

turboFei · 2021-04-09T15:49:00Z

or transfer it in connection string

turboFei · 2021-04-09T16:03:06Z

kyuubi-main/src/test/scala/org/apache/kyuubi/operation/WithKyuubiServer.scala

@@ -72,5 +72,6 @@ trait WithKyuubiServer extends KyuubiFunSuite {
    super.afterAll()
  }

-  protected def getJdbcUrl: String = s"jdbc:hive2://${server.connectionUrl}/;"
+  protected def getJdbcUrl: String =
+    s"jdbc:hive2://${server.connectionUrl}/;${KyuubiConf.FRONTEND_BIND_HOST.key}=127.0.0.1"


I am not sure whether it could work

this will work... but OMG... this is a potential bug..

yes, it should be forbid in engine side

Let create an Issue ticket for it, and mark it todo here

pan3793 · 2021-04-09T16:19:48Z

build/dependency.sh doesn't seem to work as expected, after this get in, I will try to fix it.

…n when testing with Github Action

…t to 127.0.0.1 in scalatest-maven-plugin

.github/workflows/master.yml

pan3793 · 2021-04-09T17:36:27Z

Spark switched 127.0.0.1 to localhost, see apache/spark#32102

pan3793 · 2021-04-09T18:46:28Z

Thanks! Merged into master for v1.2.0.

pan3793 · 2021-04-14T17:13:47Z

The issue is reported at actions/runner-images#3185

yaooqinn approved these changes Apr 9, 2021

View reviewed changes

yaooqinn assigned turboFei Apr 9, 2021

yaooqinn added the Test label Apr 9, 2021

yaooqinn added this to the v1.2.0 milestone Apr 9, 2021

turboFei force-pushed the KYUUBI-502 branch 2 times, most recently from 5244e44 to 022e357 Compare April 9, 2021 13:32

yaooqinn reviewed Apr 9, 2021

View reviewed changes

turboFei commented Apr 9, 2021

View reviewed changes

turboFei force-pushed the KYUUBI-502 branch from be97b8c to 6312742 Compare April 9, 2021 16:29

turboFei added 8 commits April 10, 2021 00:36

[KYUUBI apache#502][SPARK-35002][INFRA] Fix the java.net.BindExceptio…

7d70819

…n when testing with Github Action

also set KYUUBI_FRONTEND_BIND_HOST env to 127.0.0.1

5eb7cdb

involve KYUUBI_FRONTEND_BIND_HOST

e8b2372

add install shaded jars in release.yml

d5ba05a

update doc

717a992

revert env KYUUBI_FRONTEND_BIND_HOST and set kyuubi.frontend.bind.hos…

8bcd5a0

…t to 127.0.0.1 in scalatest-maven-plugin

try set frontend bind host in connection string

457ce2f

revert install shade

1e549c1

turboFei force-pushed the KYUUBI-502 branch 2 times, most recently from cd3092b to 6d1ef18 Compare April 9, 2021 16:42

pan3793 reviewed Apr 9, 2021

View reviewed changes

.github/workflows/master.yml Show resolved Hide resolved

address comments

c104ce3

turboFei force-pushed the KYUUBI-502 branch from 6d1ef18 to c104ce3 Compare April 9, 2021 16:45

pan3793 approved these changes Apr 9, 2021

View reviewed changes

use localhost instead of 127.0.0.1

1b10253

pan3793 closed this in f2ba452 Apr 9, 2021

[KYUUBI #502][SPARK-35002][INFRA] Fix the java.net.BindException when testing with Github Action #503

[KYUUBI #502][SPARK-35002][INFRA] Fix the java.net.BindException when testing with Github Action #503

Uh oh!

Conversation

turboFei commented Apr 9, 2021 • edited by pull-request-badge bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

How was this patch tested?

Uh oh!

codecov bot commented Apr 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

turboFei commented Apr 9, 2021

Uh oh!

turboFei commented Apr 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaooqinn Apr 9, 2021

Choose a reason for hiding this comment

Uh oh!

turboFei Apr 9, 2021

Choose a reason for hiding this comment

Uh oh!

turboFei commented Apr 9, 2021

Uh oh!

turboFei commented Apr 9, 2021

Uh oh!

turboFei Apr 9, 2021

Choose a reason for hiding this comment

Uh oh!

yaooqinn Apr 9, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

turboFei Apr 9, 2021

Choose a reason for hiding this comment

Uh oh!

yaooqinn Apr 9, 2021

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Apr 9, 2021

Uh oh!

Uh oh!

pan3793 commented Apr 9, 2021

Uh oh!

pan3793 commented Apr 9, 2021

Uh oh!

pan3793 commented Apr 14, 2021

Uh oh!

Uh oh!

turboFei commented Apr 9, 2021 •

edited by pull-request-badge bot

Loading

codecov bot commented Apr 9, 2021 •

edited

Loading

turboFei commented Apr 9, 2021 •

edited

Loading

yaooqinn Apr 9, 2021 •

edited

Loading