Manually terminate the Py4JServer during engine shutdown

[cfmcgrady]

Why are the changes needed?

Due to the Py4JServer initiating with a non-daemon thread, there is a possibility of it impeding the engine's termination. Therefore, it is imperative to manually terminate the Py4JServer during engine shutdown.

"Thread-23" #96 prio=5 os_prio=0 cpu=7.93ms elapsed=187532.67s tid=0x00007fee840cf000 nid=0x8f runnable  [0x00007fedca6bf000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(java.base@11.0.16/Native Method)
	at java.net.AbstractPlainSocketImpl.accept(java.base@11.0.16/Unknown Source)
	at java.net.ServerSocket.implAccept(java.base@11.0.16/Unknown Source)
	at java.net.ServerSocket.accept(java.base@11.0.16/Unknown Source)
	at py4j.GatewayServer.run(GatewayServer.java:685)
	at java.lang.Thread.run(java.base@11.0.16/Unknown Source)

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before make a pull request

[yaooqinn]

the start and stop need to be syncronized

[cfmcgrady]

the start and stop need to be syncronized

The start function has already been synchronized in the caller's code, shall we need to refine this?

kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecutePython.scala

Lines 240 to 253 in d9e14f2

	private val isPythonGatewayStart = new AtomicBoolean(false)
	private val kyuubiPythonPath = Utils.createTempDir()
	def init(): Unit = {
	if (!isPythonGatewayStart.get()) {
	synchronized {
	if (!isPythonGatewayStart.get()) {
	KyuubiPythonGatewayServer.start()
	writeTempPyFile(kyuubiPythonPath, "execute_python.py")
	writeTempPyFile(kyuubiPythonPath, "kyuubi_util.py")
	isPythonGatewayStart.set(true)
	}
	}
	}
	}

[cfmcgrady]

the start and stop need to be syncronized

updated.

[pan3793]

Thanks, merged to master/1.7