Skip to content

Commit

Permalink
[KYUUBI apache#1652] Support Flink yarn application mode
Browse files Browse the repository at this point in the history 
### _Why are the changes needed?_
Flink yarn application mode is crucial for the production usage of Flink engine.

To test this PR locally, we should:

1) set `flink.execution.target=yarn-application` in `kyuubi-defaults.conf`.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes apache#4604 from link3280/KYUUBI-1652.

Closes apache#1652

49b454f [Paul Lin] [KYUUBI apache#1652] Delay access to thrift services to stablize tests
b91b64b [Paul Lin] Revert "[KYUUBI apache#1652] Avoid hadoop conf injecting into kyuubi conf"
c9f710b [Paul Lin] [KYUUBI apache#1652] Avoid hadoop conf injecting into kyuubi conf
cde8a54 [Paul Lin] [KYUUBI apache#1652] Improve docs
edba0ec [Paul Lin] [KYUUBI apache#1652] Improve codestyle
e03e055 [Paul Lin] [KYUUBI apache#1652] Update docs according to the comments
490559c [Paul Lin] [KYUUBI apache#1652] Update docs
769d1a8 [Paul Lin] [KYUUBI apache#1652] Move zookeeper to test scope
bafb3f5 [Paul Lin] [KYUUBI apache#1652] Fix flink-it test
dd40c72 [Paul Lin] [KYUUBI apache#1652] Update docs
36c993f [Paul Lin] [KYUUBI apache#1652] Fix javax.activation not found in flink-it
2a751bd [Paul Lin] [KYUUBI apache#1652] Introduce EmbeddedZookeeper in Flink yarn tests
0933b70 [Paul Lin] [KYUUBI apache#1652] Fix spotless issue
b858f7d [Paul Lin] [KYUUBI apache#1652] Fix Flink submit timeout because failing to find hadoop conf
15801b5 [Paul Lin] [KYUUBI apache#1652] Replace unused jaxb
b210615 [Paul Lin] Update externals/kyuubi-flink-sql-engine/pom.xml
24b23da [Paul Lin] [KYUUBI apache#1652] Update jaxb scope to test
240efae [Paul Lin] [KYUUBI apache#1652] Update jaxb scope to runtime
0e9a508 [Paul Lin] [KYUUBI apache#1652] Update jaxb scope to runtime
b5dbd33 [Paul Lin] [KYUUBI apache#1652] Fix jdk11 jaxb ClassNotFoundException
72ba3ee [Paul Lin] [KYUUBI apache#1652] Update tm memory to 1gb
4e10ea2 [Paul Lin] [KYUUBI apache#1652] Refactor flink engin tests
e9cec4a [Paul Lin] [KYUUBI apache#1652] Add flink-it tests
6eb9fd3 [Paul Lin] [KYUUBI apache#1652] Fix ProcessBuilder tests
6aca061 [Paul Lin] [KYUUBI apache#1652] Fix ClassNotFoundException
7581a2a [Paul Lin] [KYUUBI apache#1652] Fix missing minicluster
412c345 [Paul Lin] [KYUUBI apache#1652] Remove flink-yarn dependencies
0eafbd7 [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/result/ResultSet.scala
ee2c64d [Paul Lin] [KYUUBI apache#1652] Add Flink YARN application tests
a726273 [Paul Lin] [KYUUBI apache#1652] Avoid flink-yarn dependencies
a75cb25 [Paul Lin] [KYUUBI apache#1652] Fix test issue
b7e173f [Paul Lin] [KYUUBI apache#1652] Replace file-based Kyuubi conf with cli args
693ad65 [Paul Lin] [KYUUBI apache#1652] Removed unused imports
68e0081 [Paul Lin] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilder.scala
ba021de [Paul Lin] [KYUUBI apache#1652] Search flink-sql.* jars and add them to pipeline jars
0846bab [Paul Lin] [KYUUBI apache#1652] Avoid Scala bug
56413fe [Paul Lin] [KYUUBI apache#1652] Improve tmp files cleanup
8bdb672 [Paul Lin] [KYUUBI apache#1652] Explicitly load Kyuubi conf on Flink engine start
0b63250 [Paul Lin] [KYUUBI apache#1652] Fix test failures
0ba03e4 [Paul Lin] [KYUUBI apache#1652] Fix wrong Flink args
00f036b [Paul Lin] [KYUUBI apache#1652] Remove unused util methods
dfd2777 [Paul Lin] [KYUUBI #apache#1652] Support Flink yarn application mode

Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
  • Loading branch information
@link3280 @pan3793
link3280 authored and pan3793 committed Apr 7, 2023
1 parent 871dd1f commit b315123
Show file tree
Hide file tree
Showing 25 changed files with 1,190 additions and 139 deletions.
7 changes: 4 additions & 3 deletions docs/deployment/settings.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,10 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| kyuubi.engine.deregister.job.max.failures | 4 | Number of failures of job before deregistering the engine. | int | 1.2.0 |
| kyuubi.engine.event.json.log.path | file:///tmp/kyuubi/events | The location where all the engine events go for the built-in JSON logger.<ul><li>Local Path: start with 'file://'</li><li>HDFS Path: start with 'hdfs://'</li></ul> | string | 1.3.0 |
| kyuubi.engine.event.loggers | SPARK | A comma-separated list of engine history loggers, where engine/session/operation etc events go.<ul> <li>SPARK: the events will be written to the Spark listener bus.</li> <li>JSON: the events will be written to the location of kyuubi.engine.event.json.log.path</li> <li>JDBC: to be done</li> <li>CUSTOM: User-defined event handlers.</li></ul> Note that: Kyuubi supports custom event handlers with the Java SPI. To register a custom event handler, the user needs to implement a subclass of `org.apache.kyuubi.events.handler.CustomEventHandlerProvider` which has a zero-arg constructor. | seq | 1.3.0 |
| kyuubi.engine.flink.extra.classpath | &lt;undefined&gt; | The extra classpath for the Flink SQL engine, for configuring the location of hadoop client jars, etc | string | 1.6.0 |
| kyuubi.engine.flink.java.options | &lt;undefined&gt; | The extra Java options for the Flink SQL engine | string | 1.6.0 |
| kyuubi.engine.flink.memory | 1g | The heap memory for the Flink SQL engine | string | 1.6.0 |
| kyuubi.engine.flink.application.jars | &lt;undefined&gt; | A comma-separated list of the local jars to be shipped with the job to the cluster. For example, SQL UDF jars. Only effective in yarn application mode. | string | 1.8.0 |
| kyuubi.engine.flink.extra.classpath | &lt;undefined&gt; | The extra classpath for the Flink SQL engine, for configuring the location of hadoop client jars, etc. Only effective in yarn session mode. | string | 1.6.0 |
| kyuubi.engine.flink.java.options | &lt;undefined&gt; | The extra Java options for the Flink SQL engine. Only effective in yarn session mode. | string | 1.6.0 |
| kyuubi.engine.flink.memory | 1g | The heap memory for the Flink SQL engine. Only effective in yarn session mode. | string | 1.6.0 |
| kyuubi.engine.hive.event.loggers | JSON | A comma-separated list of engine history loggers, where engine/session/operation etc events go.<ul> <li>JSON: the events will be written to the location of kyuubi.engine.event.json.log.path</li> <li>JDBC: to be done</li> <li>CUSTOM: to be done.</li></ul> | seq | 1.7.0 |
| kyuubi.engine.hive.extra.classpath | &lt;undefined&gt; | The extra classpath for the Hive query engine, for configuring location of the hadoop client jars and etc. | string | 1.6.0 |
| kyuubi.engine.hive.java.options | &lt;undefined&gt; | The extra Java options for the Hive query engine | string | 1.6.0 |
Expand Down
38 changes: 38 additions & 0 deletions externals/kyuubi-flink-sql-engine/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -126,11 +126,49 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-zookeeper_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-test-utils</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-minicluster</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk15on</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcpkix-jdk15on</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>jakarta.activation</groupId>
<artifactId>jakarta.activation-api</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>jakarta.xml.bind</groupId>
<artifactId>jakarta.xml.bind-api</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down
Loading

0 comments on commit b315123

Please sign in to comment.