forked from DataLinkDC/dinky
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
32 changed files
with
2,412 additions
and
0 deletions.
There are no files selected for viewing
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
* [首页](/zh-CN/introduce.md) | ||
|
||
* 文档 | ||
* [快速开始](/guide/quickstart.md) | ||
* [发展计划](/guide/roadmap.md) | ||
* [编译部署](/guide/deploy.md) | ||
* [核心功能](/guide/functions.md) | ||
* [开发调试](/guide/debug.md) | ||
|
||
* 下载 | ||
* [扩展语法补全](/extend/completion.md) | ||
* [扩展自定义函数](/extend/udf.md) | ||
* [扩展连接器](/extend/connector.md) | ||
* [扩展数据源](/extend/datasource.md) | ||
* [扩展 Flink 版本](/extend/flinkversion.md) | ||
* [Flink-CDC 集成](/extend/flinkcdc.md) | ||
* [DolphinScheduler 集成](/extend/dolphinscheduler.md) | ||
* [DataSphereStudio 集成](/extend/dataspherestudio.md) | ||
* [Hive 集成](/extend/hive.md) | ||
* [Doris 集成](/extend/doris.md) | ||
* [Clickhouse 集成](/extend/clickhouse.md) | ||
* [Hudi 集成](/extend/hudi.md) | ||
* [Iceberg 集成](/extend/hudi.md) | ||
* [Flink CDC和Kafka进行多源合并和下游同步更新](/extend/Flink_CDC_kafka_Multi_source_merger.md) | ||
|
||
* 开发者 | ||
* [Dlink Yarn 的三种提交实践](/share/yarnsubmit.md) | ||
* [Dlink AGGTABLE 表值聚合的实践](/share/aggtable.md) | ||
* [Dlink 核心概念及实现原理详解](/share/principle.md) | ||
|
||
* 用户案例 | ||
* [OpenAPI](/api/openapi.md) | ||
|
||
* 语言 | ||
* [中文](/zh-CN/) | ||
* [En](/en-US/) | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,63 @@ | ||
<!-- docs/zh-CN/_sidebar.md --> | ||
|
||
|
||
- [Dinky简介](/zh-CN/introduce.md) | ||
- 概念和架构 | ||
- [系统架构](/zh-CN/architecture.md) | ||
- [基本概念](/zh-CN/concept.md) | ||
- [功能](/zh-CN/feature.md) | ||
- 入门 | ||
- [下载](/zh-CN/quick_start/download.md) | ||
- [编译](/zh-CN/quick_start/build.md) | ||
- [部署](/zh-CN/quick_start/deploy.md) | ||
- 基本使用指南 | ||
- 参考手册 | ||
- FlinkSQL Studio | ||
- 作业和目录创建 | ||
- 作业开发 | ||
- 作业配置 | ||
- 作业管理 | ||
- 会话管理 | ||
- 注册中心 | ||
- 集群实例 | ||
- 集群配置 | ||
- jar管理 | ||
- 数据源管理 | ||
- 文档管理 | ||
- 系统设置 | ||
- 用户管理 | ||
- Flink设置 | ||
- 运维中心 | ||
- 生命周期管理 | ||
- 作业监控 | ||
- api | ||
- 最佳实践 | ||
- [Yarn提交实践指南](/zh-CN/practice/yarnsubmit.md) | ||
- [Dlink 核心概念及实现原理详解](/zh-CN/practice/principle.md) | ||
- [AGGTABLE 表值聚合的实践](/zh-CN/practice/aggtable.md) | ||
- 扩展 | ||
- 集成 | ||
- [Flink-CDC集成](/zh-CN/extend/flinkcdc.md) | ||
- [Flink-CDC-Kafka多源合并](/zh-CN/extend/Flink_CDC_kafka_Multi_source_merger.md) | ||
- [hive集成](/zh-CN/extend/hive.md) | ||
- [clickhouse集成](/zh-CN/extend/clickhouse.md) | ||
- [Doris集成](/zh-CN/extend/doris.md) | ||
- [Hudi集成](/zh-CN/extend/hudi.md) | ||
- [Iceberg集成](/zh-CN/extend/iceberg.md) | ||
- [Flink UDF集成](/zh-CN/extend/udf.md) | ||
- [DolphinScheduler集成](/zh-CN/extend/dolphinscheduler.md) | ||
- [DataSphereStudio集成](/zh-CN/extend/dataspherestudio.md) | ||
- 其他 | ||
- [扩展Flink版本](/zh-CN/extend/flinkversion.md) | ||
- [扩展连接器](/zh-CN/extend/connector.md) | ||
- [扩展数据源](/zh-CN/extend/datasource.md) | ||
- [FlinkSQL 编辑器自动补全函数](/zh-CN/extend/completion.md) | ||
- 开发者指南 | ||
- 常见问题 | ||
- 历史版本 | ||
- [近期计划](/zh-CN/others/plans.md) | ||
- [致谢](/zh-CN/others/thanks.md) | ||
- [交流与贡献](/zh-CN/others/comminicate.md) | ||
- [roadmap](/zh-CN/roadmap.md) | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
## 敬请期待 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
![](https://gitee.com/DataLinkDC/dlink/raw/main/dlink-doc/images/main/dinky_principle.png) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
<big>**JobManager**</big> | ||
|
||
JobManager 作为 Dinky 的作业管理的统一入口,负责 Flink 的各种作业执行方式及其他功能的调度。 | ||
|
||
<big>**Executor**</big> | ||
|
||
Executor 是 Dinky 定制的 FlinkSQL 执行器,来模拟真实的 Flink 执行环境,负责 FlinkSQL 的 Catalog 管理、UDF管理、片段管理、配置管理、语句集管理、语法校验、逻辑验证、计划优化、生成 JobGraph、本地执行、远程提交、SELECT 及 SHOW 预览等核心功能。 | ||
|
||
<big>**Interceptor**</big> | ||
|
||
Interceptor 是 Dinky 的 Flink 执行拦截器,负责对其进行片段解析、UDF注册、SET 和 AGGTABLE 等增强语法解析。 | ||
|
||
<big>**Gateway**</big> | ||
|
||
Gateway 并非是开源项目 flink-sql-gateway,而是 Dinky 自己定制的 Gateway,负责进行基于 Yarn 环境的任务提交与管理,主要有Yarn-Per-Job 和 Yarn-Application 的 FlinkSQL 提交、停止、SavePoint 以及配置测试,而 User Jar 目前只开放了 Yarn-Application 的提交。 | ||
|
||
<big>**Flink SDK**</big> | ||
|
||
Dinky 主要通过调用 flink-client 和 flink-table 模块进行二次开发。 | ||
|
||
<big>**Yarn SDK**</big> | ||
|
||
Dinky 通过调用 flink-yarn 模块进行二次开发。 | ||
|
||
<big>**Flink API**</big> | ||
|
||
Dinky 也支持通过调用 JobManager 的 RestAPI 对任务进行管理等操作,系统配置可以控制开启和停用。 | ||
|
||
<big>**Yarn-Session**</big> | ||
|
||
Dinky 通过已注册的 Flink Session 集群实例可以对 Standalone 和 Yarn-Session 两种集群进行 FlinkSQL 的提交、Catalog 的交互式管理以及对 SELECT 和 SHOW 等语句的执行结果预览。 | ||
|
||
<big>**Yarn-Per-Job**</big> | ||
|
||
Dinky 通过已注册的集群配置来获取对应的 YarnClient 实例,然后将本地解析生产的 JobGraph 与 Configuration 提交至 Yarn 来创建 Flink-Per-Job 应用。 | ||
|
||
<big>**Yarn-Application**</big> | ||
|
||
Dinky 通过已注册的集群配置来获取对应的 YarnClient 实例。对于 User Jar,将 Jar 相关配置与 Configuration 提交至 Yarn 来创建 Flink-Application 应用;对于 Flink SQL,Dinky 则将作业 ID 及数据库连接配置作为 Main 入参和 dlink-app.jar 以及 Configuration 提交至 Yarn 来创建 Flink-Application 应用。 |
Oops, something went wrong.