From 7a95fd21651a5f07601ab120f373649b6e5b21bd Mon Sep 17 00:00:00 2001 From: liutianyou Date: Tue, 7 May 2024 08:04:50 +0800 Subject: [PATCH] [doc] Add documentation for nebulaGraph cluster monitoring and custom monitoring using NGQL, and clean up useless parameters (#1923) --- home/docs/advanced/extend-ngql.md | 167 +++++++++++++++++ home/docs/help/guide.md | 1 + home/docs/help/nebulagraph_cluster.md | 91 ++++++++++ .../current/advanced/extend-ngql.md | 171 ++++++++++++++++++ .../current/help/guide.md | 1 + .../current/help/nebulagraph_cluster.md | 91 ++++++++++ home/sidebars.json | 9 +- .../define/app-nebula_graph_cluster.yml | 17 +- 8 files changed, 533 insertions(+), 15 deletions(-) create mode 100644 home/docs/advanced/extend-ngql.md create mode 100644 home/docs/help/nebulagraph_cluster.md create mode 100644 home/i18n/zh-cn/docusaurus-plugin-content-docs/current/advanced/extend-ngql.md create mode 100644 home/i18n/zh-cn/docusaurus-plugin-content-docs/current/help/nebulagraph_cluster.md diff --git a/home/docs/advanced/extend-ngql.md b/home/docs/advanced/extend-ngql.md new file mode 100644 index 00000000000..c07c5ae8ca0 --- /dev/null +++ b/home/docs/advanced/extend-ngql.md @@ -0,0 +1,167 @@ +--- +id: extend-ngql +title: NGQL Custom Monitoring +sidebar_label: NGQL Custom Monitoring +--- + +> From [Custom Monitoring](extend-point), you are familiar with how to customize types, Metrics, protocols, etc. Here we will introduce in detail how to use JDBC(support mysql,mariadb,postgresql,sqlserver at present) to customize Metric monitoring. +> NGQL custom monitoring allows us to easily query metric data from the NebulaGraph graph database using NGQL or OpenCypher, supporting NebulaGraph 3.X versions. + +### Data Parsing Methods + +Mapping the fields returned by NGQL queries to the metrics we need allows us to obtain corresponding metric data. Currently, there are four mapping and parsing methods: filterCount, oneRow, multiRow, columns. + +#### **filterCount** + +> Counts the number of results returned by a query based on specified fields, usually used in `SHOW ...` statements. If NGQL statements can directly return the count, it is recommended to use NGQL statements for counting. +> Syntax for the `commands` field: aliasField#NGQL#filterName#filterValue +> `aliasField`: corresponds to the value in the `aliasFields` in the monitoring template +> `NGQL`: query statement +> `filterName`: filter attribute name (optional) +> `filterValue`: filter attribute value (optional) + +For example: +- online_meta_count#SHOW HOSTS META#Status#ONLINE + Counts the number of rows returned by `SHOW HOSTS META` where Status equals ONLINE. +- online_meta_count#SHOW HOSTS META## + Counts the number of rows returned by `SHOW HOSTS META`. + +#### **oneRow** + +> Queries a single row of data by mapping the column names of the query result set to the queried fields. + +For example: +- Metrics fields: a, b +- NGQL query: match (v:metrics) return v.metrics.a as a, v.metrics.b as b; + +Here, the metric fields can be mapped to the response data row by row. + +Notes: +- When using the `oneRow` method, if a single query statement returns multiple rows of results, only the first row of results will be mapped to the metric fields. +- When the `commands` field contains two or more query statements and the returned fields of multiple query statements are the same, the fields returned by the subsequent statement will overwrite those returned by the previous statement. +- It is recommended to use the limit statement to limit the number of rows returned in the result set when defining `commands`. + +#### **multiRow** + +> Queries multiple rows of data by mapping the column names of the query result set to the queried fields. + +For example: +- Metrics fields: a, b +- NGQL query: match (v:metrics) return v.metrics.a as a, v.metrics.b as b; + +Here, the metric fields can be mapped to the response data row by row. +Notes: +- When using the `multiRow` method, the `commands` field can only contain one query statement. + +#### **columns** + +> Collects a single row of metric data by mapping two columns of data (key-value), where the key matches the queried fields and the value is the value of the queried field. + +Notes: +- When using the `columns` method, the first two columns of the result set are mapped to collect data by default, where the first column corresponds to the metric name and the second column corresponds to the metric value. +- When the `commands` field contains two or more query statements and the first column of data returned by multiple query statements is duplicated, the result of the last statement will be retained. + +### Customization Steps + +**HertzBeat Page** -> **Monitoring Template Menu** -> **Add Monitoring Type** -> **Configure Custom Monitoring Template YML** -> **Click Save Application** -> **Use the New Monitoring Type to Add Monitoring** + +![HertzBeat Page](/img/docs/advanced/extend-point-1.png) + +------- +Configuration usages of the monitoring templates yml are detailed below. + +### Monitoring Template YML + +> We define all monitoring collection types (mysql,jvm,k8s) as yml monitoring templates, and users can import these templates to support corresponding types of monitoring. +> Monitoring template is used to define the name of monitoring type(international), request parameter mapping, index information, collection protocol configuration information, etc. + +eg: Customize a monitoring type named example_ngql, which collects metric data using NGQL. + +```yaml +# Monitoring category: service-application service program-application program db-database custom-custom os-operating system bigdata-big data mid-middleware webserver-web server cache-cache cn-cloud native network-network monitoring, etc. +category: db +# Monitoring application type (consistent with the file name) eg: linux windows tomcat mysql aws... +app: example_ngql +name: + zh-CN: NGQL Custom Monitoring Application + en-US: NGQL Custom APP +# Monitoring parameter definition. These are input parameter variables, which can be written in the format of ^_^host^_^ to be replaced by system variable values in the later configuration +# This part is usually not modified +params: + # field-param field key + - field: host + name: + zh-CN: Target Host + en-US: Target Host + type: host + required: true + - field: graphPort + name: + zh-CN: Graph Port + en-US: Graph Port + type: number + range: '[0,65535]' + required: true + defaultValue: 9669 + - field: username + name: + zh-CN: Username + en-US: Username + type: text + required: true + - field: password + name: + zh-CN: Password + en-US: Password + type: password + required: true + - field: spaceName + name: + zh-CN: Space Name + en-US: Space Name + type: text + required: false + - field: timeout + name: + zh-CN: Connect Timeout(ms) + en-US: Connect Timeout(ms) + type: number + unit: ms + range: '[0,100000]' + required: true + defaultValue: 6000 +# Metric collection configuration list +metrics: + - name: base_info + i18n: + zh-CN: Vertex statistics + en-US: Vertex statistics + priority: 0 + fields: + - field: tag1 + type: 1 + i18n: + zh-CN: tag1 + en-US: tag1 + - field: tag1 + type: 1 + i18n: + zh-CN: tag2 + en-US: tag2 + aliasFields: + - tag1 + - tag2 + protocol: ngql + ngql: + host: ^_^host^_^ + username: ^_^username^_^ + password: ^_^password^_^ + port: ^_^graphPort^_^ + spaceName: ^_^spaceName^_^ + parseType: columns + # Define the query statements used to collect data + commands: + - match (v:tag1) return "tag1" as name ,count(v) as cnt + - match (v:tag2) return "tag2" as name ,count(v) as cnt + timeout: ^_^timeout^_^ +``` diff --git a/home/docs/help/guide.md b/home/docs/help/guide.md index ae78ec39a31..6949aa147d4 100644 --- a/home/docs/help/guide.md +++ b/home/docs/help/guide.md @@ -38,6 +38,7 @@ sidebar_label: Help Center  👉 [IoTDB database monitoring](iotdb)
 👉 [TiDB database monitoring](tidb)
 👉 [MongoDB database monitoring](mongodb)
+ 👉 [NebulaGraph cluster monitoring](nebulagraph_cluster)
### Operating system monitoring diff --git a/home/docs/help/nebulagraph_cluster.md b/home/docs/help/nebulagraph_cluster.md new file mode 100644 index 00000000000..d0da21a7adb --- /dev/null +++ b/home/docs/help/nebulagraph_cluster.md @@ -0,0 +1,91 @@ +--- +id: nebulagraph_cluster +title: Monitoring NebulaGraph Cluster +sidebar_label: NebulaGraph Cluster +keywords: [ Open Source Monitoring System, Open Source Database Monitoring, Open Source Graph Database Monitoring, NebulaGraph Cluster Monitoring ] +--- + +> Monitor basic information, nodes, tasks, etc., of the NebulaGraph cluster. + +**Protocol Use：ngql** + +### Configuration parameters + +| Parameter Name | Parameter help description | +|---------------------|--------------------------------------------------------------------------------------------------------------------| +| Target Host | The IPv4, IPv6, or domain name of the monitored peer. Note ⚠️ without the protocol header (eg: https://, http://). | +| Task Name | Identifies the name of this monitor, ensuring uniqueness of the name. | +| Graph Port | The port where the graph service is open, default is 9669. | +| Connection Timeout | Timeout for connecting to the graph service, in milliseconds, default is 6000 milliseconds. | +| Username | Database connection username. | +| Password | Database connection password. | +| Collection Interval | Interval for periodically collecting data, in seconds. The minimum interval that can be set is 30 seconds. | +| Bind Labels | Used to categorize and manage monitored resources. | +| Description | Additional information for identifying and describing this monitor. Users can add remarks here. | + +### Collection Metric + +#### Metric Set: Base info + +| Metric Name | Metric Unit | Metric help description | +|--------------|-------------|-------------------------| +| responseTime | None | Response time | +| charset | None | Character set | +| collation | None | Character set collation | + +#### Metric Set: Session + +| Metric Name | Metric Unit | Metric help description | +|---------------------|-------------|----------------------------------| +| session | None | Number of sessions | +| running_query_count | None | Number of queries being executed | + +#### Metric Set: Jobs + +| Metric Name | Metric Unit | Metric help description | +|--------------|-------------|-------------------------------------------| +| queue_jobs | None | Number of pending background tasks | +| running_jobs | None | Number of background tasks being executed | + +#### Metric Set: Cluster node info + +| Metric Name | Metric Unit | Metric help description | +|----------------------|-------------|---------------------------------| +| total_storage_node | None | Number of storage nodes | +| offline_storage_node | None | Number of offline storage nodes | +| total_meta_node | None | Number of meta nodes | +| offline_meta_node | None | Number of offline meta nodes | +| total_graph_node | None | Number of graph nodes | +| offline_graph_node | None | Number of offline graph nodes | + +#### Metric Set: Storage Nodes + +| Metric Name | Metric Unit | Metric help description | +|-----------------------|-------------|-------------------------------------------------------| +| host | None | Node address | +| port | None | Port | +| status | None | Status (ONLINE/OFFLINE) | +| leaderCount | None | Number of leader partitions on the current node | +| leaderDistribution | None | Distribution of leader partitions on the current node | +| partitionDistribution | None | Distribution of partitions on the current node | +| version | None | Version | + +#### Metric Set: Meta Nodes + +| Metric Name | Metric Unit | Metric help description | +|-------------|-------------|-------------------------| +| host | None | Node address | +| port | None | Port | +| status | None | Status (ONLINE/OFFLINE) | +| version | None | Version | + +#### Metric Set: Graph Nodes + +| Metric Name | Metric Unit | Metric help description | +|-------------|-------------|-------------------------| +| host | None | Node address | +| port | None | Port | +| status | None | Status (ONLINE/OFFLINE) | +| version | None | Version | + +> If you need to customize monitoring templates to collect data from NebulaGraph clusters, please refer to: [NGQL Custom Monitoring](../advanced/extend-ngql.md) diff --git a/home/i18n/zh-cn/docusaurus-plugin-content-docs/current/advanced/extend-ngql.md b/home/i18n/zh-cn/docusaurus-plugin-content-docs/current/advanced/extend-ngql.md new file mode 100644 index 00000000000..b5685eb3b16 --- /dev/null +++ b/home/i18n/zh-cn/docusaurus-plugin-content-docs/current/advanced/extend-ngql.md @@ -0,0 +1,171 @@ +--- +id: extend-ngql +title: NQGL自定义监控 +sidebar_label: NGQL自定义监控 +--- + +> 从[自定义监控](extend-point)了解熟悉了怎么自定义类型，指标，协议等，这里我们来详细介绍下用NGQL自定义指标监控。 +> NGQL自定义监控可以让我们很方便的使用NGQL或者OpenCypher从NebulaGraph图数据库中查询指标数据,支持NebulaGraph 3.X版本。 + +### 数据解析方式 + +NGQL查询回来的数据字段和我们需要的指标映射，就能获取对应的指标数据，目前映射解析方式有四种： filterCount, oneRow, multiRow, columns + +#### **filterCount** + +> 对查询返回的结果按照指定字段进行过滤后，统计数量，通常用于 `SHOW ...` 语句中，如果可以使用NGQL语句直接返回数量的，建议使用NGQL语句统计数量。 +> commands字段语法格式: aliasField#NGQL#filterName#filterValue +> `aliasField`: 对应监控模板中的`aliasFields`中的值 +> `NGQL`: 查询语句 +> `filterName`: 过滤属性名称（可选） +> `filterValue`: 过滤属性值（可选） + +例如： +- online_meta_count#SHOW HOSTS META#Status#ONLINE + 对 `SHOW HOSTS META` 返回的结果中统计滤Status==ONLINE的数量 +- online_meta_count#SHOW HOSTS META## + 统计 `SHOW HOSTS META` 返回的行数 + +#### **oneRow** + +> 查询一行数据, 通过查询返回结果集的列名称，和查询的字段映射 + +例如： + +- 查询的指标字段为：a,b +- 查询NGQL：match (v:metrics) return v.metrics.a as a,v.metrics.b as b; + +这里指标字段就能和响应数据一一映射为一行采集数据。 + +注意事项： + +- 使用 `oneRow` 方式时单条查询语句返回多行结果时只会使用第一行结果映射到指标字段； +- 当 `commands` 字段包含两条及两条以上查询语句时，如果多条查询语句返回的字段相同时，后一条语句返回的字段会覆盖前一条； +- 定义 `commands` 时建议使用 limit 语句限制返回结果集的行数； + +#### **multiRow** + +> 查询多行数据, 通过查询返回结果集的列名称，和查询的字段映射 + +例如： +- 查询的指标字段为：a,b +- 查询NGQL：match (v:metrics) return v.metrics.a as a,v.metrics.b as b; +这里指标字段就能和响应数据一一映射为多行采集数据。 + +注意事项： + +- 使用 `multiRow` 方式时，`commands` 字段只能包含一条查询语句 + +#### **columns** + +> 采集一行指标数据, 通过查询的两列数据(key-value)，key和查询的字段匹配，value为查询字段的值 + +注意事项： + +- 使用 `columns` 方式时，会默认使用结果集的前两列映射采集数据，第一列数据对应指标名称，第二列对应指标值； +- 当 `commands` 字段包含两条及两条以上查询语句时，如果多条查询语句返回的第一列数据有重复，会使用保留最后一条的结果； + +### 自定义步骤 + +**HertzBeat页面** -> **监控模版菜单** -> **新增监控类型** -> **配置自定义监控模版YML** -> **点击保存应用** -> **使用新监控类型添加监控** + +![](/img/docs/advanced/extend-point-1.png) + +------- +下面详细介绍下文件的配置用法，请注意看使用注释。 + +### 监控模版YML + +> 监控配置定义文件用于定义 *监控类型的名称(国际化), 请求参数结构定义(前端页面根据配置自动渲染UI), 采集指标信息, 采集协议配置* 等。 +> 即我们通过自定义这个YML，配置定义什么监控类型，前端页面需要输入什么参数，采集哪些性能指标，通过什么协议去采集。 + +样例：自定义一个名称为example_ngql的自定义监控类型，其使用NGQL采集指标数据。 + +```yaml +# 监控类型所属类别：service-应用服务 program-应用程序 db-数据库 custom-自定义 os-操作系统 bigdata-大数据 mid-中间件 webserver-web服务器 cache-缓存 cn-云原生 network-网络监控等等 +category: db +# 监控应用类型(与文件名保持一致) eg: linux windows tomcat mysql aws... +app: example_ngql +name: + zh-CN: NGQL自定义监控应用 + en-US: NGQL Custom APP +# 监控参数定义. field 这些为输入参数变量，即可以用^_^host^_^的形式写到后面的配置中，系统自动变量值替换 +# 这个部分通常不用修改 +params: + # field-param field key + - field: host + name: + zh-CN: 目标Host + en-US: Target Host + type: host + required: true + - field: graphPort + name: + zh-CN: graph端口 + en-US: graphPort + type: number + range: '[0,65535]' + required: true + defaultValue: 9669 + - field: username + name: + zh-CN: 用户名 + en-US: Username + type: text + required: true + - field: password + name: + zh-CN: 密码 + en-US: Password + type: password + required: true + - field: spaceName + name: + zh-CN: 图空间 + en-US: Space Name + type: text + required: false + - field: timeout + name: + zh-CN: 连接超时时间(ms) + en-US: Connect Timeout(ms) + type: number + unit: ms + range: '[0,100000]' + required: true + defaultValue: 6000 +# 采集指标配置列表 +metrics: + - name: base_info + i18n: + zh-CN: 节点统计 + en-US: Vertex statistics + priority: 0 + fields: + - field: tag1 + type: 1 + i18n: + zh-CN: tag1 + en-US: tag1 + - field: tag1 + type: 1 + i18n: + zh-CN: tag2 + en-US: tag2 + aliasFields: + - tag1 + - tag2 + protocol: ngql + ngql: + host: ^_^host^_^ + username: ^_^username^_^ + password: ^_^password^_^ + port: ^_^graphPort^_^ + spaceName: ^_^spaceName^_^ + parseType: columns + # 定义采集数据使用的查询语句 + commands: + - match (v:tag1) return "tag1" as name ,count(v) as cnt + - match (v:tag2) return "tag2" as name ,count(v) as cnt + timeout: ^_^timeout^_^ +``` diff --git a/home/i18n/zh-cn/docusaurus-plugin-content-docs/current/help/guide.md b/home/i18n/zh-cn/docusaurus-plugin-content-docs/current/help/guide.md index 187df8cf439..5dd7df23437 100644 --- a/home/i18n/zh-cn/docusaurus-plugin-content-docs/current/help/guide.md +++ b/home/i18n/zh-cn/docusaurus-plugin-content-docs/current/help/guide.md @@ -38,6 +38,7 @@ sidebar_label: 帮助入门  👉 [IoTDB数据库监控](iotdb)
 👉 [TiDB数据库监控](tidb)
 👉 [MongoDB数据库监控](mongodb)
+  👉 [NebulaGraph集群监控](nebulagraph_cluster)
### 操作系统监控 diff --git a/home/i18n/zh-cn/docusaurus-plugin-content-docs/current/help/nebulagraph_cluster.md b/home/i18n/zh-cn/docusaurus-plugin-content-docs/current/help/nebulagraph_cluster.md new file mode 100644 index 00000000000..851f6dd7946 --- /dev/null +++ b/home/i18n/zh-cn/docusaurus-plugin-content-docs/current/help/nebulagraph_cluster.md @@ -0,0 +1,91 @@ +--- +id: nebulagraph_cluster +title: 监控：NebulaGraph集群监控 +sidebar_label: NebulaGraph集群 +keywords: [ 开源监控系统, 开源数据库监控, 开源图数据库监控, NebulaGraph集群监控 ] +--- + +> 对NebulaGraph集群的基础信息、节点，任务等进行监控。 + +**协议使用：ngql** + +### 配置参数 + +| 参数名称 | 参数帮助描述 | +|---------|------------------------------------------------------| +| 目标Host | 被监控的对端IPV4，IPV6或域名。注意⚠️不带协议头(eg: https://, http://)。 | +| 任务名称 | 标识此监控的名称，名称需要保证唯一性。 | +| graph端口 | graph服务开放的端口，默认为9669。 | +| 连接超时时间 | 连接graph服务超时时间，单位ms毫秒，默认6000毫秒。 | +| 用户名 | 数据库连接用户名。 | +| 密码 | 数据库连接密码。 | +| 采集间隔 | 监控周期性采集数据间隔时间，单位秒，可设置的最小间隔为30秒 | +| 绑定标签 | 用于对监控资源进行分类管理 | +| 描述备注 | 更多标识和描述此监控的备注信息，用户可以在这里备注信息 | + +### 采集指标 + +#### 指标集合：基础信息 + +| 指标名称 | 指标单位 | 指标帮助描述 | +|--------------|------|---------| +| responseTime | 无 | 响应时间 | +| charset | 无 | 字符集 | +| collation | 无 | 字符集排序规则 | + +#### 指标集合：Session + +| 指标名称 | 指标单位 | 指标帮助描述 | +|---------------------|------|------------| +| session | 无 | session的数量 | +| running_query_count | 无 | 正在执行的查询的数量 | + +#### 指标集合：后台任务 + +| 指标名称 | 指标单位 | 指标帮助描述 | +|--------------|------|--------------| +| queue_jobs | 无 | 等待中的后台任务 | +| running_jobs | 无 | 正在执行的后台任务的数量 | + +#### 指标集合：节点信息 + +| 指标名称 | 指标单位 | 指标帮助描述 | +|----------------------|------|-----------------| +| total_storage_node | 无 | storage节点的数量 | +| offline_storage_node | 无 | 离线的storage节点的数量 | +| total_meta_node | 无 | meta节点的数量 | +| offline_meta_node | 无 | 离线的meta节点的数量 | +| total_graph_node | 无 | graph节点数量 | +| offline_graph_node | 无 | 离线graph节点数量 | + +#### 指标集合：Storage节点 + +| 指标名称 | 指标单位 | 指标帮助描述 | +|-----------------------|------|---------------------| +| host | 无 | 节点地址 | +| port | 无 | 端口 | +| status | 无 | 状态 （ONLINE/OFFLINE） | +| leaderCount | 无 | 当前节点leader分片的数量 | +| leaderDistribution | 无 | 当前节点leader分片分布 | +| partitionDistribution | 无 | 当前节点上分片分布 | +| version | 无 | 版本 | + +#### 指标集合：Meta节点 + +| 指标名称 | 指标单位 | 指标帮助描述 | +|---------|------|---------------------| +| host | 无 | 节点地址 | +| port | 无 | 端口 | +| status | 无 | 状态 （ONLINE/OFFLINE） | +| version | 无 | 版本 | + +#### 指标集合：Graph节点 + +| 指标名称 | 指标单位 | 指标帮助描述 | +|---------|------|---------------------| +| host | 无 | 节点地址 | +| port | 无 | 端口 | +| status | 无 | 状态 （ONLINE/OFFLINE） | +| version | 无 | 版本 | + +> 如果需要自定义监控模板采集NebulaGraph集群的数据，请参考： [NGQL自定义监控](../advanced/extend-ngql.md) diff --git a/home/sidebars.json b/home/sidebars.json index 69e484d189c..6925994d99a 100755 --- a/home/sidebars.json +++ b/home/sidebars.json @@ -79,6 +79,13 @@ "items": [ "advanced/extend-push" ] + }, + { + "type": "category", + "label": "NGQL", + "items": [ + "advanced/extend-ngql" + ] } ] }, @@ -170,9 +177,9 @@ "help/dm", "help/opengauss", "help/nebulaGraph", + "help/nebulagraph_cluster", "help/tidb", "help/mongodb" - ] }, { diff --git a/manager/src/main/resources/define/app-nebula_graph_cluster.yml b/manager/src/main/resources/define/app-nebula_graph_cluster.yml index 157150fcadd..9f7458c349a 100644 --- a/manager/src/main/resources/define/app-nebula_graph_cluster.yml +++ b/manager/src/main/resources/define/app-nebula_graph_cluster.yml @@ -77,17 +77,6 @@ params: # when type is number, range is required # required-true or false required: true - - field: spaceName - # name-param field display i18n name - name: - zh-CN: 图空间 - en-US: Space Name - # type-param field type(most mapping the html input type) - type: text - # when type is number, range is required - # required-true or false - required: false - # field-param field key - field: timeout # name-param field display i18n name @@ -293,7 +282,7 @@ metrics: - name: storage i18n: zh-CN: Storage节点 - en-US: Storage Node + en-US: Storage Nodes priority: 0 fields: - field: host @@ -362,7 +351,7 @@ metrics: - name: meta i18n: zh-CN: Meta节点 - en-US: Meta Node + en-US: Meta Nodes priority: 0 fields: - field: host @@ -410,7 +399,7 @@ metrics: - name: graph i18n: zh-CN: Graph节点 - en-US: Graph Node + en-US: Graph Nodes priority: 0 fields: - field: host