Skip to content

Commit

Permalink
[KYUUBI #5381] Change the default metrics reporter to Prometheus
Browse files Browse the repository at this point in the history 
### _Why are the changes needed?_

Close #5381

change default metrics reporter to prometheus since Kyuubi 1.8

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

Closes #5344 from zhaohehuhu/Improvement-0928.

Closes #5381

84f4c82 [hezhao2] reset METRICS_REPORTERS for test case
b9ee5f7 [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/events/handler/ServerJsonLoggingEventHandlerSuite.scala
86165a6 [Cheng Pan] Update kyuubi-server/src/test/scala/org/apache/kyuubi/events/handler/ServerJsonLoggingEventHandlerSuite.scala
a3605b6 [hezhao2] set METRICS_PROMETHEUS_PORT to 0 for test cases
f1a4d28 [hezhao2] restore version number for kyuubi.metrics.reporters in doc
dae40e1 [hezhao2] change default metrics reporter to prometheus

Lead-authored-by: hezhao2 <hezhao2@cisco.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
  • Loading branch information
@zhaohehuhu @pan3793
zhaohehuhu and pan3793 committed Oct 16, 2023
1 parent 1b229b6 commit 4bb67bd
Show file tree
Hide file tree
Showing 4 changed files with 16 additions and 11 deletions.
20 changes: 10 additions & 10 deletions docs/configuration/settings.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -352,16 +352,16 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co

### Metrics

| Key | Default | Meaning | Type | Since |
|---------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
| kyuubi.metrics.console.interval | PT5S | How often should report metrics to console | duration | 1.2.0 |
| kyuubi.metrics.enabled | true | Set to true to enable kyuubi metrics system | boolean | 1.2.0 |
| kyuubi.metrics.json.interval | PT5S | How often should report metrics to JSON file | duration | 1.2.0 |
| kyuubi.metrics.json.location | metrics | Where the JSON metrics file located | string | 1.2.0 |
| kyuubi.metrics.prometheus.path | /metrics | URI context path of prometheus metrics HTTP server | string | 1.2.0 |
| kyuubi.metrics.prometheus.port | 10019 | Prometheus metrics HTTP server port | int | 1.2.0 |
| kyuubi.metrics.reporters | JSON | A comma-separated list for all metrics reporters<ul> <li>CONSOLE - ConsoleReporter which outputs measurements to CONSOLE periodically.</li> <li>JMX - JmxReporter which listens for new metrics and exposes them as MBeans.</li> <li>JSON - JsonReporter which outputs measurements to json file periodically.</li> <li>PROMETHEUS - PrometheusReporter which exposes metrics in Prometheus format.</li> <li>SLF4J - Slf4jReporter which outputs measurements to system log periodically.</li></ul> | set | 1.2.0 |
| kyuubi.metrics.slf4j.interval | PT5S | How often should report metrics to SLF4J logger | duration | 1.2.0 |
| Key | Default | Meaning | Type | Since |
|---------------------------------|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
| kyuubi.metrics.console.interval | PT5S | How often should report metrics to console | duration | 1.2.0 |
| kyuubi.metrics.enabled | true | Set to true to enable kyuubi metrics system | boolean | 1.2.0 |
| kyuubi.metrics.json.interval | PT5S | How often should report metrics to JSON file | duration | 1.2.0 |
| kyuubi.metrics.json.location | metrics | Where the JSON metrics file located | string | 1.2.0 |
| kyuubi.metrics.prometheus.path | /metrics | URI context path of prometheus metrics HTTP server | string | 1.2.0 |
| kyuubi.metrics.prometheus.port | 10019 | Prometheus metrics HTTP server port | int | 1.2.0 |
| kyuubi.metrics.reporters | PROMETHEUS | A comma-separated list for all metrics reporters<ul> <li>CONSOLE - ConsoleReporter which outputs measurements to CONSOLE periodically.</li> <li>JMX - JmxReporter which listens for new metrics and exposes them as MBeans.</li> <li>JSON - JsonReporter which outputs measurements to json file periodically.</li> <li>PROMETHEUS - PrometheusReporter which exposes metrics in Prometheus format.</li> <li>SLF4J - Slf4jReporter which outputs measurements to system log periodically.</li></ul> | set | 1.2.0 |
| kyuubi.metrics.slf4j.interval | PT5S | How often should report metrics to SLF4J logger | duration | 1.2.0 |

### Operation

Expand Down
3 changes: 3 additions & 0 deletions docs/deployment/migration-guide.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@
To restore previous behavior, set `kyuubi.metadata.store.jdbc.database.type=DERBY` and
`kyuubi.metadata.store.jdbc.url=jdbc:derby:memory:kyuubi_state_store_db;create=true`.

* Since Kyuubi 1.8, PROMETHEUS is changed as the default metrics reporter. To restore previous behavior,
set `kyuubi.metrics.reporters=JSON`.

## Upgrading from Kyuubi 1.7.1 to 1.7.2

* Since Kyuubi 1.7.2, for Kyuubi BeeLine, please use `--python-mode` option to run python code or script.
Expand Down
2 changes: 1 addition & 1 deletion kyuubi-metrics/src/main/scala/org/apache/kyuubi/metrics/MetricsConf.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ object MetricsConf {
.transformToUpperCase
.toSet()
.checkValues(ReporterType)
.createWithDefault(Set(JSON.toString))
.createWithDefault(Set(PROMETHEUS.toString))

val METRICS_CONSOLE_INTERVAL: ConfigEntry[Long] = buildConf("kyuubi.metrics.console.interval")
.doc("How often should report metrics to console")
Expand Down
2 changes: 2 additions & 0 deletions .../src/test/scala/org/apache/kyuubi/events/handler/ServerJsonLoggingEventHandlerSuite.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ import org.apache.kyuubi._
import org.apache.kyuubi.client.util.BatchUtils._
import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.events.ServerEventHandlerRegister
import org.apache.kyuubi.metrics.MetricsConf
import org.apache.kyuubi.operation.HiveJDBCTestHelper
import org.apache.kyuubi.operation.OperationState._
import org.apache.kyuubi.server.KyuubiServer
Expand All @@ -56,6 +57,7 @@ class ServerJsonLoggingEventHandlerSuite extends WithKyuubiServer with HiveJDBCT
.set(KyuubiConf.SERVER_EVENT_JSON_LOG_PATH, serverLogRoot)
.set(KyuubiConf.ENGINE_SPARK_EVENT_LOGGERS, Seq("JSON"))
.set(KyuubiConf.ENGINE_EVENT_JSON_LOG_PATH, engineLogRoot)
.set(MetricsConf.METRICS_REPORTERS, Set.empty[String])
}

override protected def jdbcUrl: String = getJdbcUrl
Expand Down

0 comments on commit 4bb67bd

Please sign in to comment.