tikv, server: turn on grpc channelz to help diagnosis and add wait conn establish metric #16263
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lysu
changed the title
|
/bench |
lysu
changed the title
Benchmark Report
@@ Benchmark Diff @@
================================================================================
--- tidb: 2a2bf377aed3bc16ce5873a30f4c6d9b15dc6c0e
+++ tidb: 784d997e3faeb595da4e2a23502c0410778ad6eb
tikv: a00b00877db7ecf9ac60a1a99d1bcc7a6b84cc24
pd: 8b8f19f2de7ae75d5d41d2c932cf6c05a7f45e10
================================================================================
oltp_update_index:
* QPS: 4341.03 ± 0.55% (std=19.64) delta: -0.09% (p=0.593)
* Latency p50: 29.46 ± 0.46% (std=0.10) delta: -0.02%
* Latency p99: 54.86 ± 5.49% (std=1.87) delta: 2.78%
oltp_insert:
* QPS: 6943.63 ± 0.38% (std=20.51) delta: -0.81% (p=0.425)
* Latency p50: 18.43 ± 0.37% (std=0.05) delta: 0.81%
* Latency p99: 32.53 ± 1.81% (std=0.59) delta: -0.51%
oltp_read_write:
* QPS: 14257.47 ± 0.18% (std=18.50) delta: -0.36% (p=0.846)
* Latency p50: 179.95 ± 0.21% (std=0.27) delta: 0.38%
* Latency p99: 341.12 ± 2.70% (std=9.21) delta: -0.27%
oltp_point_select:
* QPS: 41613.22 ± 0.21% (std=64.98) delta: 0.57% (p=0.814)
* Latency p50: 3.07 ± 0.22% (std=0.00) delta: -0.62%
* Latency p99: 10.79 ± 2.27% (std=0.16) delta: 0.00%
oltp_update_non_index:
* QPS: 4749.22 ± 0.03% (std=1.14) delta: -0.23% (p=0.304)
* Latency p50: 26.95 ± 0.04% (std=0.01) delta: 0.24%
* Latency p99: 42.36 ± 1.20% (std=0.36) delta: 3.06%
|
|
LGTM |
|
It is better to upgrade grafana script in the same PR. |
Codecov Report
@@ Coverage Diff @@
## master #16263 +/- ##
================================================
- Coverage 80.6338% 80.3976% -0.2362%
================================================
Files 506 506
Lines 137730 136137 -1593
================================================
- Hits 111057 109451 -1606
- Misses 18147 18164 +17
+ Partials 8526 8522 -4 |
|
/run-monitor-test |
|
Visit the grafana server at: http://172.16.5.5:31411, it will last for 5 hours |
|
/build |
|
@jackysp monitor fixed, PTAL~ |
|
/merge |
sre-bot
added
the
status/can-merge
|
/run-all-tests |
|
@lysu merge failed. |
|
/merge |
|
Your auto merge job has been accepted, waiting for 16008, 16009, 16175, 16232, 16233, 15738, 16132, 16133, 16218, 15585 |
|
/run-all-tests |
|
@lysu merge failed. |
|
/build |
|
/run-cherry-picker |
|
cherry pick to release-4.0 in PR #20077 |
lysu
added a commit
to ti-srebot/tidb
that referenced
this pull request
Sep 23, 2020
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Problem Summary:
channelz is one of two officially recommended tool to debug grpc question(another is log, tidb enable it by #14140).
and officially test said that turn it on by default is very low overhead, so maybe we can try it.
official view tool need install npm and start other server.....very hard to use
but there are a unofficial tool https://github.com/kazegusuri/channelzcli seems be useful.
after this Patch, TiDB status-port(default 10080) will export channelz service, then we can use https://github.com/kazegusuri/channelzcli:
127.0.0.1:40500is tikv and127.0.0.1:2379is pdwe can see fail/success count and connection event log without change tidb code
and read more about tools in channelzcli's Readme.
and maybe we can integrate channelzcli https://github.com/kazegusuri/channelzcli/blob/master/channelz/client.go into TiDB-Dashboard in future
What is changed and how it works?
What's Changed:
How it Works:
Related changes
Check List
Tests
Side effects
Release note
This change is