store/tikv: enlarge default committer-concurrency to avoid queuing #23720
Conversation
Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
youjiali1995
added
sig/transaction
ti-chi-bot
added
the
size/XS
|
/run-monitor-test |
|
Visit the grafana server at: http://172.16.5.5:31868, it will last for 5 hours |
| @@ -1644,7 +1644,7 @@ func (batchExe *batchExecutor) process(batches []batchMutations) error { | |||
| } | |||
| } | |||
| close(exitCh) | |||
| metrics.TiKVTokenWaitDuration.Observe(batchExe.tokenWaitDuration.Seconds()) | |||
| metrics.TiKVTokenWaitDuration.Observe(float64(batchExe.tokenWaitDuration.Nanoseconds())) | |||
There was a problem hiding this comment.
The tokenWaitDuration is a duration type which has nano second count, but in #17268 the observed value is changed to seconds(see the grafana picture in 17268). I'm a bit confused which is correct.
There was a problem hiding this comment.
The bucket is defined to be Buckets: prometheus.ExponentialBuckets(1, 2, 34), // 1ns ~ 8s. The base is 1, so it should be a nanosecond.
There was a problem hiding this comment.
It's expected to be observed in nanoseconds. #17268 doesn't fix this. The root cause is the mismatch between the metric and the grafana panel.
TiKVTokenWaitDuration = prometheus.NewHistogram(
prometheus.HistogramOpts{
Namespace: namespace,
Subsystem: subsystem,
Name: "batch_executor_token_wait_duration",
Buckets: prometheus.ExponentialBuckets(1, 2, 34), // 1ns ~ 8s
Help: "tidb txn token wait duration to process batches",
})
|
@cfzjywxk PTAL |
|
@cfzjywxk: Please use If you have approved this PR, please ignore this reply. This reply is being used as a temporary reply during the migration of the new bot and will be removed on April 1. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by writing |
ti-chi-bot
added
status/LGT2
|
/merge |
|
This pull request has been accepted and is ready to merge. Commit hash: d3b1543
|
ti-chi-bot
added
the
status/can-merge
|
cherry pick to release-4.0 in PR #23775 |
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
|
cherry pick to release-5.0 in PR #23776 |
Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
Signed-off-by: youjiali1995 zlwgx1023@gmail.com
What problem does this PR solve?
Issue Number: close #22516
Problem Summary:
The default committer-concurrency(16) is too small. It results in queuing in TiDB if a transaction has more than 16 requests.
What is changed and how it works?
What's Changed:
Enlarge the default value to 128. Committer concurrency is introduced in v4.0 to reduce pressure(server is busy) on TiKV. However, I have met several times the performance is bad due to queuing in TiDB. I prefer enlarging the default value to making TiKV less pressure.
Related changes
pingcap/docs/pingcap/docs-cn:Check List
Tests
Side effects
Release note