statistics: add some doc for the exp feature #9891

winoros · 2022-08-09T06:57:59Z

First-time contributors' checklist

I've signed Contributor License Agreement that's required for repo owners to accept my contribution.

What is changed, added or deleted? (Required)

add docs for the missing part of statistics of the optimizer
close part of #3155

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions.

What is the related PR or file link(s)?

This PR is translated from:
Other reference link(s):

Do your changes match any of the following descriptions?

Delete files
Change aliases
Need modification after applied to another branch
Might cause conflicts after applied to another branch

ti-chi-bot · 2022-08-09T06:58:01Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

lilin90

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

extended-statistics.md

time-and-fate

We need to specify that this feature is experimental and is not a GA feature.

time-and-fate · 2022-08-12T12:53:20Z

extended-statistics.md

+CREATE TABLE t(col1 INT, col2 INT, KEY(col1), KEY(col2));
+```
+
+Suppose that the `col1` and `col2` of the table `t` both obey monotonically increasing constraints in row order, i.e., the values of `col1` and `col2` are strictly correlated in order (correlation value of 1):


(correlation value of 1) seems a bit confusing.

time-and-fate · 2022-08-12T12:58:00Z

extended-statistics.md

+- `stats_type` is the type of the extended statistics. Now it only has one possible value `correlation`.
+- `column_name` specifies the column group. It can be multiple columns. For `correlation` type, there should be and only be two columns.
+
+The extended statistics will be collected if the `mysql.stats_extended` has the corresponding record when we run the `ANALYZE` command. And the `status` column will be set to `1`, and the `version` column will be set to the new timestamp.


I'm afraid that if we are exposing too many implementation details to the doc.

+1. We can have a separated paragraph to show how to check valid extended statistics, in which mysql.stats_extended is described.

time-and-fate · 2022-08-12T12:58:35Z

extended-statistics.md

+## The cache of the Extended Statistics
+
+Each TiDB node will maintain a cache for the extended statistics to improve the efficiency of visiting the extended statistics. TiDB will load the table `mysql.stats_extended` periodically to ensure that the cache is kept the same as the data in the table. Each row in the table `mysql.stats_extended` records a column `version`. Once the row is updated, the value of the column `version` will be increased so that we can load the table into the memory incrementally instead of a full loading.
+To delete a record of the extended statistics, TiDB provides the following command:
+
+{{< copyable "sql" >}}
+
+```sql
+ALTER TABLE table_name DROP STATS_EXTENDED stats_name;
+```
+
+This command will mark the value of the corresponding record in the table `mysql.stats_extended`'s column `status` to `2`(meaning that the record is deleted) instead of deleting the record directly. Other TiDBs will read this change and delete the record in their memory cache. The background garbage collection will delete the record eventually.


extended-statistics.md

songrijie · 2022-08-16T06:51:52Z

extended-statistics.md

+
+The way mentioned in the chapter [Introduction to Statistics](/statistics.md) is also suitable for extended statistics. The dump result is in the same JSON file as the normal statistics.
+
+## The switch


Since extended statistics is disabled by default, it makes more sense to put "how to enable extended statistics" to the beginning of this page.

TomShawn · 2022-09-23T07:44:49Z

/remove-status LGT1
/status LGT2

TomShawn · 2022-09-23T07:44:54Z

/merge

ti-chi-bot · 2022-09-23T07:44:58Z

This pull request has been accepted and is ready to merge.

Commit hash: 820992a

ti-chi-bot · 2022-09-23T07:46:21Z

In response to a cherrypick label: new pull request created: #10533.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2022-09-23T07:46:59Z

In response to a cherrypick label: new pull request created: #10534.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

* This is an automated cherry-pick of #10534 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> * Apply suggestions from code review * resolve * Update extended-statistics.md Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com>

statistics: add some doc for the exp feature

dc242b1

ti-chi-bot requested a review from shichun-0415 August 9, 2022 06:58

ti-chi-bot added missing-translation-status This PR does not have translation status info. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 9, 2022

add the new file

fd59fb2

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 9, 2022

TomShawn self-assigned this Aug 9, 2022

TomShawn reviewed Aug 9, 2022

View reviewed changes

extended-statistics.md Outdated Show resolved Hide resolved

TomShawn self-requested a review August 9, 2022 07:23

Update extended-statistics.md

7020004

TomShawn reviewed Aug 9, 2022

View reviewed changes

extended-statistics.md Outdated Show resolved Hide resolved

Update extended-statistics.md

36c4e78

TomShawn added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 9, 2022

TomShawn requested review from time-and-fate and removed request for shichun-0415 August 10, 2022 06:27

TomShawn removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 10, 2022

time-and-fate reviewed Aug 12, 2022

View reviewed changes

TomShawn added the needs-cherry-pick-release-6.2 label Aug 15, 2022

songrijie reviewed Aug 16, 2022

View reviewed changes

extended-statistics.md Outdated Show resolved Hide resolved

songrijie reviewed Aug 16, 2022

View reviewed changes

extended-statistics.md Outdated Show resolved Hide resolved

songrijie reviewed Aug 16, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 22, 2022

TomShawn mentioned this pull request Sep 22, 2022

add extended statistics doc pingcap/docs-cn#11384

Closed

12 tasks

TomShawn added translation/done This PR has been translated from English into Chinese and updated to pingcap/docs-cn in a PR. and removed translation/doing This PR's assignee is translating this PR. labels Sep 22, 2022

add tidb_optimizer_selectivity_level

820992a

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 23, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 23, 2022

TomShawn added the needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. label Sep 23, 2022

ti-chi-bot merged commit fad6434 into pingcap:master Sep 23, 2022

ti-chi-bot mentioned this pull request Sep 23, 2022

statistics: add some doc for the exp feature (#9891) #10533

Merged

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/docs that referenced this pull request Sep 23, 2022

This is an automated cherry-pick of pingcap#9891

8184c49

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Sep 23, 2022

statistics: add some doc for the exp feature (#9891) #10534

Merged

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/docs that referenced this pull request Sep 23, 2022

This is an automated cherry-pick of pingcap#9891

194e5ad

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot added a commit that referenced this pull request Sep 23, 2022

statistics: add some doc for the exp feature (#9891) (#10534)

b270933

ti-chi-bot added a commit that referenced this pull request Sep 23, 2022

statistics: add some doc for the exp feature (#9891) (#10533)

1b1f499

ti-chi-bot mentioned this pull request Sep 23, 2022

statistics: add some doc for the exp feature (#9891) (#10534) #10537

Merged

13 tasks

shichun-0415 mentioned this pull request Feb 2, 2023

Document missing system variables #3155

Closed

22 tasks

lilin90 assigned lilin90 and unassigned TomShawn May 13, 2024

lilin90 mentioned this pull request May 14, 2024

*: add extended statistics doc and update related pages pingcap/docs-cn#17339

Merged

16 tasks

lilin90 mentioned this pull request May 29, 2024

Add warning and update wording for extended statistics #17670

Merged

17 tasks

ti-chi-bot mentioned this pull request May 31, 2024

*: add extended statistics doc and update related pages (#17339) pingcap/docs-cn#17486

Merged

16 tasks

ti-chi-bot mentioned this pull request Jun 28, 2024

Add warning and update wording for extended statistics (#17670) #18063

Merged

17 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statistics: add some doc for the exp feature #9891

statistics: add some doc for the exp feature #9891

winoros commented Aug 9, 2022 •

edited by shichun-0415

Loading

ti-chi-bot commented Aug 9, 2022 •

edited

Loading

time-and-fate left a comment

time-and-fate Aug 12, 2022

time-and-fate Aug 12, 2022

songrijie Aug 16, 2022

time-and-fate Aug 12, 2022

songrijie Aug 16, 2022

TomShawn Aug 18, 2022

TomShawn commented Sep 23, 2022

TomShawn commented Sep 23, 2022

ti-chi-bot commented Sep 23, 2022

ti-chi-bot commented Sep 23, 2022

ti-chi-bot commented Sep 23, 2022


		The way mentioned in the chapter [Introduction to Statistics](/statistics.md) is also suitable for extended statistics. The dump result is in the same JSON file as the normal statistics.

		## The switch

statistics: add some doc for the exp feature #9891

statistics: add some doc for the exp feature #9891

Conversation

winoros commented Aug 9, 2022 • edited by shichun-0415 Loading

First-time contributors' checklist

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

What is the related PR or file link(s)?

Do your changes match any of the following descriptions?

ti-chi-bot commented Aug 9, 2022 • edited Loading

time-and-fate left a comment

Choose a reason for hiding this comment

time-and-fate Aug 12, 2022

Choose a reason for hiding this comment

time-and-fate Aug 12, 2022

Choose a reason for hiding this comment

songrijie Aug 16, 2022

Choose a reason for hiding this comment

time-and-fate Aug 12, 2022

Choose a reason for hiding this comment

songrijie Aug 16, 2022

Choose a reason for hiding this comment

TomShawn Aug 18, 2022

Choose a reason for hiding this comment

TomShawn commented Sep 23, 2022

TomShawn commented Sep 23, 2022

ti-chi-bot commented Sep 23, 2022

ti-chi-bot commented Sep 23, 2022

ti-chi-bot commented Sep 23, 2022

winoros commented Aug 9, 2022 •

edited by shichun-0415

Loading

ti-chi-bot commented Aug 9, 2022 •

edited

Loading