ddl: support pre-split index regions before creating index #57553

tangenta · 2024-11-20T08:34:33Z

What problem does this PR solve?

Issue Number: close #57552, close #57551

Problem Summary:

See #57552.

What changed and how does it work?

Support specifying the split option in the index option at parser.
Split the target index region by keys or ranges before data reorganization when adding an index.
Extract common code and tests from pkg/executor/split.go to pkg/util/split.go.
Add two functional tests.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-11-20T08:34:50Z

Hi @tangenta. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-11-20T08:55:06Z

Codecov Report

Attention: Patch coverage is 25.18519% with 303 lines in your changes missing coverage. Please review.

Project coverage is 74.9483%. Comparing base (4ec2b23) to head (d86e2ef).
Report is 123 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #57553        +/-   ##
================================================
+ Coverage   73.1893%   74.9483%   +1.7589%     
================================================
  Files          1675       1722        +47     
  Lines        461929     471007      +9078     
================================================
+ Hits         338083     353012     +14929     
+ Misses       103081      95784      -7297     
- Partials      20765      22211      +1446

Flag	Coverage Δ
integration	`49.1991% <17.0940%> (?)`
unit	`72.2659% <25.1851%> (-0.0525%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6910% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`61.7405% <ø> (+15.7437%)`	⬆️

…it-index

pkg/parser/parser.y

pkg/parser/ast/ddl.go

pkg/meta/model/job_args.go

pkg/parser/parser_test.go

pkg/ddl/index.go

pkg/meta/model/job_args.go

pkg/ddl/index_presplit.go

… messages

pkg/ddl/index_presplit.go

pkg/executor/split_test.go

Benjamin2037

LGTM

Benjamin2037 · 2024-12-13T05:46:08Z

pkg/ddl/index_presplit.go

+	if err != nil {
+		return errors.Trace(err)
+	}
+	splitOnTempIdx := reorgMeta.ReorgTp == model.ReorgTypeLitMerge ||


Does the txnMerge method still used? I thought you have disable this path already.

ti-chi-bot · 2024-12-13T07:15:32Z

[LGTM Timeline notifier]

Timeline:

2024-11-25 08:21:19.905921193 +0000 UTC m=+451867.525575708: ☑️ agreed by D3Hunter.
2024-12-13 07:15:31.46834041 +0000 UTC m=+595521.557142953: ☑️ agreed by Benjamin2037.

ti-chi-bot · 2024-12-18T02:48:04Z

@Frank945946: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tangenta · 2024-12-18T03:04:12Z

/retest

tiprow · 2024-12-18T03:04:32Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tangenta · 2024-12-18T03:04:44Z

/lgtm

ti-chi-bot · 2024-12-18T03:04:46Z

@tangenta: you cannot LGTM your own PR.

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2024-12-18T03:25:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Benjamin2037, benmeadowcroft, D3Hunter, Frank945946, tangenta, yudongusa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Benjamin2037,D3Hunter,tangenta]
~~pkg/ddl/OWNERS~~ [Benjamin2037,D3Hunter,tangenta]
~~pkg/meta/OWNERS~~ [Benjamin2037,D3Hunter,tangenta]
~~pkg/parser/OWNERS~~ [D3Hunter,benmeadowcroft,tangenta,yudongusa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-12-26T01:51:52Z

In response to a cherrypick label: new pull request created to branch release-7.5: #58541.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> resolve conflict update bazel update bazel

tangenta added 3 commits November 20, 2024 12:35

partial update

2a25d86

partial update

4966fce

fix json marshal don't support exprNode

e43c37b

ti-chi-bot bot added do-not-merge/invalid-title do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 20, 2024

tangenta changed the title ~~Support pre-split index regions before creating index~~ ddl: support pre-split index regions before creating index Nov 20, 2024

ti-chi-bot bot removed do-not-merge/invalid-title do-not-merge/needs-tests-checked labels Nov 20, 2024

tangenta added 3 commits November 20, 2024 17:33

update bazel

0345bd2

fix br integration tests

ed73d83

Merge branch 'presplit-index' of github.com:tangenta/tidb into prespl…

873a4cf

…it-index

lance6716 reviewed Nov 21, 2024

View reviewed changes

pkg/parser/parser.y Outdated Show resolved Hide resolved

pkg/parser/ast/ddl.go Outdated Show resolved Hide resolved

pkg/parser/ast/ddl.go Outdated Show resolved Hide resolved

pkg/meta/model/job_args.go Show resolved Hide resolved

tangenta added 6 commits November 21, 2024 15:22

address comments

fddb904

support parsing simple expression

56628be

fix TestSplitIndex

b8704ac

support temp index

171c001

Merge remote-tracking branch 'upstream/master' into presplit-index

7b11889

fix TestAddIndexPresplitIndexRegions

23188ee

D3Hunter reviewed Nov 25, 2024

View reviewed changes

pkg/parser/parser_test.go Show resolved Hide resolved

D3Hunter reviewed Nov 25, 2024

View reviewed changes

pkg/ddl/index.go Outdated Show resolved Hide resolved

pkg/meta/model/job_args.go Show resolved Hide resolved

pkg/ddl/index_presplit.go Outdated Show resolved Hide resolved

pkg/ddl/index_presplit.go Outdated Show resolved Hide resolved

pkg/ddl/index_presplit.go Show resolved Hide resolved

tangenta added 2 commits November 25, 2024 15:20

enhance index presplit logic: validate region count and improve error…

c96e2e3

… messages

address comment

d64f584

D3Hunter approved these changes Nov 25, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Nov 25, 2024

add functional test for index presplit regions with validation checks

abd295c

bb7133 reviewed Dec 10, 2024

View reviewed changes

pkg/ddl/index_presplit.go Outdated Show resolved Hide resolved

tangenta added 2 commits December 11, 2024 16:50

Merge remote-tracking branch 'upstream/master' into HEAD

2c538b7

address comment

d86e2ef

tangenta requested a review from Benjamin2037 December 12, 2024 07:33

Benjamin2037 reviewed Dec 12, 2024

View reviewed changes

pkg/executor/split_test.go Show resolved Hide resolved

Benjamin2037 approved these changes Dec 13, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 13, 2024

Frank945946 approved these changes Dec 18, 2024

View reviewed changes

benmeadowcroft approved these changes Dec 18, 2024

View reviewed changes

ti-chi-bot bot added the approved label Dec 18, 2024

yudongusa approved these changes Dec 18, 2024

View reviewed changes

ti-chi-bot bot merged commit 177a03c into pingcap:master Dec 18, 2024
26 checks passed

tangenta added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Dec 26, 2024

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Dec 26, 2024

This is an automated cherry-pick of pingcap#57553

72892b4

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Dec 26, 2024

ddl: support pre-split index regions before creating index (#57553) #58541

Open

13 tasks

tangenta added a commit to tangenta/tidb that referenced this pull request Jan 7, 2025

This is an automated cherry-pick of pingcap#57553

e34f0a0

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> resolve conflict update bazel update bazel

tangenta pushed a commit to tangenta/tidb that referenced this pull request Jan 7, 2025

Cherry-pick pingcap#57044, pingcap#57145 and pingcap#57553 to v7.5.1

a850faa

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: support pre-split index regions before creating index #57553

ddl: support pre-split index regions before creating index #57553

tangenta commented Nov 20, 2024 •

edited

Loading

tiprow bot commented Nov 20, 2024

codecov bot commented Nov 20, 2024 •

edited

Loading

Benjamin2037 left a comment

Benjamin2037 Dec 13, 2024

ti-chi-bot bot commented Dec 13, 2024

ti-chi-bot bot commented Dec 18, 2024

tangenta commented Dec 18, 2024

tiprow bot commented Dec 18, 2024

tangenta commented Dec 18, 2024

ti-chi-bot bot commented Dec 18, 2024

ti-chi-bot bot commented Dec 18, 2024

ti-chi-bot commented Dec 26, 2024

ddl: support pre-split index regions before creating index #57553

ddl: support pre-split index regions before creating index #57553

Conversation

tangenta commented Nov 20, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Nov 20, 2024

codecov bot commented Nov 20, 2024 • edited Loading

Codecov Report

Benjamin2037 left a comment

Choose a reason for hiding this comment

Benjamin2037 Dec 13, 2024

Choose a reason for hiding this comment

ti-chi-bot bot commented Dec 13, 2024

[LGTM Timeline notifier]

ti-chi-bot bot commented Dec 18, 2024

tangenta commented Dec 18, 2024

tiprow bot commented Dec 18, 2024

tangenta commented Dec 18, 2024

ti-chi-bot bot commented Dec 18, 2024

ti-chi-bot bot commented Dec 18, 2024

ti-chi-bot commented Dec 26, 2024

tangenta commented Nov 20, 2024 •

edited

Loading

codecov bot commented Nov 20, 2024 •

edited

Loading