ddl: support batch create table #28763

xhebox · 2021-10-12T13:34:23Z

What problem does this PR solve?

Issue Number: refer #30272

Problem Summary: add an internal batch create table api for br, all happens in one schema version.

Some notes on the behavior:

This API does not set query since it is not from parser. One must set ctx.Value(sessionctx.QueryString) manually for displaying admin show ddl queries.
admin show ddl jobs can not show multiple ids in tableID column, but will show comma separated list of table names in tableName column.

What is changed and how it works?

What's Changed:

add CreateTablesWithInfo, the new batch api.
Common part in CreateTableWithInfo and CreateTablesWithInfo are extracted.
Common part in onCreateTable and onCreateTables are extracted.
add simple unit tests in db_test.go and table_test.go.
add Affected []HistoryInfoAffected in binlog.HistoryInfo for display multiple table names.
When the job is cancelled, its arguments will be deleted. Since someone may issued a super large job that does not fit into the history queue.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2021-10-12T13:34:25Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

AilinKid
wjhuang2016

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

infoschema/builder.go

fengou1 · 2021-10-18T02:05:10Z

/cc @wjhuang2016

ddl/ddl.go

fengou1 · 2021-10-19T01:53:26Z

/run-check_dev_2

fengou1 · 2021-10-19T03:11:27Z

/run-check_dev_2

xhebox · 2021-10-19T04:26:18Z

/run-check_dev_2

It is a problem of the unit test code, should be fixed now.

ddl/ddl.go

ddl/ddl_api.go

ddl/table.go

fengou1 · 2021-10-20T02:27:01Z

A integration test show the number of tables > 10000, the function call CreateTablesWithInfo got stuck. as private communication with xhebox, it looks CreateTablesWithInfo hit the raft entry max size limitation ( the sum of batch tables shall less than 500 if raft-entry-max-size = 8M (by default))

This PR shall return a error when we hit the max raft limitation instead of stuck
BR batch size set to 500 (possible restore table speed 50 per-seconds)

fengou1 · 2021-10-20T10:05:36Z

BR batch size set to 500, restore successful. I think we are good to merge this PR to master.

fengou1 · 2021-12-21T07:34:02Z

/run-unit-test

ti-chi-bot · 2021-12-31T06:03:50Z

This pull request has been accepted and is ready to merge.

Commit hash: 319fbda

xhebox · 2021-12-31T06:05:25Z

/hold

xhebox · 2021-12-31T06:11:21Z

/unhold

xhebox · 2021-12-31T09:21:20Z

/cherry-pick release-5.4

ti-srebot · 2021-12-31T09:21:26Z

cherry pick to release-5.4 in PR #31234

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

* topsql: make topsql enable only be controlled by pub/sub sink (pingcap#31209) * ddl: support batch create table (pingcap#28763) * executor: fix data race in IndexMergeReaderExec (pingcap#31230) close pingcap#31229 * server: filter the EOF error for normal closed at handshake (pingcap#31081) close pingcap#31063 * expression: change date add function return type (pingcap#28133) close pingcap#27573 * support create interval partition Signed-off-by: crazycs520 <crazycs520@gmail.com> * support create interval partition (support int/timestamp partition key) Signed-off-by: crazycs520 <crazycs520@gmail.com> * parser: support alter table partitions move engine statement Signed-off-by: crazycs520 <crazycs520@gmail.com> * support ddl operation Signed-off-by: crazycs520 <crazycs520@gmail.com> * support interval partition manager Signed-off-by: crazycs520 <crazycs520@gmail.com> * support interval partition manager handle job framwork Signed-off-by: crazycs520 <crazycs520@gmail.com> * support auto create interval partition when insert meet no partition suitable error Signed-off-by: crazycs520 <crazycs520@gmail.com> * fix bug Signed-off-by: crazycs520 <crazycs520@gmail.com> * fix cancel job and load old job then continue to do Signed-off-by: crazycs520 <crazycs520@gmail.com> * make partition readonly work(not allow to insert/update/delete) Signed-off-by: crazycs520 <crazycs520@gmail.com> * add begin,end time in tables Signed-off-by: crazycs520 <crazycs520@gmail.com> * tiny fix for auto create interval partition in concurrent case Signed-off-by: crazycs520 <crazycs520@gmail.com> * init Signed-off-by: crazycs520 <crazycs520@gmail.com> * init Signed-off-by: crazycs520 <crazycs520@gmail.com> * todo: remove flag Signed-off-by: crazycs520 <crazycs520@gmail.com> * fix dumpling Signed-off-by: crazycs520 <crazycs520@gmail.com> * remove data in aws s3 when drop/truncate table/partition Signed-off-by: crazycs520 <crazycs520@gmail.com> * make hello world work Signed-off-by: crazycs520 <crazycs520@gmail.com> * remove debug info Signed-off-by: crazycs520 <crazycs520@gmail.com> Co-authored-by: xhe <xw897002528@gmail.com> Co-authored-by: guo-shaoge <shaoge1994@163.com> Co-authored-by: knull-cn <hu__haifeng@163.com> Co-authored-by: Meng Xin <tregoldmeng@gmail.com>

xhebox requested review from zimulala and AilinKid October 12, 2021 13:34

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 12, 2021

xhebox commented Oct 12, 2021

View reviewed changes

infoschema/builder.go Outdated Show resolved Hide resolved

ti-chi-bot requested a review from wjhuang2016 October 18, 2021 02:05

AilinKid reviewed Oct 18, 2021

View reviewed changes

ddl/ddl.go Outdated Show resolved Hide resolved

wjhuang2016 reviewed Oct 19, 2021

View reviewed changes

ddl/ddl.go Outdated Show resolved Hide resolved

ddl/ddl_api.go Show resolved Hide resolved

ddl/ddl_api.go Show resolved Hide resolved

ddl/ddl_api.go Outdated Show resolved Hide resolved

ddl/ddl_api.go Show resolved Hide resolved

ddl/table.go Show resolved Hide resolved

xhebox requested review from wjhuang2016 and AilinKid October 20, 2021 10:40

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 26, 2021

xhebox force-pushed the batch_1 branch from 9cd696a to f047704 Compare October 26, 2021 09:21

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 26, 2021

xhebox force-pushed the batch_1 branch from 7b2b8b6 to d21bc5c Compare October 27, 2021 08:52

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 27, 2021

xhebox force-pushed the batch_1 branch from fa313fc to 84bdeaf Compare December 15, 2021 13:02

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 15, 2021

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 22, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 31, 2021

Merge branch 'master' into batch_1

ed97b5b

ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 31, 2021

ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 31, 2021

ti-chi-bot added 7 commits December 31, 2021 14:17

Merge branch 'master' into batch_1

5c69dd2

Merge branch 'master' into batch_1

9bb0f9b

Merge branch 'master' into batch_1

97287c5

Merge branch 'master' into batch_1

599b287

Merge branch 'master' into batch_1

27ccc34

Merge branch 'master' into batch_1

75559f9

Merge branch 'master' into batch_1

de56dc3

ti-chi-bot merged commit 4f30a14 into pingcap:master Dec 31, 2021

xhebox added the needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. label Dec 31, 2021

ti-srebot mentioned this pull request Dec 31, 2021

ddl: support batch create table (#28763) #31234

Closed

12 tasks

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Dec 31, 2021

cherry pick pingcap#28763 to release-5.4

e8c03db

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

GMHDBJD mentioned this pull request Jan 12, 2022

DM: batch create table for schema tracker pingcap/tiflow#4309

Closed

xhebox mentioned this pull request Jan 24, 2022

A new TiDB batch ddl interface to create tables #30272

Closed

jiaqizho mentioned this pull request Mar 8, 2022

SchemaBuilder unable to apply diff from BR pingcap/tiflash#4193

Closed

JaySon-Huang mentioned this pull request Mar 10, 2022

Support ddl job CreateTables in TiFlash pingcap/tiflash#4201

Merged

12 tasks

fengou1 removed the needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. label Mar 10, 2022

xhebox deleted the batch_1 branch August 16, 2022 16:26

xhebox restored the batch_1 branch August 16, 2022 16:26

kennytm mentioned this pull request Mar 21, 2023

Proposal: add an option PREFER_TABLE_ID to CREATE TABLE to set the preferred Table ID #14030

Open

D3Hunter mentioned this pull request Jul 31, 2024

ddl: integrate fast create table into normal general DDL workflow #55025

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: support batch create table #28763

ddl: support batch create table #28763

xhebox commented Oct 12, 2021 •

edited by fengou1

Loading

ti-chi-bot commented Oct 12, 2021 •

edited

Loading

fengou1 commented Oct 18, 2021

fengou1 commented Oct 19, 2021

fengou1 commented Oct 19, 2021

xhebox commented Oct 19, 2021

fengou1 commented Oct 20, 2021 •

edited

Loading

fengou1 commented Oct 20, 2021

fengou1 commented Dec 21, 2021

ti-chi-bot commented Dec 31, 2021

xhebox commented Dec 31, 2021

xhebox commented Dec 31, 2021

xhebox commented Dec 31, 2021

ti-srebot commented Dec 31, 2021

ddl: support batch create table #28763

ddl: support batch create table #28763

Conversation

xhebox commented Oct 12, 2021 • edited by fengou1 Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Oct 12, 2021 • edited Loading

fengou1 commented Oct 18, 2021

fengou1 commented Oct 19, 2021

fengou1 commented Oct 19, 2021

xhebox commented Oct 19, 2021

fengou1 commented Oct 20, 2021 • edited Loading

fengou1 commented Oct 20, 2021

fengou1 commented Dec 21, 2021

ti-chi-bot commented Dec 31, 2021

xhebox commented Dec 31, 2021

xhebox commented Dec 31, 2021

xhebox commented Dec 31, 2021

ti-srebot commented Dec 31, 2021

xhebox commented Oct 12, 2021 •

edited by fengou1

Loading

ti-chi-bot commented Oct 12, 2021 •

edited

Loading

fengou1 commented Oct 20, 2021 •

edited

Loading