bootstrap: use reserved ID for system schemas/tables in nextgen by D3Hunter · Pull Request #62570 · pingcap/tidb

D3Hunter · 2025-07-23T05:13:56Z

What problem does this PR solve?

Issue Number: ref #61702

Problem Summary:

What changed and how does it work?

for nextgen, we initialize system dbs/tables in the same way we init DDL related tables, without going through DDL, so we can use reserved ID to create them.

with reserved ID, we can load related Database or schema without scanning all of them, this also can make cross keyspace loading faster

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

after bootstrap with this PR

mysql> admin show ddl jobs;
+--------+---------+-----------------------+---------------+--------------+-----------------+----------+-----------+----------------------------+----------------------------+----------------------------+--------+----------+
| JOB_ID | DB_NAME | TABLE_NAME            | JOB_TYPE      | SCHEMA_STATE | SCHEMA_ID       | TABLE_ID | ROW_COUNT | CREATE_TIME                | START_TIME                 | END_TIME                   | STATE  | COMMENTS |
+--------+---------+-----------------------+---------------+--------------+-----------------+----------+-----------+----------------------------+----------------------------+----------------------------+--------+----------+
|      6 | test    |                       | create schema | public       |               5 |        0 |         0 | 2025-07-23 15:47:30.589000 | 2025-07-23 15:47:30.589000 | 2025-07-23 15:47:30.640000 | synced |          |
|      4 | sys     | schema_unused_indexes | create view   | public       | 281474976710595 |        3 |         0 | 2025-07-23 15:47:30.540000 | 2025-07-23 15:47:30.589000 | 2025-07-23 15:47:30.589000 | synced |          |
|      2 | mysql   | tidb_mdl_view         | create view   | public       | 281474976710655 |        1 |         0 | 2025-07-23 15:47:30.489000 | 2025-07-23 15:47:30.540000 | 2025-07-23 15:47:30.540000 | synced |          |
+--------+---------+-----------------------+---------------+--------------+-----------------+----------+-----------+----------------------------+----------------------------+----------------------------+--------+----------+
3 rows in set (0.01 sec)

mysql> select tidb_table_id, table_name from information_schema.tables where table_schema='mysql';
+-----------------+----------------------------------+
| tidb_table_id   | table_name                       |
+-----------------+----------------------------------+
|               1 | tidb_mdl_view                    |
| 281474976710596 | tidb_workload_values             |
| 281474976710597 | tidb_kernel_options              |
| 281474976710598 | index_advisor_results            |
| 281474976710599 | tidb_restore_registry            |
| 281474976710600 | tidb_pitr_id_map                 |
| 281474976710601 | request_unit_by_group            |
| 281474976710602 | dist_framework_meta              |
| 281474976710603 | tidb_runaway_watch_done          |
| 281474976710604 | tidb_timers                      |
| 281474976710605 | tidb_runaway_queries             |
| 281474976710606 | tidb_runaway_watch               |
| 281474976710607 | tidb_import_jobs                 |
| 281474976710608 | tidb_global_task_history         |
| 281474976710609 | tidb_global_task                 |
| 281474976710610 | tidb_ttl_job_history             |
| 281474976710611 | tidb_ttl_task                    |
| 281474976710612 | tidb_ttl_table_status            |
| 281474976710613 | stats_table_locked               |
| 281474976710614 | plan_replayer_task               |
| 281474976710615 | plan_replayer_status             |
| 281474976710616 | advisory_locks                   |
| 281474976710617 | analyze_jobs                     |
| 281474976710618 | stats_meta_history               |
| 281474976710619 | stats_history                    |
| 281474976710620 | analyze_options                  |
| 281474976710621 | table_cache_meta                 |
| 281474976710622 | column_stats_usage               |
| 281474976710623 | capture_plan_baselines_blacklist |
| 281474976710624 | global_grants                    |
| 281474976710625 | stats_fm_sketch                  |
| 281474976710626 | stats_extended                   |
| 281474976710627 | opt_rule_blacklist               |
| 281474976710628 | expr_pushdown_blacklist          |
| 281474976710629 | stats_top_n                      |
| 281474976710630 | bind_info                        |
| 281474976710631 | default_roles                    |
| 281474976710632 | role_edges                       |
| 281474976710633 | stats_feedback                   |
| 281474976710634 | gc_delete_range_done             |
| 281474976710635 | gc_delete_range                  |
| 281474976710636 | stats_buckets                    |
| 281474976710637 | stats_histograms                 |
| 281474976710638 | stats_meta                       |
| 281474976710639 | help_topic                       |
| 281474976710640 | tidb                             |
| 281474976710641 | global_variables                 |
| 281474976710642 | columns_priv                     |
| 281474976710643 | tables_priv                      |
| 281474976710644 | db                               |
| 281474976710645 | global_priv                      |
| 281474976710646 | password_history                 |
| 281474976710647 | user                             |
| 281474976710648 | tidb_ddl_notifier                |
| 281474976710649 | tidb_background_subtask_history  |
| 281474976710650 | tidb_background_subtask          |
| 281474976710651 | tidb_mdl_info                    |
| 281474976710652 | tidb_ddl_history                 |
| 281474976710653 | tidb_ddl_reorg                   |
| 281474976710654 | tidb_ddl_job                     |
+-----------------+----------------------------------+
60 rows in set (0.01 sec)

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2025-07-23T05:14:10Z

Hi @D3Hunter. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-07-23T05:32:18Z

Codecov Report

Attention: Patch coverage is 52.57732% with 46 lines in your changes missing coverage. Please review.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #62570        +/-   ##
================================================
+ Coverage   72.8737%   74.9689%   +2.0952%     
================================================
  Files          1763       1813        +50     
  Lines        485946     498146     +12200     
================================================
+ Hits         354127     373455     +19328     
+ Misses       110165     101589      -8576     
- Partials      21654      23102      +1448

Flag	Coverage Δ
integration	`48.9060% <37.1134%> (?)`
unit	`72.3173% <52.5773%> (+0.1749%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7804% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`63.2512% <ø> (+16.9158%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

D3Hunter · 2025-07-23T07:25:33Z

/hold

the bootstrap became slower, as we scatter region in serial

D3Hunter · 2025-07-23T07:58:04Z

/unhold

ti-chi-bot · 2025-07-23T08:19:52Z

[LGTM Timeline notifier]

Timeline:

2025-07-23 06:53:59.256770048 +0000 UTC m=+147595.994196258: ☑️ agreed by wjhuang2016.
2025-07-23 08:19:50.509130065 +0000 UTC m=+152747.246556272: ☑️ agreed by tangenta.

lcwangchao

Other part LGTM

But I think this PR makes the boostrap procedure more complex. Do we need to involve more guys to have a review?

lcwangchao · 2025-07-23T09:24:28Z

 )

+// BootTableVersion is the version of nextgen bootstrapping.
+type BootTableVersion int


Is NextGenBootTableVersion better? It is only used by next-gen now.

lcwangchao · 2025-07-23T09:26:17Z

+
+var versionedBootstrapSchemas = []versionedBootstrapSchema{
+	{ver: meta.BaseBootTableVersion, databases: []DatabaseBasicInfo{
+		{ID: metadef.SystemDatabaseID, Name: mysql.SystemDB, Tables: tablesInSystemDatabase[:52]},


Could you add some comments here to explain why use 52 as a slice? I think it's also better to make 52 as a const.

lcwangchao · 2025-07-23T09:27:53Z

+	databases []DatabaseBasicInfo
+}
+
+var versionedBootstrapSchemas = []versionedBootstrapSchema{


It's better to add more tests here to test the below case:

If in the future a developer add a new table in tablesInSystemDatabase, it should also add a new item in versionedBootstrapSchemas, otherwise, in next-gen bootstrap, the new table will be lost.

TestMySQLDBTables and TestVersionedBootstrapSchemas are used to check those constraint, to avoid potential human error

D3Hunter · 2025-07-23T11:42:13Z

/retest

tiprow · 2025-07-23T11:42:35Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-07-23T12:54:30Z

/retest

tiprow · 2025-07-23T12:54:53Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-07-24T02:22:50Z

/unhold

D3Hunter · 2025-07-24T03:08:41Z

/retest

tiprow · 2025-07-24T03:09:03Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-07-24T03:25:53Z

/retest

tiprow · 2025-07-24T03:26:16Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-07-24T03:47:08Z

/retest

tiprow · 2025-07-24T03:47:31Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-07-24T05:17:54Z

/retest

tiprow · 2025-07-24T05:18:17Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-07-24T05:33:34Z

/retest

tiprow · 2025-07-24T05:33:57Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-07-24T06:19:41Z

/retest

tiprow · 2025-07-24T06:20:03Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-07-24T06:49:48Z

/retest

tiprow · 2025-07-24T06:50:11Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-07-24T07:38:36Z

/retest

tiprow · 2025-07-24T07:39:01Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-07-24T08:19:45Z

/retest

tiprow · 2025-07-24T08:20:08Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-07-24T08:35:39Z

/retest

tiprow · 2025-07-24T08:36:03Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-07-24T09:34:28Z

/retest

tiprow · 2025-07-24T09:34:51Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-07-24T15:41:22Z

/retest

tiprow · 2025-07-24T15:41:44Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

…cap#62570) ref pingcap#61702

D3Hunter added 2 commits July 23, 2025 13:08

change

e3d3ec0

change

564e93a

ti-chi-bot Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 23, 2025

D3Hunter added 2 commits July 23, 2025 14:39

test

f59ed82

change

1e44fee

D3Hunter changed the title ~~[wip]bootstrap: use reserved ID for system schema in nextgen~~ bootstrap: use reserved ID for system schema in nextgen Jul 23, 2025

ti-chi-bot Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 23, 2025

D3Hunter changed the title ~~bootstrap: use reserved ID for system schema in nextgen~~ bootstrap: use reserved ID for system schemas/tables in nextgen Jul 23, 2025

wjhuang2016 approved these changes Jul 23, 2025

View reviewed changes

ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jul 23, 2025

ti-chi-bot Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 23, 2025

opt split

f43df0d

ti-chi-bot Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 23, 2025

tangenta approved these changes Jul 23, 2025

View reviewed changes

ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 23, 2025

lcwangchao reviewed Jul 23, 2025

View reviewed changes

fix comments

d80bebe

fix test

bcbe8e0

ti-chi-bot Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 24, 2025

ti-chi-bot Bot merged commit cfe9c12 into pingcap:master Jul 24, 2025
29 checks passed

D3Hunter deleted the fix-sys-obj-id-nextgen branch July 25, 2025 01:05

JaySon-Huang pushed a commit to JaySon-Huang/tidb that referenced this pull request Aug 10, 2025

bootstrap: use reserved ID for system schemas/tables in nextgen (ping…

92edc75

…cap#62570) ref pingcap#61702

D3Hunter mentioned this pull request Aug 12, 2025

DXF service on nextgen kernel #61702

Open

Uh oh!

Conversation

D3Hunter commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow Bot commented Jul 23, 2025

Uh oh!

codecov Bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

D3Hunter commented Jul 23, 2025

Uh oh!

D3Hunter commented Jul 23, 2025

Uh oh!

ti-chi-bot Bot commented Jul 23, 2025

[LGTM Timeline notifier]

Uh oh!

lcwangchao left a comment

Choose a reason for hiding this comment

Uh oh!

lcwangchao Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

lcwangchao Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

lcwangchao Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

D3Hunter Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

D3Hunter commented Jul 23, 2025

Uh oh!

tiprow Bot commented Jul 23, 2025

Uh oh!

D3Hunter commented Jul 23, 2025

Uh oh!

tiprow Bot commented Jul 23, 2025

Uh oh!

D3Hunter commented Jul 24, 2025

Uh oh!

D3Hunter commented Jul 24, 2025

Uh oh!

tiprow Bot commented Jul 24, 2025

Uh oh!

D3Hunter commented Jul 24, 2025

Uh oh!

tiprow Bot commented Jul 24, 2025

Uh oh!

D3Hunter commented Jul 24, 2025

Uh oh!

tiprow Bot commented Jul 24, 2025

Uh oh!

D3Hunter commented Jul 24, 2025

Uh oh!

tiprow Bot commented Jul 24, 2025

Uh oh!

D3Hunter commented Jul 24, 2025

Uh oh!

tiprow Bot commented Jul 24, 2025

Uh oh!

D3Hunter commented Jul 24, 2025

Uh oh!

tiprow Bot commented Jul 24, 2025

Uh oh!

D3Hunter commented Jul 24, 2025

Uh oh!

tiprow Bot commented Jul 24, 2025

Uh oh!

D3Hunter commented Jul 24, 2025

Uh oh!

tiprow Bot commented Jul 24, 2025

Uh oh!

D3Hunter commented Jul 24, 2025

D3Hunter commented Jul 23, 2025 •

edited

Loading

codecov Bot commented Jul 23, 2025 •

edited

Loading