Support distributed processing of DDL reorg phase

[zimulala]

Feature Request

Is your feature request related to a problem? Please describe:

TiDB currently supports the parallel processing of DDL jobs on the owner. However, the resources of a single TiDB are limited. Even if the parallel framework is supported, the speed of DDL execution is limited, and the competition for resources will affect the daily operations of TiDB, such as TPS.

Describe the feature you'd like:

At present, the time-consuming and resource-consuming phase is the reorg phase. Considering the problems that can significantly improve DDL performance and TiDB resource utilization, we will implement distributed processing in the DDL reorg phase.

Feature branch: distribute-reorg

Tech design

• Distributed Processing of DDL Reorg Phase Design #37129

Development Plan

• Step 1（sprint 11-12）

  • Global variable controls the switch of distribution reorg #37120
  • Upgrade specifies related adaptations, including creating and using new tables interfaces #37122
  • Decoupling of Backfill worker and Reorg worker codes, add backfill workerPool #37124
  • Do demo and simple performance test

• Step 2 （sprint 13）

  • Implement the write-reorg state split task related functions, and the related interfaces of backfill worker #37123
  • Implement the normal flow of distributed processing in DDL reorg phase  #37125
    • Lighting accelerates Add Index adaptation
  • Handle the failed, canceled and rollback states of the backfill job #37172
  • Added switching logic for old and new architectures #37127
  • Change the table structure to be more general #41002
  • Support for dist-reorg on partitioned tables  #41144
  • Add telemetry for distributed reorg tasks #41266
  • Test-related work

• Step 3（sprint 14）

  • Clean the tidb_ddl_backfill_history regularly #40298
  • Adjust the number of workers corresponding to backfillWorkerPool by adjusting tidb_ddl_reorg_worker_cnt #40933
  • Support distributed processing LitMerge in the BackfillStateMerging
  • Compatible with admin cancel ddl jobs
  • Compatible with the correct error and warning messages for DDL
  • Improved sync mode of txn-merge and lit-merge
  • Compatible with DDL retry methods (control retries based on tidb_ddl_error_count_limit)
  • Specification dist-reorg related log
  • Performance tuning
  • Improve the metrics for DDL dist-reorg #37128
  • Obtain the store ID for each region range #38686

• Step 4

• Backfill processes the Backfill job scheduling policy, Admin show DDL jobs supports reorg progress display, get rid of the DDL owner design, etc.

• Related Issues

  • Delete unnecessary error logs with TiDBEnableDistributeReorg #41106
  • Return a panic error when other TiDBs exit earlier than the owner #41208
  • It mayn't cleanup subtasks after cancel add index with dist-reorg #41371
  • Move some dist-reorg code to dist_* files and add the generate prefix key function #41576
  • Incorrect the remain ranges check for adding index for dist-reorg #42176
  • Backfill job needs self ReorgCtx, otherwise may affect the tmp merge state #42199

• Unified table structure related issues

  • The like in RemoveBackfillJob in job_table.go is incorrect #41579
  • Forget to use physical ID in checkBackfillJobCount #42175

Note
Above sprint information is not delievery plan，this indicate dev time schedule，Maybe changed according real project situation

[YangKeao]

Will it notify the workers after inserting tasks into the table? Or just waiting for the workers to poll the table?

I'm implementing this for TTL (in #40705) , maybe we should also consider this in the unified worker framework 🤔

[zimulala]

Will it notify the workers after inserting tasks into the table? Or just waiting for the workers to poll the table?

I'm implementing this for TTL (in #40705) , maybe we should also consider this in the unified worker framework 🤔

We do notify in https://github.com/pingcap/tidb/pull/40542/files#diff-a185df836b8926c4afe351d4def1b56d1e282c65aabe6677c2fa2c54590cc4b2R1063.

[YangKeao]

Will it notify the workers after inserting tasks into the table? Or just waiting for the workers to poll the table?
I'm implementing this for TTL (in #40705) , maybe we should also consider this in the unified worker framework thinking

We do notify in https://github.com/pingcap/tidb/pull/40542/files#diff-a185df836b8926c4afe351d4def1b56d1e282c65aabe6677c2fa2c54590cc4b2R1063.

Good. Thanks for pointing out. I think what I actually need (and how TTL's currently implemented) is something like addingBackfillJob 👍 .