ddl: arg v2 for modify column and multi schema change #56536

joechenrh · 2024-10-10T09:09:39Z

What problem does this PR solve?

Issue Number: ref #54436

Problem Summary:

What changed and how does it work?

Refactor args for Modify Columns.

Since all job arguments are refactored, multi schema change will be refactored in this PR too.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-10-10T09:09:54Z

Hi @joechenrh. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-10-10T09:26:09Z

Codecov Report

Attention: Patch coverage is 90.15544% with 19 lines in your changes missing coverage. Please review.

Project coverage is 57.1304%. Comparing base (aade0ee) to head (a023ee6).
Report is 9 commits behind head on master.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #56536         +/-   ##
=================================================
- Coverage   73.4169%   57.1304%   -16.2866%     
=================================================
  Files          1629       1778        +149     
  Lines        450073     638238     +188165     
=================================================
+ Hits         330430     364628      +34198     
- Misses        99369     248653     +149284     
- Partials      20274      24957       +4683

Flag	Coverage Δ
integration	`39.2143% <69.9481%> (?)`
unit	`72.5296% <89.6373%> (-0.0187%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.9478% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`63.1318% <ø> (+17.5688%)`	⬆️

joechenrh · 2024-10-10T11:38:57Z

/retest

tiprow · 2024-10-10T11:39:18Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2024-10-10T12:16:14Z

/retest-required

tiprow · 2024-10-10T12:16:36Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2024-10-10T12:30:17Z

/retest

tiprow · 2024-10-10T12:30:38Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter

seems this pr need to change, as we have merged the args inside #56130, will review after that

ti-chi-bot · 2024-10-15T09:41:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, wjhuang2016

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/ddl/OWNERS~~ [D3Hunter,wjhuang2016]
~~pkg/meta/OWNERS~~ [D3Hunter,wjhuang2016]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-10-15T09:42:02Z

[LGTM Timeline notifier]

Timeline:

2024-10-15 09:24:13.804059134 +0000 UTC m=+349450.952968966: ☑️ agreed by D3Hunter.
2024-10-15 09:42:01.097465037 +0000 UTC m=+350518.246374869: ☑️ agreed by wjhuang2016.

D3Hunter · 2024-10-15T09:45:00Z

/retest

tiprow · 2024-10-15T09:45:22Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2024-10-15T09:53:51Z

/retest

tiprow · 2024-10-15T09:54:13Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2024-10-15T10:02:43Z

panics

[2024/10/15 09:57:53.538 +00:00] [ERROR] [misc.go:115] ["panic in the recoverable goroutine"] [label=ddl] [funcInfo=submitLoop] [r="runtime error: invalid memory address or nil pointer dereference"] [stack="github.com/pingcap/tidb/pkg/util.Recover
  pkg/util/misc.go:119
runtime.gopanic
  GOROOT/src/runtime/panic.go:785
runtime.panicmem
  GOROOT/src/runtime/panic.go:262
runtime.sigpanic
  GOROOT/src/runtime/signal_unix.go:900
github.com/pingcap/tidb/pkg/meta/model.(*Job).FillArgs
  pkg/meta/model/job.go:489
github.com/pingcap/tidb/pkg/ddl.(*JobWrapper).FillArgsWithSubJobs
  pkg/ddl/ddl.go:239
github.com/pingcap/tidb/pkg/ddl.insertDDLJobs2Table
  pkg/ddl/job_submitter.go:686
github.com/pingcap/tidb/pkg/ddl.(*JobSubmitter).GenGIDAndInsertJobsWithRetry.func1
  pkg/ddl/job_submitter.go:444
github.com/pingcap/tidb/pkg/ddl.genGIDAndCallWithRetry.func1
  pkg/ddl/job_submitter.go:614
github.com/pingcap/tidb/pkg/ddl.genGIDAndCallWithRetry
  pkg/ddl/job_submitter.go:618
github.com/pingcap/tidb/pkg/ddl.(*JobSubmitter).GenGIDAndInsertJobsWithRetry
  pkg/ddl/job_submitter.go:424
github.com/pingcap/tidb/pkg/ddl.(*JobSubmitter).addBatchDDLJobs2Table
  pkg/ddl/job_submitter.go:351
github.com/pingcap/tidb/pkg/ddl.(*JobSubmitter).addBatchDDLJobs
  pkg/ddl/job_submitter.go:119
github.com/pingcap/tidb/pkg/ddl.(*JobSubmitter).submitLoop
  pkg/ddl/job_submitter.go:93
github.com/pingcap/tidb/pkg/ddl.(*ddl).Start.func1
  pkg/ddl/ddl.go:735
github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1
  pkg/util/wait_group_wrapper.go:157"] [stack="github.com/pingcap/tidb/pkg/util.Recover
  pkg/util/misc.go:115
runtime.gopanic
  GOROOT/src/runtime/panic.go:785
runtime.panicmem
  GOROOT/src/runtime/panic.go:262
runtime.sigpanic
  GOROOT/src/runtime/signal_unix.go:900
github.com/pingcap/tidb/pkg/meta/model.(*Job).FillArgs
  pkg/meta/model/job.go:489
github.com/pingcap/tidb/pkg/ddl.(*JobWrapper).FillArgsWithSubJobs
  pkg/ddl/ddl.go:239
github.com/pingcap/tidb/pkg/ddl.insertDDLJobs2Table
  pkg/ddl/job_submitter.go:686
github.com/pingcap/tidb/pkg/ddl.(*JobSubmitter).GenGIDAndInsertJobsWithRetry.func1
  pkg/ddl/job_submitter.go:444
github.com/pingcap/tidb/pkg/ddl.genGIDAndCallWithRetry.func1
  pkg/ddl/job_submitter.go:614
github.com/pingcap/tidb/pkg/ddl.genGIDAndCallWithRetry
  pkg/ddl/job_submitter.go:618
github.com/pingcap/tidb/pkg/ddl.(*JobSubmitter).GenGIDAndInsertJobsWithRetry
  pkg/ddl/job_submitter.go:424
github.com/pingcap/tidb/pkg/ddl.(*JobSubmitter).addBatchDDLJobs2Table
  pkg/ddl/job_submitter.go:351
github.com/pingcap/tidb/pkg/ddl.(*JobSubmitter).addBatchDDLJobs
  pkg/ddl/job_submitter.go:119
github.com/pingcap/tidb/pkg/ddl.(*JobSubmitter).submitLoop
  pkg/ddl/job_submitter.go:93
github.com/pingcap/tidb/pkg/ddl.(*ddl).Start.func1
  pkg/ddl/ddl.go:735
github.com/pingcap/tidb/pkg/util.(*WaitGroupWrapper).Run.func1
  pkg/util/wait_group_wrapper.go:157"]

joechenrh · 2024-10-15T15:00:36Z

/retest

tiprow · 2024-10-15T15:00:57Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2024-10-16T03:48:27Z

/retest

tiprow · 2024-10-16T03:48:50Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2024-10-16T03:51:16Z

/ok-to-test

D3Hunter · 2024-10-16T04:22:11Z

/retest

D3Hunter · 2024-10-16T04:33:37Z

/retest

joechenrh · 2024-10-16T05:16:01Z

/retest

joechenrh · 2024-10-16T06:14:11Z

/retest

D3Hunter · 2024-10-16T07:05:35Z

/retest

joechenrh · 2024-10-16T08:38:21Z

/retest

D3Hunter · 2024-10-16T09:26:50Z

/retest

joechenrh added 3 commits October 10, 2024 11:39

Add code

b3ec874

Update argument

510d354

Add test

0ec4c86

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 10, 2024

joechenrh added 2 commits October 10, 2024 19:15

Fix

d3386da

Fix

ef23486

Remove comment

fffff65

joechenrh added 2 commits October 10, 2024 20:38

Update

6c9ea74

Fix build

0bd3775

D3Hunter reviewed Oct 11, 2024

View reviewed changes

joechenrh added 2 commits October 14, 2024 09:52

Merge branch 'master' into addargs-modify-columns

1448637

Modify multi schema change

b6be3b8

joechenrh changed the title ~~ddl: arg v2 for modify column~~ ddl: arg v2 for modify column and multi schema change Oct 14, 2024

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 14, 2024

Fix

e08b5ba

D3Hunter mentioned this pull request Oct 14, 2024

improve the usability of DDL Job.Args #53930

Closed

26 tasks

joechenrh added 3 commits October 14, 2024 15:47

Fix

258d883

Revert some changes

3478158

Revert some changes

4d82823

ti-chi-bot bot removed the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 14, 2024

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Oct 15, 2024

joechenrh added 2 commits October 15, 2024 19:11

Fix TestMultiSchemaChangeDropIndexesParallel

16fed77

Fix test

ab46813

joechenrh added 4 commits October 16, 2024 00:32

Fix test

d6fdd3b

Fix test after multischemachange

059bf1d

Fix test after multischemachange

2f64488

Fix test after multischemachange

9941650

ti-chi-bot bot added the ok-to-test Indicates a PR is ready to be tested. label Oct 16, 2024

Fix test for v1

a023ee6

ti-chi-bot bot merged commit c18eaa7 into pingcap:master Oct 16, 2024
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: arg v2 for modify column and multi schema change #56536

ddl: arg v2 for modify column and multi schema change #56536

joechenrh commented Oct 10, 2024 •

edited

Loading

tiprow bot commented Oct 10, 2024

codecov bot commented Oct 10, 2024 •

edited

Loading

joechenrh commented Oct 10, 2024

tiprow bot commented Oct 10, 2024

joechenrh commented Oct 10, 2024

tiprow bot commented Oct 10, 2024

joechenrh commented Oct 10, 2024

tiprow bot commented Oct 10, 2024

D3Hunter left a comment

ti-chi-bot bot commented Oct 15, 2024

ti-chi-bot bot commented Oct 15, 2024

D3Hunter commented Oct 15, 2024

tiprow bot commented Oct 15, 2024

joechenrh commented Oct 15, 2024

tiprow bot commented Oct 15, 2024

D3Hunter commented Oct 15, 2024

joechenrh commented Oct 15, 2024

tiprow bot commented Oct 15, 2024

joechenrh commented Oct 16, 2024

tiprow bot commented Oct 16, 2024

joechenrh commented Oct 16, 2024

D3Hunter commented Oct 16, 2024

D3Hunter commented Oct 16, 2024

joechenrh commented Oct 16, 2024

joechenrh commented Oct 16, 2024

D3Hunter commented Oct 16, 2024

joechenrh commented Oct 16, 2024

D3Hunter commented Oct 16, 2024

ddl: arg v2 for modify column and multi schema change #56536

ddl: arg v2 for modify column and multi schema change #56536

Conversation

joechenrh commented Oct 10, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Oct 10, 2024

codecov bot commented Oct 10, 2024 • edited Loading

Codecov Report

joechenrh commented Oct 10, 2024

tiprow bot commented Oct 10, 2024

joechenrh commented Oct 10, 2024

tiprow bot commented Oct 10, 2024

joechenrh commented Oct 10, 2024

tiprow bot commented Oct 10, 2024

D3Hunter left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Oct 15, 2024

ti-chi-bot bot commented Oct 15, 2024

[LGTM Timeline notifier]

D3Hunter commented Oct 15, 2024

tiprow bot commented Oct 15, 2024

joechenrh commented Oct 15, 2024

tiprow bot commented Oct 15, 2024

D3Hunter commented Oct 15, 2024

joechenrh commented Oct 15, 2024

tiprow bot commented Oct 15, 2024

joechenrh commented Oct 16, 2024

tiprow bot commented Oct 16, 2024

joechenrh commented Oct 16, 2024

D3Hunter commented Oct 16, 2024

D3Hunter commented Oct 16, 2024

joechenrh commented Oct 16, 2024

joechenrh commented Oct 16, 2024

D3Hunter commented Oct 16, 2024

joechenrh commented Oct 16, 2024

D3Hunter commented Oct 16, 2024

joechenrh commented Oct 10, 2024 •

edited

Loading

codecov bot commented Oct 10, 2024 •

edited

Loading