disttask: rename dispatcher to scheduler #49182

okJiang · 2023-12-05T06:23:13Z

What problem does this PR solve?

Issue Number: close #49125

Problem Summary:

What changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

okJiang · 2023-12-05T06:24:11Z

/ok-to-test

ywqzzy · 2023-12-05T06:29:05Z

pkg/ddl/backfilling_dispatcher_test.go

@@ -153,7 +153,7 @@ func TestBackfillingDispatcherGlobalSortMode(t *testing.T) {
 	ctx = util.WithInternalSourceType(ctx, "handle")
 	mgr := storage.NewTaskManager(pool)
 	storage.SetTaskManager(mgr)
-	dspManager, err := dispatcher.NewManager(util.WithInternalSourceType(ctx, "dispatcher"), mgr, "host:port")
+	dspManager, err := scheduler.NewManager(util.WithInternalSourceType(ctx, "scheduler"), mgr, "host:port")


Suggested change

dspManager, err := scheduler.NewManager(util.WithInternalSourceType(ctx, "scheduler"), mgr, "host:port")

scheManager, err := scheduler.NewManager(util.WithInternalSourceType(ctx, "scheduler"), mgr, "host:port")

pkg/ddl/backfilling_dispatcher.go

pkg/disttask/framework/scheduler/rebalance_test.go

ywqzzy · 2023-12-05T06:30:09Z

pkg/disttask/framework/scheduler/scheduler_manager.go

@@ -51,11 +51,11 @@ func (dm *Manager) getRunningTaskCnt() int {
 	return len(dm.runningTasks.taskIDs)
 }

-func (dm *Manager) setRunningTask(task *proto.Task, dispatcher Dispatcher) {
+func (dm *Manager) setRunningTask(task *proto.Task, scheduler Scheduler) {
 	dm.runningTasks.Lock()


ywqzzy · 2023-12-05T06:31:35Z

pkg/disttask/framework/scheduler/scheduler.go

 }

 // refreshTask fetch task state from tidb_global_task table.
-func (d *BaseDispatcher) refreshTask() error {
+func (d *BaseScheduler) refreshTask() error {


Suggested change

func (d *BaseScheduler) refreshTask() error {

func (s *BaseScheduler) refreshTask() error {

ywqzzy · 2023-12-05T06:32:18Z

pkg/disttask/framework/storage/task_table.go

@@ -975,7 +975,7 @@ func (stm *TaskManager) TransferTasks2History(ctx context.Context, tasks []*prot
 	return stm.WithNewTxn(ctx, func(se sessionctx.Context) error {
 		insertSQL := new(strings.Builder)
 		if err := sqlescape.FormatSQL(insertSQL, "replace into mysql.tidb_global_task_history"+
-			"(id, task_key, type, dispatcher_id, state, priority, start_time, state_update_time,"+
+			"(id, task_key, type, scheduler_id, state, priority, start_time, state_update_time,"+


won't pass the test

ywqzzy · 2023-12-05T06:33:13Z

pkg/disttask/importinto/scheduler_test.go

@@ -61,7 +61,7 @@ func (s *importIntoSuite) TestDispatcherGetEligibleInstances() {
 	}
 	mockedAllServerInfos := makeFailpointRes(serverInfoMap)

-	dsp := ImportDispatcherExt{}
+	dsp := ImportSchedulerExt{}


codecov · 2023-12-05T06:34:42Z

Codecov Report

Merging #49182 (3d6ce4c) into master (9322568) will increase coverage by 0.6815%.
The diff coverage is 13.8702%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #49182        +/-   ##
================================================
+ Coverage   71.0517%   71.7332%   +0.6815%     
================================================
  Files          1368       1414        +46     
  Lines        401578     414951     +13373     
================================================
+ Hits         285328     297658     +12330     
- Misses        96418      98491      +2073     
+ Partials      19832      18802      -1030

Flag	Coverage Δ
integration	`43.7506% <13.8702%> (?)`
unit	`71.0561% <ø> (+0.0044%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`53.9663% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`47.9117% <ø> (-4.9818%)`	⬇️

ti-chi-bot · 2023-12-11T13:56:45Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

okJiang · 2023-12-12T08:36:24Z

pkg/session/bootstrap.go

@@ -573,7 +573,7 @@ const (
 		id BIGINT(20) NOT NULL AUTO_INCREMENT PRIMARY KEY,
    	task_key VARCHAR(256) NOT NULL,
 		type VARCHAR(256) NOT NULL,
-		dispatcher_id VARCHAR(256),
+		scheduler_id VARCHAR(256),


the table field will update later, better not in this pr

…nto rename-dispatcher

D3Hunter

rest lgtm

D3Hunter · 2023-12-12T10:47:27Z

pkg/errno/logredaction.md

@@ -25,9 +25,9 @@ ERROR 1062 (23000): Duplicate entry '?' for key '?'

 And its corresponding log is：
 ```
-[2020/10/20 11:45:37.796 +08:00] [INFO] [conn.go:800] ["command dispatched failed"] [conn=5] [connInfo="id:5, addr:127.0.0.1:57222 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="insert into t values (1),(1)"] [txn_mode=OPTIMISTIC] [err="[kv:1062]Duplicate entry '1' for key 'a'"]
+[2020/10/20 11:45:37.796 +08:00] [INFO] [conn.go:800] ["command scheduled failed"] [conn=5] [connInfo="id:5, addr:127.0.0.1:57222 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="insert into t values (1),(1)"] [txn_mode=OPTIMISTIC] [err="[kv:1062]Duplicate entry '1' for key 'a'"]


this is not part of disttask

…dispatcher

D3Hunter · 2023-12-13T02:40:26Z

pkg/ddl/job_table.go

@@ -50,13 +50,13 @@ import (

 var (
 	addingDDLJobConcurrent      = "/tidb/ddl/add_ddl_job_general"
-	dispatchLoopWaitingDuration = 1 * time.Second


seems this is not part of framework

ti-chi-bot · 2023-12-13T02:45:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, ywqzzy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [D3Hunter]
~~pkg/ddl/OWNERS~~ [ywqzzy]
~~pkg/disttask/OWNERS~~ [D3Hunter,ywqzzy]
~~pkg/domain/OWNERS~~ [D3Hunter]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2023-12-13T02:45:54Z

[LGTM Timeline notifier]

Timeline:

2023-12-12 08:45:42.292789551 +0000 UTC m=+345833.330016463: ☑️ agreed by ywqzzy.
2023-12-13 02:45:52.649976827 +0000 UTC m=+410643.687203754: ☑️ agreed by D3Hunter.

ti-chi-bot bot added do-not-merge/needs-triage-completed release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 5, 2023

ti-chi-bot bot added the ok-to-test Indicates a PR is ready to be tested. label Dec 5, 2023

ywqzzy reviewed Dec 5, 2023

View reviewed changes

pkg/ddl/backfilling_dispatcher.go Outdated Show resolved Hide resolved

ywqzzy reviewed Dec 5, 2023

View reviewed changes

ti-chi-bot bot removed the do-not-merge/needs-triage-completed label Dec 5, 2023

ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 11, 2023

okJiang force-pushed the rename-dispatcher branch from 667e1c2 to c4285c9 Compare December 12, 2023 08:12

okJiang closed this Dec 12, 2023

okJiang force-pushed the rename-dispatcher branch from c4285c9 to 28cb579 Compare December 12, 2023 08:14

ti-chi-bot bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 12, 2023

rename

4dcfe3b

okJiang reopened this Dec 12, 2023

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Dec 12, 2023

rename

2873fbe

ti-chi-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 12, 2023

fix comment: revert scheduler_id

3c6f4e3

okJiang commented Dec 12, 2023

View reviewed changes

ywqzzy approved these changes Dec 12, 2023

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Dec 12, 2023

fix bazel

88f8bfd

okJiang added 3 commits December 12, 2023 17:51

fix bazel

294b465

Merge branch 'rename-dispatcher' of https://github.com/okJiang/tidb i…

cbfb75f

…nto rename-dispatcher

fix lint

c821bc2

D3Hunter reviewed Dec 12, 2023

View reviewed changes

okJiang added 2 commits December 13, 2023 10:21

fix comment

84f5168

Merge branch 'master' of https://github.com/pingcap/tidb into rename-…

1bcb2c8

…dispatcher

D3Hunter reviewed Dec 13, 2023

View reviewed changes

okJiang added 2 commits December 13, 2023 10:41

fix comment

26569a7

fix comment

1bca975

D3Hunter approved these changes Dec 13, 2023

View reviewed changes

ti-chi-bot bot added approved lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 13, 2023

fix ci, update license

3d6ce4c

ti-chi-bot bot merged commit 64430c3 into pingcap:master Dec 13, 2023
16 checks passed

okJiang deleted the rename-dispatcher branch December 13, 2023 03:41

ywqzzy mentioned this pull request Jan 16, 2024

Enhance distributed task execution framework 2.0 #48795

Closed

31 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

disttask: rename dispatcher to scheduler #49182

disttask: rename dispatcher to scheduler #49182

okJiang commented Dec 5, 2023

okJiang commented Dec 5, 2023

ywqzzy Dec 5, 2023

ywqzzy Dec 5, 2023

ywqzzy Dec 5, 2023

ywqzzy Dec 5, 2023

ywqzzy Dec 5, 2023

codecov bot commented Dec 5, 2023 •

edited

Loading

ti-chi-bot bot commented Dec 11, 2023

okJiang Dec 12, 2023

D3Hunter left a comment

D3Hunter Dec 12, 2023

D3Hunter Dec 13, 2023

ti-chi-bot bot commented Dec 13, 2023

ti-chi-bot bot commented Dec 13, 2023

	dspManager, err := scheduler.NewManager(util.WithInternalSourceType(ctx, "scheduler"), mgr, "host:port")
	scheManager, err := scheduler.NewManager(util.WithInternalSourceType(ctx, "scheduler"), mgr, "host:port")

	func (d *BaseScheduler) refreshTask() error {
	func (s *BaseScheduler) refreshTask() error {

disttask: rename dispatcher to scheduler #49182

disttask: rename dispatcher to scheduler #49182

Conversation

okJiang commented Dec 5, 2023

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

okJiang commented Dec 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 5, 2023 • edited Loading

Codecov Report

ti-chi-bot bot commented Dec 11, 2023

Choose a reason for hiding this comment

D3Hunter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Dec 13, 2023

ti-chi-bot bot commented Dec 13, 2023

[LGTM Timeline notifier]

codecov bot commented Dec 5, 2023 •

edited

Loading