Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

master/scheduler: support rebalancer #5078

Open
wants to merge 6 commits into
base: release-multi-source
Choose a base branch
from
Open

master/scheduler: support rebalancer #5078

wants to merge 6 commits into from

Conversation

lichunzhu
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #5053

What is changed and how it works?

support rebalancer interface and tableNumberRebalancer in dm. This interface will periodically pick victim sources, sort them in orders and try to pick up workers for these sources.

Check List

Tests

  • Unit test

Code changes

  • Has exported function/method change

Side effects

  • Possible performance regression
  • Increased code complexity

Related changes

  • Need to update the documentation

Release note 

support table number rebalancer for dm-master

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Mar 30, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • WizardXiao

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Mar 30, 2022
@lichunzhu lichunzhu mentioned this pull request Mar 30, 2022
Closed
@lichunzhu lichunzhu marked this pull request as draft March 30, 2022 12:58
@ti-chi-bot ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 30, 2022
@lichunzhu lichunzhu linked an issue Apr 6, 2022 that may be closed by this pull request
abstract scheduling interface, support source number balancing algorithm #5053
Open
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 20, 2022
@lichunzhu lichunzhu force-pushed the rebalanceSupport2 branch from ef6a114 to 4d242ec Compare May 26, 2022 02:49
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 26, 2022
@lichunzhu lichunzhu marked this pull request as ready for review May 26, 2022 02:52
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 26, 2022
@lichunzhu
Copy link
Contributor Author

/run-verify-ci

@lichunzhu
Copy link
Contributor Author

/run-unit-test

@lichunzhu lichunzhu requested a review from WizardXiao June 1, 2022 08:01
@lichunzhu lichunzhu added the area/dm Issues or PRs related to DM. label Jun 2, 2022
WizardXiao
WizardXiao reviewed Jun 6, 2022
View reviewed changes
dm/dm/master/scheduler/balancer.go Outdated
case hasLoadTaskByWorkerAndSource(w.BaseInfo().Name, source):
score = hasLoadTaskWeight
case hasRelay:
score = 100 - float32(len(relayWorkers[source])) + rand.Float32()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you give some simple explanations as note about score, please?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in 232f2b2

dm/dm/master/scheduler/scheduler.go Outdated
s.updateStatusToUnbound(bound.Source)
unbounds = append(unbounds, bound.Source)
// 4. unbound for the source.
boundSourcesByWeight := s.balance.GetWorkerBoundsByWeight(w, s.relayWorkers, s.hasLoadTaskByWorkerAndSource)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this bounds need to be sorted by score? is it for rebound source in order?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because I want to try to rebound for high priority sources first.

dm/dm/master/scheduler/scheduler.go Outdated
@@ -2219,8 +2230,6 @@ func (s *Scheduler) tryBoundForWorker(w *Worker) (bounded bool, err error) {
// caller should update the s.unbounds.
// caller should make sure this source has source config.
func (s *Scheduler) tryBoundForSource(source string) (bool, error) {
var worker *Worker

// TODO: change this to pick a worker which has the least load.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// TODO: change this to pick a worker which has the least load.

dm/dm/master/scheduler/scheduler.go Outdated
@@ -2251,6 +2281,7 @@ func (s *Scheduler) tryBoundForSource(source string) (bool, error) {
}
}
if worker == nil {
boundSourcesNum := sourceNum
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here seems that will pick a relayworker which len(w.Bounds()) is least, Maybe we can put the log outside the loop at here

dm/dm/master/scheduler/scheduler.go Outdated
}

if worker == nil {
boundSourcesNum := sourceNum
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 9, 2022
@lichunzhu lichunzhu mentioned this pull request Jun 27, 2022
Draft
@codecov
Copy link

codecov bot commented Oct 9, 2023

Codecov Report

❗ No coverage uploaded for pull request base (release-multi-source@c07996d). Click here to learn what that means.
The diff coverage is n/a.

Additional details and impacted files
Flag Coverage Δ
cdc 61.9648% <0.0000%> (?)
dm 52.1231% <0.0000%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

@@                    Coverage Diff                    @@
##             release-multi-source      #5078   +/-   ##
=========================================================
  Coverage                        ?   56.4107%           
=========================================================
  Files                           ?        527           
  Lines                           ?      69813           
  Branches                        ?          0           
=========================================================
  Hits                            ?      39382           
  Misses                          ?      26692           
  Partials                        ?       3739

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WizardXiao WizardXiao WizardXiao approved these changes

Assignees
No one assigned
Labels
area/dm Issues or PRs related to DM. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/LGT1 Indicates that a PR has LGTM 1.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

abstract scheduling interface, support source number balancing algorithm
3 participants
@lichunzhu @ti-chi-bot @WizardXiao