[to #48] try to fix scheduler problem #90
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: zeminzhou <zhouzemin@ping.com>
Contributor
Author
|
TODO: Unit Test. |
Contributor
Author
zeminzhou
changed the title
added 4 commits
April 21, 2022 11:57
Signed-off-by: zeminzhou <zhouzemin@ping.com>
Signed-off-by: zeminzhou <zhouzemin@ping.com>
Codecov Report
@@ Coverage Diff @@
## main #90 +/- ##
================================================
+ Coverage 53.4580% 61.2819% +7.8239%
================================================
Files 197 180 -17
Lines 19838 14275 -5563
================================================
- Hits 10605 8748 -1857
+ Misses 8265 4910 -3355
+ Partials 968 617 -351
Flags with carried forward coverage won't be shown. Click here to find out more.
|
pingyu
reviewed
May 6, 2022
| return shouldUpdateState, nil | ||
| } | ||
|
|
||
| func (s *oldScheduler) diffCurrentKeySpans(currentKeySpans map[model.KeySpanID]regionspan.Span) (map[model.KeySpanID]struct{}, []model.KeySpanID) { |
Collaborator
There was a problem hiding this comment.
Please add unit test for this method.
| // syncKeySpansWithCurrentKeySpans iterates all current keyspans to check whether it should be listened or not. | ||
| // this function will return schedulerJob to make sure all keyspans will be listened. | ||
| func (s *oldScheduler) syncKeySpansWithCurrentKeySpans() ([]*schedulerJob, error) { | ||
| func (s *oldScheduler) syncKeySpansWithCurrentKeySpans(newKeySpans map[model.KeySpanID]struct{}, needRemoveKeySpans []model.KeySpanID) ([]*schedulerJob, error) { |
Collaborator
There was a problem hiding this comment.
Suggest to add unit test for this method.
| return nil | ||
| } | ||
|
|
||
| func (p *processor) checkRelatedKeyspans(relatedKeySpans []model.KeySpanLocation) bool { |
Collaborator
There was a problem hiding this comment.
Please add unit test for this method.
Signed-off-by: zeminzhou <zhouzemin@ping.com>
Signed-off-by: zeminzhou <zhouzemin@ping.com>
Signed-off-by: zeminzhou <zhouzemin@pingcap.com>
haojinming
reviewed
Jun 14, 2022
cdc/cdc/model/owner.go
Outdated
| // WorkloadInfo records the workload info of a keyspan | ||
| type WorkloadInfo struct { | ||
| Workload uint64 `json:"workload"` | ||
| Workload int64 `json:"workload"` |
Contributor
There was a problem hiding this comment.
Why change to int64? Can it be negative?
Signed-off-by: zeminzhou <zhouzemin@pingcap.com>
haojinming
reviewed
Jun 15, 2022
cdc/pkg/scheduler/workload.go
Outdated
|
|
||
| func (w workloads) SelectIdleCapture() model.CaptureID { | ||
| minWorkload := uint64(math.MaxUint64) | ||
| minWorkload := uint64(math.MaxInt64) |
Contributor
There was a problem hiding this comment.
maybe revert this change?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: zeminzhou zhouzemin@ping.com
What problem does this PR solve?
Keyspans to add & keyspans to remove, must be overlapped. So to be correct, I think we must finish remove jobs first, and then dispatch the add jobs.here
Issue Number: to #48
Problem Description:
Keyspans to add & keyspans to remove, must be overlapped. So to be correct, I think we must finish remove jobs first, and then dispatch the add jobs
What is changed and how does it work?
After each acquisition of the current keyspan, compare the current keyspan with the keyspan of the previous period to establish a corresponding relationship newKeySpan -> needRemoveKeySpan. The running and deployment of the newKeySpan task needs to wait for all the needRemoveKeySpan tasks to stop. This check will be done in Processor.
Code changes