worker(dm): fix Server Start/Close race (#6213)

[ti-chi-bot]

This is an automated cherry-pick of #6213

Signed-off-by: lance6716 lance6716@gmail.com

What problem does this PR solve?

Issue Number: close #5836 close #5089

What is changed and how it works?

for #5836 : Server.Start is blocking, so before it we must spawn a goroutine to watch terminate signal. Original in that goroutine we call Server.Close, but Start/Close is not concurrent-safe. Now we use context to do cancellation.

for #5089 : when worker Server is about to close, it will hold the lock, enter doClose, then blocked on waitgroup

tiflow/dm/dm/worker/server.go

Lines 438 to 439 in 9bce8cc

	s.cancel()
	s.wg.Wait()

but other goroutine may in the same time stuck at s.Lock.

Check List

Tests

• hard to test these functions 🤔

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix a bug that start DM-worker and `kill` it immediately will not let process stop.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (release-6.1@8e289e4). Click here to learn what that means.
The diff coverage is n/a.

Flag	Coverage Δ
cdc	62.2852% <0.0000%> (?)
dm	52.0170% <0.0000%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             release-6.1      #6292   +/-   ##
================================================
  Coverage               ?   56.5524%           
================================================
  Files                  ?        532           
  Lines                  ?      70241           
  Branches               ?          0           
================================================
  Hits                   ?      39723           
  Misses                 ?      26744           
  Partials               ?       3774           

[lance6716]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: ff8a4f2