store/tikv: Fix goroutine leak in tikv client

[MyonKeminta]

Signed-off-by: MyonKeminta MyonKeminta@users.noreply.github.com

What problem does this PR solve?

Issue Number: close #20654

Problem Summary: There's a goroutine leak found in tikv client, and it's confirmed to be caused by forgetting closing the etcd client in EtcdSafePointKV.

The bug doesn't harm TiDB, since the tikvStore longs for the whole lifetime of the TiDB process. However, it affects other projects that make use of TiKV client. For example, CDC may open and close tikv client many times.

What is changed and how it works?

This PR fixes the goroutine leak by adding a Close interface to SafePointKV, and invoke it in Close method of tikvStore.

Related changes

• Need to cherry-pick to the release branch
  • release 4.0
  • release 3.0
    The problem should also exists in release 2.x, but I'm not sure if it's necessary. Nor do I know if we still have release plan of 2.1.x.

Check List

Tests

• Manual test (add detailed scripts or steps below)

The code segment of the test program:

func main() {
	go func() {
		log.Println(http.ListenAndServe("localhost:6060", nil))
	}()

	_ = store.Register("tikv", tikv.Driver{})

	uri := "tikv://xxx.xxx.xxx.xxx:2379?disableGC=true"

	for {
		var times int
		fmt.Scanf("%d", &times)

		for i := 0; i < times; i++ {
			tiStore, err := store.New(uri)
			if err != nil {
				panic(err)
			}
			err = tiStore.Close()
			if err != nil {
				panic(err)
			}
		}
	}
}

Run and type 100 (create and close tikv client 100 times), dump the goroutines by pprof after the log stopps:

goroutine profile: total 606

After the fix and run again:

goroutine profile: total 5

Side effects

-

Release note

• Fix an goroutine leak issue in TiKV client.

[MyonKeminta]

@cfzjywxk @tiancaiamao PTAL
cc @leoppro

[zier-one]

LGTM

[cfzjywxk]

LGTM

[cfzjywxk]

/merge

[ti-srebot]

/run-all-tests

[ti-srebot]

@MyonKeminta merge failed.

[MyonKeminta]

/merge

[ti-srebot]

Your auto merge job has been accepted, waiting for:

• 20813
• 20551

[MyonKeminta]

/merge

[ti-srebot]

Your auto merge job has been accepted, waiting for:

• 20551

[ti-srebot]

/run-all-tests

[ti-srebot]

cherry pick to release-3.0 in PR #20863

[ti-srebot]

cherry pick to release-4.0 failed

[MyonKeminta]

/run-cherry-picker

[ti-srebot]

cherry pick to release-4.0 in PR #20870