Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky Test TestKeyspacesServedByNonDefaultKeyspaceGroups #8055

Closed
nolouch opened this issue Apr 11, 2024 · 0 comments · Fixed by #8077
Closed

Flaky Test TestKeyspacesServedByNonDefaultKeyspaceGroups #8055

nolouch opened this issue Apr 11, 2024 · 0 comments · Fixed by #8077
Labels
type/ci The issue is related to CI.

Comments

@nolouch
Copy link
Contributor

nolouch commented Apr 11, 2024

Flaky Test

2024-04-11T06:44:44.7066047Z ==================
2024-04-11T06:44:44.7066466Z WARNING: DATA RACE
2024-04-11T06:44:44.7066972Z Write at 0x00c00c35c160 by goroutine 270038:
2024-04-11T06:44:44.7068001Z   github.com/tikv/pd/client.(*tsoBatchController).finishCollectedRequests()
2024-04-11T06:44:44.7069383Z       /home/runner/work/pd/pd/client/tso_batch_controller.go:142 +0x1ad
2024-04-11T06:44:44.7070413Z   github.com/tikv/pd/client.(*tsoClient).processRequests()
2024-04-11T06:44:44.7071706Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:828 +0x8fc
2024-04-11T06:44:44.7072736Z   github.com/tikv/pd/client.(*tsoClient).handleDispatcher()
2024-04-11T06:44:44.7074121Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:538 +0x1f67
2024-04-11T06:44:44.7075224Z   github.com/tikv/pd/client.(*tsoClient).createTSODispatcher.func1()
2024-04-11T06:44:44.7076434Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:358 +0x79
2024-04-11T06:44:44.7077033Z 
2024-04-11T06:44:44.7077367Z Previous read at 0x00c00c35c160 by goroutine 270555:
2024-04-11T06:44:44.7078201Z   github.com/tikv/pd/client.(*tsoRequest).Wait()
2024-04-11T06:44:44.7079217Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:138 +0x845
2024-04-11T06:44:44.7080106Z   github.com/tikv/pd/client.(*client).GetTS()
2024-04-11T06:44:44.7081017Z       /home/runner/work/pd/pd/client/client.go:844 +0xe1
2024-04-11T06:44:44.7082312Z   github.com/tikv/pd/tests/integrations/mcs.WaitForMultiKeyspacesTSOAvailable.func1.1()
2024-04-11T06:44:44.7083888Z       /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:116 +0x75
2024-04-11T06:44:44.7084984Z   github.com/stretchr/testify/assert.Eventually.func1()
2024-04-11T06:44:44.7086458Z       /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/assert/assertions.go:1852 +0x33
2024-04-11T06:44:44.7087328Z 
2024-04-11T06:44:44.7087560Z Goroutine 270038 (running) created at:
2024-04-11T06:44:44.7088427Z   github.com/tikv/pd/client.(*tsoClient).createTSODispatcher()
2024-04-11T06:44:44.7089611Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:358 +0x84d
2024-04-11T06:44:44.7090738Z   github.com/tikv/pd/client.(*tsoClient).updateTSODispatcher.func1()
2024-04-11T06:44:44.7092023Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:155 +0xad
2024-04-11T06:44:44.7092811Z   sync.(*Map).Range()
2024-04-11T06:44:44.7093670Z       /opt/hostedtoolcache/go/1.21.9/x64/src/sync/map.go:476 +0x1db
2024-04-11T06:44:44.7094740Z   github.com/tikv/pd/client.(*tsoClient).updateTSODispatcher()
2024-04-11T06:44:44.7095878Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:152 +0x109
2024-04-11T06:44:44.7096790Z   github.com/tikv/pd/client.(*tsoClient).Setup()
2024-04-11T06:44:44.7097801Z       /home/runner/work/pd/pd/client/tso_client.go:127 +0xa4
2024-04-11T06:44:44.7098802Z   github.com/tikv/pd/client.(*client).resetTSOClientLocked()
2024-04-11T06:44:44.7099829Z       /home/runner/work/pd/pd/client/client.go:647 +0x85e
2024-04-11T06:44:44.7100704Z   github.com/tikv/pd/client.(*client).setServiceMode()
2024-04-11T06:44:44.7101710Z       /home/runner/work/pd/pd/client/client.go:609 +0x36e
2024-04-11T06:44:44.7102800Z   github.com/tikv/pd/client.(*client).setServiceMode-fm()
2024-04-11T06:44:44.7103586Z       <autogenerated>:1 +0x3b
2024-04-11T06:44:44.7104492Z   github.com/tikv/pd/client.(*pdServiceDiscovery).checkServiceModeChanged()
2024-04-11T06:44:44.7105863Z       /home/runner/work/pd/pd/client/pd_service_discovery.go:880 +0x570
2024-04-11T06:44:44.7106954Z   github.com/tikv/pd/client.(*pdServiceDiscovery).Init()
2024-04-11T06:44:44.7108254Z       /home/runner/work/pd/pd/client/pd_service_discovery.go:508 +0x317
2024-04-11T06:44:44.7109331Z   github.com/tikv/pd/client.(*client).setup()
2024-04-11T06:44:44.7110269Z       /home/runner/work/pd/pd/client/client.go:573 +0x14e
2024-04-11T06:44:44.7111205Z   github.com/tikv/pd/client.createClientWithKeyspace()
2024-04-11T06:44:44.7112206Z       /home/runner/work/pd/pd/client/client.go:397 +0x7c4
2024-04-11T06:44:44.7113312Z   github.com/tikv/pd/client.NewClientWithKeyspace()
2024-04-11T06:44:44.7114514Z       /home/runner/work/pd/pd/client/client.go:358 +0x26a
2024-04-11T06:44:44.7115632Z   github.com/tikv/pd/tests/integrations/mcs.SetupClientWithKeyspaceID()
2024-04-11T06:44:44.7117047Z       /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:41 +0x11e
2024-04-11T06:44:44.7118418Z   github.com/tikv/pd/tests/integrations/mcs.WaitForMultiKeyspacesTSOAvailable()
2024-04-11T06:44:44.7120113Z       /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:109 +0x2b1
2024-04-11T06:44:44.7122092Z   github.com/tikv/pd/tests/integrations/mcs/tso.(*tsoKeyspaceGroupManagerTestSuite).TestKeyspacesServedByNonDefaultKeyspaceGroups()
2024-04-11T06:44:44.7124411Z       /home/runner/work/pd/pd/tests/integrations/mcs/tso/keyspace_group_manager_test.go:232 +0xc24
2024-04-11T06:44:44.7125564Z   runtime.call16()
2024-04-11T06:44:44.7126431Z       /opt/hostedtoolcache/go/1.21.9/x64/src/runtime/asm_amd64.s:747 +0x42
2024-04-11T06:44:44.7127311Z   reflect.Value.Call()
2024-04-11T06:44:44.7128224Z       /opt/hostedtoolcache/go/1.21.9/x64/src/reflect/value.go:380 +0xb5
2024-04-11T06:44:44.7129162Z   github.com/stretchr/testify/suite.Run.func1()
2024-04-11T06:44:44.7130467Z       /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x766
2024-04-11T06:44:44.7131506Z   testing.tRunner()
2024-04-11T06:44:44.7132433Z       /opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:1595 +0x261
2024-04-11T06:44:44.7133295Z   testing.(*T).Run.func1()
2024-04-11T06:44:44.7134243Z       /opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:1648 +0x44
2024-04-11T06:44:44.7134930Z 
2024-04-11T06:44:44.7135171Z Goroutine 270555 (finished) created at:
2024-04-11T06:44:44.7135932Z   github.com/stretchr/testify/assert.Eventually()
2024-04-11T06:44:44.7137320Z       /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/assert/assertions.go:1852 +0x3b5
2024-04-11T06:44:44.7138537Z   github.com/stretchr/testify/require.Eventually()
2024-04-11T06:44:44.7139922Z       /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/require/require.go:401 +0xb5
2024-04-11T06:44:44.7141218Z   github.com/stretchr/testify/require.(*Assertions).Eventually()
2024-04-11T06:44:44.7142814Z       /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/require/require_forward.go:323 +0xbe
2024-04-11T06:44:44.7144128Z   github.com/tikv/pd/pkg/utils/testutil.Eventually()
2024-04-11T06:44:44.7145774Z       /home/runner/work/pd/pd/pkg/utils/testutil/testutil.go:67 +0x189
2024-04-11T06:44:44.7147193Z   github.com/tikv/pd/tests/integrations/mcs.WaitForMultiKeyspacesTSOAvailable.func1()
2024-04-11T06:44:44.7148797Z       /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:115 +0x1b1
2024-04-11T06:44:44.7149662Z ==================
2024-04-11T06:44:44.7150055Z ==================
2024-04-11T06:44:44.7150464Z WARNING: DATA RACE
2024-04-11T06:44:44.7150965Z Write at 0x00c00c35c168 by goroutine 270038:
2024-04-11T06:44:44.7152005Z   github.com/tikv/pd/client.(*tsoBatchController).finishCollectedRequests()
2024-04-11T06:44:44.7153584Z       /home/runner/work/pd/pd/client/tso_batch_controller.go:142 +0x1ca
2024-04-11T06:44:44.7154662Z   github.com/tikv/pd/client.(*tsoClient).processRequests()
2024-04-11T06:44:44.7155792Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:828 +0x8fc
2024-04-11T06:44:44.7156845Z   github.com/tikv/pd/client.(*tsoClient).handleDispatcher()
2024-04-11T06:44:44.7158096Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:538 +0x1f67
2024-04-11T06:44:44.7159232Z   github.com/tikv/pd/client.(*tsoClient).createTSODispatcher.func1()
2024-04-11T06:44:44.7160638Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:358 +0x79
2024-04-11T06:44:44.7161437Z 
2024-04-11T06:44:44.7161767Z Previous read at 0x00c00c35c168 by goroutine 270555:
2024-04-11T06:44:44.7162595Z   github.com/tikv/pd/client.(*tsoRequest).Wait()
2024-04-11T06:44:44.7163654Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:138 +0x824
2024-04-11T06:44:44.7164583Z   github.com/tikv/pd/client.(*client).GetTS()
2024-04-11T06:44:44.7165704Z       /home/runner/work/pd/pd/client/client.go:844 +0xe1
2024-04-11T06:44:44.7166971Z   github.com/tikv/pd/tests/integrations/mcs.WaitForMultiKeyspacesTSOAvailable.func1.1()
2024-04-11T06:44:44.7168584Z       /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:116 +0x75
2024-04-11T06:44:44.7169713Z   github.com/stretchr/testify/assert.Eventually.func1()
2024-04-11T06:44:44.7171302Z       /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/assert/assertions.go:1852 +0x33
2024-04-11T06:44:44.7172220Z 
2024-04-11T06:44:44.7172459Z Goroutine 270038 (running) created at:
2024-04-11T06:44:44.7173358Z   github.com/tikv/pd/client.(*tsoClient).createTSODispatcher()
2024-04-11T06:44:44.7174519Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:358 +0x84d
2024-04-11T06:44:44.7175658Z   github.com/tikv/pd/client.(*tsoClient).updateTSODispatcher.func1()
2024-04-11T06:44:44.7176897Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:155 +0xad
2024-04-11T06:44:44.7177703Z   sync.(*Map).Range()
2024-04-11T06:44:44.7178529Z       /opt/hostedtoolcache/go/1.21.9/x64/src/sync/map.go:476 +0x1db
2024-04-11T06:44:44.7179577Z   github.com/tikv/pd/client.(*tsoClient).updateTSODispatcher()
2024-04-11T06:44:44.7180771Z       /home/runner/work/pd/pd/client/tso_dispatcher.go:152 +0x109
2024-04-11T06:44:44.7181728Z   github.com/tikv/pd/client.(*tsoClient).Setup()
2024-04-11T06:44:44.7182682Z       /home/runner/work/pd/pd/client/tso_client.go:127 +0xa4
2024-04-11T06:44:44.7183660Z   github.com/tikv/pd/client.(*client).resetTSOClientLocked()
2024-04-11T06:44:44.7184719Z       /home/runner/work/pd/pd/client/client.go:647 +0x85e
2024-04-11T06:44:44.7185640Z   github.com/tikv/pd/client.(*client).setServiceMode()
2024-04-11T06:44:44.7186610Z       /home/runner/work/pd/pd/client/client.go:609 +0x36e
2024-04-11T06:44:44.7187687Z   github.com/tikv/pd/client.(*client).setServiceMode-fm()
2024-04-11T06:44:44.7188506Z       <autogenerated>:1 +0x3b
2024-04-11T06:44:44.7189457Z   github.com/tikv/pd/client.(*pdServiceDiscovery).checkServiceModeChanged()
2024-04-11T06:44:44.7190807Z       /home/runner/work/pd/pd/client/pd_service_discovery.go:880 +0x570
2024-04-11T06:44:44.7191938Z   github.com/tikv/pd/client.(*pdServiceDiscovery).Init()
2024-04-11T06:44:44.7193333Z       /home/runner/work/pd/pd/client/pd_service_discovery.go:508 +0x317
2024-04-11T06:44:44.7194301Z   github.com/tikv/pd/client.(*client).setup()
2024-04-11T06:44:44.7195223Z       /home/runner/work/pd/pd/client/client.go:573 +0x14e
2024-04-11T06:44:44.7196139Z   github.com/tikv/pd/client.createClientWithKeyspace()
2024-04-11T06:44:44.7197163Z       /home/runner/work/pd/pd/client/client.go:397 +0x7c4
2024-04-11T06:44:44.7198075Z   github.com/tikv/pd/client.NewClientWithKeyspace()
2024-04-11T06:44:44.7199038Z       /home/runner/work/pd/pd/client/client.go:358 +0x26a
2024-04-11T06:44:44.7200127Z   github.com/tikv/pd/tests/integrations/mcs.SetupClientWithKeyspaceID()
2024-04-11T06:44:44.7201548Z       /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:41 +0x11e
2024-04-11T06:44:44.7202957Z   github.com/tikv/pd/tests/integrations/mcs.WaitForMultiKeyspacesTSOAvailable()
2024-04-11T06:44:44.7204475Z       /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:109 +0x2b1
2024-04-11T06:44:44.7206435Z   github.com/tikv/pd/tests/integrations/mcs/tso.(*tsoKeyspaceGroupManagerTestSuite).TestKeyspacesServedByNonDefaultKeyspaceGroups()
2024-04-11T06:44:44.7208739Z       /home/runner/work/pd/pd/tests/integrations/mcs/tso/keyspace_group_manager_test.go:232 +0xc24
2024-04-11T06:44:44.7209908Z   runtime.call16()
2024-04-11T06:44:44.7210969Z       /opt/hostedtoolcache/go/1.21.9/x64/src/runtime/asm_amd64.s:747 +0x42
2024-04-11T06:44:44.7211931Z   reflect.Value.Call()
2024-04-11T06:44:44.7212845Z       /opt/hostedtoolcache/go/1.21.9/x64/src/reflect/value.go:380 +0xb5
2024-04-11T06:44:44.7213789Z   github.com/stretchr/testify/suite.Run.func1()
2024-04-11T06:44:44.7215111Z       /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x766
2024-04-11T06:44:44.7216274Z   testing.tRunner()
2024-04-11T06:44:44.7217194Z       /opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:1595 +0x261
2024-04-11T06:44:44.7218093Z   testing.(*T).Run.func1()
2024-04-11T06:44:44.7219058Z       /opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:1648 +0x44
2024-04-11T06:44:44.7219716Z 
2024-04-11T06:44:44.7219937Z Goroutine 270555 (finished) created at:
2024-04-11T06:44:44.7220822Z   github.com/stretchr/testify/assert.Eventually()
2024-04-11T06:44:44.7222254Z       /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/assert/assertions.go:1852 +0x3b5
2024-04-11T06:44:44.7223495Z   github.com/stretchr/testify/require.Eventually()
2024-04-11T06:44:44.7224860Z       /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/require/require.go:401 +0xb5
2024-04-11T06:44:44.7226177Z   github.com/stretchr/testify/require.(*Assertions).Eventually()
2024-04-11T06:44:44.7227775Z       /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/require/require_forward.go:323 +0xbe
2024-04-11T06:44:44.7229071Z   github.com/tikv/pd/pkg/utils/testutil.Eventually()
2024-04-11T06:44:44.7230243Z       /home/runner/work/pd/pd/pkg/utils/testutil/testutil.go:67 +0x189
2024-04-11T06:44:44.7231676Z   github.com/tikv/pd/tests/integrations/mcs.WaitForMultiKeyspacesTSOAvailable.func1()
2024-04-11T06:44:44.7233480Z       /home/runner/work/pd/pd/tests/integrations/mcs/testutil.go:115 +0x1b1
2024-04-11T06:44:44.7234383Z ==================

Which jobs are failing

13

CI link

https://github.com/tikv/pd/actions/runs/8642563139/job/23693875551?pr=8053

Reason for failure (if possible)

data race

Anything else
@nolouch nolouch added the type/ci The issue is related to CI. label Apr 11, 2024
This was referenced Apr 16, 2024
Closed
Merged
ti-chi-bot bot added a commit that referenced this issue Apr 17, 2024
@JmPotato @ti-chi-bot
client/tso: use the TSO request pool at the tsoClient level to avoid …
db88a43 
…data race (#8077)

close #8055, ref #8076

Use the TSO request pool at the `tsoClient` level to avoid data race.

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot ti-chi-bot mentioned this issue Apr 30, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue Apr 30, 2024
@ti-chi-bot @JmPotato
client/tso: use the TSO request pool at the tsoClient level to avoid …
13eaf8a 
…data race (#8077) (#8141)

close #8055, ref #8076

Use the TSO request pool at the `tsoClient` level to avoid data race.

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: JmPotato <ghzpotato@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/ci The issue is related to CI.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

client/tso: use the TSO request pool at the tsoClient level to avoid data race JmPotato/pd
1 participant
@nolouch