Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestLabelerRuleTTL is flaky #8687

Open
okJiang opened this issue Oct 10, 2024 · 4 comments · May be fixed by #8723
Open

TestLabelerRuleTTL is flaky #8687

okJiang opened this issue Oct 10, 2024 · 4 comments · May be fixed by #8723
Labels
type/ci The issue is related to CI.

Comments

@okJiang
Copy link
Member

okJiang commented Oct 10, 2024

Flaky Test

Which jobs are failing


[FAIL]  pkg/schedule/labeler TestLabelerRuleTTL
=== RUN   TestLabelerRuleTTL
==================
WARNING: DATA RACE
Read at 0x00c000433d10 by goroutine [135](https://github.com/tikv/pd/actions/runs/11271034543/job/31343067657?pr=8685#step:4:136):
  runtime.mapaccess1_faststr()
      /opt/hostedtoolcache/go/1.23.2/x64/src/runtime/map_faststr.go:13 +0x0
  github.com/tikv/pd/pkg/schedule/labeler.checkRuleInMemoryAndStorage()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:405 +0x90
  github.com/tikv/pd/pkg/schedule/labeler.TestLabelerRuleTTL()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:393 +0x1891
  github.com/pingcap/failpoint.parseTerm()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:149 +0x233
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x296
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0x174a
  github.com/tikv/pd/pkg/schedule/labeler.TestLabelerRuleTTL()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:383 +0x174b
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1743 +0x44

Previous write at 0x00c000433d10 by goroutine [137](https://github.com/tikv/pd/actions/runs/11271034543/job/31343067657?pr=8685#step:4:138):
  runtime.mapdelete_faststr()
      /opt/hostedtoolcache/go/1.23.2/x64/src/runtime/map_faststr.go:321 +0x0
  github.com/tikv/pd/pkg/schedule/labeler.(*RegionLabeler).DeleteLabelRuleLocked()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:265 +0x1e8
  github.com/tikv/pd/pkg/schedule/labeler.(*RegionLabeler).checkAndClearExpiredLabels()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:92 +0x417
  github.com/tikv/pd/pkg/schedule/labeler.(*RegionLabeler).doGC()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:67 +0x165
  github.com/tikv/pd/pkg/schedule/labeler.NewRegionLabeler.gowrap1()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:55 +0x44

Goroutine 135 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1743 +0x825
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:2168 +0x85
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1690 +0x226
  testing.runTests()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:2034 +0xf17
  main.main()
      _testmain.go:73 +0x164

Goroutine 137 (running) created at:
  github.com/tikv/pd/pkg/schedule/labeler.NewRegionLabeler()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:55 +0x297
  github.com/tikv/pd/pkg/schedule/labeler.TestLabelerRuleTTL()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:344 +0x48e
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1743 +0x44
==================
    labeler_test.go:405: 
        	Error Trace:	/home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:405
        	            				/home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:393
        	Error:      	Not equal: 
        	            	expected: true
        	            	actual  : false
        	Test:       	TestLabelerRuleTTL
    testing.go:[139](https://github.com/tikv/pd/actions/runs/11271034543/job/31343067657?pr=8685#step:4:140)9: race detected during execution of test
--- FAIL: TestLabelerRuleTTL (0.10s)

CI link

https://github.com/tikv/pd/actions/runs/11271034543/job/31343067657?pr=8685

Reason for failure (if possible)

Anything else
@okJiang okJiang added the type/ci The issue is related to CI. label Oct 10, 2024
@okJiang
Copy link
Member Author

okJiang commented Oct 11, 2024

https://github.com/tikv/pd/actions/runs/11271449522/job/31344283482?pr=8685

@okJiang
Copy link
Member Author

okJiang commented Oct 11, 2024

https://github.com/tikv/pd/actions/runs/11285918824/job/31389457320?pr=8685

@okJiang
Copy link
Member Author

okJiang commented Oct 15, 2024

https://github.com/tikv/pd/actions/runs/11341558936/job/31540093117?pr=8707

@okJiang
Copy link
Member Author

okJiang commented Oct 16, 2024 •
edited
Loading

pd/pkg/schedule/labeler/labeler_test.go

Line 344 in b70107e

labeler, err := NewRegionLabeler(context.Background(), store, time.Millisecond*10)

gcInterval(10ms) is so short that it has been GC before L393(check the status) and after L387(update its ttl).

@okJiang okJiang linked a pull request Oct 16, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/ci The issue is related to CI.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

test: make TestLabelerRuleTTL stable okJiang/pd
1 participant
@okJiang