Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TiDB CI data race (once) #23280

Closed
LittleFall opened this issue Mar 12, 2021 · 3 comments · Fixed by #24257
Closed

TiDB CI data race (once) #23280

LittleFall opened this issue Mar 12, 2021 · 3 comments · Fixed by #24257
Assignees
Labels
severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.

Comments

@LittleFall
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

see:
https://internal.pingcap.net/idc-jenkins/blue/rest/organizations/jenkins/pipelines/tidb_ghpr_unit_test/runs/67419/nodes/82/steps/323/log/?start=0

2. What did you expect to see? (Required)

test success

3. What did you see instead (Required)

(*ParallelNestedLoopApplyExec).innerWorker\n\t/home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/parallel_apply.go:238"]
[2021-03-12T07:05:37.307Z] ==================
[2021-03-12T07:05:37.307Z] WARNING: DATA RACE
[2021-03-12T07:05:37.308Z] Write at 0x00c08f9bf300 by goroutine 332:
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/statistics.(*QueryFeedback).Update()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/statistics/feedback.go:256 +0x8e4
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/distsql.(*selectResult).fetchResp()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/distsql/select_result.go:153 +0xaae
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/distsql.(*selectResult).Next()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/distsql/select_result.go:180 +0x2b3
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*tableResultHandler).nextChunk()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/table_reader.go:329 +0xaa
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*TableReaderExecutor).Next()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/table_reader.go:206 +0x21e
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.Next()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:277 +0x27d
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*ParallelNestedLoopApplyExec).outerWorker()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/parallel_apply.go:202 +0x5a9
[2021-03-12T07:05:37.308Z] 
[2021-03-12T07:05:37.308Z] Previous read at 0x00c08f9bf300 by goroutine 407:
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/session.(*session).StoreQueryFeedback()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:372 +0x92e
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*TableReaderExecutor).Close()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/table_reader.go:226 +0x121
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*ParallelNestedLoopApplyExec).Close()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/parallel_apply.go:162 +0xa8
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*baseExecutor).Close()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:172 +0xad
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*ProjectionExec).Close()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/projection.go:320 +0x255
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*recordSet).Close()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/adapter.go:166 +0x59
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/session.(*execStmtResult).Close()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/session/session.go:1565 +0x7b
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/util/testkit.(*TestKit).QueryToErr()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/testkit/testkit.go:306 +0x56a
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor_test.(*testSuite).TestApplyGoroutinePanic()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/parallel_apply_test.go:587 +0x546
[2021-03-12T07:05:37.308Z]   runtime.call32()
[2021-03-12T07:05:37.308Z]       /usr/local/go/src/runtime/asm_amd64.s:539 +0x3a
[2021-03-12T07:05:37.308Z]   reflect.Value.Call()
[2021-03-12T07:05:37.308Z]       /usr/local/go/src/reflect/value.go:321 +0xd3
[2021-03-12T07:05:37.308Z]   github.com/pingcap/check.(*suiteRunner).forkTest.func1()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:850 +0x9aa
[2021-03-12T07:05:37.308Z]   github.com/pingcap/check.(*suiteRunner).forkCall.func1()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:739 +0x113
[2021-03-12T07:05:37.308Z] 
[2021-03-12T07:05:37.308Z] Goroutine 332 (running) created at:
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*ParallelNestedLoopApplyExec).Next()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/parallel_apply.go:137 +0xdb
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.Next()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/executor.go:277 +0x27d
[2021-03-12T07:05:37.308Z]   github.com/pingcap/tidb/executor.(*projectionInputFetcher).run()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/executor/projection.go:373 +0x48f
[2021-03-12T07:05:37.308Z] 
[2021-03-12T07:05:37.308Z] Goroutine 407 (running) created at:
[2021-03-12T07:05:37.308Z]   github.com/pingcap/check.(*suiteRunner).forkCall()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:734 +0x4a3
[2021-03-12T07:05:37.308Z]   github.com/pingcap/check.(*suiteRunner).forkTest()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:832 +0x1b9
[2021-03-12T07:05:37.308Z]   github.com/pingcap/check.(*suiteRunner).doRun()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:666 +0x13a
[2021-03-12T07:05:37.308Z]   github.com/pingcap/check.(*suiteRunner).asyncRun.func1()
[2021-03-12T07:05:37.308Z]       /home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/pkg/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:650 +0xf7
[2021-03-12T07:05:37.308Z] ==================

4. What is your TiDB version? (Required)

2021.3.12 master

@LittleFall LittleFall added the type/bug The issue is confirmed as a bug. label Mar 12, 2021
@jebter jebter added this to the v5.0.0 ga milestone Mar 21, 2021
@zimulala zimulala removed this from the v5.0.0 ga milestone Mar 22, 2021
@guo-shaoge
Copy link
Collaborator

/assign

@guo-shaoge
Copy link
Collaborator

guo-shaoge commented Apr 18, 2021

As @eurekaka suggested, this data race happens when Apply operator panic and its' Close() is called, which will read feedback. At the same time, its outer worker(TableReaderExecutor) runs normally, which will write feedback.

So we may need to wait Apply operator's outer worker to finish, then read feedback.

@ti-srebot
Copy link
Contributor

Please edit this comment or add a new comment to complete the following information

Not a bug

  1. Remove the 'type/bug' label
  2. Add notes to indicate why it is not a bug

Duplicate bug

  1. Add the 'type/duplicate' label
  2. Add the link to the original bug

Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: #20100

1. Root Cause Analysis (RCA) (optional)

2. Symptom (optional)

3. All Trigger Conditions (optional)

4. Workaround (optional)

5. Affected versions

6. Fixed versions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants