Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lightning import failed after inject pdleader io delay 1s #47930

Closed
Lily2025 opened this issue Oct 23, 2023 · 2 comments · Fixed by #47959
Closed

lightning import failed after inject pdleader io delay 1s #47930

Lily2025 opened this issue Oct 23, 2023 · 2 comments · Fixed by #47959
Assignees
Labels
affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. affects-6.1 affects-6.5 affects-7.1 affects-7.5 component/lightning This issue is related to Lightning of TiDB. severity/major type/bug The issue is confirmed as a bug.

Comments

@Lily2025
Copy link

Lily2025 commented Oct 23, 2023

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、lightning import
- storage: s3://benchmark/location-dumpling
s3Endpoint: http://minio.pingcap.net:9000
db: location
options: --enable-checkpoint false
config: |-
[lightning]
max-error = 9999999999
[tikv-importer]
duplicate-resolution = "record"
incremental-import = true
crc32Checksum: 2645404411
adminCheck:
- table: AreaMeshLake
indexes:
- areameshlakeid
backend: local
sortedKvDir: /tmp/sorted-kv-dir
- storage: s3://brie/lightning-mhy-100G-csv
s3Endpoint: http://minio.pingcap.net:9001
db: sysbench
backend: local
sortedKvDir: /tmp/sorted-kv-dir
options: "--enable-checkpoint false"
config: "[[mydumper.files]]\npattern = "sysbench-schema-create.sql"\nschema = "sysbench"\ntype = "schema-schema"\n[[mydumper.files]]\npattern = "sysbench.user_data1-schema.sql"\nschema = "sysbench"\ntable = "user_data1"\ntype = "table-schema"\n[[mydumper.files]]\npattern = "sysbench.user_data1.000000[0-3][0-9][0-9].csv"\nschema = "sysbench"\ntable = "user_data1"\ntype = "csv""
crc32Checksum: 619903217
2、inject pdleader io delay 1s
- name: ha_pdleader_io_delay(1s)
faultType: iochaos_io_delay(volumePath:/var/lib/pd,path:/var/lib/pd/data/**/*,delay:1s,percent:100)
selector: pdleader
warmUpTime: 1m
period: "@every 3m"
faultDuration: 1m
faultTotalRunTime: 20m
lightningList: *id001

2. What did you expect to see? (Required)

lightning import can success

3. What did you see instead (Required)

lightning import failed after inject pdleader io delay 1s

tidb lightning encountered error: [Lightning:Restore:ErrRestoreTable]restore table sysbench.user_data1 failed: Post "http://tc-pd:2379/pd/api/v1/config/region-label/rule": http: ContentLength=274 with Body length 0

github.com/pingcap/errors.AddStack
	/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174
github.com/pingcap/errors.Trace
	/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15
github.com/pingcap/tidb/br/pkg/pdutil.pdRequestWithCode
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:203
github.com/pingcap/tidb/br/pkg/pdutil.pdRequest
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:154
github.com/pingcap/tidb/br/pkg/pdutil.(*PdController).CreateOrUpdateRegionLabelRule
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:983
github.com/pingcap/tidb/br/pkg/pdutil.(*PdController).pauseSchedulerByKeyRangeWithTTL
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:1048
github.com/pingcap/tidb/br/pkg/pdutil.(*PdController).PauseSchedulersByKeyRange
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:1027
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).ImportEngine
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1523
github.com/pingcap/tidb/br/pkg/lightning/backend.(*ClosedEngine).Import
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/backend.go:365
github.com/pingcap/tidb/br/pkg/lightning/importer.(*TableImporter).importKV
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/table_import.go:1242
github.com/pingcap/tidb/br/pkg/lightning/importer.(*TableImporter).importEngine
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/table_import.go:891
github.com/pingcap/tidb/br/pkg/lightning/importer.(*TableImporter).importEngines.func3
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/table_import.go:504
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v7.5.0-alpha
Edition: Community
Git Commit Hash: af7b32c
Git Branch: heads/refs/tags/v7.5.0-alpha
UTC Build Time: 2023-10-19 11:43:11
GoVersion: go1.21.3
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2023-10-20T08:01:59.525+0800

@Lily2025 Lily2025 added the type/bug The issue is confirmed as a bug. label Oct 23, 2023
@Lily2025
Copy link
Author

/severity major
/assign lance6716

@lance6716 lance6716 added affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. affects-6.1 affects-6.5 affects-7.1 affects-7.5 labels Oct 23, 2023
@ti-chi-bot ti-chi-bot bot removed may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 labels Oct 23, 2023
@lance6716
Copy link
Contributor

lance6716 commented Oct 23, 2023

I guess it's caused by

tidb/br/pkg/pdutil/pd.go

Lines 167 to 174 in 05b50a1

req, err := http.NewRequestWithContext(ctx, method, reqURL, body)
if err != nil {
return 0, nil, errors.Trace(err)
}
var resp *http.Response
count := 0
for {
resp, err = cli.Do(req) //nolint:bodyclose

client.Do will close req.Body, so in second retry body size is 0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects 5.4.x versions. affects-6.1 affects-6.5 affects-7.1 affects-7.5 component/lightning This issue is related to Lightning of TiDB. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants