Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lightning: DBaaS import csv fails with InvalidRange if CSV file size is about 256MB when strict-format is true #27763

Closed
fubinzh opened this issue Sep 2, 2021 · 3 comments · Fixed by #27769
Closed

Lightning: DBaaS import csv fails with InvalidRange if CSV file size is about 256MB when strict-format is true #27763

fubinzh opened this issue Sep 2, 2021 · 3 comments · Fixed by #27769
Assignees
@lonng
Labels
affects-4.0 This bug affects 4.0.x versions. affects-5.0 This bug affects 5.0.x versions. affects-5.1 This bug affects 5.1.x versions. affects-5.2 This bug affects 5.2.x versions. component/lightning This issue is related to Lightning of TiDB. severity/major sig/migrate type/bug The issue is confirmed as a bug.

Comments

@fubinzh
Copy link

fubinzh commented Sep 2, 2021

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

On DBaaS, try to import csv from S3, S3 bucket files like below:
[root@centos76_vm ~]# du -shl /minio/fubin/dbgen/xxx/*
257M /minio/fubin/dbgen/aaaa/aaaa.bbbb.000000000.csv
257M /minio/fubin/dbgen/aaaa/aaaa.bbbb.000000001.csv
257M /minio/fubin/dbgen/aaaa/aaaa.bbbb.000000002.csv
257M /minio/fubin/dbgen/aaaa/aaaa.bbbb.000000003.csv
257M /minio/fubin/dbgen/aaaa/aaaa.bbbb.000000004.csv
257M /minio/fubin/dbgen/aaaa/aaaa.bbbb.000000005.csv
257M /minio/fubin/dbgen/aaaa/aaaa.bbbb.000000006.csv
257M /minio/fubin/dbgen/aaaa/aaaa.bbbb.000000007.csv
257M /minio/fubin/dbgen/aaaa/aaaa.bbbb.000000008.csv
257M /minio/fubin/dbgen/aaaa/aaaa.bbbb.000000009.csv
4.0K /minio/fubin/dbgen/aaaa/aaaa.bbbb-schema.sql
4.0K /minio/fubin/dbgen/aaaa/aaaa-schema-create.sql
4.0K /minio/fubin/dbgen/aaaa/metadata

2. What did you expect to see? (Required)

Import should succeed

3. What did you see instead (Required)

Import failed.

4. What is your TiDB version? (Required)

Lightning: 5.2.0
[release-version=v5.2.0] [git-hash=05d2210647d6a1503a8d772477e43b14a024f609] [git-branch=heads/refs/tags/v5.2.0] [go-version=go1.16.4]
@fubinzh fubinzh added type/bug The issue is confirmed as a bug. severity/critical sig/migrate component/lightning This issue is related to Lightning of TiDB. labels Sep 2, 2021
@jebter jebter added the affects-5.2 This bug affects 5.2.x versions. label Sep 3, 2021
@fubinzh fubinzh added affects-5.1 This bug affects 5.1.x versions. affects-5.0 This bug affects 5.0.x versions. affects-4.0 This bug affects 4.0.x versions. labels Sep 6, 2021
@glorv glorv mentioned this issue Sep 6, 2021
Merged
4 tasks
@fubinzh fubinzh changed the title Lightning: DBaaS import csv fails with InvalidRange if CSV file size is about 256MB Lightning: DBaaS import csv fails with InvalidRange if CSV file size is about 256MB when strict-format is true Sep 6, 2021
@github-actions
Copy link

github-actions bot commented Sep 8, 2021

Please check whether the issue should be labeled with 'affects-x.y' or 'backport-x.y.z',
and then remove 'needs-more-info' label.

@fubinzh
Copy link
Author

fubinzh commented Sep 18, 2021

On DBaaS, strict-format is default to False, and this issue is not seen when strict-format is False.

This was referenced Feb 21, 2022
Merged
Merged
@ti-srebot ti-srebot mentioned this issue Apr 12, 2022
Closed
4 tasks
@VelocityLight
Copy link

update

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lonng lonng

Labels
affects-4.0 This bug affects 4.0.x versions. affects-5.0 This bug affects 5.0.x versions. affects-5.1 This bug affects 5.1.x versions. affects-5.2 This bug affects 5.2.x versions. component/lightning This issue is related to Lightning of TiDB. severity/major sig/migrate type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

br: fix lightning split large csv file error and adjust s3 seek result glorv/tidb
4 participants
@lonng @jebter @fubinzh @VelocityLight