Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LOAD DATA statement doesn't respect tidb_dml_batch_size variable #10524

Closed
alex-lx opened this issue May 17, 2019 · 3 comments
Closed

LOAD DATA statement doesn't respect tidb_dml_batch_size variable #10524

alex-lx opened this issue May 17, 2019 · 3 comments
Labels
sig/transaction SIG:Transaction

Comments

@alex-lx
Copy link
Contributor

alex-lx commented May 17, 2019

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?

We use LOAD DATA statement to load csv to tidb.
Because the size of each row is quite large, the tidb-server complains about too large transaction, then we tried using set @@tidb_dml_batch_size=100 to make the batch smaller.

  1. What did you expect to see?

All the data loaded to tidb with batch size 100

  1. What did you see instead?

the batch size is still 20000.
We use another session console to monitor the count of the target table, it increases by 20000 for every step.

  1. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?
Release Version: v2.1.2-1-gb4cc58c8c
Git Commit Hash: b4cc58c8c8c03ee2c9a462c31c7ee4f0df81b9bf
Git Branch: v2.1.2-xm
UTC Build Time: 2019-01-14 09:06:14
GoVersion: go version go1.11.1 darwin/amd64
Race Enabled: false
TiKV Min Version: 2.1.0-alpha.1-ff3dd160846b7d1aed9079c389fc188f7f5ea13e
Check Table Before Drop: false
@alex-lx
Copy link
Contributor Author

alex-lx commented May 17, 2019

tidb/server/conn.go

Lines 1079 to 1080 in 3177328

// TODO: Make the loadDataRowCnt settable.
loadDataInfo.SetMaxRowsInBatch(defaultLoadDataBatchCnt)

It looks like your guys need to complete the TODO

@ghost ghost added the type/bug The issue is confirmed as a bug. label Jul 23, 2020
@jebter jebter added the sig/transaction SIG:Transaction label Nov 16, 2020
@youjiali1995
Copy link
Contributor

Fixed by #11533

@ti-srebot
Copy link
Contributor

ti-srebot commented Jan 21, 2021

Please edit this comment or add a new comment to complete the following information

Not a bug

It's a TODO item, not a bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sig/transaction SIG:Transaction
Projects
None yet
Development

No branches or pull requests

5 participants