Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: add memory tracker for InsertExec and ReplaceExec #14179

Merged
merged 28 commits into from
Dec 30, 2019
Merged

*: add memory tracker for InsertExec and ReplaceExec #14179

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
e4bffdb
*: add memory tracker for InsertExec and ReplaceExec
XuHuaiyu Dec 23, 2019
99a6574
test
XuHuaiyu Dec 23, 2019
c6f9e29
fix ci
XuHuaiyu Dec 23, 2019
1bb2640
fix ci
XuHuaiyu Dec 23, 2019
a9f2e5b
Merge branch 'master' of https://github.com/pingcap/tidb into trace-w…
XuHuaiyu Dec 24, 2019
ddfbbe1
remove useless code
XuHuaiyu Dec 24, 2019
04d4dad
refine
XuHuaiyu Dec 24, 2019
554f22c
remove useless code
XuHuaiyu Dec 24, 2019
d430227
add some tests
XuHuaiyu Dec 24, 2019
1f6ed6e
refine code
XuHuaiyu Dec 25, 2019
13edb1f
remove useless code
XuHuaiyu Dec 25, 2019
b55a29e
add test
XuHuaiyu Dec 25, 2019
dbc820c
fix ci
XuHuaiyu Dec 25, 2019
0b61a15
test
XuHuaiyu Dec 25, 2019
4797740
fix ci
XuHuaiyu Dec 25, 2019
c498234
address comment
XuHuaiyu Dec 26, 2019
46e698d
fix ci
XuHuaiyu Dec 26, 2019
80926b3
Merge branch 'master' into trace-write-memory
XuHuaiyu Dec 26, 2019
3fb96e9
address comment
XuHuaiyu Dec 26, 2019
d69dd2c
Merge branch 'trace-write-memory' of https://github.com/XuHuaiyu/tidb…
XuHuaiyu Dec 26, 2019
6419d09
address comment
XuHuaiyu Dec 27, 2019
1df692f
address comment
XuHuaiyu Dec 27, 2019
7b9ed07
address comment
XuHuaiyu Dec 27, 2019
438ee6c
fix ci
XuHuaiyu Dec 27, 2019
0fd2084
address comment
XuHuaiyu Dec 30, 2019
0b63c28
address comment
XuHuaiyu Dec 30, 2019
4859fa3
Merge branch 'master' of https://github.com/pingcap/tidb into trace-w…
XuHuaiyu Dec 30, 2019
7625390
remove useless code
XuHuaiyu Dec 30, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address comment
  • Loading branch information
@XuHuaiyu
XuHuaiyu committed Dec 26, 2019
commit c498234a4c5831066284eda1d0bd340eb7380d52
4 changes: 4 additions & 0 deletions executor/insert_common.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,8 @@ func insertRows(ctx context.Context, base insertCommon) (err error) {
memTracker = x.memTracker
case *ReplaceExec:
memTracker = x.memTracker
default:
return errors.Errorf("unexpected executor type %v", x)
}
sessVars := e.ctx.GetSessionVars()
batchInsert := sessVars.BatchInsert && !sessVars.InTxn() && config.GetGlobalConfig().EnableBatchDML
Expand Down Expand Up @@ -405,6 +407,8 @@ func insertRowsFromSelect(ctx context.Context, base insertCommon) error {
memTracker = x.memTracker
case *ReplaceExec:
memTracker = x.memTracker
default:
return errors.Errorf("unexpected executor type %v", x)
}

sessVars := e.ctx.GetSessionVars()
Expand Down
8 changes: 6 additions & 2 deletions session/txn.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -438,8 +438,12 @@ func (s *session) getTxnFuture(ctx context.Context) *txnFuture {

// StmtCommit implements the sessionctx.Context interface.
func (s *session) StmtCommit(memTracker *memory.Tracker) error {
Copy link
Member

@jackysp jackysp Dec 26, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The statement is finished and the memory is allocated when StmtCommit, I think it is meanless. Consume it at

tidb/session/txn.go

Line 260 in d666ed4

return st.buf.Set(k, v)
.

Copy link
Contributor Author

@XuHuaiyu XuHuaiyu Dec 26, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But stmtBuffer can be GCed after be written to txnBuffer?

defer s.txn.cleanup()
defer func() {
s.txn.cleanup()
memTracker.Consume(int64(-s.txn.Size()))
}()
st := &s.txn
txnSize := st.Transaction.Size()
var count int
err := kv.WalkMemBuffer(st.buf, func(k kv.Key, v []byte) error {
failpoint.Inject("mockStmtCommitError", func(val failpoint.Value) {
Expand All @@ -462,7 +466,7 @@ func (s *session) StmtCommit(memTracker *memory.Tracker) error {
return err
}
if memTracker != nil {
memTracker.Consume(int64(st.Transaction.Size()))
memTracker.Consume(int64(st.Transaction.Size() - txnSize))
}

// Need to flush binlog.
Expand Down