Unbounded memory usage of `BatchGet`'s cache

[bobotu]

Bug Report

The internal cache used by BatchGet need a size limit, it is OK to randomly discard some entry in it.

1. Minimal reproduce step (Required)

Start a transaction, do a lot of batch get.

2. What did you expect to see? (Required)

Low memory usage.

3. What did you see instead (Required)

Unbounded memory usage.

4. What is your TiDB version? (Required)

v4.0, master

[time-and-fate]

/label sig/transaction

[ichn-hu]

/assign @ichn-hu

[ichn-hu]

/unassign @ichn-hu

[ti-srebot]

Please edit this comment or add a new comment to complete the following information

Bug

1. Root Cause Analysis (RCA) (optional)

The internal cache used by BatchGet does not have a size limit. If the user starts a transaction and query lots of data without commit/rollback, the cache may use too much memory and result in an OOM error.

2. Symptom (optional)

TiDB OOM.

3. All Trigger Conditions (optional)

Do a large amount of query in one transaction and leave it valid forever (without commit/rollback).

4. Workaround (optional)

Find this session and kill it.

5. Affected versions

[v4.0.0:v4.0.8]

6. Fixed versions

v4.0.9