memory_usage_limit is deduced much less then expected

[CalvinNeo]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

If not set, memory_usage_limit is inferred by block_cache_cap

            let mut limit =
                (block_cache_cap.0 as f64 / BLOCK_CACHE_RATE * MEMORY_USAGE_LIMIT_RATE) as u64;

https://github.com/pingcap/tidb-engine-ext/blob/521fd9dbc55e58646045d88f91c3c35db50b5981/src/config/mod.rs#L3597-L3598
which is actually set by(I mean it's original value is not this, but the following code will adjust and take final effect)

        if let Some(a) = self.rocksdb.defaultcf.block_cache_size
            && let Some(b) = self.rocksdb.writecf.block_cache_size
            && let Some(c) = self.rocksdb.lockcf.block_cache_size
        {
            let d = self
                .raftdb
                .defaultcf
                .block_cache_size
                .map(|s| s.0)
                .unwrap_or_default();
            let sum = a.0 + b.0 + c.0 + d;
            self.storage.block_cache.capacity = Some(ReadableSize(sum));
        }

https://github.com/pingcap/tidb-engine-ext/blob/521fd9dbc55e58646045d88f91c3c35db50b5981/src/config/mod.rs#L3795-L3805

and modified by

    config.raftdb.defaultcf.block_cache_size = proxy_config.raftdb.defaultcf.block_cache_size;
    config.rocksdb.defaultcf.block_cache_size = proxy_config.rocksdb.defaultcf.block_cache_size;
    config.rocksdb.writecf.block_cache_size = proxy_config.rocksdb.writecf.block_cache_size;
    config.rocksdb.lockcf.block_cache_size = proxy_config.rocksdb.lockcf.block_cache_size;

https://github.com/pingcap/tidb-engine-ext/blob/521fd9dbc55e58646045d88f91c3c35db50b5981/proxy_components/proxy_server/src/config.rs#L405-L408

See pingcap/tidb-engine-ext@2f2900a.

---

And tiflash proxy limit the memory for CF to a small number.

pub fn memory_limit_for_cf(is_raft_db: bool, cf: &str, total_mem: u64) -> ReadableSize {
    let (ratio, min, max) = match (is_raft_db, cf) {
        (true, CF_DEFAULT) => (0.05, 256 * MIB as usize, usize::MAX),
        (false, CF_DEFAULT) => (0.25, 0, 128 * MIB as usize),
        (false, CF_LOCK) => (0.02, 0, 128 * MIB as usize),
        (false, CF_WRITE) => (0.15, 0, 128 * MIB as usize),
        _ => unreachable!(),
    };
    let mut size = (total_mem as f64 * ratio) as usize;
    size = size.clamp(min, max);
    ReadableSize::mb(size as u64 / MIB)
}

https://github.com/pingcap/tidb-engine-ext/blob/521fd9dbc55e58646045d88f91c3c35db50b5981/proxy_components/proxy_server/src/config.rs#L123-L134

---

The logic affects all released versions including LTS 6.1/6.5/7.1/7.5/8.1/8.5

As a result, consider an enough big memory, the limit could be:

• kv cf: 128MB * 3 = 0.375GiB
• raft cf: 0.05 * total available machine memory

So basicly, the memory limit is 0.05 * total available machine memory / 0.45 * 0.75.

• If the machine memory is 376 GiB, then memory limit is 31.3GiB.
• If the machine memory is 32 GiB, then memory limit is 2.66 GiB.

Note, even if the raft-engine is used, we still take the raft_db memory size into account.

And because there is a memory high water machanism, the memory usage on proxy will time another 0.1 factor which calls memory_usage_high_water.

#[allow(clippy::derivable_impls)]
impl Default for ProxyConfig {
    fn default() -> Self {
        ProxyConfig {
            raft_store: RaftstoreConfig::default(),
            server: ServerConfig::default(),
            rocksdb: RocksDbConfig::default(),
            raftdb: RaftDbConfig::default(),
            storage: StorageConfig::default(),
            enable_io_snoop: false,
            memory_usage_high_water: 0.1,
            readpool: ReadPoolConfig::default(),
            import: ImportConfig::default(),
            engine_store: EngineStoreConfig::default(),
            memory: MemoryConfig::default(),
        }
    }
}

So as a result,

• If the machine memory is 376 GiB, then memory limit is 31.3GiB. TiFlash would complain if the memory is more than 3.1GiB.
• If the machine memory is 32 GiB, then memory limit is 2.66 GiB. TiFlash would complain if the memory usage is more than 0.26 GiB.