PD can't redistribute the hot write regions among TiFlash nodes

[JaySon-Huang]

Bug Report

What did you do?

I deployed a cluster with 1 TiDB + 1 PD + 1 TiKV + 2 TiFlash base on version v4.0.8.

The TiFlash and its proxy branch are "store_stats_4.0". These two branches fix the problem that the written bytes and written keys at the store level are not reported to PD.

What did you expect to see?

If the write-pressure between TiFlash nodes is not balanced, PD can generate some move-hot-write-region operators and make it balanced.

What did you see instead?

By adding a sysbench workload on this cluster, I found that:

• move-hot-write-region between TiFlash store rarely happen.
• In the phase of running the "oltp_update_index" workload, the writes-pressure between two TiFlash nodes is imbalanced, one is about 10 times to another node. But PD still did not generate move-hot-write-region between TiFlash stores.

Another problem, maybe related or not:
I use the PD API: /pd/api/v1/hotspot/regions/write to check the stats of hot write regions. In the TiFlash node, the flow bytes by summing all regions is about 4 times to the TiFlash reported.

What version of PD are you using (pd-server -V)?

>  ./pd-server -V
Release Version: v4.0.8
Edition: Community
Git Commit Hash: 775b6a5ef517f8ab2f43fef6418bbfc7d6c9c9dc
Git Branch: heads/refs/tags/v4.0.8
UTC Build Time:  2020-10-30 08:15:09

[JaySon-Huang]

message StoreStats {
    uint64 store_id = 1;
    // Capacity for the store.
    uint64 capacity = 2;
    // Available size for the store.
    uint64 available = 3;
    // Total region count in this store.
    uint32 region_count = 4;
    // Current sending snapshot count.
    uint32 sending_snap_count = 5;
    // Current receiving snapshot count.
    uint32 receiving_snap_count = 6;
    // When the store is started (unix timestamp in seconds).
    uint32 start_time = 7;
    // How many region is applying snapshot.
    uint32 applying_snap_count = 8;
    // If the store is busy
    bool is_busy = 9;
    // Actually used space by db
    uint64 used_size = 10;
    // Bytes written for the store during this period.
    uint64 bytes_written = 11;
    // Keys written for the store during this period.
    uint64 keys_written = 12;
    // Bytes read for the store during this period.
    uint64 bytes_read = 13;
    // Keys read for the store during this period.
    uint64 keys_read = 14;
    // Actually reported time interval
    TimeInterval interval = 15;
    // Threads' CPU usages in the store
    repeated RecordPair cpu_usages = 16;
    // Threads' read disk I/O rates in the store
    repeated RecordPair read_io_rates = 17;
    // Threads' write disk I/O rates in the store
    repeated RecordPair write_io_rates = 18;
    // Operations' latencies in the store
    repeated RecordPair op_latencies = 19;
}

Does not like leader and followers in TiKV, all TiFlash learner peers are ready for reading. And TiDB chooses TiFlash peer by using round-robin. So I think that reporting bytes_read, keys_read for hot read regions redistributing is not very meaningful for TiFlash.

Are there any key metrics for PD scheduling besides capacity, available, used_size, region_count, bytes_written, keys_written?

[ilovesoup]

Will this be a fix in 4.0.x version? ZTO (one of our largest cluster user is trying TiFlash on 4.0). This issue is a troublesome one.

[JaySon-Huang]

Done in #3671, related PR: #3900