Too many empty region after restoration of many small tables

[YuJuncen]

1. What did you do?
   Restore a workload with 190G, 6000 tables, according to the backup meta, there are about 42K files backed up.

2. What did you expect to see?
   Restore done, and the region count should be less than 42K (or we nearly must get empty regions), and the number of empty region should be a reasonable value.

3. What did you see instead?
   See figures below.

Figure 1. too many unhealthy regions

Figure 2. too many regions

4. What version of BR and TiDB/TiKV/PD are you using?

Nightly (2020-7-20)

5. Operation logs

(The log is too large to be updated... Too many write conflicts with crating tables concurrently...)

[YuJuncen]

Regions with size less than or equal to 1M bytes would be treated as an empty region.

According the log from TiKV, with the workload of many-small-tables, the reported table size is very small (only 2 of 10 regions are greater than 1M):

[YuJuncen]

Currently, We split region in two class of keys: the new key of rewrite rules (t{new_table_id}) and the end key of each files. Generally, this did two things:

1. Split at the start and the last key backed up of the table, so there isn't any records from other tables share the region with these records.

0|-----|----------------------|---------|∞
       ^                      ^(t{new_table_id}_r{last_record_key_backed_up})
       +(t{new_new_table_id})

2. Then, according to the file size of this table, split internally.

0|-[1]-|-96M-|-96M-|-96M-|-35M|---[2]---|∞
       ^                      ^(t{new_table_id}_r{last_record_key_backed_up})
       (t{new_new_table_id})

This method make best safety: because for performance, each download RPC can take only one rewrite rule, no table overlapping means each download RPC can always choose the rewrite rule for the table (or the index) to restore.

However, this method make more empty regions, in the example, there are two regions [0,t{new_new_table_id})([1]) and [t{new_table_id}_r{last_record_key_backed_up}, ∞)([2]) become empty.

Maybe the last key of each table or the first key can be omitted. In the many-table workload, if the former table has split at the end, then the latter can reuse the region at [2]. and vice versa.

0|-----|-T1--|-T1--|-T1--|-T1----|-T2-|∞
       ^                      ^  ^(t2)
       |                      +(t1_r{last_record_key_backed_up})
       +(t1)

This figure shows an example that after omitting split the last key of each table, the property of each table owns one region can probably be kept.