collect table statistics when importing data by Lightning

[xuyifangreeneyes]

Enhancement

Currently, after we import data to the cluster, we need to analyze the table, which is time-consuming since it needs to scan the whole table. Collecting table statistics can be done when iterating all the data. There are several steps in Lightning that iterate all the data, such as scanning CSV files, passing KV values streamingly from Lightning to TiKV, and checksum. Hence collecting table statistics can be done in any of those steps to avoid another table full scan.

[lance6716]

Write some notes about my understanding. Lightning should integrate these actions (take adding index as example):

1. maintain Count-Min Sketch, Flajolet–Martin Sketch, Histogram for each import chunk
   https://github.com/tikv/tikv/blob/321aa833ca5ec0fd5dcec7fa8c01f65116d72ba6/src/coprocessor/statistics/analyze.rs#L145-L188

2. update statistics for whole table from multiple per-chunk statistics

   tidb/executor/analyze_idx.go

   Line 175 in 338fd30

   func (e *AnalyzeIndexExec) buildStatsFromResult(result distsql.SelectResult, needCMS bool) (*statistics.Histogram, *statistics.CMSketch, *statistics.FMSketch, *statistics.TopN, error) {