Enhance heartbeat processing capabilities

[nolouch]

Development Task

Sigle node processing capacity

Improve the diagnosis process and implement hierarchical processing for Heartbeat logic

• Add Heartbeat trace statistics help analyze performance bottlenecks #7868
• Async the logic not the relative metadata process
  • Introduce a common async method with the resource limiter. pkg/ratelimit: introduce an executor that can run with a limiter #8024
  • async the statistical processing
    • *: optimize heartbeat process with concurrent runner - part 1 #8053
    • *: optimize heartbeat process with concurrent runner - part 2 #8052

Optimize Lock and Reduce lock holding time

• getOverlapRegionFromSubTreeLocked consume many cpu hold lock #8136
• Reduce operator controller locking competition pkg/schedule: optimize the lock usage of operator controller #8032
• operator: make additional information thread safe #8104

Improve the memory alloc

• once there are 10M regions, the GC and memory issues will be significant.
  • pkg: improve get forwarded host #8182
  • pkg: improve region path #8180
  • pkg: improve region statistics observe #8181
  • pkg: reduce the allocation of observe #8188
  • reduce the memory alloc in hot statics #8207
  • *: batch process peer task #8213
  • *: optimize memory usage #8164

Scalability

• [TBD] Proposal
  • only metadata(lookup) part, statics part not consider