Hierarchical Caching Refactoring and Fixing TP issue

[xiezhq-hermann]

Motivation

This is a follow-up PR to #2693, primarily focus on two things:

1. Fixing concurrency issue in the case of tp_size > 1, which was discussed by [Bug] HiCacheController gets stuck when testing with multiple long text documents #3998 and Hierarchical Caching supports MLA #4009
2. Introducing layer-wise KV cache loading and synchronization

Modifications

1. A new completion checking of hierarchical cache activities with order enforcement across TP workers
2. New event and counters to enable layer-wise KV cache loading, more policies to be added
3. Some style changes

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[zhyncs]

Do we have unit test for this?

[xiezhq-hermann]

Do we have unit test for this?

I will add a set of unit tests for hierarchical caching and incorporate them into CI in a separate PR. This PR is a fix for a rather emergent issue of this feature.