Handle orphan write cf key from a raftstore v2 tablet snapshot

[CalvinNeo]

What problem does this PR solve?

Issue Number: ref #7568

Problem Summary:

Raftstore v2 snapshot may contain extra write key which is from later raft log index than the snapshot has claimed. We decide to suppress this error rather than throwing an exception.

We will keep track of these orphan key, and throw an exception later if we are sure these keys are orphan because of data loss rather than an eager-replicated data.

What is changed and how it works?

The suppress part:

1. Deduce TiKV cluster's raftstore version by the first /engine_type response from TiKV.
   1. Note that TiKV later supports coexistence of v1 and v2 node in a cluster. After that, they can provide us a mechanism to fix this.
2. Store all orphan write key in OrphanKeyInfo struct when prehandling snapshot.
3. Remove a key from OrphanKeyInfo when insert into write cf.
4. When meet orphan key in a row2col job which is not triggered by prehandling snapshot, check OrphanKeyInfo first. If the key is registered in OrphanKeyInfo, consider it as a soft error.

The async checking part:

1. Log a deadline_index sent from TiKV's snapshot which must GE than snapshot_index in in OrphanKeyInfo.
2. If the applied_index is proceeded exceed the deadline_index, check if the OrphanKeyInfo is cleared. Otherwise raise a hard error.

What is not included:

1. PreHandleSnapshot requires deadline_index being sent from TiKV. So we can have async check only after TiKV support this.
2. Although, we have this PR merged first to avoid blocking test.

NOTE
Orphan keys will not be iterated by region read scanner, so it will not be presented in data_list_read, so it will not be removed as a committed entry. We don't eagerly do that, since it brings extra cost. These keys will either be removed when its default counterpart arrives in the following raft log, or being reported as an error when exceeds deadline_index.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
  • Tested the cluster version part

  [2023/06/12 12:41:25.170 +08:00] [INFO] [run.rs:147] ["start probing cluster's raftstore version"]
  [2023/06/12 12:41:25.267 +08:00] [INFO] [run.rs:150] ["cluster's raftstore version is V2"]
  [2023/06/12 14:14:17.974 +08:00] [INFO] [run.rs:147] ["start probing cluster's raftstore version"]
  [2023/06/12 14:14:18.072 +08:00] [INFO] [run.rs:150] ["cluster's raftstore version is V1"]
  

• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Handle orphan write cf key from a raftstore v2 tablet snapshot

[Lloyd-Pottiger]

How about converting to draft before ready to avoid seeds messages to reviewers?

[CalvinNeo]

This will be done when tikv support a deadline_index.

[CalvinNeo]

/run-all-tests

[CalvinNeo]

/run-all-tests

[CalvinNeo]

/run-all-tests

[JaySon-Huang]

LGTM

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, lidezhu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,lidezhu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2023-06-20 03:00:34.336236373 +0000 UTC m=+71799.737486822: ☑️ agreed by JaySon-Huang.
• 2023-06-20 06:53:45.049481776 +0000 UTC m=+85790.450732224: ☑️ agreed by lidezhu.

[JaySon-Huang]

@CalvinNeo can we unhold this PR and get it merged?

[CalvinNeo]

/unhold