redo(ticdc): use memory backend in writer

[CharlesCheung96]

What problem does this PR solve?

Issue Number: close #8086

What is changed and how it works?

1. Split out MetaManager to manage meta and gc. After this pr, cdc will only perform garbage colletion in owner.
2. Add and implement redo DDLManager.
3. Refactor writer interface.
4. Implemet memoryLogWriter.

Check List

Tests

• Unit test
• Integration test
• test_infra test
  1. use memory backend
     • cdc image: redo_test_master_memory_writer_07a32d
     • tcms execution ID: 1621572
  2. use file backend
     • cdc image: redo_test_master_file_writer_77bd6dd
     • tcms execution ID: 1621573
• Manual test (add detailed scripts or steps below)
  Run benchmark tests with storage = "file:///tmp/testredo", write 300k events to each of the 20 tables

1. set 50 nanosecond lag for marshalling operation (from 181k ==> 202k)

goos: linux
goarch: amd64
pkg: [github.com/pingcap/tiflow/cdc/redo](http://github.com/pingcap/tiflow/cdc/redo)
cpu: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
BenchmarkRedoManagerWaitFlush_memory_50NanoLag
[2023/01/30 15:48:40.485 +08:00] [INFO] [manager.go:528] ["redo manager bgUpdateLog is running"] [namespace=] [changefeed=] [flushIntervalInMs=2000]
[2023/01/30 15:48:40.485 +08:00] [INFO] [encoding_worker.go:168] ["redo log encoding workers started"] [workerNum=16]
[2023/01/30 15:48:40.485 +08:00] [INFO] [file_worker.go:139] ["redo file workers started"] [workerNum=8]
BenchmarkRedoManagerWaitFlush_memory_50NanoLag-16              1        296588198108 ns/op      105787869240 B/op       473722266 allocs/op

goos: linux
goarch: amd64
pkg: [github.com/pingcap/tiflow/cdc/redo](http://github.com/pingcap/tiflow/cdc/redo)
cpu: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
BenchmarkRedoManagerWaitFlush_file_50NanoLag
[2023/01/30 15:55:35.836 +08:00] [INFO] [manager.go:528] ["redo manager bgUpdateLog is running"] [namespace=] [changefeed=] [flushIntervalInMs=2000]
BenchmarkRedoManagerWaitFlush_file_50NanoLag-16                1        331281670719 ns/op      48071539928 B/op        229195602 allocs/op

5. set 300 nanosecond lag for marshalling operation (from 129k ==> 205k)

goos: linux
goarch: amd64
pkg: [github.com/pingcap/tiflow/cdc/redo](http://github.com/pingcap/tiflow/cdc/redo)
cpu: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
BenchmarkRedoManagerWaitFlush_memory_300NanoLag
[2023/01/30 16:43:34.852 +08:00] [INFO] [manager.go:528] ["redo manager bgUpdateLog is running"] [namespace=] [changefeed=] [flushIntervalInMs=2000]
[2023/01/30 16:43:34.852 +08:00] [INFO] [encoding_worker.go:168] ["redo log encoding workers started"] [workerNum=16]
[2023/01/30 16:43:34.852 +08:00] [INFO] [file_worker.go:139] ["redo file workers started"] [workerNum=8]
BenchmarkRedoManagerWaitFlush_memory_300NanoLag-16              1        292145861334 ns/op      95117248024 B/op        441196050 allocs/op

goos: linux
goarch: amd64
pkg: [github.com/pingcap/tiflow/cdc/redo](http://github.com/pingcap/tiflow/cdc/redo)
cpu: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
BenchmarkRedoManagerWaitFlush_file_300NanoLag
[2023/01/30 16:42:45.791 +08:00] [INFO] [manager.go:528] ["redo manager bgUpdateLog is running"] [namespace=] [changefeed=] [flushIntervalInMs=2000]
BenchmarkRedoManagerWaitFlush_file_300NanoLag-16                1        463626690994 ns/op      47229759264 B/op        208209940 allocs/op

Questions

Do you need to update user documentation, design documentation or monitoring documentation?

Add new config consistent.use-file-backend, which is false by default.

Release note

`Add a memory backend writer to redo log`.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• asddongmen
• zhaoxinyu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 564daaa

[CharlesCheung96]

/run-integration-test

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-6.5: #8410.