Build table snapshot on write node for disaggregated read from S3 #6919

JaySon-Huang · 2023-02-28T12:26:21Z

What problem does this PR solve?

Issue Number: ref #6827

Problem Summary:

What is changed and how it works?

Depends on:

Add rpc interface for tiflash disaggregated architecture kvproto#1068

Changes:

Add DisaggregatedTaskId for the task id, it is based on MPPId and executor_id
Add DAGContext::is_disaggregated_task DAGContext::disaggregated_id
Add DisaggregatedReadSnapshot to hold the segment snapshot with DisaggregatedTaskId
Add DisaggregatedSnapshotManager to hold the snapshots of logical table in the write node
FlashService::EstablishDisaggregatedTask
- run into DAGStorageInterpreter::buildLocalStreams
- build segment snapshots from StorageDeltaMerge
- register the snapshots into DisaggregatedSnapshotManager in the write node
the compute node will build the read task and try to read most data from S3 according to the EstablishDisaggregatedTaskResponse
FlashService::FetchDisaggregatedPages
- compute nodes will use this PRC to fetch the delta-layer data from write nodes
Remove the dangling snapshot from DisaggregatedSnapshotManager if the read node doesn't try to finish/cancel the request

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2023-02-28T12:26:23Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

breezewish
guo-shaoge

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

JaySon-Huang · 2023-03-02T08:38:13Z

/run-all-tests

JaySon-Huang · 2023-03-02T10:50:43Z

/run-all-tests

breezewish

Only some minor suggestions

breezewish · 2023-03-02T12:22:26Z

dbms/src/Flash/Coprocessor/DAGContext.h

    /* const */ bool is_mpp_task = false;
    /* const */ bool is_root_mpp_task = false;
    /* const */ bool is_batch_cop = false;
+    /* const */ bool is_disaggregated_task = false;


Do we use this flag for both: requests in rn and establish requests in wn?

The rn does not need to create a DAGContext with is_disaggregated_task==true as it only needs to send disaggregated task. This flag is only set when a wn receives a disaggregated task

dbms/src/Flash/FlashService.h

breezewish · 2023-03-03T03:37:33Z

dbms/src/Flash/FlashService.cpp

+    db_context->setMockStorage(mock_storage);
+    db_context->setMockMPPServerInfo(mpp_test_info);
+
+    RUNTIME_CHECK(context->isDisaggregatedStorageMode());


Off topic: When will we use context and when will we use db_context?

context - actually it is a pointer to the global context
db_context - a temporary context that its lifetime lasts only until the end of this request

db_context is a session-only context, some value of db_context->getSettings() is not the same as the global settings.

dbms/src/Flash/FlashService.cpp

dbms/src/Storages/DeltaMerge/Remote/Serializer.cpp

dbms/src/Storages/DeltaMerge/Remote/DisaggregatedSnapshotManager.h

dbms/src/Core/Defines.h

dbms/src/Storages/DeltaMerge/Remote/DisaggSnapshot.cpp

JaySon-Huang · 2023-03-03T13:16:03Z

/run-all-tests

JaySon-Huang · 2023-03-03T16:34:07Z

dbms/src/Flash/Coprocessor/DAGStorageInterpreter.cpp

+    // Note that `buildRemoteRequests` must be called after `buildLocalStreams` because
+    // `buildLocalStreams` will setup `region_retry_from_local_region` and we must
+    // retry those regions or there will be data lost.


detect by test case "tests/fullstack-test/fault-inject/exception_after_read_from_storage.test"
/cc @breezewish

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Signed-off-by: JaySon-Huang <tshent@qq.com>

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

JaySon-Huang · 2023-03-06T06:43:55Z

/merge

ti-chi-bot · 2023-03-06T06:43:56Z

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-03-06T06:43:58Z

This pull request has been accepted and is ready to merge.

Commit hash: c886f7d

JaySon-Huang · 2023-03-06T07:42:31Z

/run-integration-test

SeaRise · 2023-03-06T08:01:31Z

/run-integration-test

JaySon-Huang · 2023-03-06T08:45:00Z

/run-all-tests

JaySon-Huang · 2023-03-06T09:41:01Z

/run-all-tests

JaySon-Huang · 2023-03-06T10:03:11Z

/run-integration-test

JaySon-Huang force-pushed the disagg_read_snap branch 4 times, most recently from 3072251 to 61471a5 Compare March 2, 2023 08:19

JaySon-Huang changed the title ~~[WIP] Serialize table snapshot for disaggregated read~~ Build table snapshot on write node for disaggregated read from S3 Mar 2, 2023

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 2, 2023

JaySon-Huang requested review from windtalker, breezewish, guo-shaoge and flowbehappy March 2, 2023 08:37

ti-chi-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed do-not-merge/needs-linked-issue labels Mar 2, 2023

JaySon-Huang force-pushed the disagg_read_snap branch from 61471a5 to 5ade3a4 Compare March 2, 2023 10:42

JaySon-Huang removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 2, 2023

breezewish reviewed Mar 3, 2023

View reviewed changes

JaySon-Huang force-pushed the disagg_read_snap branch from 74249e6 to 772ed15 Compare March 3, 2023 07:56

breezewish approved these changes Mar 3, 2023

View reviewed changes

dbms/src/Core/Defines.h Outdated Show resolved Hide resolved

dbms/src/Storages/DeltaMerge/Remote/DisaggSnapshot.cpp Show resolved Hide resolved

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 3, 2023

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 3, 2023

JaySon-Huang commented Mar 3, 2023

View reviewed changes

ti-chi-bot requested a review from breezewish March 3, 2023 16:34

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 3, 2023

guo-shaoge approved these changes Mar 6, 2023

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 6, 2023

JaySon-Huang and others added 13 commits March 6, 2023 14:42

Serialize table snapshot for disaggreagated read

c93a444

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

clean useless

817305f

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

establish disagg read task on write node

a142f5d

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

clean

7ca96ed

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Rename func name

290fb2e

Signed-off-by: JaySon-Huang <tshent@qq.com>

Address comment

786cc65

Signed-off-by: JaySon-Huang <tshent@qq.com>

Update kvproto

3c3ed11

Signed-off-by: JaySon-Huang <tshent@qq.com>

Clean expired snapshots

7c63d58

Signed-off-by: JaySon-Huang <tshent@qq.com>

Address comment

0d07c8e

Signed-off-by: JaySon-Huang <tshent@qq.com>

Fix bug for retry remote streams

807a4b3

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Make debug more easy

daa3512

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Address comment

35f87a8

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Add comment

c886f7d

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

JaySon-Huang force-pushed the disagg_read_snap branch from d41b5f3 to c886f7d Compare March 6, 2023 06:43

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 6, 2023

Merge branch 'master' into disagg_read_snap

88d4cb7

Merge branch 'master' into disagg_read_snap

392a0c3

ti-chi-bot merged commit 5604724 into pingcap:master Mar 6, 2023

JaySon-Huang deleted the disagg_read_snap branch March 9, 2023 09:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build table snapshot on write node for disaggregated read from S3 #6919

Build table snapshot on write node for disaggregated read from S3 #6919

JaySon-Huang commented Feb 28, 2023 •

edited

Loading

ti-chi-bot commented Feb 28, 2023 •

edited

Loading

JaySon-Huang commented Mar 2, 2023

JaySon-Huang commented Mar 2, 2023

breezewish left a comment

breezewish Mar 2, 2023

JaySon-Huang Mar 3, 2023

breezewish Mar 3, 2023

JaySon-Huang Mar 3, 2023

JaySon-Huang Mar 3, 2023

JaySon-Huang commented Mar 3, 2023

JaySon-Huang Mar 3, 2023

JaySon-Huang commented Mar 6, 2023

ti-chi-bot commented Mar 6, 2023

ti-chi-bot commented Mar 6, 2023

JaySon-Huang commented Mar 6, 2023

SeaRise commented Mar 6, 2023

JaySon-Huang commented Mar 6, 2023

JaySon-Huang commented Mar 6, 2023

JaySon-Huang commented Mar 6, 2023

Build table snapshot on write node for disaggregated read from S3 #6919

Build table snapshot on write node for disaggregated read from S3 #6919

Conversation

JaySon-Huang commented Feb 28, 2023 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Feb 28, 2023 • edited Loading

JaySon-Huang commented Mar 2, 2023

JaySon-Huang commented Mar 2, 2023

breezewish left a comment

Choose a reason for hiding this comment

breezewish Mar 2, 2023

Choose a reason for hiding this comment

JaySon-Huang Mar 3, 2023

Choose a reason for hiding this comment

breezewish Mar 3, 2023

Choose a reason for hiding this comment

JaySon-Huang Mar 3, 2023

Choose a reason for hiding this comment

JaySon-Huang Mar 3, 2023

Choose a reason for hiding this comment

JaySon-Huang commented Mar 3, 2023

JaySon-Huang Mar 3, 2023

Choose a reason for hiding this comment

JaySon-Huang commented Mar 6, 2023

ti-chi-bot commented Mar 6, 2023

ti-chi-bot commented Mar 6, 2023

JaySon-Huang commented Mar 6, 2023

SeaRise commented Mar 6, 2023

JaySon-Huang commented Mar 6, 2023

JaySon-Huang commented Mar 6, 2023

JaySon-Huang commented Mar 6, 2023

JaySon-Huang commented Feb 28, 2023 •

edited

Loading

ti-chi-bot commented Feb 28, 2023 •

edited

Loading