GC data on S3 #6899

JaySon-Huang · 2023-02-25T17:26:44Z

What problem does this PR solve?

Issue Number: ref #6827

Problem Summary: run GC on S3 data to reclaim the storage usage

What is changed and how it works?

Depends on

Components

CheckpointManifestS3Set - parse the manifests list info from S3
S3GCManager. For each store_id on S3
- Get the latest manifest from S3
- Parse the locks hold by the latest manifest (depend on Manifest uploading)
- Scan and remove the expired locks, try add delmark when the data file is not locked by any store (depend on S3LockService)
- Remove the data file with expired delmark

TODOs:

Remove the sub files for DTFile
Collect valid locks from multiple manifest
Write a mark to skip useless LIST for tombstone store files
Calculate the valid size of CheckpointDataFile and compact them
Use DataStoreS3 for downloading manifest file

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2023-02-25T17:26:46Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

flowbehappy
lidezhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

JaySon-Huang · 2023-02-28T17:07:17Z

/run-all-tests

dbms/src/Storages/Page/V3/Universal/UniversalWriteBatchImpl.h

dbms/src/Storages/S3/S3GCManager.cpp

dbms/src/Storages/S3/CheckpointManifestS3Set.cpp

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

lidezhu

LGTM with minor comments

dbms/src/Storages/S3/CheckpointManifestS3Set.cpp

dbms/src/Storages/S3/CheckpointManifestS3Set.h

lidezhu · 2023-03-03T09:25:11Z

dbms/src/Storages/S3/S3GCManager.h

+
+    void removeOutdatedManifest(const CheckpointManifestS3Set & manifests, const Aws::Utils::DateTime * const timepoint); // NOLINT(readability-avoid-const-params-in-decls)
+
+    String getTemporaryDownloadFile(String s3_key);


Suggested change

String getTemporaryDownloadFile(String s3_key);

String getTemporaryDownloadFile(const String & s3_key);

lidezhu · 2023-03-03T09:25:31Z

dbms/src/Storages/S3/S3GCManager.cpp

+    }
+}
+
+String S3GCManager::getTemporaryDownloadFile(String s3_key)


Suggested change

String S3GCManager::getTemporaryDownloadFile(String s3_key)

String S3GCManager::getTemporaryDownloadFile(const String & s3_key)

this can not be const & because I replace '/' with '_' in the s3_key for a download name

dbms/src/Storages/S3/S3GCManager.cpp

dbms/src/Storages/S3/S3GCManager.h

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

JaySon-Huang · 2023-03-03T11:07:05Z

/merge

ti-chi-bot · 2023-03-03T11:07:06Z

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-03-03T11:07:08Z

This pull request has been accepted and is ready to merge.

Commit hash: dfaaee5

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

JaySon-Huang · 2023-03-03T13:01:47Z

/merge

ti-chi-bot · 2023-03-03T13:01:48Z

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-03-03T13:01:50Z

This pull request has been accepted and is ready to merge.

Commit hash: 6c7a875

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

JaySon-Huang · 2023-03-03T13:12:41Z

/merge

ti-chi-bot · 2023-03-03T13:12:42Z

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-03-03T13:12:44Z

This pull request has been accepted and is ready to merge.

Commit hash: fbd3dc3

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 25, 2023

JaySon-Huang force-pushed the s3gc branch 2 times, most recently from c5d057c to 7dbf01a Compare February 25, 2023 17:53

ti-chi-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Feb 25, 2023

JaySon-Huang force-pushed the s3gc branch from 77bd0cf to 5499315 Compare February 27, 2023 07:39

This was referenced Feb 27, 2023

The task in BackgroundProcessingPool is not run by fixed interval #6908

Closed

pd_client: Add getAllStore API tikv/client-c#126

Merged

JaySon-Huang force-pushed the s3gc branch from 46f5ae4 to 337bf4f Compare February 28, 2023 15:56

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 28, 2023

flowbehappy reviewed Mar 1, 2023

View reviewed changes

dbms/src/Storages/Page/V3/Universal/UniversalWriteBatchImpl.h Outdated Show resolved Hide resolved

flowbehappy reviewed Mar 1, 2023

View reviewed changes

dbms/src/Storages/S3/S3GCManager.cpp Outdated Show resolved Hide resolved

flowbehappy reviewed Mar 1, 2023

View reviewed changes

dbms/src/Storages/S3/CheckpointManifestS3Set.cpp Outdated Show resolved Hide resolved

JaySon-Huang added 13 commits March 2, 2023 17:48

add new type

5a92bbb

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

v3

da8ba34

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Add S3GCManager

44c2362

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Clean outdated manifest

b077dbb

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

expired with same timepoint

30d648e

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

One gc config

c8526e1

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Add ut test

36009fd

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Register s3gc manager to TMTContext

9ec8e98

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

init with mock lock client

4de4a03

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Add todo for remove dmfile objects

bb5ae7c

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Clean files of tombstoned store

b197f02

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Add shutdown

9bf0e1c

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Check with owner manager

3858440

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 3, 2023

lidezhu approved these changes Mar 3, 2023

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 3, 2023

JaySon-Huang added 3 commits March 3, 2023 18:23

Fix ut

6a9eee2

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Merge remote-tracking branch 'upstream/master' into s3gc

d23c21c

Address comment

dfaaee5

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 3, 2023

ti-chi-bot and others added 3 commits March 3, 2023 19:07

Merge branch 'master' into s3gc

9fbd561

Merge branch 'master' into s3gc

9856036

Fix bug

9a3f415

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Mar 3, 2023

Add ut

6c7a875

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 3, 2023

ti-chi-bot and others added 2 commits March 3, 2023 21:01

Merge branch 'master' into s3gc

df165f2

Format files

fbd3dc3

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Mar 3, 2023

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 3, 2023

Merge branch 'master' into s3gc

60166fe

ti-chi-bot merged commit 20c6dda into pingcap:master Mar 3, 2023

JaySon-Huang deleted the s3gc branch March 3, 2023 14:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GC data on S3 #6899

GC data on S3 #6899

JaySon-Huang commented Feb 25, 2023 •

edited

Loading

ti-chi-bot commented Feb 25, 2023 •

edited

Loading

JaySon-Huang commented Feb 28, 2023

lidezhu left a comment

lidezhu Mar 3, 2023

lidezhu Mar 3, 2023

JaySon-Huang Mar 3, 2023

JaySon-Huang commented Mar 3, 2023

ti-chi-bot commented Mar 3, 2023

ti-chi-bot commented Mar 3, 2023

JaySon-Huang commented Mar 3, 2023

ti-chi-bot commented Mar 3, 2023

ti-chi-bot commented Mar 3, 2023

JaySon-Huang commented Mar 3, 2023

ti-chi-bot commented Mar 3, 2023

ti-chi-bot commented Mar 3, 2023


		void removeOutdatedManifest(const CheckpointManifestS3Set & manifests, const Aws::Utils::DateTime * const timepoint); // NOLINT(readability-avoid-const-params-in-decls)

		String getTemporaryDownloadFile(String s3_key);

	String getTemporaryDownloadFile(String s3_key);
	String getTemporaryDownloadFile(const String & s3_key);

	String S3GCManager::getTemporaryDownloadFile(String s3_key)
	String S3GCManager::getTemporaryDownloadFile(const String & s3_key)

GC data on S3 #6899

GC data on S3 #6899

Conversation

JaySon-Huang commented Feb 25, 2023 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Feb 25, 2023 • edited Loading

JaySon-Huang commented Feb 28, 2023

lidezhu left a comment

Choose a reason for hiding this comment

lidezhu Mar 3, 2023

Choose a reason for hiding this comment

lidezhu Mar 3, 2023

Choose a reason for hiding this comment

JaySon-Huang Mar 3, 2023

Choose a reason for hiding this comment

JaySon-Huang commented Mar 3, 2023

ti-chi-bot commented Mar 3, 2023

ti-chi-bot commented Mar 3, 2023

JaySon-Huang commented Mar 3, 2023

ti-chi-bot commented Mar 3, 2023

ti-chi-bot commented Mar 3, 2023

JaySon-Huang commented Mar 3, 2023

ti-chi-bot commented Mar 3, 2023

ti-chi-bot commented Mar 3, 2023

JaySon-Huang commented Feb 25, 2023 •

edited

Loading

ti-chi-bot commented Feb 25, 2023 •

edited

Loading