Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support more efficient way to persist manifest of TimeMergeStorage #1592

Open
baojinri opened this issue Nov 12, 2024 · 0 comments · May be fixed by #1596
Open

Support more efficient way to persist manifest of TimeMergeStorage #1592

baojinri opened this issue Nov 12, 2024 · 0 comments · May be fixed by #1596
Assignees

Comments

@baojinri
Copy link
Contributor

baojinri commented Nov 12, 2024

Describe This Problem

Now, when meta information of TimeMergeStorage changes, the efficiency of persisting manifest sst files is very low. Each time, the new sst file will be added to the manifest and then persisted it whole.

Proposal

Each time, the new sst file will be persisted instantly, then add the new sst into snapshot background.
The manifest file is organized as follows:

/root/manifest/snapshot
/root/manifest/new_sst_0
/root/manifest/new_sst_1
/root/manifest/new_sst_2
...

Additional Context

Process manifest files:

  • When initializing the manifest, we will list all the sst files in the manifest directory and then add them into the snapshot
  • Every time a new file is added, we will count the new files. When the number of new files reaches 50, we will list all the sst files in the manifest directory and then add them into the snapshot
@baojinri baojinri self-assigned this Nov 12, 2024
@baojinri baojinri linked a pull request Nov 18, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant