Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(eds/store): IPLD blockstore cleanup #2593

Open
distractedm1nd opened this issue Aug 21, 2023 · 2 comments
Open

fix(eds/store): IPLD blockstore cleanup #2593

distractedm1nd opened this issue Aug 21, 2023 · 2 comments
Assignees
Labels
area:ipld IPLD plugin area:shares Shares and samples kind:fix Attached to bug-fixing PRs

Comments

@distractedm1nd
Copy link
Collaborator

Individual IPLD blocks are stored in the edsstore's blockstore as of #2532. These blocks need to eventually be cleaned up. There are two approaches we briefly discussed:

  1. GC approach: Every X minutes/hours all blocks under the datastore prefix are purged. The non-trivial part here is that it may remove blocks that are currently being used for reconstruction.
  2. Post-reconstruction removal: Trigger removal of all of the IPLD blocks of an EDS upon successful retrieval. This can be done by adding a hook somewhere where the CIDs are known (such as inside retriever.go or after indexing). The non-trivial part here is not making it extremely ugly
@distractedm1nd distractedm1nd added area:shares Shares and samples area:ipld IPLD plugin kind:fix Attached to bug-fixing PRs labels Aug 21, 2023
@distractedm1nd distractedm1nd self-assigned this Aug 21, 2023
@ramin
Copy link
Contributor

ramin commented Feb 29, 2024

shwap

@ramin ramin closed this as completed Feb 29, 2024
@walldiss
Copy link
Member

Initially, shwap aimed to address this by implementing an enhanced retriever. However, following an internal discussion on shwap's scope, we decided against including retriever improvements. Therefore, this issue will need to be addressed separately from shwap.

@walldiss walldiss reopened this Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:ipld IPLD plugin area:shares Shares and samples kind:fix Attached to bug-fixing PRs
Projects
None yet
Development

No branches or pull requests

3 participants