Estuary Cleaner

[alvin-reyes]

Request

There are a lot of stale records on Estuary database and we need to clean this up somehow.

Proposal

Create a script the cleans up estuary database with the following criteria:

1 - get all ContentID with Unwalkable CID
2 - get all CID that cannot be provided
3 - get all CID with content deadline exceeded
4 - get all blockstore: block not found CIDs
5 - create a failure_log table to log all failed CIDs. We can use this information to investigate each CID.

For each of the criteria above, log the records content on a file and delete.

The script should run on a periodic basis (weekly) to ensure that any stale data is purged to prevent any unnecessary re-processing.