Open
Description
When DS gets overwhelmed with too many catalog requests, catalog can fail due to ES timeouts, DB timeouts, lambda errors, which results in data that are in S3 but not cataloged and unavailable for search.
Fixing the root cause is something DS will work on i.e. setting up DLQ, retrying failed DB writes, processing a batch of messages with a single lambda instead of invoking hundreds of concurrent lambdas, etc. but providing users with an API endpoint to catalog data is still useful.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
No status