Skip to content

[New Feature]: Users need an easy way to catalog data that exists in S3 but failed to catalog #539

Open
@ngachung

Description

@ngachung

When DS gets overwhelmed with too many catalog requests, catalog can fail due to ES timeouts, DB timeouts, lambda errors, which results in data that are in S3 but not cataloged and unavailable for search.

Fixing the root cause is something DS will work on i.e. setting up DLQ, retrying failed DB writes, processing a batch of messages with a single lambda instead of invoking hundreds of concurrent lambdas, etc. but providing users with an API endpoint to catalog data is still useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions