[Feature Request] Error handling for pull-based ingestion

[varunbharadwaj]

Is your feature request related to a problem? Please describe

Handle failure scenarios in pull-based ingestion. Failures during pull-based ingestion could be due to multiple reasons, like a malformed message or a poison pill. Handle failure scenarios with sufficient observability into the errors.

Describe the solution you'd like

Create ingestion error handling strategy interface and support following two error handling modes to begin with:

1. Drop: This will be used if failures can be ignored and it is possible to continue processing remaining updates in the ingestion source.
2. Block: This will be used if failures cannot be tolerated and ingestion must be halted till the issue is fixed.

The error handler strategies will also provide flexibility to log errors, record bad messages and emit metrics. In future, it can be extended to write to a DLQ when required.

Related component

Indexing

Describe alternatives you've considered

No response

Additional context

No response

[andrross]

Catch All Triage - 1, 2, 3, 4