internal_logs: Capture Failed Payload For Sink Errors

[rlazimi-dev]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Current Vector Version

vector 0.20.0 (x86_64-unknown-linux-gnu)

Use-cases

• re-uploading failed payloads to another sink for further inspection
• re-uploading failed payloads to a dead letter queue

Attempted Solutions

ideas i've had:

• create an env var with the original payload when processing data, then access that env var when handling errors generated from internal_logs
• reference one event from another unrelated event

Proposal

In internal_log events triggered by a sink error: record the payload that caused the sink to fail.

Rationale for Proposal

At a high level, I believe the solution is to pair a sink's error response with the vector event payload that caused that error response. This yields a vector event that is actionable – an event that contains all the information needed to fix pipeline logic issues.

References

• observability: log more context during errors #7750 #(my proposed solution may also solve this problem)

[jszwedko]

Thanks @rlazimi-dev . We also have a related issue for dead-letter queues #1772 that I think could also resolve this by attaching the failure message to the event being routed to the dead letter queue.

[PavelGloba]

DLQ is more solid solution for this, but it's been 4 years since this feature was proposed. Maybe just logging the payload will be easier to implement as a sort of workaround and will solve a lot of issues, because currently we're blind in terms of figuring out which exact message caused the issue