fix: treat webhook connection errors as delivery errors (retry) instead of system errors

[alexluong]

fixes #571

Fix: Treat webhook connection errors as delivery errors (retry) instead of system errors (DLQ)

Problem

Connection errors (DNS failures, connection refused, etc.) were being sent to DLQ instead of being retried. This happened because these errors were classified as "pre-delivery errors" rather than "delivery errors".

Solution

Network errors are now treated as delivery errors, which means they will be acknowledged and scheduled for retry instead of being nacked and sent to DLQ.

Error Classification

Code	Cause	Type	Behavior
dns_error	Domain doesn't exist	Delivery Error	ack + retry
connection_refused	Server not running	Delivery Error	ack + retry
connection_reset	Connection dropped	Delivery Error	ack + retry
network_unreachable	Network unavailable	Delivery Error	ack + retry
timeout	I/O timeout or deadline	Delivery Error	ack + retry
tls_error	Certificate/TLS failure	Delivery Error	ack + retry
redirect_error	Too many redirects	Delivery Error	ack + retry
network_error	Other network issues	Delivery Error	ack + retry
canceled	Context canceled (shutdown)	System Error	nack + requeue

The fix applies to both webhook and webhook_standard destination types.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Review	Updated (UTC)
outpost-docs	Ready	Preview, Comment	Dec 16, 2025 0:54am
outpost-website	Ready	Preview, Comment	Dec 16, 2025 0:54am

[alexbouchardd]

The fix applies to both webhook and webhook_standard destination types.

Can you clarify what the behavior is for say RabbitMQ if the queue is unavailable / has incorrect credentials? That should also retry. Anything that is outside of Outpost control really, should retry and not go to DLQ

[alexluong]

Can you clarify what the behavior is for say RabbitMQ if the queue is unavailable / has incorrect credentials?

I can look into RabbitMQ later. What do you think if we move forward with this PR and have RabbitMQ as a follow-up?

As a whole we need to be a bit more intentional about the error handling for every destination type. This PR focuses on webhook for now.

If you prefer we handle everything in this PR, happy to do so as well.

[alexbouchardd]

I'm just using RabbitMQ as an example of all other destination types here but no, I think that's fundamental design for the error handling where destination specific config shouldn't cause message to go to DQL and that's a generalized implementation rather then destination per destination

[alexluong]

Here's the delivery logic

deliverymq message handler:
  pre_delivery_err, delivery_err = deliver(destination, event)
  if pre_delivery_err -> nack
  if delivery_err -> schedule retry
  ack

It just happens that in webhook, the HTTP request is splitted into 2 steps. I am also just now realized that this PR basically will ignore cases where the system loses network, or if its DNS resolver has issue. That's why I said it's not that simple, at least from what I'm seeing.

For example, this is the step in RabbitMQ that would qualify as PreDelivery

This could be because auth issue. Or RabbitMQ destination is down. But what if it's because some system issue? How can we differentiate?

[alexbouchardd]

If we don't have a connection to rabbitMQ (or any destination for that matter) it's a delivery error, not "pre delivery". It's not "Outpost" fault. The only things that would outpost fault would be unexpected data in the message, errors fetch the destination config (Redis calls), parsing or application logic errors

[alexbouchardd]

Ultimately it's about who's responsible for the issue, the operator or the tenant

[alexluong]

I get that, I guess it doesn't quite click for me what's the simplest way to differentiate.

For example, when we see an error as "DNS resolution" or "connection reset" error or "generic network error", is that 100% of the time destination error?

[alexbouchardd]

Yes it is, or least we treat it that way since its transient

[alexluong]

okay noted, let me noodle on this and apply this idea to the other destinations.

For now, I'd like to move forward with v0.10.0 without this PR, then focus on the CH implementation, then I'll get back to this. Do let me know if you prefer I prioritize this instead.