cdc: clarify that sink-level retry configuration doesn't affect other retries

[HonoreDB]

Aaron Zinger (honoredb) commented:

The webhook_sink_config parameter supports a Retry.Max option, but this only affects low-level HTTP retries, not other sources of duplicate messages. Setting it to a low number will not prevent the changefeed from retrying indefinitely.

Jira Issue: DOC-8305

[theodore-hyman]

Dropping additional notes from TSE from the ticket to give this docs request a bit more detailed context:

Discussing further with engineering, the max retries is somewhat misleading. The changefeed will continue to attempt to retry a failed message repeatedly until it is delivered. After the max retries is initially met, the error will bubble up to the changefeed coordinator which will retry again. In essence, the changefeed is designed to not drop messages, so if an error in delivery occurs it will keep attempting to retry delivery until it succeeds.

In regards to even 200 messages being resent, this is most likely the result of the changefeed encountering a non-recoverable error and causing the changefeed to retry in its entirety. This would include re-sending messages that were previously delivered with a 200 response.

Zendesk ticket number is 17416 just for internal tracking purposes in case that is helpful