adding tcp_user_timeout setting for aws components

[alextreichler]

In certain situations it is possible where a connection is ESTABLISHED, but for some reason, such as a network disconnect, the data is still sending to an aws service such as sqs or dynamodb, but those services are not responding back. Typically in this case, it will continue to retry sending the data up to the number set by the kernel setting for tcp_retries2, typically 15, which means it takes ~ 15-16mins before the connection is terminated.

This can cause the input, such as consumers to stop consuming from a topic/partition from Redpanda until that connection is killed and redpanda connect re-establishes a new connection.

Instead of modifying a systemwide kernel setting, it is possible to pass a value on the application level using TCP_USER_TIMEOUT. This will take president over the tcp_retries2. This will make sure users don't have to modify a systemwide kernel setting and have more control specifically for redpanda connect in these cases.

[CLAassistant]

All committers have signed the CLA.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Jeffail]

Thanks @alextreichler, just a few comments, the docs look great. One thing we'd need to be careful of is the possibility of other fields in future also wanting to override the HTTP client. We could be proactive here and refactor this setting to be more of a general purpose HTTP customization function that other fields could contribute to, something like httpClientFromConfig(p *service.ParsedConfig) so it's more clear that all HTTP client fields go there.

[alextreichler]

Based on Ash's earlier mention, I create an http_client.go file that is used to create the custom http client, which is called within the session.go file.

[mmatczuk]

Could we have it not limited to AWS? I wonder could we perhaps have a tcp section in the main config and then tcp config object in components that support this for overwriting? This would allow to group and add more fields in the future instead of adding ad-hoc fields in multiple places.

We should have a dedicated library component that decorates / creates net.Dialer.

Possibly event this should be part of benthos?

[alextreichler]

Based on Michal M, comment - I ended up changing this PR to "draft" and I added a PR in benthos framwork to add a new package called "netclient" in order for other components to use that within their project. So, this is why I switched it to draft as the other one needs to be reviewed first.

redpanda-data/benthos#291

[mmatczuk]

Closing this as work moved to Benthos.