Add request auto-retry options

[petermasking]

When scaling an application onto multiple servers we want to be fail safe. Currently we have a health check system that is used for monitoring the availability of nodes. In case one of the nodes is not healthy or doesn't respond, the gateway will stop using it. But when the gateway already forwarded a request to such a node, the request will fail.

As an improvement we need to add extends the gateway with a retry strategy. When a node has become unavailable, the gateway should look for an alternative node for the requested procedure and try it again on that node before it fails.

[basmasking]

I read some pages on MDN and for adding retry options, we will need to add an abort controller to our fetch() implementations.

But we also need to handle rejected fetch requests, as these indicate connectivity issues.

The Promise returned from fetch() won't reject on HTTP error status even if the response is an HTTP 404 or 500. Instead, as soon as the server responds with headers, the Promise will resolve normally (with the ok property of the response set to false if the response isn't in the range 200–299), and it will only reject on network failure or if anything prevented the request from completing.

We already check if the response code is what we expect and throw an error otherwise. But we need to think about handling 4xx and 5xx responses. These are probably issues we still want to reject and don't want to retry ?