Add retries to worker health check

[SantiagoPittella]

What should be done?

In the proxy, we should be able to define a number of retries for the health check before removing it from the list of available.

Also, health checks are being performed sequentially, we need to change the implementation to use parallel execution for this.

How should it be done?

• Add a parameter that can be configured from the CLI with the amount of retries.
• Implement the retry logic.
• Refactor health check calls to run in parallel.

When is this task done?

The task is done when a retry policy is in place and check calls are not sequential anymore.

Additional context

No response

[SantiagoPittella]

As part of this issue, we can move the Backend instantiation inside the Worker::new method as commented here

[SantiagoPittella]

As part of this issue, we can change a bit the strategy for selecting available workers in order to get a more even distribution of requests: #1017 (comment)

[bobbinth]

As part of this issue, we can change a bit the strategy for selecting available workers in order to get a more even distribution of requests: #1017 (comment)

In a couple of sentences, could you describe how we could change the strategy?

[SantiagoPittella]

Currently we are using last in first out for the workers, a change to first in first out should reduce this. We don't need to change the structure used to contain the workers, but how we get workers.

[bobbinth]

Closed by #1255.