Otel Collector Being Offline for a While Causes Exporting to Stop Working

[dnimon]

I'm running the otel collector on a server via docker, with the ports forwarded to the host. A while back it stopped working, and it did not have the proper restart logic, so it did not recover on its own.

I noticed once I restarted it, the python processes we have connected to that otel collector did not recover on there own. It's been up and running for a few days, but I'm still seeing this error:

Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4317, retrying in 1s.
Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4317, retrying in 2s.
Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4317, retrying in 4s.
Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4317, retrying in 8s.
Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4317, retrying in 16s.
Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4317, retrying in 32s.

If I restart one of these processes, then it starts exporting to the otel collector and it works fine, but for some reason the existing processes that were running while the otel collector was going through those issues do not recover on their own.

I can restart the processes to fix this particular issue, but would like to understand better why the python SDK did not recover on its own, and if there's something different I should be doing to improve that behavior.