Monitoring instance logs not showing any connectivity errors if instance cannot connect to broker

[ramonsmits]

Describe the bug

Description

Monitoring instance logs not showing any connectivity errors if instance cannot connect to broker

For example, by default ASBS connects using AMQP ports but some environments have these blocked.

Expected behavior

Monitoring instance to fail to start when broker is not available or at mimimum report ERROR or FATAL log events indicating broker is not available

Actual behavior

Monitoring instance remains running without indicating failure but in the Azure Portal it shows that the instance is not ingesting messages from the monitoring queue.

Versions

6.6.3 with ASB (other transports might also be affected)

Steps to reproduce

1. Setup monitoring instance with ASB
2. Block ports:

• sudo iptables -A OUTPUT -p tcp -d 57.153.239.59 --dport 5671 -j REJECT
  sudo iptables -A OUTPUT -p tcp -d 57.153.239.59 --dport 5672 -j REJECT
  

3. Launch instance

Observe it is running without reporting failures.

Relevant log output

Additional Information

No response

[danielmarbach]

We've reviewed this bug and added it to our backlog. It is not currently prioritized, but it may be fixed along with other future work in the repository. We will update this issue when work begins.