Healthcheck for zero ingress connection count

[yacovm]

Why this should be merged

This PR adds a new health check that monitors if the node is a primary network validator and has no node connecting to it.

Our current health checks only monitor whether the node is connected to enough stake. However, a node may be connected to enough stake by opening egress connections to all other validators, but also at the same time, not be reachable from any validator or non-validator node.

This health check does just that - it alerts when:

1. The node is a primary network validator.
2. The node has zero ingress connections to it.
3. More than 10 minutes (configurable) passed since its startup.

How this works

A node can establish a connection through two pathways:

1. It initiates a connection with a remote node
2. A remote node initiates connection to it.

We count the number of connections of the latter type via an atomic counter, and add a health check that checks the three conditions mentioned earlier.

How this was tested

Added unit tests and also did a manual test:

I launched a validator node on Fuji (testnet) and I added it to the validators:

03-03|17:34:09.380] INFO <P Chain> validators/logger.go:50 node added to validator set {"subnetID": "11111111111111111111111111111111LpoYY", "nodeID": "NodeID-49DuEYJ8Kub8pdHz6yucyVqKBhEw5urMZ", "publicKey": "0x80f073c922fbfe9fe8ee22a5c4d47f8b1c30d6c95574d7f5fb9d7d673f5170141751d51b2c9e6ca0a1ca2f2acde2477b", "txID": "2XuGeB5LeKcrR5N2yjPvWMJj19NVE7kTiLk6gzU437sL9qjDZN", "weight": 2991109255}

~/avalanchego$ curl -s  -H 'Content-Type: application/json' --data '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"health.health",
    "params": {
        "tags": ["11111111111111111111111111111111LpoYY", "29uVeLPJB1eQJkzRemU8g8wZDw5uJRqpab5U2mX9euieVwiEbL"]
    }
}' 'http://localhost:9650/ext/health' | jq '.result.checks.network.message["primary network validator health"]'
{
  "ingressConnectionCount": 1708,
  "primary network validator": true
}

I then caused all other nodes to disconnect from it by blocking port 9651:

sudo iptables -I INPUT -p tcp --dport 9651 -j DROP

After a short time, the health check started failing:

[03-03|19:08:12.853] INFO health/worker.go:261 check started passing {"name": "health", "name": "bootstrapped", "tags": ["application"]}
[03-03|19:08:12.853] INFO health/worker.go:261 check started passing {"name": "readiness", "name": "bootstrapped", "tags": ["application"]}
[03-03|19:10:42.853] WARN health/worker.go:252 check started failing {"name": "health", "name": "network", "tags": ["application"], "error": "network layer is unhealthy reason: primary network validator is unreachable"}

and probing the health check API showed it:

~/avalanchego$ curl -s  -H 'Content-Type: application/json' --data '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"health.health",
    "params": {
        "tags": ["11111111111111111111111111111111LpoYY", "29uVeLPJB1eQJkzRemU8g8wZDw5uJRqpab5U2mX9euieVwiEbL"]
    }
}' 'http://localhost:9650/ext/health' | jq '.result.checks.network'
{
  "message": {
    "connectedPeers": 227,
    "primary network validator health": {
      "ingressConnectionCount": 0,
      "primary network validator": true
    },
    "sendFailRate": 0,
    "timeSinceLastMsgReceived": "853.896432ms",
    "timeSinceLastMsgSent": "853.896432ms"
  },
  "error": "network layer is unhealthy reason: primary network validator is unreachable",
  "timestamp": "2025-03-03T19:11:12.853915018Z",
  "duration": 25832,
  "contiguousFailures": 2,
  "timeOfFirstFailure": "2025-03-03T19:10:42.853036554Z"
}

I then restored connectivity via deleting the iptables rule:

sudo iptables -D INPUT -p tcp --dport 9651 -j DROP

and observed that the health check recovered:

[03-03|19:12:12.853] INFO health/worker.go:261 check started passing {"name": "health", "name": "network", "tags": ["application"]}

~/avalanchego$ curl -s  -H 'Content-Type: application/json' --data '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"health.health",
    "params": {
        "tags": ["11111111111111111111111111111111LpoYY", "29uVeLPJB1eQJkzRemU8g8wZDw5uJRqpab5U2mX9euieVwiEbL"]
    }
}' 'http://localhost:9650/ext/health' | jq '.result.healthy'
true

Need to be documented in RELEASES.md?

Added a section about the health check added.

[tsachiherman]

Overall, looks great. I've added few comments.

[yacovm]

Overall, looks great. I've added few comments.

Thanks, addressed/responded to your comments.

[tsachiherman]

Thanks. Will re-review shortly.

…

On Tue, Feb 18, 2025, 2:31 PM yacovm ***@***.***> wrote: Overall, looks great. I've added few comments. Thanks, addressed/responded to your comments. — Reply to this email directly, view it on GitHub <#3719 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF2OOHY34J6BCLZ6OD4ZA532QODB7AVCNFSM6AAAAABXDA2QMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRWG42DSNJRG4> . You are receiving this because you commented.Message ID: ***@***.***> [image: yacovm]*yacovm* left a comment (ava-labs/avalanchego#3719) <#3719 (comment)> Overall, looks great. I've added few comments. Thanks, addressed/responded to your comments. — Reply to this email directly, view it on GitHub <#3719 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF2OOHY34J6BCLZ6OD4ZA532QODB7AVCNFSM6AAAAABXDA2QMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRWG42DSNJRG4> . You are receiving this because you commented.Message ID: ***@***.***>

[yacovm]

Parking this as a draft before I manually test it again after this code change

[yacovm]

@StephenButtolph the PR is ready again for re-review.