[Enhancement] Improve Namesrv health check: auto-close unhealthy channel on repeated failures

[Kris20030907]

Before Creating the Enhancement Request

• I have confirmed that this should be classified as an enhancement rather than a bug/feature.

Summary

Solution: Add a configurable switch to the Client, add health detection for namesrv, and record the number of consecutive failures. When the number reaches the set value, actively call nettyRemotingClient to disconnect the channel of the currently selected namesrv to avoid the situation where the channel is hung for a long time and refresh the connection. At the same time, it can also ensure that in the event of an abnormality in the network of a certain namesrv, re-initiate the connection to other available namesrv in the namesrv list to work normally.

Motivation

Problem scenario: After the Client (Producer, Consumer) establishes a connection with Namesrv, it will regularly pull TopicRouteInfo from namesrv. However, once the machine where namesrv is located is disconnected from the Internet (TCP level), or the network fluctuates for a long time, or the firewall jitters, the application layer channel cannot be detected as abnormal and will not be disconnected. All requests based on this connection will not be able to obtain data normally due to timeout.

Describe the Solution You'd Like

1. Add a client-level switch (such as enableNamesrvCheck) to turn on or off the Namesrv health check function.
2. After calling updateTopicRouteInfoFromNameServer() in the scheduled task, add logic:
   • If the call is successful, reset the continuous failure count (namesrvHealthCheckFailCount) to 0.
   • If the call throws an exception, increase the failure count; when the failure count reaches the threshold set by clientConfig.getMaxClientNamesrvCheckFailedCnt(),
     actively call NettyRemotingClient.closeUnHealthyNamesrvChannel() to disconnect the currently selected Namesrv channel and reset the failure count to 0.
3. This improvement ensures that only continuous failures will trigger the disconnection operation, and the failure count will be immediately reset to zero after Namesrv returns to normal, ensuring that the disconnection logic will not be triggered by mistake.
4. In addition, when a Namesrv exception is detected, the mechanism can automatically refresh the routing information and try to connect to other available nodes in the Namesrv list, thereby improving the availability of the overall system.

Describe Alternatives You've Considered

1. Rely only on the existing IdleStateHandler and TCP layer SO_KEEPALIVE: Since the default OS keepalive time interval is too long, and IdleStateHandler cannot directly detect TCP disconnection, it is difficult to meet real-time requirements by relying on these mechanisms alone.
2. Force the use of short-timeout RPC calls to detect anomalies: Although this method can detect anomalies to a certain extent, it may increase the misjudgment rate under normal network fluctuations and affect the success rate of business requests.
3. Use TCP_USER_TIMEOUT: Although this parameter can detect network disconnection faster, it is not available or inconvenient to modify in the current environment, and may introduce cross-platform compatibility issues.
   Therefore, adding a health detection switch and recording the number of consecutive failures, and then actively disconnecting unhealthy Namesrv connections, becomes a more reliable and flexible solution.

Additional Context

This improvement has been verified through local testing. During Namesrv abnormalities, it can accurately record the number of consecutive failures and actively disconnect unhealthy channels after reaching the set threshold. When Namesrv recovers, the scheduled task can reset the failure count normally and continue to update the routing information.

[gaoyf]

However, once the machine where namesrv is located is disconnected from the Internet (TCP level), or the network fluctuates for a long time, or the firewall jitters, the application layer channel cannot be detected as abnormal and will not be disconnected.
I don't think so! You can refrence to the code: NettyRemotingClient.java:

If there is an abnormality in the communication between the client and NS, the client will actively close the link. By default, the next NS will be selected to establish a connection.
So，I don't think need NameServer health check.

[Kris20030907]

However, once the machine where namesrv is located is disconnected from the Internet (TCP level), or the network fluctuates for a long time, or the firewall jitters, the application layer channel cannot be detected as abnormal and will not be disconnected.
I don't think so! You can refrence to the code: NettyRemotingClient.java:

If there is an abnormality in the communication between the client and NS, the client will actively close the link. By default, the next NS will be selected to establish a connection. So，I don't think need NameServer health check.

Makes sense! Maybe I'm overthinking it. I'll verify again later whether turning on the isClientCloseSocketIfTimeout switch can solve this problem when the network is abnormal.

[Kris20030907]

@gaoyf Hey, I found the problem. When calling invokeSync, the addr passed in is null, just like the method I modified now:

The remotingClinet method will be called in the end:

According to the existing logic, you can see that Channel channel = this.getAndCreateChannel(addr); will eventually get the selected namesrv address and further get the channel:

Although the RemotingTimeOut exception can be captured at this time, addr is null and the corresponding channel cannot be closed:

In other words, the connection established with the unhealthy namesrv will not be closed, resulting in a hang.

[gaoyf]

In other words, the connection established with the unhealthy namesrv will not be closed, resulting in a hang.

Even if addr is null, channel can be closed also, FYI closeChannel:

[Kris20030907]

You are right, I understand why I encountered this problem, because I am using version 4.8.0, and the clientCloseSocketIfTimeout configuration item of this version is false by default, which makes it impossible to solve.

I will further confirm whether it works after changing it to true.

[Kris20030907]

@gaoyf I have verified this by disconnecting the cloud server from the Internet. In version 4.8.0, I changed the default configuration of clientCloseSocketIfTimeout to true, recompiled, packaged, and ran it. I found that the connection to namesrv can be closed normally when the network is disconnected (disabling the network card or disconnecting the network through nmcli).