Revert "DNS: Fix lack of nameserver failover in low use circumstances…

[bneradt]

…. (#7843)"

This reverts commit 2fbec37.

I'm reverting this to test whether this commit introduced CI regression
test instability.

[SolidWallOfCode]

That patch fixed a Yahoo! production bug (where an ATS is sitting directly in front of a Jetty application on the same box, and only ever resolves for the FQDN of that box). Flaky DNS would eventually disable all of the nameservers, requiring a restart. The interesting question is, why would changes in HostDB have this effect?

[ywkaras]

Can we make a fake PR that turns on certain debug tags in the regression test runs in CI? Would that perhaps provide better info?

[ywkaras]

In failed regression tests in #8662 like https://ci.trafficserver.apache.org/job/Github_Builds/job/centos/508/console we see:

REGRESSION TEST SDK_API_TSHttpConnectServerIntercept started
[Feb 10 15:53:20.173] [ET_NET 0] WARNING: connection to DNS server 10.255.250.30 lost, marking as down
[Feb 10 15:53:20.173] [ET_NET 0] WARNING: connection to DNS server 10.255.251.30 lost, marking as down
[Feb 10 15:53:20.173] [ET_NET 0] WARNING: connection to all DNS servers lost, retrying
TEST 158 *** STARTING ***
*** TEST 158 *** PASSED ***
...
Regression test(SDK_API_TSHttpConnectServerIntercept) still in progress
Regression test(SDK_API_TSHttpConnectServerIntercept) still in progress
[Feb 10 15:53:50.174] [ET_NET 2] WARNING: Synserver failed to bind to port 65535.
Fatal: traffic_server/InkAPITest.cc:910: failed assertion `!"Synserver must be able to bind to a port, check system netstat"`

In successful tests, we'll eventually see:

[Feb 10 15:52:11.589] [ET_NET 0] WARNING: connection to DNS server 10.255.250.30 restored

What's weird is that I don't think the ConnectServerIntercept test should really need any DNS lookup, it's testing the APIs for intercept plugins.

[bneradt]

This is interesting debug output from your https://ci.trafficserver.apache.org/job/Github_Builds/job/centos/508/console run @ywkaras :

[Feb 10 15:53:50.174] [ET_NET 2] WARNING: Synserver failed to bind to port 65535.
Fatal: traffic_server/InkAPITest.cc:910: failed assertion !"Synserver must be able to bind to a port, check system netstat"

The port it tried to bind to, for some reason, is the 16 bit max value (SYNSERVER_DUMMY_PORT). I don't get the impression that it should actually try to listen on that port? Not sure. Still trying to figure out the test.

[bneradt]

What confuses me is that, given this message:

[Feb 10 15:53:50.174] [ET_NET 2] WARNING: Synserver failed to bind to port 65535.

That failure is associated with the #define SYNSERVER_DUMMY_PORT -1 value. However, that occurs in the synserver_vc_refuse which is only associated with a continuation in a single place:

trafficserver/src/traffic_server/InkAPITest.cc

Line 7335 in 92d238a

ptest->os = synserver_create(SYNSERVER_LISTEN_PORT, TSContCreate(synserver_vc_refuse, TSMutexCreate()));

How did that function get called back with the 65535 (-1 for 16 bits) rather than the #define SYNSERVER_LISTEN_PORT 3300 value?

[bneradt]

What confuses me is that, given this message:

[Feb 10 15:53:50.174] [ET_NET 2] WARNING: Synserver failed to bind to port 65535.

That failure is associated with the #define SYNSERVER_DUMMY_PORT -1 value. However, that occurs in the synserver_vc_refuse which is only associated with a continuation in a single place:

trafficserver/src/traffic_server/InkAPITest.cc

Line 7335 in 92d238a

ptest->os = synserver_create(SYNSERVER_LISTEN_PORT, TSContCreate(synserver_vc_refuse, TSMutexCreate()));

How did that function get called back with the 65535 (-1 for 16 bits) rather than the #define SYNSERVER_LISTEN_PORT 3300 value?

Oh, I see. There are two messages with that exact same Synserver failed to bind to port warning, one in synserver_vc_refuse and another in synserver_vc_accept. I was looking at the wrong message location.

[ywkaras]

With this reverted, there are no longer any warnings during regression tests about the DNS server being down: https://ci.trafficserver.apache.org/job/Github_Builds/job/centos/523/console .

[zwoop]

Cherry-picked to v9.2.x