DNS: Fix lack of nameserver failover in low use circumstances. #7843
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maskit
reviewed
May 17, 2021
Member
Author
|
Something is still not quite right. Further testing reveals some oddities I need to track down. At this point I think the problem is that in the case where the nameservers are not round robin, a down nameserver is never actually marked down. |
Member
Author
|
Also, timeouts don't seem to be working. The DNS layer never times out the request and the HostDB layer just keeps stacking up the queue for that host. AFAICT it's the generic inactivity timeout that terminates the connection. |
SolidWallOfCode
force-pushed
the
dns-failover-fix
branch
from
7364d1e to
d491f50
Compare
SolidWallOfCode
force-pushed
the
dns-failover-fix
branch
from
d491f50 to
9ad33fd
Compare
Member
Author
|
I think I've tracked down all the required changes. Fundamentally the issue was
As minor other cleanups
|
SolidWallOfCode
commented
Jun 1, 2021
| cur_con.num = icon; | ||
| Debug("dns", "opening connection %s SUCCEEDED for %d", ip_text, icon); | ||
| cur_con.num = icon; | ||
| ns_down[icon] = 0; |
Member
Author
There was a problem hiding this comment.
Note - don't clear the down flag if this nameserver connection isn't in the epoll.
Contributor
|
@randall is going to take a look at this. |
Contributor
|
@SolidWallOfCode is going to pull this into our internal tree on the 9.1.x branch. |
Member
Author
|
Looks some one else might be seeing this - Nir Finkel on the chat. He's currently going to do some tests to hopefully verify this is the problem and then maybe try out this fix. |
Contributor
|
[approve ci] |
1 similar comment
Contributor
|
[approve ci] |
SolidWallOfCode
force-pushed
the
dns-failover-fix
branch
from
9ad33fd to
cb76fa5
Compare
Member
|
[approve ci] |
bneradt
pushed a commit
to bneradt/trafficserver
that referenced
this pull request
Feb 10, 2022
apache#7843)" This reverts commit 2fbec37. I'm reverting this to test whether this commit introduced CI regression test instability.
bneradt
added a commit
that referenced
this pull request
Feb 11, 2022
zwoop
pushed a commit
that referenced
this pull request
Feb 17, 2022
(cherry picked from commit 2fbec37)
Contributor
|
Cherry-picked to v9.2.x |
Contributor
|
This was reverted in 92d238a |
bneradt
pushed a commit
to bneradt/trafficserver
that referenced
this pull request
Feb 19, 2022
…mstances. (apache#7843)" (apache#8663)" This reverts commit 92d238a.
moonchen
pushed a commit
to moonchen/trafficserver
that referenced
this pull request
Mar 17, 2022
* asf/9.2.x: Revert "DNS: Fix lack of nameserver failover in low use circumstances. (apache#7843)" (apache#8663) Fix strategies to initialize scheme (apache#8650) DNS: Fix lack of nameserver failover in low use circumstances. (apache#7843) Cleanup strategy debug logs (apache#8656) Making 9.2.x backwards compatible with 9.1.x (apache#8661) Adds two overridable config variables to control parent mark downs. (apache#8595) Fix plugin parent_select missing hostname len (apache#8649) Ports apache#7925 apache#8365 core to parent_select plugin (apache#8590) Ports apache#7897 from core strategies to parent_select plugin. (apache#8580) Adding clangd language server files to .gitignore (apache#8640) Make TsSharedMutex.h compile on MacOS. (apache#8645) In TsSharedMutex.h, make error reporting thread-safe. (apache#8636) Revert "body factory does not respect runroot (apache#8388)" (apache#8654) doc: Convert miscased Traffic Server references to |TS| macro (apache#8543) Add a new --enable-event-tracker configure option (apache#8179) Add parent_select plugin strategy caching (apache#8651) TLS Session Resumption: fix timed out session (apache#8667) Fix to allow running from outside top_srcdir (apache#8673) Send diags output to stderr when running regression tests. (apache#8662) Default proxy.config.http.strict_uri_parsing to "2" (apache#8632)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In situations where DNS is lightly used, or when there is only one FQDN requested, if a nameserver fails there is no failover to another nameserver.
The underlying cause is recovery depends on actions triggered from
DNSHandler::mainEvent. This means if a nameserver fails on a request, it takes another request for a different FQDN to trigger recovery. In a sidecar situation where only one FQDN is every requested, new requests for that FQDN stack up on the collapsing queue for the failed nameserver and ATS appears to become unresponsive. This fix makes sure that if there is a nameserver failure,DNSHandler::mainEventis invoked at least once after the failure. If that happens, recovery becomes self sustaining recovery actions cause subsequent invocations ofDNSHandler::mainEvent.This is probably a fix for #4514.