NodeLocal DNS container hung on SIGTERM

[rtheis]

We are still hitting the same problem reported by #394. The test failure occurred on Kubernetes version 1.21 with NodeLocal DNS cache version 1.17.3.

To recap, NodeLocal DNS container occasionally hangs on termination causing Kubernetes to kill the container using SIGTERM after the grace period has expired. This leaves left over iptables rules on the node thus breaking DNS resolution. Our theory is that there is iptables lock contention between NodeLocal DNS, Calico and/or Kubernetes.

[prameshj]

Doesn't nodelocaldns startup right away though? The DNS downtime should be O(seconds). Is that what you observe?

It is possible for nodelocaldns to run into lock contention, but that usually has a log message. Anything in the logs?

[rtheis]

@prameshj I was unable to collect any valuable logs at the time of the latest failure. I assume there is some type of lock contention that causes the pod to hang on termination.

NodeLocal DNS does startup right away, but our test failure comes when we verify that DNS works after disabling NodeLocal DNS. If we restart NodeLocal DNS then stop it again, that usually fixes the node.

[prameshj]

NodeLocal DNS does startup right away, but our test failure comes when we verify that DNS works after disabling NodeLocal DNS.

Ah I see. Just to confirm - 1) test disables nodelocaldns 2) nodelocaldns pod is stuck handling sigterm and is killed by kubelet with iptables rules getting left over. 3) test times out with DNS failure?

Do you see a log line of the sigterm being handled -

dns/vendor/github.com/coredns/caddy/sigtrap_posix.go

Line 42 in 3b17e06

log.Println("[INFO] SIGTERM: Shutting down servers then terminating")

?

It should call teardown in that case -

dns/cmd/node-cache/main.go

Line 53 in 3b17e06

caddy.OnProcessExit = append(caddy.OnProcessExit, func() { cache.TeardownNetworking() })

It is possible that the pod handles sigterm and tries cleaning up iptables, but cannot get the lock. We do expose a metric on port 9353 for nodelocaldns lock errors, but we do not increment it for delete errors. We should check errors in

dns/cmd/node-cache/app/cache_app.go

Line 157 in 3b17e06

c.iptables.DeleteRule(rule.table, rule.chain, rule.args...)

and update the metric.

[rtheis]

@prameshj That is correct. We've updated the termination grace period to 900 seconds and still see this problem. Although, the failure rate has been much lower than it has been in the past. Given the long termination, it seems like there is a hang somewhere. We'll continue trying to collect more data when this problem occurs.

[rtheis]

We hit the problem again on Kubernetes version 1.20 with NodeLocal DNS version 1.17.3. Unfortunately, I don't have any additional debug data to provide.

[rtheis]

We hit this problem again on Kubernetes version 1.22 with NodeLocal DNS version 1.21.1. We are collecting debug data now to determine if we can find the root cause.

[prameshj]

Thanks. I have also opened #488 to count errors from rule deletions at teardown, in case that provides some hints.

[rtheis]

We hit the problem again Kubernetes version 1.22 with NodeLocal DNS version 1.21.1. Here is the end of the log captured during pod termination:

[INFO] SIGTERM: Shutting down servers then terminating

[prameshj]

We hit the problem again Kubernetes version 1.22 with NodeLocal DNS version 1.21.1. Here is the end of the log captured during pod termination:

[INFO] SIGTERM: Shutting down servers then terminating

Any metrics from node-cache?

[rtheis]

@prameshj unfortunately, I don't have any metrics captured when the failure occurred. What would you like us to collect?

[prameshj]

#488

The "setup_errors_total" metric which was modified in #488 to also increment during deletions. That PR is included in 1.21.2 and later tags.

[rtheis]

Thanks, we'll update our test to collect metrics once we pull in the NodeLocal DNS cache latest version.

[rtheis]

We were able to recreate the problem on NodeLocal DNS version 1.21.3. Here are the logs and metrics.

[INFO] SIGTERM: Shutting down servers then terminating

# HELP coredns_build_info A metric with a constant '1' value labeled by version, revision, and goversion from which CoreDNS was built.
# TYPE coredns_build_info gauge
coredns_build_info{goversion="go1.16.10",revision="",version="1.7.0"} 1
# HELP coredns_cache_entries The number of elements in the cache.
# TYPE coredns_cache_entries gauge
coredns_cache_entries{server="dns://169.254.20.10:53",type="denial"} 2
coredns_cache_entries{server="dns://169.254.20.10:53",type="success"} 0
coredns_cache_entries{server="dns://172.21.0.10:53",type="denial"} 8
coredns_cache_entries{server="dns://172.21.0.10:53",type="success"} 1
# HELP coredns_cache_hits_total The count of cache hits.
# TYPE coredns_cache_hits_total counter
coredns_cache_hits_total{server="dns://172.21.0.10:53",type="denial"} 882
coredns_cache_hits_total{server="dns://172.21.0.10:53",type="success"} 131
# HELP coredns_cache_misses_total The count of cache misses.
# TYPE coredns_cache_misses_total counter
coredns_cache_misses_total{server="dns://169.254.20.10:53"} 8
coredns_cache_misses_total{server="dns://172.21.0.10:53"} 51
# HELP coredns_dns_request_duration_seconds Histogram of the time (in seconds) each request took.
# TYPE coredns_dns_request_duration_seconds histogram
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="0.00025"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="0.0005"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="0.001"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="0.002"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="0.004"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="0.008"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="0.016"} 1
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="0.032"} 1
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="0.064"} 1
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="0.128"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="0.256"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="0.512"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="1.024"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="2.048"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="4.096"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="8.192"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone=".",le="+Inf"} 2
coredns_dns_request_duration_seconds_sum{server="dns://169.254.20.10:53",type="other",zone="."} 0.12080307600000001
coredns_dns_request_duration_seconds_count{server="dns://169.254.20.10:53",type="other",zone="."} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="0.00025"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="0.0005"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="0.001"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="0.002"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="0.004"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="0.008"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="0.016"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="0.032"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="0.064"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="0.128"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="0.256"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="0.512"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="1.024"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="2.048"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="4.096"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="8.192"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="cluster.local.",le="+Inf"} 2
coredns_dns_request_duration_seconds_sum{server="dns://169.254.20.10:53",type="other",zone="cluster.local."} 0.002413771
coredns_dns_request_duration_seconds_count{server="dns://169.254.20.10:53",type="other",zone="cluster.local."} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="0.00025"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="0.0005"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="0.001"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="0.002"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="0.004"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="0.008"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="0.016"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="0.032"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="0.064"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="0.128"} 1
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="0.256"} 1
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="0.512"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="1.024"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="2.048"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="4.096"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="8.192"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa.",le="+Inf"} 2
coredns_dns_request_duration_seconds_sum{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa."} 0.391487433
coredns_dns_request_duration_seconds_count{server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa."} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="0.00025"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="0.0005"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="0.001"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="0.002"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="0.004"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="0.008"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="0.016"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="0.032"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="0.064"} 0
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="0.128"} 1
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="0.256"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="0.512"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="1.024"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="2.048"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="4.096"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="8.192"} 2
coredns_dns_request_duration_seconds_bucket{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa.",le="+Inf"} 2
coredns_dns_request_duration_seconds_sum{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa."} 0.256938208
coredns_dns_request_duration_seconds_count{server="dns://169.254.20.10:53",type="other",zone="ip6.arpa."} 2
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="0.00025"} 508
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="0.0005"} 518
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="0.001"} 530
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="0.002"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="0.004"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="0.008"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="0.016"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="0.032"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="0.064"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="0.128"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="0.256"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="0.512"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="1.024"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="2.048"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="4.096"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="8.192"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="A",zone="cluster.local.",le="+Inf"} 532
coredns_dns_request_duration_seconds_sum{server="dns://172.21.0.10:53",type="A",zone="cluster.local."} 0.047663295000000064
coredns_dns_request_duration_seconds_count{server="dns://172.21.0.10:53",type="A",zone="cluster.local."} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="0.00025"} 500
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="0.0005"} 519
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="0.001"} 531
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="0.002"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="0.004"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="0.008"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="0.016"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="0.032"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="0.064"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="0.128"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="0.256"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="0.512"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="1.024"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="2.048"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="4.096"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="8.192"} 532
coredns_dns_request_duration_seconds_bucket{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local.",le="+Inf"} 532
coredns_dns_request_duration_seconds_sum{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local."} 0.046383935999999994
coredns_dns_request_duration_seconds_count{server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local."} 532
# HELP coredns_dns_request_size_bytes Size of the EDNS0 UDP buffer in bytes (64K for TCP).
# TYPE coredns_dns_request_size_bytes histogram
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="0"} 0
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="100"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="200"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="300"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="400"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="511"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="1023"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="2047"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="4095"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="8291"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="16000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="32000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="48000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="64000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="+Inf"} 2
coredns_dns_request_size_bytes_sum{proto="udp",server="dns://169.254.20.10:53",zone="."} 114
coredns_dns_request_size_bytes_count{proto="udp",server="dns://169.254.20.10:53",zone="."} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="0"} 0
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="100"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="200"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="300"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="400"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="511"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="1023"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="2047"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="4095"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="8291"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="16000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="32000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="48000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="64000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="+Inf"} 2
coredns_dns_request_size_bytes_sum{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local."} 142
coredns_dns_request_size_bytes_count{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local."} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="0"} 0
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="100"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="200"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="300"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="400"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="511"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="1023"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="2047"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="4095"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="8291"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="16000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="32000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="48000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="64000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="+Inf"} 2
coredns_dns_request_size_bytes_sum{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa."} 140
coredns_dns_request_size_bytes_count{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa."} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="0"} 0
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="100"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="200"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="300"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="400"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="511"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="1023"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="2047"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="4095"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="8291"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="16000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="32000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="48000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="64000"} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="+Inf"} 2
coredns_dns_request_size_bytes_sum{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa."} 132
coredns_dns_request_size_bytes_count{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa."} 2
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="0"} 0
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="100"} 1064
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="200"} 1064
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="300"} 1064
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="400"} 1064
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="511"} 1064
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="1023"} 1064
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="2047"} 1064
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="4095"} 1064
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="8291"} 1064
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="16000"} 1064
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="32000"} 1064
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="48000"} 1064
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="64000"} 1064
coredns_dns_request_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="+Inf"} 1064
coredns_dns_request_size_bytes_sum{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local."} 73948
coredns_dns_request_size_bytes_count{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local."} 1064
# HELP coredns_dns_requests_total Counter of DNS requests made per zone, protocol and family.
# TYPE coredns_dns_requests_total counter
coredns_dns_requests_total{family="1",proto="udp",server="dns://169.254.20.10:53",type="other",zone="."} 2
coredns_dns_requests_total{family="1",proto="udp",server="dns://169.254.20.10:53",type="other",zone="cluster.local."} 2
coredns_dns_requests_total{family="1",proto="udp",server="dns://169.254.20.10:53",type="other",zone="in-addr.arpa."} 2
coredns_dns_requests_total{family="1",proto="udp",server="dns://169.254.20.10:53",type="other",zone="ip6.arpa."} 2
coredns_dns_requests_total{family="1",proto="udp",server="dns://172.21.0.10:53",type="A",zone="cluster.local."} 532
coredns_dns_requests_total{family="1",proto="udp",server="dns://172.21.0.10:53",type="AAAA",zone="cluster.local."} 532
# HELP coredns_dns_response_size_bytes Size of the returned response in bytes.
# TYPE coredns_dns_response_size_bytes histogram
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="0"} 0
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="100"} 0
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="200"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="300"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="400"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="511"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="1023"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="2047"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="4095"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="8291"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="16000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="32000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="48000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="64000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone=".",le="+Inf"} 2
coredns_dns_response_size_bytes_sum{proto="udp",server="dns://169.254.20.10:53",zone="."} 264
coredns_dns_response_size_bytes_count{proto="udp",server="dns://169.254.20.10:53",zone="."} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="0"} 0
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="100"} 0
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="200"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="300"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="400"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="511"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="1023"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="2047"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="4095"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="8291"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="16000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="32000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="48000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="64000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local.",le="+Inf"} 2
coredns_dns_response_size_bytes_sum{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local."} 328
coredns_dns_response_size_bytes_count{proto="udp",server="dns://169.254.20.10:53",zone="cluster.local."} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="0"} 0
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="100"} 0
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="200"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="300"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="400"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="511"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="1023"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="2047"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="4095"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="8291"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="16000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="32000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="48000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="64000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa.",le="+Inf"} 2
coredns_dns_response_size_bytes_sum{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa."} 308
coredns_dns_response_size_bytes_count{proto="udp",server="dns://169.254.20.10:53",zone="in-addr.arpa."} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="0"} 0
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="100"} 0
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="200"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="300"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="400"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="511"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="1023"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="2047"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="4095"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="8291"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="16000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="32000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="48000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="64000"} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa.",le="+Inf"} 2
coredns_dns_response_size_bytes_sum{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa."} 284
coredns_dns_response_size_bytes_count{proto="udp",server="dns://169.254.20.10:53",zone="ip6.arpa."} 2
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="0"} 0
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="100"} 0
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="200"} 1064
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="300"} 1064
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="400"} 1064
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="511"} 1064
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="1023"} 1064
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="2047"} 1064
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="4095"} 1064
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="8291"} 1064
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="16000"} 1064
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="32000"} 1064
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="48000"} 1064
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="64000"} 1064
coredns_dns_response_size_bytes_bucket{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local.",le="+Inf"} 1064
coredns_dns_response_size_bytes_sum{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local."} 167447
coredns_dns_response_size_bytes_count{proto="udp",server="dns://172.21.0.10:53",zone="cluster.local."} 1064
# HELP coredns_dns_responses_total Counter of response status codes.
# TYPE coredns_dns_responses_total counter
coredns_dns_responses_total{rcode="NOERROR",server="dns://172.21.0.10:53",zone="cluster.local."} 266
coredns_dns_responses_total{rcode="NXDOMAIN",server="dns://169.254.20.10:53",zone="."} 2
coredns_dns_responses_total{rcode="NXDOMAIN",server="dns://169.254.20.10:53",zone="cluster.local."} 2
coredns_dns_responses_total{rcode="NXDOMAIN",server="dns://169.254.20.10:53",zone="in-addr.arpa."} 2
coredns_dns_responses_total{rcode="NXDOMAIN",server="dns://169.254.20.10:53",zone="ip6.arpa."} 2
coredns_dns_responses_total{rcode="NXDOMAIN",server="dns://172.21.0.10:53",zone="cluster.local."} 798
# HELP coredns_forward_max_concurrent_rejects_total Counter of the number of queries rejected because the concurrent queries were at maximum.
# TYPE coredns_forward_max_concurrent_rejects_total counter
coredns_forward_max_concurrent_rejects_total 0
# HELP coredns_forward_request_duration_seconds Histogram of the time each request took.
# TYPE coredns_forward_request_duration_seconds histogram
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="0.00025"} 0
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="0.0005"} 0
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="0.001"} 0
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="0.002"} 0
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="0.004"} 0
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="0.008"} 0
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="0.016"} 0
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="0.032"} 0
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="0.064"} 0
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="0.128"} 1
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="0.256"} 1
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="0.512"} 1
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="1.024"} 1
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="2.048"} 1
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="4.096"} 1
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="8.192"} 1
coredns_forward_request_duration_seconds_bucket{to="10.0.80.11:53",le="+Inf"} 1
coredns_forward_request_duration_seconds_sum{to="10.0.80.11:53"} 0.112503827
coredns_forward_request_duration_seconds_count{to="10.0.80.11:53"} 1
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="0.00025"} 1
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="0.0005"} 34
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="0.001"} 49
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="0.002"} 53
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="0.004"} 53
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="0.008"} 53
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="0.016"} 53
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="0.032"} 53
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="0.064"} 53
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="0.128"} 55
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="0.256"} 56
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="0.512"} 57
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="1.024"} 57
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="2.048"} 57
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="4.096"} 57
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="8.192"} 57
coredns_forward_request_duration_seconds_bucket{to="172.21.156.115:53",le="+Inf"} 57
coredns_forward_request_duration_seconds_sum{to="172.21.156.115:53"} 0.6737926949999999
coredns_forward_request_duration_seconds_count{to="172.21.156.115:53"} 57
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="0.00025"} 0
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="0.0005"} 0
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="0.001"} 0
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="0.002"} 0
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="0.004"} 0
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="0.008"} 0
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="0.016"} 1
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="0.032"} 1
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="0.064"} 1
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="0.128"} 1
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="0.256"} 1
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="0.512"} 1
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="1.024"} 1
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="2.048"} 1
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="4.096"} 1
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="8.192"} 1
coredns_forward_request_duration_seconds_bucket{to="8.8.8.8:53",le="+Inf"} 1
coredns_forward_request_duration_seconds_sum{to="8.8.8.8:53"} 0.008086901
coredns_forward_request_duration_seconds_count{to="8.8.8.8:53"} 1
# HELP coredns_forward_requests_total Counter of requests made per upstream.
# TYPE coredns_forward_requests_total counter
coredns_forward_requests_total{to="10.0.80.11:53"} 1
coredns_forward_requests_total{to="172.21.156.115:53"} 57
coredns_forward_requests_total{to="8.8.8.8:53"} 1
# HELP coredns_forward_responses_total Counter of requests made per upstream.
# TYPE coredns_forward_responses_total counter
coredns_forward_responses_total{rcode="NOERROR",to="172.21.156.115:53"} 9
coredns_forward_responses_total{rcode="NXDOMAIN",to="10.0.80.11:53"} 1
coredns_forward_responses_total{rcode="NXDOMAIN",to="172.21.156.115:53"} 48
coredns_forward_responses_total{rcode="NXDOMAIN",to="8.8.8.8:53"} 1
# HELP coredns_health_request_duration_seconds Histogram of the time (in seconds) each request took.
# TYPE coredns_health_request_duration_seconds histogram
coredns_health_request_duration_seconds_bucket{le="0.00025"} 0
coredns_health_request_duration_seconds_bucket{le="0.0005"} 20
coredns_health_request_duration_seconds_bucket{le="0.001"} 1074
coredns_health_request_duration_seconds_bucket{le="0.002"} 1107
coredns_health_request_duration_seconds_bucket{le="0.004"} 1117
coredns_health_request_duration_seconds_bucket{le="0.008"} 1119
coredns_health_request_duration_seconds_bucket{le="0.016"} 1119
coredns_health_request_duration_seconds_bucket{le="0.032"} 1119
coredns_health_request_duration_seconds_bucket{le="0.064"} 1119
coredns_health_request_duration_seconds_bucket{le="0.128"} 1119
coredns_health_request_duration_seconds_bucket{le="0.256"} 1119
coredns_health_request_duration_seconds_bucket{le="0.512"} 1119
coredns_health_request_duration_seconds_bucket{le="1.024"} 1119
coredns_health_request_duration_seconds_bucket{le="2.048"} 1119
coredns_health_request_duration_seconds_bucket{le="4.096"} 1119
coredns_health_request_duration_seconds_bucket{le="8.192"} 1119
coredns_health_request_duration_seconds_bucket{le="+Inf"} 1119
coredns_health_request_duration_seconds_sum 0.793689844
coredns_health_request_duration_seconds_count 1119
# HELP coredns_panics_total A metrics that counts the number of panics.
# TYPE coredns_panics_total counter
coredns_panics_total 0
# HELP coredns_plugin_enabled A metric that indicates whether a plugin is enabled on per server and zone basis.
# TYPE coredns_plugin_enabled gauge
coredns_plugin_enabled{name="cache",server="dns://169.254.20.10:53",zone="."} 1
coredns_plugin_enabled{name="cache",server="dns://169.254.20.10:53",zone="cluster.local."} 1
coredns_plugin_enabled{name="cache",server="dns://169.254.20.10:53",zone="in-addr.arpa."} 1
coredns_plugin_enabled{name="cache",server="dns://169.254.20.10:53",zone="ip6.arpa."} 1
coredns_plugin_enabled{name="cache",server="dns://172.21.0.10:53",zone="."} 1
coredns_plugin_enabled{name="cache",server="dns://172.21.0.10:53",zone="cluster.local."} 1
coredns_plugin_enabled{name="cache",server="dns://172.21.0.10:53",zone="in-addr.arpa."} 1
coredns_plugin_enabled{name="cache",server="dns://172.21.0.10:53",zone="ip6.arpa."} 1
coredns_plugin_enabled{name="errors",server="dns://169.254.20.10:53",zone="."} 1
coredns_plugin_enabled{name="errors",server="dns://169.254.20.10:53",zone="cluster.local."} 1
coredns_plugin_enabled{name="errors",server="dns://169.254.20.10:53",zone="in-addr.arpa."} 1
coredns_plugin_enabled{name="errors",server="dns://169.254.20.10:53",zone="ip6.arpa."} 1
coredns_plugin_enabled{name="errors",server="dns://172.21.0.10:53",zone="."} 1
coredns_plugin_enabled{name="errors",server="dns://172.21.0.10:53",zone="cluster.local."} 1
coredns_plugin_enabled{name="errors",server="dns://172.21.0.10:53",zone="in-addr.arpa."} 1
coredns_plugin_enabled{name="errors",server="dns://172.21.0.10:53",zone="ip6.arpa."} 1
coredns_plugin_enabled{name="forward",server="dns://169.254.20.10:53",zone="."} 1
coredns_plugin_enabled{name="forward",server="dns://169.254.20.10:53",zone="cluster.local."} 1
coredns_plugin_enabled{name="forward",server="dns://169.254.20.10:53",zone="in-addr.arpa."} 1
coredns_plugin_enabled{name="forward",server="dns://169.254.20.10:53",zone="ip6.arpa."} 1
coredns_plugin_enabled{name="forward",server="dns://172.21.0.10:53",zone="."} 1
coredns_plugin_enabled{name="forward",server="dns://172.21.0.10:53",zone="cluster.local."} 1
coredns_plugin_enabled{name="forward",server="dns://172.21.0.10:53",zone="in-addr.arpa."} 1
coredns_plugin_enabled{name="forward",server="dns://172.21.0.10:53",zone="ip6.arpa."} 1
coredns_plugin_enabled{name="log",server="dns://169.254.20.10:53",zone="."} 1
coredns_plugin_enabled{name="log",server="dns://169.254.20.10:53",zone="cluster.local."} 1
coredns_plugin_enabled{name="log",server="dns://169.254.20.10:53",zone="in-addr.arpa."} 1
coredns_plugin_enabled{name="log",server="dns://169.254.20.10:53",zone="ip6.arpa."} 1
coredns_plugin_enabled{name="log",server="dns://172.21.0.10:53",zone="."} 1
coredns_plugin_enabled{name="log",server="dns://172.21.0.10:53",zone="cluster.local."} 1
coredns_plugin_enabled{name="log",server="dns://172.21.0.10:53",zone="in-addr.arpa."} 1
coredns_plugin_enabled{name="log",server="dns://172.21.0.10:53",zone="ip6.arpa."} 1
coredns_plugin_enabled{name="loop",server="dns://169.254.20.10:53",zone="."} 1
coredns_plugin_enabled{name="loop",server="dns://169.254.20.10:53",zone="cluster.local."} 1
coredns_plugin_enabled{name="loop",server="dns://169.254.20.10:53",zone="in-addr.arpa."} 1
coredns_plugin_enabled{name="loop",server="dns://169.254.20.10:53",zone="ip6.arpa."} 1
coredns_plugin_enabled{name="loop",server="dns://172.21.0.10:53",zone="."} 1
coredns_plugin_enabled{name="loop",server="dns://172.21.0.10:53",zone="cluster.local."} 1
coredns_plugin_enabled{name="loop",server="dns://172.21.0.10:53",zone="in-addr.arpa."} 1
coredns_plugin_enabled{name="loop",server="dns://172.21.0.10:53",zone="ip6.arpa."} 1
coredns_plugin_enabled{name="prometheus",server="dns://169.254.20.10:53",zone="."} 1
coredns_plugin_enabled{name="prometheus",server="dns://169.254.20.10:53",zone="cluster.local."} 1
coredns_plugin_enabled{name="prometheus",server="dns://169.254.20.10:53",zone="in-addr.arpa."} 1
coredns_plugin_enabled{name="prometheus",server="dns://169.254.20.10:53",zone="ip6.arpa."} 1
coredns_plugin_enabled{name="prometheus",server="dns://172.21.0.10:53",zone="."} 1
coredns_plugin_enabled{name="prometheus",server="dns://172.21.0.10:53",zone="cluster.local."} 1
coredns_plugin_enabled{name="prometheus",server="dns://172.21.0.10:53",zone="in-addr.arpa."} 1
coredns_plugin_enabled{name="prometheus",server="dns://172.21.0.10:53",zone="ip6.arpa."} 1
# HELP coredns_reload_failed_total Counter of the number of failed reload attempts.
# TYPE coredns_reload_failed_total counter
coredns_reload_failed_total 0
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.5841e-05
go_gc_duration_seconds{quantile="0.25"} 8.2532e-05
go_gc_duration_seconds{quantile="0.5"} 0.000124131
go_gc_duration_seconds{quantile="0.75"} 0.000154354
go_gc_duration_seconds{quantile="1"} 0.000278483
go_gc_duration_seconds_sum 0.003190547
go_gc_duration_seconds_count 25
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 45
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.16.10"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 8.073456e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 5.320392e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.457767e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 302576
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 1.504496677254594e-05
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.355248e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 8.073456e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.6008704e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 1.0051584e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 36502
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.246976e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.6060288e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.6427311328710146e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 339078
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 4800
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 123216
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 147456
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 1.2238992e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 708273
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.048576e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.048576e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.4793992e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 11
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 2.5
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 20
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.8047744e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.64273003965e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.54192384e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19

[prameshj]

Thanks for sharing this. However this does not include the "setup_errors_total" metric. This metric is exposed on 9353 port. The other coredns metrics from prometheus plugin are exposed on 9253.

By any chance, would you be able to export these metrics to a dashboard, so we can see the values as a function of time?

However, the logs don't have an entry like "Failed deleting iptables rule" - so it does not look like an iptables lock error :(