Improve kubedns QPS to be closer to GCE DNS server performance

[girishkalele]

Ran a DNS performance test against the kubedns pod vs the native GCE DNS server.
kubedns performance was only around 10% that of going directly to the GCE DNS at "169.254.169.254".

The single kubedns pod seems to max out at around the same performance as provided by the GCE DNS to each unique IP address. Since all 10 Pods are running the same query-file, we would expect the dnsmasq caching would dramatically speed up performance.

Running against DNS cluster ip 10.0.0.10 is around 109 QPS

  Queries per second:   137.991371
  Queries per second:   112.373670
  Queries per second:   97.577085
  Queries per second:   131.381920
  Queries per second:   35.410735
  Queries per second:   136.044372
  Queries per second:   49.806700
  Queries per second:   128.092776
  Queries per second:   133.084200
  Queries per second:   133.843242

Running directly against 169.254.169.254 is 1256 QPS

  Queries per second:   1233.658841
  Queries per second:   1292.178002
  Queries per second:   1204.961019
  Queries per second:   1278.054183
  Queries per second:   1297.116902
  Queries per second:   1209.742660
  Queries per second:   1250.562897
  Queries per second:   1233.336819
  Queries per second:   1303.879214
  Queries per second:   1255.794775

Methodology:

2 n1-standard-2 instances as schedulable nodes.
Hairpin_veth mode is on.
1 kubedns pod.
10 DNSPerf client pods running as 10 parallel completions under one "Job".

[girishkalele]

cc @bprashanth @thockin @ArtfulCoder

[thockin]

It might just be SkyDNS. Weave also has a DNS server that they claim performs really well. It's very different, and would be pretty much a rewrite again, but maybe we should consider it medium term?

@errordeveloper

[girishkalele]

@thockin Yes, it looks like kubedns/skydns performance is the bottleneck.

Because, even when the queryfile contained only local service records and dnsmasq cache was set to 1, QPS didn't increase a lot. This case should have skydns servicing all requests with no forwarding required.

@ArtfulCoder and I did some more measurements.

1. Bump up dnsmasq cache size to 64K - the kubedns performance increased to ~ 220 QPS.
2. Query file contains only local service A records ~ 5000 QPS (all dnsmasq-caching)
3. Query file contains only local service A records and dnsmasq cache size = 1 ~ 300 QPS.

Side-Note: Cluster IP vs Endpoint/Pod IP had no visible impact on the QPS.

[girishkalele]

@errordeveloper Is there a Docker image with the Weave DNS server that we can drop in and do a test ? It won't resolve service DNS IPs, but we will get an idea of external forwarding performance.

Another option is to use the DNS server built into Consul. There is a new kubernetes/kube2consul project starting up.

https://www.consul.io/docs/agent/dns.html

[bprashanth]

2 n1-standard-2 instances as schedulable nodes.

What else was sharing that machine? did you try giving it unlimited cpu/memory?

[girishkalele]

@bprashanth

Just the standard set to pods scheduled on minions - they shouldn't have been very busy.
One noticeable thing is that kubedns is logging very verbosely, I am going to build a test container with all that logging removed, I am not sure if the logging framework logs asynchronously or blocks and flushes the log stream synchronously.

[bprashanth]

There's a -quiet option
Make sure you aren't maxing out on the 1/10th of a cpu core and how much ever memory that pod manifest gives you on a populated node.

[bprashanth]

https://github.com/kubernetes/contrib/blob/master/exec-healthz/exechealthz.go#L43
you should be able to specify that through the "args" field in yaml directly

[girishkalele]

@bprashanth Good tip - let me crank up the cpu limits on the DNS containers.

The exechealth -q option is already there - this is the kubedns daemon going crazy printing every query and datastructure like so:

I0701 19:23:27.115855       1 dns.go:583] Received ReverseRecord Request:1.0.0.10.in-addr.arpa.
I0701 19:23:29.114784       1 dns.go:439] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0701 19:23:29.114833       1 dns.go:539] records:[0xc8202b97a0], retval:[{10.0.0.1 0 10 10  false 30 0  /skydns/local/cluster/svc/default/kubernetes/3234633364383235}], path:[local cluster svc default 

[bprashanth]

you should be able to bring its log level down with --v

[girishkalele]

@bprashanth

Eliminated logs from kubedns and bumped up CPU to 1000m.
Marginally improved QPS from ~ 290 to ~ 341.

Will leave testing this where it is, and if we get Weave or Consul DNS available, we can rerun the tests.

[thockin]

I think ConsulDNS requires running Consul, which sounds pretty
heavy-weight. We could also engage with skydns folks on expected perf
(@bketelsen). We could also look at CoreDNS (@miekg) which is much newer
and riskier, but maybe a better fit longer term?

@smarterclayton - have you guys done DNS perf numbers?

On Fri, Jul 1, 2016 at 4:40 PM, Girish Kalele notifications@github.com
wrote:

@bprashanth https://github.com/bprashanth

Eliminated logs from kubedns and bumped up CPU to 1000m.
Marginally improved QPS from ~ 290 to ~ 341.

Will leave testing this where it is, and if we get Weave or Consul DNS
available, we can rerun the tests.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#28366 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AFVgVJA6pxFwnO6SU3fdf65OuZ3qs8w3ks5qRaVWgaJpZM4JDWO9
.

[miekg]

[ Quoting notifications@github.com in "Re: [kubernetes/kubernetes] Improve..." ]

I think ConsulDNS requires running Consul, which sounds pretty
heavy-weight. We could also engage with skydns folks on expected perf
(@bketelsen). We could also look at CoreDNS (@miekg) which is much newer
and riskier, but maybe a better fit longer term?

@smarterclayton - have you guys done DNS perf numbers?

~500 qps sounds really slow. Is every request going to etcd?

CoreDNS has much better caching, in the (non optimized version) I can easily do
15K qps, but SkyDNS should be able to get to those as well.

Longer term I think CoreDNS makes much more sense. Note the "SkyDNS" middleware
in CoreDNS is a cleaned up version from CoreDNS - but yeah the project itself is
barely 2 months old.

Is more information available on where the time is spend.

On Fri, Jul 1, 2016 at 4:40 PM, Girish Kalele notifications@github.com
wrote:

@bprashanth https://github.com/bprashanth

Eliminated logs from kubedns and bumped up CPU to 1000m.
Marginally improved QPS from ~ 290 to ~ 341.

Will leave testing this where it is, and if we get Weave or Consul DNS
available, we can rerun the tests.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#28366 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AFVgVJA6pxFwnO6SU3fdf65OuZ3qs8w3ks5qRaVWgaJpZM4JDWO9
.

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#28366 (comment)
/Miek

Miek Gieben

[bketelsen]

I've been out of the loop, but happy to help. Also: There's a container "Hack Room" in the 13th of July at GopherCon - would be a good time to sit down and work through things.

10:00am - 5:00pm
Room: 4f
Container Technologies
Hack on and discuss the various container technologies written in and powered by Go

[thockin]

Girish, have you take a profile?

On Fri, Jul 1, 2016 at 4:54 PM, Brian Ketelsen notifications@github.com
wrote:

I've been out of the loop, but happy to help. Also: There's a container
"Hack Room" in the 13th of July at GopherCon - would be a good time to sit
down and work through things.

10:00am - 5:00pm
Room: 4f
Container Technologies
Hack on and discuss the various container technologies written in and
powered by Go

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#28366 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AFVgVCKDDlJirb9Mj0MQMQ6t8OeHPlakks5qRai6gaJpZM4JDWO9
.