Agent v6: kube-dns metrics are constantly increasing despite constant traffic

[jonmoter]

I recently deployed v6.1.0 of the datadog agent to one of my Kubernetes clusters. I added the annotations to my instances of kube-dns so that the agent would collect information from them. I've let that run for the better part of two days.

When I look at the metrics for kubedns.request_count, it appears that the number of DNS requests has been increasing steadily over the past 2 days. When I know for a fact the traffic and activity on the cluster has been steady. (It's the weekend, so not much is going on.)

I know that kube-dns exports its metrics in the prometheus format, so that counter metrics are always increasing. (See metrics formats.) But it's supposed to be the job of the scraper to take the difference between the values at T(n) and T(n-1) to calculate the count of events in that interval.

My assumption is that the prometheus scraper in datadog is supposed to do that subtraction. So I'd expect the graph above to be a flat line with a slope of 0, rather than a steadily increasing line.

Datadog Agent Version: 6.1.0
KubeDNS based on: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5
Kubernetes cluster version: 1.5.7 (though I don't think that should affect the prometheus scraping)

[antoinepouille]

Hi Jon,

Thanks for getting to us with this!
Yes indeed, this metrics will show the total count of requests, and not the rate. Actually, this is the way kube-state-metrics is exposing this data through prometheus, and this is why it is submitted like this in Datadog.
This isn't straightforward to compute a rate in the current prometheus rate, but it is to do it in a graph or an alert to get the correct rate, for the period you would be interested in: you would have to use the timeshift function to get an older value, and to substract it from the current value.
You can see an example of this on this screenshot. Please note that you would have to specify timeshift by editing the query, it isn't in the dropdowns.

I'll close the issue, but feel free to drop a question to our support that will be glad to help you out!

Regards

[jonmoter]

Wait, is this the intended behavior of all prometheus-style metrics? In the prometheus metrics format, it says:

A counter is a cumulative metric that represents a single numerical value that only ever goes up. A counter is typically used to count requests served, tasks completed, errors occurred, etc. Counters should not be used to expose current counts of items whose number can also go down, e.g. the number of currently running goroutines. Use gauges for this use case.

Or this article about how prometheus counters work.

(The metrics for kube-dns aren't coming from kube-state-metrics. The datadog agent is scraping the metrics from the kube-dns instances.)

Your timeshift option also won't work if I restart the kube-dns containers, because those metrics will return to zero on newly created instances. That's covered in the blog post above:

A common question is what happens when the process restarts and the counter is reset to 0? rate() will automatically handle this. Any time a counter appears to decrease it’ll be treated as though there was a reset to 0 right after the first data point.

If your agent (or the datadog UI) doesn't support that logic, I'll have places in my graph where it will look like I received -100,000 requests in a timeframe.

Every other count metric in datadog is the count of events that happened in that timeframe, not the total # of events that happened. If you don't normalize this data coming from prometheus-formatted metrics, those metrics are going to be useless to me.

[antoinepouille]

@jonmoter
Sorry about the confusion between kube-dns and kube-state-metrics. The situation is very similar as they are both prometheus checks.
There are actually other integration metrics that give always increasing counters like this one, so this metric isn't an isolated case with this matter.

You're right about this issue with very low values. A workaround could be to play with the query to make it ignore low values, and have 0 instead. With the same example as before, use the query (a-b + abs(a-b))/2 instead of a-b, and you will then see 0 instead of -100,000.
You won't lose any points compared to a computation that would be made by the agent.

Would that be a acceptable solution for your purpose?

[jonmoter]

Right. But in the kubernetes_state check, you already have logic that handles this case:

integrations-core/kubernetes_state/datadog_checks/kubernetes_state/kubernetes_state.py

Lines 191 to 201 in abc5455

	# Job counters are monotonic: they increase at every run of the job
	# We want to send the delta via the `monotonic_count` method
	self.job_succeeded_count = defaultdict(int)
	self.job_failed_count = defaultdict(int)
	self.process(endpoint, send_histograms_buckets=send_buckets, instance=instance)
	for job_tags, job_count in self.job_succeeded_count.iteritems():
	self.monotonic_count(self.NAMESPACE + '.job.succeeded', job_count, list(job_tags))
	for job_tags, job_count in self.job_failed_count.iteritems():
	self.monotonic_count(self.NAMESPACE + '.job.failed', job_count, list(job_tags))

Why can't you use similar logic for the counter prometheus metrics in kube-dns?

[antoinepouille]

@jonmoter Thanks for pulling this out from the code. I could sync up with an engineer from the metrics team to figure this out: this matter is actually causing some issues when using the metrics, so I think we can actually create a second metric that would use the monotonic_count: function to directly forward the counts instead of the current raw value.
I will forward this to the team so that we add some work to tackle this, stay tuned.

[antoinepouille]

This PR #1341 should add the new metric you are after. This is still work-in-progress, so we will update this issue once we will be able to merge it.

Cheers

[jonmoter]

Another idea to consider.

I was poking around in the PrometheusScraper implementation to see how that works. I notice that you're already getting the "type" of each metric, either from the protobuf object definition or the # TYPE comments. Depending on the metric type, you submit different metrics:

integrations-core/datadog-checks-base/datadog_checks/checks/prometheus/mixins.py

Lines 517 to 528 in 26f423c

	for metric in message.metric:
	custom_hostname = self._get_hostname(hostname, metric)
	if message.type == 4:
	self._submit_gauges_from_histogram(metric_name, metric, send_histograms_buckets, custom_tags, custom_hostname)
	elif message.type == 2:
	self._submit_gauges_from_summary(metric_name, metric, custom_tags, custom_hostname)
	else:
	val = getattr(metric, self.METRIC_TYPES[message.type]).value
	if message.name in self.rate_metrics:
	self._submit_rate(metric_name, val, metric, custom_tags, custom_hostname)
	else:
	self._submit_gauge(metric_name, val, metric, custom_tags, custom_hostname)

In that block of code, you're testing whether the metric is of type 4 (histogram) or 2 (summary), and submitting different metrics. How about adding logic there, that if the metric is of type 0 (counter) then you submit the metric as both a gauge and a monotonic count?

That way, any counter style metric would automatically get this behavior for any prometheus integration, rather than needing to add logic for individual integrations.

/cc @mfpierre, who appears to be the person who wrote a good chunk of that code. :-)

[mfpierre]

Hi @jonmoter indeed it seems sending counter metrics as monotonic counts would be more approriate, however we want to avoid sending the metrics as both gauge & monotonic counter but we also need to be careful about all integrations that already rely on the existing metric as gauge. (this would break graphs, monitors, etc...)
A setting on the prometheus scraper to enable this behavior would be a way to go, we added a task in our backlog for it.

[jonmoter]

Agreed, it would technically make more sense to just change the behavior, but that would likely break dashboards and monitors for existing users.

Thanks for responding!

[antoinepouille]

@jonmoter The PR was just merged: #1341, so the changes will be released in a future version of the agent.
Feel free to use the modified files from this PR in your integrations, or as custom integrations in the meantime in order to have the new monotonic_count metrics :-).

[jonmoter]

Great, thank you. I just tried copying kube_dns.py into my docker image and redeploying, and now I'm seeing the kubedns.request_count.count metric. That metric is staying relatively flat with some fluctuations up and down, rather than constantly growing. Just as I'd expect.

Much appreciate!

[antoinepouille]

Glad this suits your use case!
I'll then close this issue, since everything related should be solved now.

Cheers!