Support "*" wildcard in type_overrides configuration

[ahmed-mez]

What does this PR do?

Make it possible to use the wildcard * to match metric names when configuring type_overrides

fixes #6091

Motivation

FR

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached

[codecov]

Codecov Report

Merging #7071 into master will decrease coverage by 1.63%.
The diff coverage is n/a.

Impacted Files	Coverage Δ
linkerd/tests/test_linkerd.py
pgbouncer/datadog_checks/pgbouncer/__init__.py
http_check/tests/conftest.py
prometheus/tests/conftest.py
druid/tests/test_integration_e2e.py
datadog_checks_dev/datadog_checks/dev/warn.py
datadog_checks_dev/tests/conftest.py
openldap/datadog_checks/openldap/__about__.py
sqlserver/tests/common.py
kubelet/datadog_checks/kubelet/kubelet.py
... and 741 more

[L3n41c]

The goal of the config['_matched_type_overrides'] dictionary is to avoid to do an fnmatch comparison against all type_overrides for a given metric more than once.

In order to achieve that goal, we would also need a negative cache that would cache the metrics that are matching none of the type_overrides.

With the current implementation, for all the metrics that are matching none of the type_overrides, we’ll re-do the fnmatch comparison with all type_overrides at each execution.

I would suggest that we put an entry in config['_matched_type_overrides'] in all cases.
If there is a matching type_overrides parameter, we put the type. If there is no matching type_overrides parameter, we put None.

[L3n41c]

LGTM !

[ofek]

Please do this instead: #5764 (comment)

[L3n41c]

From a complexity and performance perspective, I think we can do even better.

Here, we have a map with one entry per “pattern” type_override even if several entries have the same type.
It means that, in the loop below, we’ll do several regex matching where a single regex match could be enough.

The idea would be for config['_type_override_patterns'] to have a single entry per type.

That would be something like this:

Suggested change

	for metric, type in iteritems(config['type_overrides']):
	if '*' in metric:
	config['_type_override_patterns'][compile(translate(metric))] = type
	overrides_per_type = {}
	for metric, type in iteritems(config['type_overrides']):
	if '*' in metric:
	if type not in overrides_per_type:
	overrides_per_type[type] = []
	overrides_per_type[type].append(fnmatch.translate(metric))
	del(config['type_overrides'][metric])
	for type, metric_patterns in iteritems(overrides_per_type):
	config['_type_override_patterns'][compile('|'.join(metrics_patterns))] = type

[ahmed-mez]

thanks @L3n41c for the suggestion!
I actually thought about this approach at first but I'm not sure if it's worth it, as this path is executed only once (at the class creation) and more importantly this makes the code less straight forward and more difficult to read/understand.
@ofek what do you think about the tradeoff perf/code-readability here? is it worth it?

[ofek]

If only at class creation time, then yes might not be worth it

[ahmed-mez]

thank you, mind approving the PR please if looks good then?

[ahmed-mez]

@ofek a friendly bump!

[L3n41c]

LGTM

[ruthnaebeck]

👍 for docs