Improve performance of pattern matching in OpenMetrics

[ahmed-mez]

What does this PR do?

Follow-up on https://github.com/DataDog/integrations-core/pull/5759/files#r380296940

Compile and cache wildcards instead of calling fnmatchcase to match metric names.

Motivation

Perf improvment

Additional Notes

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached

[codecov]

Codecov Report

Merging #5764 into master will decrease coverage by 26.29%.
The diff coverage is n/a.

Impacted Files	Coverage Δ
...ts/tooling/configuration/consumers/test_example.py
...checks_dev/tests/plugin/test_environment_runner.py
oracle/tests/test_oracle.py
...r/datadog_checks/openstack_controller/__about__.py
twistlock/datadog_checks/twistlock/twistlock.py
linkerd/datadog_checks/linkerd/linkerd.py
datadog_checks_base/tests/test_tracing.py
vsphere/datadog_checks/vsphere/cache.py
btrfs/tests/test_btrfs.py
...nmetrics/datadog_checks/openmetrics/openmetrics.py
... and 868 more

[mfpierre]

nit: you can also wrap if metric.name in scraper_config['_ignored_metrics'] if you change to if config['ignore_metrics']

[ofek]

Just to clarify: I was saying fnmatchcase is super slow, as is everything else in the fnmatch module

The performant way to do this is calling https://docs.python.org/3/library/fnmatch.html#fnmatch.translate on all patterns, then '|'.join(patterns), then compile that and cache it

In the end you just call .search once

[ahmed-mez]

@ofek yes I got you point and this is not related to fnmatch. Just to avoid two unnecessary lookups in case ignore_metrics wasn't configured in the first place.

[vboulineau]

Just to clarify: I was saying fnmatchcase is super slow, as is everything else in the fnmatch module

The performant way to do this is calling https://docs.python.org/3/library/fnmatch.html#fnmatch.translate on all patterns, then '|'.join(patterns), then compile that and cache it

In the end you just call .search once

Out of curiosity, I wanted to bench this good idea (on the Mac + Python 3.7) on a KSM output (320M), time for 10 runs.

./test_py.py
With 5 excludes
FNMatch: 41.420358180999756
RESearch: 16.89269208908081
With 10 excludes
FNMatch: 78.64589977264404
RESearch: 22.81194281578064

Worth changing to this suggestion!

[ahmed-mez]

pushed an attempt with translate, compile and search.

[ahmed-mez]

@ofek shouldn't we implement a similar logic to match whitelisted metrics? https://github.com/DataDog/integrations-core/blob/master/datadog_checks_base/datadog_checks/base/checks/openmetrics/mixins.py#L564
I'd be happy to implement it in this PR.

[ofek]

Sure!