Implement Python API for setting check metadata

[ofek]

What does this PR do?

API

This adds a method set_metadata to the AgentCheck base class that updates cached metadata values, which are then sent by the Agent at regular intervals.

It requires 2 arguments:

1. name - the name of the metadata
2. value - the value for the metadata. if name has no transformer defined then the raw value will be submitted and therefore it must be a str

The method also accepts arbitrary keyword arguments that are forwarded to any defined transformers.

Transformers

Transformers are defined via a class level attribute METADATA_TRANSFORMERS.

This is a mapping of metadata names to functions. When you call self.set_metadata(name, value, **options), if name is in this mapping then the corresponding function will be called with the value, and the return value(s) will be sent instead.

Transformer functions must satisfy the following signature:

def transform_<NAME>(value: Any, options: dict) -> Union[str, Dict[str, str]]:

If the return type is a string, then it will be sent as the value for name. If the return type is a mapping type, then each key will be considered a name and will be sent with its (str) value.

Initially, there are 2 default transformers: version and config

version

This transforms a version like 1.2.3-rc.4+5 to its constituent parts. In all cases, the metadata names version.raw and version.scheme will be sent.

If a scheme is defined then it will be looked up from our known schemes. If no scheme is defined then it will default to semver.

The scheme may be set to regex in which case a pattern must also be defined. Any matching named subgroups will then be sent as version.<GROUP_NAME>. In this case, the check name will be used as the value of version.scheme unless final_scheme is also set, which will take precedence.

config

This transforms a dict of arbitrary user configuration. A section must be defined indicating what the configuration represents e.g. init_config.

The metadata name submitted will become config.<section>.

The value will be a JSON str with the root being an array. There will be one map element for every allowed field. Every map may have 2 entries:

1. is_set - a boolean indicating whether or not the field exists
2. value - the value of the field. this is only set if the field exists and the value is a primitive type (None | bool | float | int | str)

The allowed fields are derived from the optional whitelist and blacklist. By default, nothing will be sent.

User configuration can override defaults allowing complete, granular control of metadata submissions. In any section, one may set metadata_whitelist and/or metadata_blacklist which will override their keyword argument counterparts. In following our standard, blacklists take precedence over whitelists.

Blacklists are special in that each item is considered a regular expression.

Motivation

DataDog/datadog-agent#4234
DataDog/datadog-agent#4158

Additional Notes

In another PR I will implement a mechanism to automatically perform config metadata collection to avoid redundant calls in every check.

[ofek]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[codecov]

Codecov Report

Merging #4686 into master will increase coverage by 2.48%.
The diff coverage is n/a.

Impacted Files	Coverage Δ
disk/tests/common.py	60% <0%> (-40%)	⬇️
disk/tests/conftest.py	78.26% <0%> (-17.4%)	⬇️
disk/tests/test_filter.py	96.42% <0%> (-3.58%)	⬇️
disk/datadog_checks/disk/disk.py	83.94% <0%> (-2.3%)	⬇️
vertica/tests/metrics.py
...checks_dev/tests/plugin/test_environment_runner.py
riakcs/datadog_checks/riakcs/riakcs.py
twistlock/datadog_checks/twistlock/twistlock.py
gitlab_runner/tests/test_integration.py
...ent_log/datadog_checks/win32_event_log/__init__.py
... and 454 more

[therve]

Looks good, some minor questions.

[FlorianVeaux]

This is great, just a couple minor points:

• I agree with Thomas, having a default blacklist to prevent passwords and other secrets to be submitted would help.
• The backend will only receive json formatted data for configs. Do we know how well that is supported? I think this is a good idea because we don't want to index every single field, but I'm not sure how the backend would handle JSON

[ofek]

@FlorianVeaux As far as I know this will be supported, but the backend parsing has yet to be written