Add ability to log warnings during pytest

[ofek]

Motivation

We need a way to output messages even for passing tests, such as when temporary directories could not be cleaned up. However, pytest captures output by default, so logs only get displayed upon failure or when using extra flags. Therefore we use their built-in warnings mechanism to elicit this behavior by default https://docs.pytest.org/en/latest/warnings.html. We also can output condensed messages so these look just as nice in terminal when not running tests.

Additional Notes

This needs to be merged before #2762, which in turn is required for #2760

[codecov-io]

Codecov Report

Merging #2764 into master will decrease coverage by 7.51%.
The diff coverage is 0%.

@@            Coverage Diff            @@
##           master   #2764      +/-   ##
=========================================
- Coverage   84.82%   77.3%   -7.52%     
=========================================
  Files         657      23     -634     
  Lines       37746     780   -36966     
  Branches     4537      81    -4456     
=========================================
- Hits        32019     603   -31416     
+ Misses       4395     149    -4246     
+ Partials     1332      28    -1304

[masci]

However, pytest captures output by default, so logs only get displayed upon failure

Not sure what you mean, if I try:

import pytest
import warnings

def test_foo():
    warnings.warn("Show me this")

warning message is actually printed when I do pytest <test_module.py> without additional flags:

tests/tooling/test_dep.py .............                                                                          [100%]

=================================================== warnings summary ===================================================
tests/tooling/test_dep.py::test_foo
  /Users/massi/dev/integrations-core/datadog_checks_dev/tests/tooling/test_dep.py:22: UserWarning: Show me this
    warnings.warn("Show me this")

-- Docs: https://docs.pytest.org/en/latest/warnings.html
======================================== 13 passed, 1 warnings in 0.15 seconds =========================================

[ofek]

Right that's why we're using warnings here. I was talking about logging

[masci]

Can't we use capsys within the single tests where we don't want pytest to capture logs? Or use capsys to get the loglines and output them as warnings?

[ofek]

Not really because the tests themselves likely wouldn't do this kind of logging but rather these warnings would come from elsewhere like utility functions or fixtures.

[masci]

I like the current solution better 👍