Coverage: warning "module-not-measured"

[florealcab]

Hello,

Thank you for your good work, tests take 2x less times to run with it comparing to unittest.

But I have a problem. I run with this command:
unittest-parallel -t . -s . -p 'test_*.py' --coverage --coverage-source XXX

where XXX is the name of my package

Just before the coverage report, i have this warning:
Coverage.py warning: Module XXX was previously imported, but not measured (module-not-measured)

Is there a way to fix this warning?

[craigahobbs]

I'm glad to hear that you find unittest-parallel useful! Sadly, I was unable to reproduce the issue. I'm running Ubuntu 20.04. Here's my venv setup:

python3 -m venv env
env/bin/pip install -U pip setuptools wheel
env/bin/pip install coverage unittest-parallel

Here are my versions:

$ env/bin/pip freeze
coverage==5.2.1
pkg-resources==0.0.0
unittest-parallel==0.8.7

I setup the following directory tree:

.
├── mypkg
│   ├── __init__.py
│   └── mypkg.py
└── tests
    ├── __init__.py
    └── test_mypkg.py

The "__init__.py" files are empty. "mypkg.py" contains the following:

def myfunc():
    return 42

"test_mypkg.py" contains:

import unittest

from mypkg.mypkg import myfunc as myfunc

class TestMypkg(unittest.TestCase):
    def test_myfunc(self):
        self.assertEqual(myfunc(), 42)

I then ran unittest-parallel like so:

env/bin/unittest-parallel -t . -s tests --coverage --coverage-source mypkg

I get the following output with no warnings:

test_myfunc (tests.test_mypkg.TestMypkg) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.001s

OK

Ran 1 tests

Name                Stmts   Miss  Cover
---------------------------------------
mypkg/__init__.py       0      0   100%
mypkg/mypkg.py          2      0   100%
---------------------------------------
TOTAL                   2      0   100%

Total coverage is 100.00%

[florealcab]

In my case, module is installed (pip install . in the source folder) inside the virtualenv. Maybe it's why it warns? With your test I have no warning too.

But if I use official unittests with coverage, I have no warning with my module

[craigahobbs]

Can you try disabling the warning using a .coveragerc file?

https://coverage.readthedocs.io/en/coverage-5.2.1/cmd.html?highlight=warnings#warnings

[ferdnyc]

@florealcab @craigahobbs

(Four years later...) I was able to reproduce this, but only by making some modifications to the example setup @craigahobbs provided above.

The changes I made were:

1. Move mypkg into a source directory:

   mkdir src
   mv mypkg src/

2. Add a pyproject.toml to make it installable:

   [build-system]
   requires = ["setuptools >= 61.0"]
   build-backend = "setuptools.build_meta"
   
   [project]
   name = 'mypkg'
   version = '0.1'
   description = "My Package"
   license = {text = 'MIT No Attribution License (MIT-0)'}
   requires-python = ">= 3.8"
   
   [project.optional-dependencies]
   tests = [
       'unittest-parallel',
   ]

3. Update the module code to make it import itself when loaded:

   # src/mypkg/__init__.py
   
   from .mypkg import *

4. Adjust the test code to match:

   # tests/test_mypkg.py
   import unittest
   from mypkg import myfunc
   
   class TestMypkg(unittest.TestCase):
       def test_myfunc(self):
           self.assertEqual(myfunc(), 42)

5. Install mypkg into the current venv (doesn't really matter if it's an editable install or one built by setuptools, same behavior either way):

   pip install -e '.[tests]'

6. Run unittest-parallel

   $ unittest-parallel -vv --coverage-branch \
     --coverage-source mypkg
    Running 1 test suites (1 total tests) across 1 processes
   
    test_myfunc (tests.test_mypkg.TestMypkg.test_myfunc) ...
    test_myfunc (tests.test_mypkg.TestMypkg.test_myfunc) ... ok
    ...venv/lib64/python3.12/site-packages/coverage/inorout.py:519: 
    CoverageWarning: Module mypkg was previously imported, but not
    measured (module-not-measured)
      self.warn(msg, slug="module-not-measured")
   
    ----------------------------------------------------------------------
    Ran 1 test in 0.205s
   
    OK
   
    Name                    Stmts   Miss Branch BrPart  Cover
    ---------------------------------------------------------
    src/mypkg/__init__.py       1      0      0      0   100%
    src/mypkg/mypkg.py          2      0      0      0   100%
    ---------------------------------------------------------
    TOTAL                       3      0      0      0   100%
    Total coverage is 100.00%

It seems to be the relative star import in the package __init__.py that causes the problem, coupled with the package being installed. (I couldn't reproduce with a local, non-installed package tree that contained a star import. Only once it was installed.)

And since that's pretty terrible code anyway, it's probably not unfair to just say, "Yeah, don't do that, it's ugly and it confuses coverage.py."

[ferdnyc]

It doesn't actually have to be a star import. I just confirmed that the same behavior occurs if src/mypkg/__init__.py contains this:

from .mypkg import myfunc

The key factor seems to be that the package imports the part of itself that's being tested from its __init__.py, as soon as the top-level package definition is imported into sys.modules, which I guess happens before unittest-parallel's _coverage wrapper is able to trigger the module's import under the controlled conditions it expects.

The src/ paths in the unittest output are just because I happened to do an editable install for that test. If it's performed with a standard pip install '.[tests]' instead, same behavior (including the Coverage warning) but now referencing the installed path:

$ pip install '.[tests]'
$ unittest-parallel -vv --coverage-branch --coverage-source mypkg
Running 1 test suites (1 total tests) across 1 processes

test_myfunc (tests.test_mypkg.TestMypkg.test_myfunc) ...
test_myfunc (tests.test_mypkg.TestMypkg.test_myfunc) ... ok
...venv/lib64/python3.12/site-packages/coverage/inorout.py:519:
CoverageWarning: Module mypkg was previously imported, but not 
measured (module-not-measured)
  self.warn(msg, slug="module-not-measured")

----------------------------------------------------------------------
Ran 1 test in 0.180s

OK

Name                                                      Stmts   Miss Branch BrPart  Cover
-------------------------------------------------------------------------------------------
venv3.12/lib/python3.12/site-packages/mypkg/__init__.py       1      0      0      0   100%
venv3.12/lib/python3.12/site-packages/mypkg/mypkg.py          2      0      0      0   100%
-------------------------------------------------------------------------------------------
TOTAL                                                         3      0      0      0   100%
Total coverage is 100.00%

[ferdnyc]

(Last update, I swear...)

Relative imports are also not required. Same warning if src/mypkg/__init__.py contains:

from mypkg.mypkg import myfunc

[craigahobbs]

@ferdnyc will take a look next week

[craigahobbs]

OK, I've reproduced it with your changes above and the following command:

env/bin/unittest-parallel -t . -s tests --coverage --coverage-source mypkg

The comment above the warning line in the coverage source for inorout.py is informative:

        # The module was in sys.modules, and seems like a module with code, but
        # we never measured it. I guess that means it was imported before
        # coverage even started.

The --coverage-source option takes a source directory or a module name. Since we've moved our package source to "src/mypkg", the --coverage-source option above ("mypkg") refers to the mypkg that is installed in the virtual environment, which is what triggers the warning.

If I change the unittest-parallel command to refer to the package source directory ("src/mypkg"), the warning goes away:

env/bin/unittest-parallel -t . -s tests --coverage --coverage-source src/mypkg

Does that fix it for you?

[florealcab]

If I do that coverage is not well measured and I have this new warning instead:

CoverageWarning: No data was collected. (no-data-collected)
self._warn("No data was collected.", slug="no-data-collected")

[craigahobbs]

I don't see that warning. I added a test repo for our test case - please update as necessary:

https://github.com/craigahobbs/unittest-parallel-issue-3

It has a makefile to build the venv and run the tests:

Reproduce the original warning

make test

No warning

make test2

Cleanup venv, etc

make clean

[florealcab]

Maybe the problem was I am using setup.py instead of pyproject.toml to configure my package. I can try to switch.

With your test, I see no warning with test2 as expected. I missed the invitation, it has expired now. I will try to switch to toml and if it does not change anything I will try to make a minimal test case that reproduce my problem

[ferdnyc]

The src/mypkg version fails if you've done a non-editable install, in which case coverage does indeed produce a "no-data-collected" error, because src/mypkg is no longer being imported by the test code:

$ build/env/bin/pip uninstall 'mypkg'
 [...]
$ build/env/bin/pip install '.[tests]'
 [...]
$ build/env/bin/unittest-parallel -t . -s tests --coverage --coverage-source src/mypkg
.../build/env/lib64/python3.12/site-packages/coverage/control.py:888: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")
Running 1 test suites (1 total tests) across 1 processes
.../build/env/lib64/python3.12/site-packages/coverage/control.py:888: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")

----------------------------------------------------------------------
Ran 1 test in 0.220s

OK

Name                    Stmts   Miss  Cover
-------------------------------------------
src/mypkg/__init__.py       1      1     0%
src/mypkg/mypkg.py          2      2     0%
-------------------------------------------
TOTAL                       3      3     0%
Total coverage is 0.00%

[ferdnyc]

If you run the above with PYTHONPATH=$(pwd)/src prepended, then it succeeds.

Getting coverage.py to work correctly in a multiprocessing environment is apparently a very difficult thing. The coverage.py docs have a lot to say about that.