Different tests were collected between workers

[pythondev786]

I see there are tickets opened and closed related to the same issue in the past. The tickets were closed with the reason saying the the issue is due to parameterized tests and that the issue is with sets bring unordered vs a sorted list not being unordered.

Here are the links to old tickets
#187
#149

In my case, my tests are not parameterized at all but still facing the same issue. I also tried having 'pythonhashseed=0' in pytest.ini file with pytest-env installed but that did not help. I feel the issue is not only with parameterized tests. I've python3.6 with latest pytest and pytest-xdist.

___________________________________________________________________________________ ERROR collecting gw23 ___________________________________________________________________________________
Different tests were collected between gw0 and gw23. The difference is:
--- gw0

+++ gw23

@@ -1,84 +0,0 @@

-test.py::test_01_verify_connectivity_to_device_vpn_zone_default_vlan_ip_from_vpn_zone_client
-test.py::test_02_verify_traffic_from_vpn_to_dmz_blackholed_by_default_perimeter_policy
-test.py::test_03_verify_traffic_from_vpn_to_dmz_allowed_by_perimeter_policy
-test.py::test_04_verify_traffic_from_vpn_to_dmz_allowed_by_nml_hit_policy
-test.py::test_05_verify_traffic_from_vpn_to_dmz_blackholed_by_user_defined_perimeter_policy
-test.py::test_06_verify_traffic_from_vpn_to_dmz_blackholed_by_user_defined_nml_hit_policy
-test.py::test_07_verify_traffic_from_vpn_to_dmz_dropped_by_unrechable_action_in_user_defined_perimeter_policy
-test.py::test_08_verify_traffic_from_vpn_to_dmz_dropped_by_unrechable_action_in_nml_hit_policy
-test.py::test_09_verify_traffic_from_vpn_to_dmz_dropped_by_prohibit_action_in_user_defined_perimeter_policy
-test.py::test_10_verify_traffic_from_vpn_to_dmz_dropped_by_prohibit_action_in_user_defined_nml_hit_policy

[RonnyPfannschmidt]

without more context we have absolutely no idea what you are doing

[pythondev786]

Hi Ronny,
I was trying to run multiple tests parallelly. The tests are independent of each other. Each test sends traffic to a device and verify the traffic is forwarded/blocked as per the policy configured for that traffic on the device. Totally I had 80 different tests, each test using separate flow (as flows were having unique source port numbers for each test) and I had 80 different policies configured on the device. The test will just send traffic and verify traffic is forwarded or not as per the policies .

When I run it parallelly, sometimes all the 80 tests were passing. But other times I see 'different tests collected' error. I also tried with downgrading python, pytest and pytest-xdist versions. But same issue exists.

Please let me know if you are looking for any specific info.

[pythondev786]

Pasting test #30 here...All other tests are similar

def test_30_Verify_traffic_from_vpn_to_public_zone_not_allowed_by_prohibit_action_by_user_defined_policy():
    # start traffic from vpn to public(Internet)
    startTraffic(client, test_30['cmd'])


    # Verify packets on lan interface
    for device, deviceType in zip(devices, deviceTypes):
        captureFile1 = start_tcpdump(device, deviceType,
                         'interface     : %s' % test_30['lan_interface'],
                         'host          : %s' % test_30['host'],
                         'port              :  %s' % test_30['port'],
                         'greater       : %s' % test_30['greater'],
                         'count         : %s' % test_30['count'])

    assert captureFile1 != False, " Unable to start tcpdump at lan interface"

    for device, deviceType in zip(devices, deviceTypes):
        captureFile2 = start_tcpdump(device, deviceType,
                         'interface     : %s' % test_30['public_interface'],
                         'host          : %s' % test_30['host'],
                         'port              :  %s' % test_30['port'],
                         'greater       : %s' % test_30['greater'],
                         'count         : %s' % test_30['count'],
                         'zero_pkts_expected : %s' % 'True',
                         'timeout            : %s' % '10')

    assert captureFile2 != False, " Unable to start tcpdump at public interface"

    # Wait until al pkts are sent out from vpn to dmz zone and check for packet loss
    time.sleep(16)
    result = verify_tcpdump_capture(device, deviceType,
                           'file          : %s' % captureFile1,
                           'count         : %s' % test_30['count'])

    assert result == True, "Expected number of %s packets not seen on lan interface" % test_30['count']

    result = verify_tcpdump_capture(device, deviceType,
                           'file          : %s' % captureFile2,
                           'zero_pkts_expected : %s' % 'True')

    assert result == True, "Expected number of %s packets not seen on public interface" % test_30['count']

[RonnyPfannschmidt]

based on just the data the issue should not happen, i do wonder if there is some kind of race condition that triggers it when you create as many workers as you have tests

[pythondev786]

I hit the same issue sometimes even when I execute the test with 'pytest -n auto' option which creates 4 workers/ 8 workers depends on the machine where I execute it.

[RonnyPfannschmidt]

i see, right now i have absolutely no idea what causes this

[avoine]

same issue here with:

pytest==3.10.0
pytest-xdist==1.24.0

[dmitry-saritasa]

I have the same issue due to the use of @pytest.mark.parametrize decorator

Example of the test

Function

def list_or_call(l: Union[List, Callable]) -> List:
    """Return list or call function if l is Callable"""
    res = l
    if callable(l):
        res = l()
    if isinstance(res, list):
        return res
    return list(res)

Test

@pytest.mark.parametrize(
    "value", [
        ('a', 'b', 'c'),
        {'a', 'b', 'c'},
        ['a', 'b', 'c'],
        {
            'a': 1,
            'b': 2,
            'c': 3
        },
        lambda: range(0, 10),
    ],
    ids=lambda param: str(param)
)
def test_list_or_call_should_return_list(value):
    """It should always return a list back"""
    assert isinstance(list_or_call(value), list)

✦7 ➜ pytest -n 3 apps
Test session starts (platform: linux, Python 3.6.8, pytest 4.3.0, pytest-sugar 0.9.2)
cachedir: .pytest_cache
metadata: {'Python': '3.6.8', 'Platform': 'Linux-4.20.7-200.fc29.x86_64-x86_64-with-fedora-29-Twenty_Nine', 'Packages': {'pytest': '4.3.0', 'py': '1.7.0', 'pluggy': '0.8.0'}, 'Plugins': {'xdist': '1.26.1', 'sugar': '0.9.2', 'regtest': '1.3.2', 'metadata': '1.8.0', 'jira': '0.3.9', 'html': '1.20.0', 'forked': '1.0.2', 'django': '3.4.7', 'cov': '2.6.1', 'cagoule': '0.3.0', 'pylama': '7.6.6', 'django-test-plus': '1.1.1', 'celery': '4.2.1'}, 'JAVA_HOME': '/usr/java/latest'}
Django settings: config.settings.test (from ini file)
================================================================================
Thank you 💛 my hero user 💛 for running the tests! 
================================================================================
rootdir: /home/dmitry/Projects/analytics/backend, inifile: pytest.ini
plugins: xdist-1.26.1, sugar-0.9.2, regtest-1.3.2, metadata-1.8.0, jira-0.3.9, html-1.20.0, forked-1.0.2, django-3.4.7, cov-2.6.1, cagoule-0.3.0, pylama-7.6.6, django-test-plus-1.1.1, celery-4.2.1
[gw0] linux Python 3.6.8 cwd: /home/dmitry/Projects/analytics/backend
[gw1] linux Python 3.6.8 cwd: /home/dmitry/Projects/analytics/backend
[gw2] linux Python 3.6.8 cwd: /home/dmitry/Projects/analytics/backend
[gw0] Python 3.6.8 (default, Jan 14 2019, 11:21:16)  -- [GCC 8.2.1 20181215 (Red Hat 8.2.1-6)]
[gw1] Python 3.6.8 (default, Jan 14 2019, 11:21:16)  -- [GCC 8.2.1 20181215 (Red Hat 8.2.1-6)]
[gw2] Python 3.6.8 (default, Jan 14 2019, 11:21:16)  -- [GCC 8.2.1 20181215 (Red Hat 8.2.1-6)]
gw0 [94] / gw1 [94] / gw2 [94]
scheduling tests via LoadScheduling

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― gw1 ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
Different tests were collected between gw0 and gw1. The difference is:
--- gw0

+++ gw1

@@ -45,10 +45,10 @@

 apps/business/metrics/tools_tests.py::test_list_or_call_should_call_function_if_passed
 apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list_with_no_modifications_if_list_is_passed
 apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[('a', 'b', 'c')]
-apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[{'c', 'a', 'b'}]
+apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[{'c', 'b', 'a'}]
 apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[['a', 'b', 'c']]
 apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[{'a': 1, 'b': 2, 'c': 3}]
-apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[<function <lambda> at 0x7f496ecabb70>]
+apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[<function <lambda> at 0x7f19ee19ab70>]
 apps/business/metrics/tools_tests.py::test_check_list_contains_only_types[[1, 2, 3]-<class 'int'>]
 apps/business/metrics/tools_tests.py::test_check_list_contains_only_types[('a', 'b', 'c')-<class 'str'>]
 apps/business/metrics/tools_tests.py::test_check_list_contains_only_types[[(1, 2), (3, 4)]-<class 'tuple'>]

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― gw2 ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
Different tests were collected between gw0 and gw2. The difference is:
--- gw0

+++ gw2

@@ -45,10 +45,10 @@

 apps/business/metrics/tools_tests.py::test_list_or_call_should_call_function_if_passed
 apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list_with_no_modifications_if_list_is_passed
 apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[('a', 'b', 'c')]
-apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[{'c', 'a', 'b'}]
+apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[{'c', 'b', 'a'}]
 apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[['a', 'b', 'c']]
 apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[{'a': 1, 'b': 2, 'c': 3}]
-apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[<function <lambda> at 0x7f496ecabb70>]
+apps/business/metrics/tools_tests.py::test_list_or_call_should_return_list[<function <lambda> at 0x7f4a89cc3b70>]
 apps/business/metrics/tools_tests.py::test_check_list_contains_only_types[[1, 2, 3]-<class 'int'>]
 apps/business/metrics/tools_tests.py::test_check_list_contains_only_types[('a', 'b', 'c')-<class 'str'>]
 apps/business/metrics/tools_tests.py::test_check_list_contains_only_types[[(1, 2), (3, 4)]-<class 'tuple'>]

--------------------------------------------------------- generated html file: /home/dmitry/Projects/analytics/backend/htmlcov/report.html ---------------------------------------------------------

----------- coverage: platform linux, python 3.6.8-final-0 -----------
Coverage HTML written to dir htmlcov

===================================================================================== short test summary info ======================================================================================
FAILED gw1
FAILED gw2

Results (5.03s):

Versions

✦7 ➜ pytest --version
This is pytest version 4.3.0, imported from /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/pytest.py
setuptools registered plugins:
  pytest-xdist-1.26.1 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/xdist/plugin.py
  pytest-xdist-1.26.1 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/xdist/looponfail.py
  pytest-sugar-0.9.2 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/pytest_sugar.py
  pytest-regtest-1.3.2 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/pytest_regtest.py
  pytest-metadata-1.8.0 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/pytest_metadata/plugin.py
  pytest-jira-0.3.9 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/pytest_jira.py
  pytest-html-1.20.0 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/pytest_html/plugin.py
  pytest-forked-1.0.2 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/pytest_forked/__init__.py
  pytest-django-3.4.7 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/pytest_django/plugin.py
  pytest-cov-2.6.1 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/pytest_cov/plugin.py
  pytest-cagoule-0.3.0 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/pytest_cagoule/plugin.py
  pylama-7.6.6 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/pylama/pytest.py
  django-test-plus-1.1.1 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/test_plus/plugin.py
  celery-4.2.1 at /home/dmitry/.pyenv/versions/3.6.8/envs/cam/lib/python3.6/site-packages/celery/contrib/pytest.py

pytest.ini

[pytest]
DJANGO_SETTINGS_MODULE = config.settings.test
python_files = tests.py test_*.py *_tests.py *\.tests.py
; if you want to parallelize tests then use -n<X> parameter from pytest-xdist
; https://pypi.org/project/pytest-xdist/
; that will distribute tests among various processes, however
; it seems it doesn't pass SKIP_DB_LOAD environment variable inside these
; processes, therefore in current situation (number of tests) they work
; around 9 seconds, with 3 seconds without it. We should consider using it
; once we see benefits from switching into -n option
; addopts = -n3 -s ...
addopts =
  -ra -s --reuse-db --verbose --nomigrations --cov=.
  --cov-report=html
  --ignore=src
  --ignore=htmlcov
  --ignore=.
  --durations=3
  --strict
  --html=htmlcov/report.html --self-contained-html

; we use strict mode via addopts above and it won't allow use
; of markers on the test functions that are not registed here
; @pytest.mark.smoke
markers =
  smoke: Run the smoke test test functions (quick)

; required min version of pytest framework
minversion = 4.0

; don't search for tests in the following directories
norecursdirs = .dev .docker .git artifacts htmlcov provision wdemo src

; testpaths
testpaths = apps libs

If I remove lambda and dict from parametrize options, then tests will pass

@pytest.mark.parametrize(
    "value", [
        ('a', 'b', 'c'),
        ['a', 'b', 'c'],
    ], ids=lambda param: str(param)
)
def test_list_or_call_should_return_list(value):
    """It should always return a list back"""
    assert isinstance(list_or_call(value), list)

[spearsem]

@RonnyPfannschmidt any movement on this? I am also seeing this error. It seems to happen differently on different operating systems with the same version of pytest and pytest-xdist as well. I can't reproduce my full example, but with the same versions and underlying code, I see this 'different workers collected different tests' error when I run in a Docker container based on Ubuntu 18.04, but I do not see it with Mac OS Mojave 10.14.4.

[yurzo]

We debugged an issue like this yesterday. Granted I haven't read the thread. I though I might just point this out.

In our case we were parametrizing with a list that was randomly drawn.
So every-time a worker woke up, the list got redrawn thus producing the collection mismatch.

Once we realized that the list gets redrawn for every process, we were able to nail down so that each process reached the same conclusion, thus collection succeeds. Hint: use a known seed to seed all the processes. first try random.seed(0) and if that works use random.seed(int(os.environ["PYTHONHASHSEED"]))

The key lays in that each process is fully independent from the other. In fact, your module as singleton stops being so. Modules ended up being loaded once per process.

@RonnyPfannschmidt: What would be a way to precompute an array or a dict that's visible (readable) to all workers?

[yurzo]

@dmitry-saritasa: did you fix it? did you try -n 2 by any chance?

[RonnyPfannschmidt]

Only answer I can give is that it depends on use case and size

[Kinzal15]

After upgrading to python 3.7 from python 2.7 below scenario are not working for @pytest.mark.parametrize decorator.
Error collected > Different tests were collected between gw0 and gw1. is shown.
Examples for which error occurs:
pytest.mark.parametrize(set(list_a) ^ {'list_b'})
or
pytest.mark.parametrize(set_a.intersection({set_b}))
or
pytest.mark.parametrize(data.dict_1)

With python 2.7, pytest==3.6.3, pytest-xdist==1.24.0 no error occurred and test were running in parallel.
Is this any alternate way to run above test successfully in parallel with python3.7

Current version used:
python 3.7
pytest 3.6.3
pytest.xdist 1.27

ERROR MSG:
Different tests were collected between gw0 and gw1. The difference is:
--- gw0

+++ gw1

@@ -23,10 +23,10 @@

desktop_tests/tests/bike/test_bike_quote_page.py::TestBikeQuotePage::()::test_pyp_blank_error
desktop_tests/tests/bike/test_bike_quote_page.py::TestBikeQuotePage::()::test_invalid_reg_number_error
desktop_tests/tests/bike/test_bike_quote_page.py::TestBikeQuotePage::()::test_reg_number_formatting_for_valid_number

Temporary solution:
Use sorted() function when using set/ dict.
eg:
pytest.mark.parametrize(sorted(set_a.intersection({set_b})))

For reference:
#149 (comment)