Skip to content

sqlite3.OperationalError: database is locked when using --testmon-nocollect with pytest-xdist #259

@jogendra-india

Description

@jogendra-india

Bug Description

When using --testmon --testmon-nocollect with pytest-xdist (-n auto or -n <num>), the test run fails intermittently with:

sqlite3.OperationalError: database is locked

This is especially common on CI systems with multiple parallel workers.

Root Cause

The TestmonXdistSync plugin is only registered inside the if should_collect: block in register_plugins():

def register_plugins(config, should_select, should_collect, cov_plugin):
    if should_select or should_collect:
        config.pluginmanager.register(TestmonSelect(...))

    if should_collect:  # BUG: TestmonXdistSync only registered here!
        config.pluginmanager.register(TestmonCollect(...))
        if config.pluginmanager.hasplugin("xdist"):
            config.pluginmanager.register(TestmonXdistSync())

When --testmon-nocollect is used:

  1. should_collect = False
  2. TestmonXdistSync is not registered
  3. The pytest_configure_node hook doesn't run
  4. Workers don't receive testmon_exec_id from the controller
  5. Each worker calls TestmonData.for_local_run() instead of TestmonData.for_worker()
  6. for_local_run()initiate_execution()fetch_or_create_environment()
  7. fetch_or_create_environment() uses BEGIN IMMEDIATE TRANSACTION (write lock)
  8. Multiple workers race for the write lock → database locked error

Steps to Reproduce

# Create a simple test file
echo 'def test_one(): pass' > test_sample.py

# First run to create .testmondata
pytest --testmon test_sample.py

# Second run with nocollect + xdist - this causes database lock
pytest --testmon --testmon-nocollect -n 4 test_sample.py

Expected Behavior

The --testmon-nocollect mode should work seamlessly with pytest-xdist. Workers should receive the exec_id from the controller and open the database in read-only mode.

Proposed Fix

  1. Register TestmonXdistSync whenever testmon is active (not just when collecting):
if should_select or should_collect:
    config.pluginmanager.register(TestmonSelect(...))
    
    # Register xdist sync even in nocollect mode
    if config.pluginmanager.hasplugin("xdist"):
        config.pluginmanager.register(TestmonXdistSync(should_collect=should_collect))
  1. Skip database sync in pytest_xdist_node_collection_finished when in nocollect mode:
class TestmonXdistSync:
    def __init__(self, should_collect=True):
        self._should_collect = should_collect
    
    def pytest_xdist_node_collection_finished(self, node, ids):
        self.await_nodes += -1
        if self.await_nodes == 0 and self._should_collect:
            node.config.testmon_data.sync_db_fs_tests(retain=set(ids))

Environment

  • pytest-testmon version: 2.2.0
  • pytest version: 7.x / 8.x
  • pytest-xdist version: 3.x
  • Python version: 3.10+
  • OS: Linux (GitHub Actions runners), but reproducible on any OS

Impact

This bug affects all users who:

  • Use --testmon-nocollect for CI pipelines (read-only mode)
  • Use pytest-xdist for parallel test execution
  • Run on systems with multiple workers

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions