pytest-xdist incompatible with generating files

[ConorMacBride]

When pytest is run over multiple processes/machines using pytest-xdist, generating files such as baseline images, baseline hashes, result and diff images, and JSON and HTML summaries may not work as expected. This is due to the tests being run in different directories in different processes, as well as information not being shared between processes which is necessary for generating a full summary or hash library.

However, the result of each test will be reported correctly by pytest, so using xdist should be fine if you are not using these pytest-mpl features.

This should be documented, and a warning should be given if using pytest-xdist. Ideally support for pytest-xdist would be added.

[cvanelteren]

I don't understand why this would not be possible? Assuming that each test is independent, why can't the process collect the tests afterwards and generate a html on a singular thread while the tests themselves are running on independent threads?

EDIT: or is it because the hash generator needs to be run from a single thread? If so a simple mutex (or equivalent) would work on this no?

[ConorMacBride]

Hi @cvanelteren, the hash generation should be okay to happen anywhere; the tests results are reported correctly when run with pytest-xdist. When I looked into this I think I struggled to find a robust way to copy the generated image files and hashes and merge them into a single HTML report. I think the pytest Stash class looked promising but maybe there is a simpler way, even if it only supports xdist without remote workers.

[cvanelteren]

Do you have a remnant of your attempt? I am not too familiar with how the parallelization is implemented in xdist, but this seems very achievable.

[cvanelteren]

This seems to work

@pytest.hookimpl(trylast=True)
def pytest_terminal_summary(terminalreporter, exitstatus, config):
    """Generate HTML report after all tests have completed"""
    # Skip on workers, only run on the main process
    if hasattr(config, "workerinput"):
        return

    print("\nGenerating HTML report for image comparison tests...")

    # Get the results directory
    results_dir = Path(getattr(config.option, "mpl_results_path", "./mpl-results"))
    if not results_dir.exists():
        print(f"Results directory not found: {results_dir}")
        return

    # Collect all image files and organize by test
    test_results = {}

    # Recursively search for all PNG files
    for image_file in results_dir.rglob("*.png"):
        # Get path relative to results directory for storing in dict
        rel_path = image_file.relative_to(results_dir)
        parent_dir = rel_path.parent if rel_path.parent != Path(".") else None

        filename = image_file.name

        # Skip hash files
        if "hash" in filename:
            continue

        # Generate a more unique test identifier that includes parent directory if available
        test_id = str(parent_dir / image_file.stem) if parent_dir else image_file.stem

        # Handle different pytest-mpl filename patterns, preserving directory structure
        if filename.endswith("-expected.png"):
            # This is a baseline image
            test_name = test_id.replace("-expected", "")
            if test_name not in test_results:
                test_results[test_name] = {
                    "baseline": None,
                    "result": None,
                    "diff": None,
                    "path": parent_dir,
                }
            test_results[test_name]["baseline"] = image_file

        elif filename.endswith("-failed-diff.png"):
            # This is a diff image for a failed test
            test_name = test_id.replace("-failed-diff", "")
            if test_name not in test_results:
                test_results[test_name] = {
                    "baseline": None,
                    "result": None,
                    "diff": None,
                    "path": parent_dir,
                }
            test_results[test_name]["diff"] = image_file

        else:
            # Try to extract test name and check if it corresponds to a real test result
            # Deal with various possible naming conventions
            possible_test_name = test_id

            # Remove common result suffixes if present
            for suffix in ["-result", "-actual", "-diff"]:
                if possible_test_name.endswith(suffix):
                    possible_test_name = possible_test_name.replace(suffix, "")

            if possible_test_name not in test_results:
                test_results[possible_test_name] = {
                    "baseline": None,
                    "result": None,
                    "diff": None,
                    "path": parent_dir,
                }

            # Assume it's a result image if not obviously something else
            if not test_results[possible_test_name]["result"]:
                test_results[possible_test_name]["result"] = image_file

    # Generate a better display name for each test
    for test_name, data in test_results.items():
        # Use directory structure to create more descriptive names
        if data["path"]:
            # Convert path to a more readable format, replacing / with .
            module_path = str(data["path"]).replace("/", ".").replace("\\", ".")
            data["display_name"] = f"{module_path}.{Path(test_name).name}"
        else:
            data["display_name"] = test_name

    # Generate the HTML
    html_output = generate_html_report(results_dir, test_results)

    # Write the HTML report
    report_path = results_dir / "mpl_comparison_report.html"
    with open(report_path, "w") as f:
        f.write(html_output)

    print(f"HTML report generated at: {report_path}")

edit: with added html (not in code)

[cvanelteren]

I made this as a plugin under conftest.py as a proof of principle, would need some polishing if we wish to integrate it.