Skip to content

chore(ci): fix test spans being incorrectly marked as failed #13574

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

vitor-de-araujo
Copy link
Contributor

@vitor-de-araujo vitor-de-araujo commented Jun 3, 2025

The repo tests/conftest.py defines a pytest_runtest_protocol which was overriding the pytest_runtest_protocol hook from the Test Optimization pytest plugin, with a number of consequences:

  • Tests marked with skip, skipif or subprocess markers would not run the Test Optimization hook, causing their spans not to be finished (which in turned caused their module/suite/session not to be finished) and left in a failed stated.
  • The way the conftest.py hook invoked the builtin pytest_runtest_protocol directly with nextitem=None forced some session fixtures to be torn down and set up again in the next test, which masked a problem with a database fixture in the Django tests.

This PR:

  • Changes the conftest.py hook to return None instead of calling the builtin pytest_runtest_protocol directly. This causes the next hook (Test Optimization) to run instead of the builtin one.
  • The Django test that uses a database in a subprocess was changed to not drop the database at the end of the subprocess, so it will still be available to the main process.

Future work:

  • The tests with the subprocess marker are still being incorrectly sent as failed to Test Optimization (the pytest session passes normally). This will be fixed in a subsequent PR.

Checklist

  • PR author has checked that all the criteria below are met
  • The PR description includes an overview of the change
  • The PR description articulates the motivation for the change
  • The change includes tests OR the PR description describes a testing strategy
  • The PR description notes risks associated with the change, if any
  • Newly-added code is easy to change
  • The change follows the library release note guidelines
  • The change includes or references documentation updates if necessary
  • Backport labels are set (if applicable)

Reviewer Checklist

  • Reviewer has checked that all the criteria below are met
  • Title is accurate
  • All changes are related to the pull request's stated goal
  • Avoids breaking API changes
  • Testing strategy adequately addresses listed risks
  • Newly-added code is easy to change
  • Release note makes sense to a user of the library
  • If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
  • Backport labels are set in a manner that is consistent with the release branch maintenance policy

Copy link
Contributor

github-actions bot commented Jun 3, 2025

CODEOWNERS have been resolved as:

tests/conftest.py                                                       @DataDog/apm-core-python
tests/contrib/django/test_django.py                                     @DataDog/apm-core-python @DataDog/apm-idm-python

@vitor-de-araujo vitor-de-araujo changed the title fix(ci_visibility): fix test spans being incorrectly marked as failed fix(ci_visibility): fix test spans being incorrectly marked as failed Jun 3, 2025
Copy link
Contributor

github-actions bot commented Jun 3, 2025

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 296 ± 6 ms.

The average import time from base is: 296 ± 5 ms.

The import time difference between this PR and base is: 0.2 ± 0.2 ms.

The difference is not statistically significant (z = 0.75).

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 1.856 ms (0.63%)
ddtrace.bootstrap.sitecustomize 1.172 ms (0.40%)
ddtrace.bootstrap.preload 1.172 ms (0.40%)
ddtrace.internal.remoteconfig.client 0.632 ms (0.21%)
ddtrace 0.684 ms (0.23%)
ddtrace.internal._unpatched 0.026 ms (0.01%)

@pr-commenter
Copy link

pr-commenter bot commented Jun 3, 2025

Benchmarks

Benchmark execution time: 2025-06-06 15:07:58

Comparing candidate commit a3e3e81 in PR branch vitor-de-araujo/SDTEST-2125/bugfix-tests-marked-as-failed with baseline commit 40eab4b in branch main.

Found 0 performance improvements and 3 performance regressions! Performance is the same for 550 metrics, 3 unstable metrics.

scenario:iastaspectsospath-ospathsplit_aspect

  • 🟥 execution_time [+779.580ns; +897.456ns] or [+16.111%; +18.546%]

scenario:iastaspectsospath-ospathsplitdrive_aspect

  • 🟥 execution_time [+316.390ns; +381.075ns] or [+8.661%; +10.431%]

scenario:iastaspectssplit-splitlines_aspect

  • 🟥 execution_time [+113.689ns; +146.025ns] or [+7.714%; +9.907%]

@vitor-de-araujo vitor-de-araujo added changelog/no-changelog A changelog entry is not required for this PR. CI App and removed backport 2.21 labels Jun 6, 2025
@vitor-de-araujo vitor-de-araujo marked this pull request as ready for review June 6, 2025 13:54
@vitor-de-araujo vitor-de-araujo requested review from a team as code owners June 6, 2025 13:54
@vitor-de-araujo vitor-de-araujo changed the title fix(ci_visibility): fix test spans being incorrectly marked as failed fix(ci): fix test spans being incorrectly marked as failed Jun 6, 2025
@vitor-de-araujo vitor-de-araujo changed the title fix(ci): fix test spans being incorrectly marked as failed ci: fix test spans being incorrectly marked as failed Jun 6, 2025
@vitor-de-araujo vitor-de-araujo changed the title ci: fix test spans being incorrectly marked as failed chore(ci): fix test spans being incorrectly marked as failed Jun 6, 2025
@vitor-de-araujo vitor-de-araujo merged commit 3ff7046 into main Jun 11, 2025
823 of 826 checks passed
@vitor-de-araujo vitor-de-araujo deleted the vitor-de-araujo/SDTEST-2125/bugfix-tests-marked-as-failed branch June 11, 2025 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
changelog/no-changelog A changelog entry is not required for this PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants