test_otel.py::TestOtelIntegration is flaky

[xBis7]

Apache Airflow version

main (development)

If "Other Airflow 2 version" selected, which one?

No response

What happened?

The test class has a 20% failure rate.

https://github.com/xBis7/airflow/actions/runs/16086980198

https://github.com/xBis7/airflow/actions/runs/16087424463

https://github.com/xBis7/airflow/actions/runs/16087596566

I'm attaching a file with the logs

fail_logs.txt

What you think should happen instead?

No response

How to reproduce

I can't reproduce it locally, it's occurring only on the remote CI. It can be reproduced by running the test on the CI on repeat. Increasing execution concurrency makes it more likely to occur. For example, for 10 iterations

max-parallel: 10

Operating System

The CI uses ubuntu-22.04.

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Assigned it to the project .

[xBis7]

@potiuk Thank you!

[potiuk]

Are you going to mark it as flaky ? If so you can even add comment and link to that issue.

[xBis7]

Are you going to mark it as flaky ? If so you can even add comment and link to that issue.

I'm trying to fix it. If it takes longer than expected then I'll mark it as flaky.

[xBis7]

I've concluded that the issue is with the 2nd test, test_scheduler_change_after_the_first_task_finishes.

The test

• uses 2 schedulers
• pauses the 1st scheduler thread so that the 2nd will pick up processing the dag_run
• once the dag_run has finished, it resumes the paused scheduler to check that it makes the needed updates to the db

Once that fails, then the rest fail as well. After commenting it out, there are no more failures.

40/40 successful executions where there used to be 12-15 failures in 40 runs.

https://github.com/xBis7/airflow/actions/runs/16096527407

https://github.com/xBis7/airflow/actions/runs/16096660040

https://github.com/xBis7/airflow/actions/runs/16096742809

https://github.com/xBis7/airflow/actions/runs/16096823166