Dynamic task mapping compatibility with multi-scheduler

[tnyz]

Apache Airflow version

Other Airflow 2 version

What happened

We noticed that tasks using the dynamic task mapping api crashes the scheduler with the error

psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "task_instance_pkey"
80
DETAIL:  Key (dag_id, task_id, run_id, map_index)=(ingestion-scheduler, check_for_success_flag, manual__2022-10-03T15:56:03.241451+00:00, 0) already exists.

We have 2 schedulers so my guess is that both try to schedule the dynamically mapped task

What you think should happen instead

it should be not crash the scheduler

How to reproduce

run a large dynamically mapped task in multi-scheduler environment

Operating System

linux

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

I seriously doubt this is the case - this is an extremely wild and unfounded guess. It's likely an error in your DAG, but I think we need more informationand reproducible case showing exactly what causes it to be sure.

I suggest you do more experimentation with it and trying to provide more information, logs, circumstances, also experimenting with decreasing the number of schedulers to 1 and running it for a while - if you can make a fully reproducible case that can help with diagnosing and result of your analysis, it would be much better.

Converting it into discussion. I am also not going to respond as I am going for 2 weeks vacation, but I hope others will.