Increase Metropolitan reingestion timeout #1293
Labels
💻 aspect: code
Concerns the software code in the repository
🛠 goal: fix
Bug fix
🟨 priority: medium
Not blocking but should be addressed soon
🧱 stack: catalog
Related to the catalog and Airflow DAGs
🔧 tech: airflow
Involves Apache Airflow
Description
We have a number of failures in the
metropolitan_reingestion_workflow
caused by AirflowTaskTimeouts during thepull_data
step.Per context in this comment, it's possible this is being caused by the entire DAG timing out, rather than the individual task. We should investigate to make sure.
Possible fixes:
pull_data
step is timing out.dagrun_timeout
of the entire DAG_list_length
fields in the (https://github.com/WordPress/openverse-catalog/blob/main/openverse_catalog/dags/providers/provider_reingestion_workflows.py#L80)The text was updated successfully, but these errors were encountered: