`airflow standalone` deployment is unable to run `DockerOperator` tasks on MacOS

[schustmi]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

Airflow version 2.4.3 running on MacOS, running a DAG using DockerOperators fails in combination with a local standalone deployment:

*** Reading local file: /Users/schustmi/airflow/logs/dag_id=test/run_id=manual__2022-12-20T09:42:32.767926+00:00/task_id=docker_task/attempt=1.log
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: test.docker_task manual__2022-12-20T09:42:32.767926+00:00 [queued]>
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: test.docker_task manual__2022-12-20T09:42:32.767926+00:00 [queued]>
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1362} INFO - 
--------------------------------------------------------------------------------
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1363} INFO - Starting attempt 1 of 1
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1364} INFO - 
--------------------------------------------------------------------------------
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1383} INFO - Executing <Task(DockerOperator): docker_task> on 2022-12-20 09:42:32.767926+00:00
[2022-12-20, 09:42:35 UTC] {standard_task_runner.py:55} INFO - Started process 98208 to run task
[2022-12-20, 09:42:35 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'test', 'docker_task', 'manual__2022-12-20T09:42:32.767926+00:00', '--job-id', '17', '--raw', '--subdir', 'DAGS_FOLDER/dag.py', '--cfg-path', '/var/folders/45/1tkl8h1d3tvf2q72t8p6bw5r0000gn/T/tmpe6h4cv9f']
[2022-12-20, 09:42:35 UTC] {standard_task_runner.py:83} INFO - Job 17: Subtask o
[2022-12-20, 09:42:35 UTC] {task_command.py:376} INFO - Running <TaskInstance: test.docker_task manual__2022-12-20T09:42:32.767926+00:00 [running]> on host schustmi-mac-work.fritz.box
[2022-12-20, 09:42:35 UTC] {taskinstance.py:1590} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=test
AIRFLOW_CTX_TASK_ID=docker_task
AIRFLOW_CTX_EXECUTION_DATE=2022-12-20T09:42:32.767926+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2022-12-20T09:42:32.767926+00:00
[2022-12-20, 09:42:35 UTC] {local_task_job.py:159} INFO - Task exited with return code Negsignal.SIGSEGV
[2022-12-20, 09:42:35 UTC] {taskinstance.py:2623} INFO - 0 downstream tasks scheduled from follow-on schedule check

What you think should happen instead

On other operating systems, the same combination of standalone deployment and DockerOperators works.

How to reproduce

pip install apache-airflow==2.4.3 apache-airflow-providers-docker==3.3.0
airflow standalone

Then run the following DAG:

from airflow import DAG
from airflow.providers.docker.operators.docker import DockerOperator
from datetime import datetime

with DAG(dag_id="test", schedule="@once", start_date=datetime.utcnow()) as dag:
    docker_task = DockerOperator(image="alpine:latest", command="echo test", task_id="docker_task")

Operating System

MacOS Ventura 13.0 (22A380)

Versions of Apache Airflow Providers

apache-airflow-providers-common-sql 1.3.1
apache-airflow-providers-docker 3.3.0
apache-airflow-providers-ftp 3.2.0
apache-airflow-providers-http 4.1.0
apache-airflow-providers-imap 3.1.0
apache-airflow-providers-sqlite 3.3.1

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

This is problem with MacOS M1 not Airflow, not even Docker. There are various workarounds for this. We've implemented a better message for this case in #27381 but due to the nature of this error, we cannot do more - because it is really problem of some of the libraries that do not work on M1 (ARM) Macbook. There are various workarouds you can apply (see the message in #27381) and you can also look for some other cases (search for SIGSEGV our issues or in SIGESEGV MacOS M1 via google).

[schustmi]

The same error happens on non-M1 macs, but the no_proxy=* workaround in the PR you linked still worked, thanks!

Is there a general option to run tasks without forking? For example setting AIRFLOW__CORE__DEFAULT_IMPERSONATION=... also resolves the issue for me, as StandardTaskRunner.start() then runs without forking.

[potiuk]

https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#execute-tasks-new-python-interpreter

[potiuk]

And yes we think that on MacOS we should enable it by default

[kajen3]

The same error happens on non-M1 macs, but the no_proxy=* workaround in the PR you linked still worked, thanks!

Is there a general option to run tasks without forking? For example setting AIRFLOW__CORE__DEFAULT_IMPERSONATION=... also resolves the issue for me, as StandardTaskRunner.start() then runs without forking.

Could you share a link to the solution? I couldn't find a workaround in the linked PR?

[potiuk]

Could you share a link to the solution? I couldn't find a workaround in the linked PR?

Because there is not one, generally accepted - SIGSEGV might be generated by various causes and libraries and if you look through linked issues (and maybe search for just SIGSEGV) and read what worked for various people - maybe you will find a solution that will work for you.

One generall thing you can attempt is to set this config to True: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#execute-tasks-new-python-interpreter

This might help in a number of similar cases but there is no guarantee - because there might be various reasons and libraries raising SIGSEGV. The linked PR might help you to see how you can try to narrow it down and possibly find a root cause.

That thing is - This is not an Airflow Problem - this is a problem that some libraries are not working on ARM and some of them might require some workarounds - but you have to find those workarounds on your own - first attempt various workarouds people found for their libraries, if it does not work, finding which library causes it and maybe finding or figuring out a new workaround you might need.

Sorry we can't help more, but other than helping you to investigate by better information and showing how you can investigate it is the best we can do.