Issue while connecting to docker daemon from containerized Airflow

[kaustubhharapanahalli]

Apache Airflow Provider(s)

docker

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==2.3.0
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-cncf-kubernetes==2.0.3
apache-airflow-providers-docker==2.2.0
apache-airflow-providers-elasticsearch==2.0.3
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-google==6.0.0
apache-airflow-providers-grpc==2.0.1
apache-airflow-providers-hashicorp==2.1.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-microsoft-azure==3.2.0
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-odbc==2.0.1
apache-airflow-providers-postgres==2.3.0
apache-airflow-providers-redis==2.0.1
apache-airflow-providers-sendgrid==2.0.1
apache-airflow-providers-sftp==2.1.1
apache-airflow-providers-slack==4.1.0
apache-airflow-providers-sqlite==2.0.1
apache-airflow-providers-ssh==2.2.0

Apache Airflow version

2.2.1 (latest released)

Operating System

PRETTY_NAME="Debian GNU/Linux 10 (buster)"

Deployment

Docker-Compose

Deployment details

version: "3.9"

services:
  postgres:
    restart: always
    image: postgres:latest
    container_name: dpg_postgres
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=airflow
      - POSTGRES_DB=airflow
    ports:
      - "5434:5432"

  webserver:
    container_name: dpg_airflow
    build:
      context: .
      args:
        PYTHON_BASE_IMAGE: "python:3.7-slim-buster"
        PYTHON_MAJOR_MINOR_VERSION: 3.7
        AIRFLOW_INSTALLATION_METHOD: "apache-airflow"
        AIRFLOW_VERSION: "2.2.1"
        AIRFLOW_INSTALL_VERSION: "==2.2.1"
        AIRFLOW_CONSTRAINTS_REFERENCE: "constraints-2-0"
        AIRFLOW_SOURCES_FROM: "empty"
        AIRFLOW_SOURCES_TO: "/empty"
        ADDITIONAL_AIRFLOW_EXTRAS: "apache-airflow-providers-amazon \
          apache-airflow-providers-http \
          apache-airflow-providers-hashicorp \
          apache-airflow-providers-docker \
          apache-airflow-providers-ftp \
          apache-airflow-providers-postgres \
          apache-airflow-providers-sftp \
          apache-airflow-providers-ssh"
        ADDITIONAL_DEV_APT_DEPS: "gcc g++"
    environment:
      key_name:
      project_group:
    hostname: webserver
    restart: always
    depends_on:
      - postgres
    env_file:
      - .env
    volumes:
      - ./dags:/opt/airflow/dags
      - ./plugins:/opt/airflow/plugins
      - data_volume:/data/project_meta
    ports:
      - "8090:8080"
    entrypoint: ./dags/scripts/airflow-entrypoint.sh
    healthcheck:
      test: ["CMD-SHELL", "[ -f ./opt/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 32

volumes:
  data_volume:

What happened

Hello, I was trying to connect to docker daemon from a containerized airflow setup. I saw this one issue: #16803 > looking at this, I tired to migrate my Airflow version to 2.2.1 and set mount_tmp_dir=False

I tried to run this using two approaches.

1. By not setting a tcp docker url
2. By setting a tcp docker url

In the first approach, I got this error:

*** Reading local file: /opt/airflow/logs/hello_world/echo_name/2021-11-10T04:14:45.866031+00:00/1.log
[2021-11-10, 04:14:47 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: hello_world.echo_name manual__2021-11-10T04:14:45.866031+00:00 [queued]>
[2021-11-10, 04:14:47 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: hello_world.echo_name manual__2021-11-10T04:14:45.866031+00:00 [queued]>
[2021-11-10, 04:14:47 UTC] {taskinstance.py:1241} INFO - 
--------------------------------------------------------------------------------
[2021-11-10, 04:14:47 UTC] {taskinstance.py:1242} INFO - Starting attempt 1 of 2
[2021-11-10, 04:14:47 UTC] {taskinstance.py:1243} INFO - 
--------------------------------------------------------------------------------
[2021-11-10, 04:14:47 UTC] {taskinstance.py:1262} INFO - Executing <Task(DockerOperator): echo_name> on 2021-11-10 04:14:45.866031+00:00
[2021-11-10, 04:14:47 UTC] {standard_task_runner.py:52} INFO - Started process 5836 to run task
[2021-11-10, 04:14:47 UTC] {standard_task_runner.py:76} INFO - Running: ['***', 'tasks', 'run', 'hello_world', 'echo_name', 'manual__2021-11-10T04:14:45.866031+00:00', '--job-id', '2', '--raw', '--subdir', 'DAGS_FOLDER/dag_creator.py', '--cfg-path', '/tmp/tmp39jdfg29', '--error-file', '/tmp/tmp2tmtb5n4']
[2021-11-10, 04:14:47 UTC] {standard_task_runner.py:77} INFO - Job 2: Subtask echo_name
[2021-11-10, 04:14:47 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: hello_world.echo_name manual__2021-11-10T04:14:45.866031+00:00 [running]> on host webserver
[2021-11-10, 04:14:47 UTC] {taskinstance.py:1429} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=***
AIRFLOW_CTX_DAG_ID=hello_world
AIRFLOW_CTX_TASK_ID=echo_name
AIRFLOW_CTX_EXECUTION_DATE=2021-11-10T04:14:45.866031+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-11-10T04:14:45.866031+00:00
[2021-11-10, 04:14:47 UTC] {taskinstance.py:1703} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
    self._execute_task_with_callbacks(context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
    result = self._execute_task(context, self.task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
    result = execute_callable(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 367, in execute
    self.cli = self._get_cli()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 397, in _get_cli
    return APIClient(base_url=self.docker_url, version=self.api_version, tls=tls_config)
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
    'Error while fetching server API version: {0}'.format(e)
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
[2021-11-10, 04:14:47 UTC] {taskinstance.py:1280} INFO - Marking task as UP_FOR_RETRY. dag_id=hello_world, task_id=echo_name, execution_date=20211110T041445, start_date=20211110T041447, end_date=20211110T041447
[2021-11-10, 04:14:47 UTC] {standard_task_runner.py:91} ERROR - Failed to execute job 2 for task echo_name
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/task/task_runner/standard_task_runner.py", line 85, in _start_by_fork
    args.func(args, dag=self.dag)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 292, in task_run
    _run_task_by_selected_method(args, dag, ti)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
    _run_raw_task(args, ti)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 184, in _run_raw_task
    error_file=args.error_file,
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
    self._execute_task_with_callbacks(context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
    result = self._execute_task(context, self.task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
    result = execute_callable(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 367, in execute
    self.cli = self._get_cli()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 397, in _get_cli
    return APIClient(base_url=self.docker_url, version=self.api_version, tls=tls_config)
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
    'Error while fetching server API version: {0}'.format(e)
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
[2021-11-10, 04:14:47 UTC] {local_task_job.py:154} INFO - Task exited with return code 1
[2021-11-10, 04:14:47 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

In the second approach, I got this error:

*** Reading local file: /opt/airflow/logs/hello_world/echo_name/2021-11-10T04:39:06.388937+00:00/1.log
[2021-11-10, 04:39:07 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: hello_world.echo_name manual__2021-11-10T04:39:06.388937+00:00 [queued]>
[2021-11-10, 04:39:07 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: hello_world.echo_name manual__2021-11-10T04:39:06.388937+00:00 [queued]>
[2021-11-10, 04:39:07 UTC] {taskinstance.py:1241} INFO - 
--------------------------------------------------------------------------------
[2021-11-10, 04:39:07 UTC] {taskinstance.py:1242} INFO - Starting attempt 1 of 2
[2021-11-10, 04:39:07 UTC] {taskinstance.py:1243} INFO - 
--------------------------------------------------------------------------------
[2021-11-10, 04:39:07 UTC] {taskinstance.py:1262} INFO - Executing <Task(DockerOperator): echo_name> on 2021-11-10 04:39:06.388937+00:00
[2021-11-10, 04:39:07 UTC] {standard_task_runner.py:52} INFO - Started process 13507 to run task
[2021-11-10, 04:39:07 UTC] {standard_task_runner.py:76} INFO - Running: ['***', 'tasks', 'run', 'hello_world', 'echo_name', 'manual__2021-11-10T04:39:06.388937+00:00', '--job-id', '2', '--raw', '--subdir', 'DAGS_FOLDER/dag_creator.py', '--cfg-path', '/tmp/tmphw9dkz6v', '--error-file', '/tmp/tmp5glruzwd']
[2021-11-10, 04:39:07 UTC] {standard_task_runner.py:77} INFO - Job 2: Subtask echo_name
[2021-11-10, 04:39:08 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: hello_world.echo_name manual__2021-11-10T04:39:06.388937+00:00 [running]> on host webserver
[2021-11-10, 04:39:08 UTC] {taskinstance.py:1429} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=***
AIRFLOW_CTX_DAG_ID=hello_world
AIRFLOW_CTX_TASK_ID=echo_name
AIRFLOW_CTX_EXECUTION_DATE=2021-11-10T04:39:06.388937+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-11-10T04:39:06.388937+00:00
[2021-11-10, 04:39:08 UTC] {taskinstance.py:1703} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=2376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=2376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
    self._execute_task_with_callbacks(context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
    result = self._execute_task(context, self.task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
    result = execute_callable(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 367, in execute
    self.cli = self._get_cli()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 397, in _get_cli
    return APIClient(base_url=self.docker_url, version=self.api_version, tls=tls_config)
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
    'Error while fetching server API version: {0}'.format(e)
docker.errors.DockerException: Error while fetching server API version: HTTPConnectionPool(host='127.0.0.1', port=2376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2021-11-10, 04:39:08 UTC] {taskinstance.py:1280} INFO - Marking task as UP_FOR_RETRY. dag_id=hello_world, task_id=echo_name, execution_date=20211110T043906, start_date=20211110T043907, end_date=20211110T043908
[2021-11-10, 04:39:08 UTC] {standard_task_runner.py:91} ERROR - Failed to execute job 2 for task echo_name
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/airflow/.local/lib/python3.7/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=2376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=2376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/task/task_runner/standard_task_runner.py", line 85, in _start_by_fork
    args.func(args, dag=self.dag)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 292, in task_run
    _run_task_by_selected_method(args, dag, ti)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
    _run_raw_task(args, ti)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 184, in _run_raw_task
    error_file=args.error_file,
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
    self._execute_task_with_callbacks(context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
    result = self._execute_task(context, self.task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
    result = execute_callable(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 367, in execute
    self.cli = self._get_cli()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/docker/operators/docker.py", line 397, in _get_cli
    return APIClient(base_url=self.docker_url, version=self.api_version, tls=tls_config)
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/home/airflow/.local/lib/python3.7/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
    'Error while fetching server API version: {0}'.format(e)
docker.errors.DockerException: Error while fetching server API version: HTTPConnectionPool(host='127.0.0.1', port=2376): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f569c1bd410>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2021-11-10, 04:39:08 UTC] {local_task_job.py:154} INFO - Task exited with return code 1
[2021-11-10, 04:39:08 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

In both the cases, the error similar saying Error while fetching server API version: HTTPConnectionPool.

I am not able to debug further. If you can guide me on how I can fix this issue, that would be of great help!

Thanks.

What you expected to happen

No response

How to reproduce

Setup a ubuntu OS on an EC2 instance. Create an airflow container using airflow version = 2.2.1
Setup docker and docker-compose
Setup tcp access to docker daemon using following commands:

sudo mkdir -p /etc/systemd/system/docker.service.d
sudo touch /etc/systemd/system/docker.service.d/options.conf
echo -e """
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H unix:// -H tcp://0.0.0.0:2375
""" >> /etc/systemd/system/docker.service.d/options.conf
sudo systemctl daemon-reload
sudo systemctl restart docker

Create a simple docker operator in a dag and run it.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[kaustubhharapanahalli]

I have tried a few things through which I will try to summarize my findings:

1. To make sure I can run docker operator, I need to upload the /var/run/docker.sock as a volume to the target path of /var/run/docker.sock
2. I need to change the permissions of docker.sock to 777 which is not an ideal way to approach it in my understanding.

Making these changes allows the dockeroperator to run with the default docker url = unix://var/run/docker.sock but still doesn't work with a TCP hosted URL.

I tried this also on my personal laptop with ubuntu 20.04.3 LTS.

[ashb]

This is generally a problem with "how do I access the host's docker daemon/socket from within a container" and isn't anything particular to Airflow, so I'm going to close out this issue.

[kaustubhharapanahalli]

@ashb isn't TCP hosted url not being accessible an issue?

[tyomik-mnemonic]

How u fix it in the end? I have the same problem

[ashb]

@kaustubhharapanahalli Nothing about that is specific to running Airflow in a docker container -- it would apply equally as much if you ran ubuntu:latest for example.

[Leoduv]

I still think this thread deserves a response with leads on what is the recommended way of dealing with deploying a DockerOperator task on Kubernetes.

I have understood that you are meant to use the KubernetesPodOperator (https://stackoverflow.com/questions/65497763/how-can-i-run-tasks-using-the-dockeroperator-from-within-a-kubernetes-deployment).

But we want our developers to be able to test their tasks locally, so the KubernetesPodOperator is really something we want to avoid because this means asking everyone to setup a minikube for example.

In my case I think I'm going to end up writing a Operator wrapper that will use DockerOperator if Airflow is running locally, and KubernetesPodOperator if it's on the deployed version but I would really like some insight on how other people deal with this problem, or if I'm missing a simple tricks allowing to have code working locally and once deployed on k8s.

[potiuk]

This is nothing Airflow specific. Running docker workloads from within Kubernetes POD is a generic Kubernetes question and it has plenty of answers and depending on Your deployment needs, limitations etc. you might choose the solution that is best for you.

If you search in google/stack overflow, you will find plenty of tutorials on how to run Docker in Kubernetes. I just googled it and found this https://applatix.com/case-docker-docker-kubernetes-part-2/ - but I can neither endorse or confirm that this is good.

There is no "recommended" way from Airflow. Adding it in Airlfow docs or threads, where you have plenty of recommendations and discussions of pros/cons available how to do it for K8S makes very little sense from our side, because there is nothing Airflow-specific there - it adds no value and any recommendation that we can make will simply be a duplication of information existing elsewhere - and additionally the "elsewhere" information get updated and evolves over time, so we would have to keep up and monitor if the recommendations change/when there are new ways/when people find that there are other reasonss.

You are managing your deployments and you should choose the way of running docker from kubernetes in the way that fits your expectations, rules, etc.