DockerOperator not working from containerized Airflow not recognizing /var/run/docker.sock
#16803
Comments
|
Thanks for opening your first issue here! Be sure to follow the issue template! |
ExecutionSpeed
changed the title
/var/run/docker.sock
|
Can you also try to mount /tmp folder into the container ? I am not telling it's the final solution yet, but it's likely to help or at least validate a hypothesis i have. It looks like the apis used by the DockerOperator create and use a temporary file to pass something between the client and engine. |
|
As a workaround, you can use a sidecar container with |
I think that's not it. The access right are good and confirmed they work with docker command (see the description). The real reason is described in #16808 - as I suspected, DockerOperator creates a folder in "/tmp" and tries to bind-mount it to the docker container it runs - which does not work because the "/tmp" file is created inside airflow container, not in the host. This also prevents to use DockerOperator from running with remote engine (because the host folder cannot be bind-mounted there). The temporary solution for docker-compose I proposed (mounting /tmp directory) should work in this case, but we have to fix it differently - DockerOperator should create a separate volume in docker and copy all the files there and mount this volume in the Container it runs, rather than mount the "/tmp" host folder. This should solve both problem - running airflow as docker container in local docker-compose and running airflow with remote docker engine. |
potiuk
added
the
priority:medium
@potiuk This ended up working with mounting Docker Compose: # Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
#
# WARNING: This configuration is for local development. Do not use it in a production deployment.
#
# This configuration supports basic configuration using environment variables or an .env file
# The following variables are supported:
#
# AIRFLOW_IMAGE_NAME - Docker image name used to run Airflow.
# Default: apache/airflow:master-python3.8
# AIRFLOW_UID - User ID in Airflow containers
# Default: 50000
# AIRFLOW_GID - Group ID in Airflow containers
# Default: 50000
#
# Those configurations are useful mostly in case of standalone testing/running Airflow in test/try-out mode
#
# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account (if requested).
# Default: airflow
# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account (if requested).
# Default: airflow
# _PIP_ADDITIONAL_REQUIREMENTS - Additional PIP requirements to add when starting all containers.
# Default: ''
#
# Feel free to modify this file to suit your needs.
---
version: '3'
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.1-python3.8}
environment: &airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: 300 # Just to have a fast load in the front-end. Do not use it in production with those configurations.
AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
AIRFLOW__CORE__ENABLE_XCOM_PICKLING: 'true' # "_run_image of the DockerOperator returns now a python string, not a byte string" Ref: https://github.com/apache/airflow/issues/13487
_PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
- '/var/run/docker.sock:/var/run/docker.sock' # We will pass the Docker Deamon as a volume to allow the webserver containers start docker images. Ref: https://stackoverflow.com/q/51342810/7024760
- '/tmp:/tmp'
user: '${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}'
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
services:
postgres:
image: postgres:13
environment:
POSTGRES_USER: airflow
POSTGRES_PASSWORD: airflow
POSTGRES_DB: airflow
volumes:
- postgres-db-volume:/var/lib/postgresql/data
healthcheck:
test: ['CMD', 'pg_isready', '-U', 'airflow']
interval: 5s
retries: 5
restart: always
redis:
image: redis:latest
ports:
- 6379:6379
healthcheck:
test: ['CMD', 'redis-cli', 'ping']
interval: 5s
timeout: 30s
retries: 50
restart: always
airflow-webserver:
<<: *airflow-common
command: webserver
ports:
- 80:8080
healthcheck:
test: ['CMD', 'curl', '--fail', 'http://localhost:80/health']
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-scheduler:
<<: *airflow-common
command: scheduler
healthcheck:
test:
[
'CMD-SHELL',
'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"',
]
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-worker:
<<: *airflow-common
command: celery worker
healthcheck:
test:
- 'CMD-SHELL'
- 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-init:
<<: *airflow-common
command: version
environment:
<<: *airflow-common-env
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
flower:
<<: *airflow-common
command: celery flower
ports:
- 5555:5555
healthcheck:
test: ['CMD', 'curl', '--fail', 'http://localhost:5555/']
interval: 10s
timeout: 10s
retries: 5
restart: always
volumes:
postgres-db-volume: |
|
Yeah. Thought so. We will likely have to fix it. |
|
@potiuk Great, thank you for the quick response and pointing me to the temporary workaround. |
|
I looked at this and turned the temp file mounting into optional feature that you might disable: see #16932 |
|
Seems it can also be workarounded by downgrading to |
|
Hi, I got the same issue when updating from 2.1.0 to 2.1.1. |
The DockerOperator by default mounts temporary folder to inside the container in order to allow to store files bigger than default size of disk for the container, however this did not work when remote Docker engine or Docker-In-Docker solution was used. This worked before the apache#15843 change, because the /tmp has been ignored, however when we change to "Mounts", the "/tmp" mount fails when using remote docker engine. This PR adds parameter that allows to disable this temporary directory mounting (and adds a note that it can be replaced with mounting existing volumes). Also it prints a warning if the directory cannot be mounted and attempts to re-run such failed attempt without mounting the temporary directory which brings back backwards-compatible behaviour for remote engines and docker-in-docker. Fixes: apache#16803 Fixes: apache#16806
…16932) * Adds option to disable mounting temporary folder in DockerOperator The DockerOperator by default mounts temporary folder to inside the container in order to allow to store files bigger than default size of disk for the container, however this did not work when remote Docker engine or Docker-In-Docker solution was used. This worked before the #15843 change, because the /tmp has been ignored, however when we change to "Mounts", the "/tmp" mount fails when using remote docker engine. This PR adds parameter that allows to disable this temporary directory mounting (and adds a note that it can be replaced with mounting existing volumes). Also it prints a warning if the directory cannot be mounted and attempts to re-run such failed attempt without mounting the temporary directory which brings back backwards-compatible behaviour for remote engines and docker-in-docker. Fixes: #16803 Fixes: #16806
|
sudo chmod 666 /var/run/docker.sock |
Thank you. It's work! |
|
Is this solution for running jobs using docker operator available in new releases? |
Apache Airflow version: 2.1.1
Docker Image:
apache/airflow:2.1.1-python3.8Kubernetes version (if you are using kubernetes) (use
kubectl version): Not running on k8s.Environment:
uname -a):Linux airflow 5.11.0-22-generic #23-Ubuntu SMP Thu Jun 17 00:34:23 UTC 2021 x86_64 x86_64 x86_64 GNU/LinuxWhat happened:
What you expected to happen:
How to reproduce it:
Running with the
DockerOperatorcauses the following error:I have even tried with Docker API v1.41 (latest) and same issue. I have bound the
/var/run/docker.sockas a bind mount into the container.Docker Compose:
DAG:
Anything else we need to know: Problem happens any time
DockerOperatoris being used. Not entirely sure why this happening given that the docker sock is fully permissive (has777) and is bind mounted into the container. When I test via docker-py client in Python shell underairflowuser inside the container, it works perfectly fine to run all docker-py operations like listing running containers and such confirming the mounted docker UNIX socket is available and working. However, even with thedocker_urlpointing to the docker socket in the above DAG, I am getting this error thrown in above trace.For whatever strange reason the logs say it's trying to connect over
http+docker://localhost/v1.30/containers/createinstead of the UNIX docker socket that's bind mounted and explicitly specified viadocker_url.The text was updated successfully, but these errors were encountered: