Apache Airflow 101

Introduction

We are going to review Apache Airflow features and use it to develop and monitor workflows. In order to examine source codes, you first need to have an instance of airflow:

just dev up

# username: airflow
# password: airflow

You can maintain DAGs in /dags and have access to its UI at 127.0.0.1:8080.

Using it with Docker Operator

For having Docker operator in Airflow we need to pass Docker's socket into docker:

/var/run/docker.sock

This socket belongs to root user and docker group, but we don't have either of these in Airflow container so, we need to give read and write access to others for this socket.

sudo chmod o+rw /var/run/docker.sock