Airflow 3: Fundamentals Before MLOps

A hands-on course based on Apache Airflow 3.3.1. Work through the lessons in order, run every example, and modify the labs instead of only reading them.

Step 0: Install Airflow

Complete this setup before Lesson 1. The commands below create an isolated Python environment inside this workspace and install Airflow with its official constraints file. Run the whole block from the first line. The explicit .venv/bin/... paths prevent python: command not found and PEP 668 errors even when the virtual environment is not active yet.

cd ~/airflow_3_study
python3.12 -m venv .venv
.venv/bin/python -m pip install --upgrade pip

export AIRFLOW_HOME="$PWD/.airflow"
export AIRFLOW_VERSION=3.3.1
export PYTHON_VERSION="$(.venv/bin/python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')"
export CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

.venv/bin/python -m pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
mkdir -p "$AIRFLOW_HOME/dags"
.venv/bin/airflow version
.venv/bin/airflow db migrate
source .venv/bin/activate

Every time you open a new terminal, restore the environment before running an Airflow command:

cd ~/airflow_3_study
source .venv/bin/activate
export AIRFLOW_HOME="$PWD/.airflow"
command -v python
command -v airflow
airflow version

If airflow: command not found appears, the virtual environment is not active. The two command -v lines should point inside ~/airflow_3_study/.venv/bin/. Do not install Airflow globally, use sudo pip, or pass --break-system-packages.

WSL: airflow info Hangs

airflow info probes external tools, including gcloud, without a timeout. In WSL, a Windows gcloud executable inherited through /mnt/c can hang. This does not indicate a broken Airflow installation. Run the diagnostic with a Linux-only PATH:

SAFE_PATH="$VIRTUAL_ENV/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
PATH="$SAFE_PATH" airflow info --anonymize

This PATH change applies only to that command. Normal Airflow commands can use your regular PATH.

Learning Path

1. Fundamentals and architecture
2. Environment and operations
3. DAG authoring
4. Scheduling and time
5. Reliability
6. Data communication
7. Scaling and concurrency
8. Observability and debugging
9. Testing and quality
10. Security and deployment

Conventions

• The examples use Linux, Python 3.12, and Airflow 3.3.1.
• Complete Step 0 before running commands from any lesson.
• With a local installation, run commands such as airflow dags list.
• With the official Docker Compose setup, replace airflow with docker compose run --rm airflow-worker airflow.
• Save the Python examples in the DAG directory described in Lesson 2.
• Dates and times in the examples use UTC unless otherwise stated.
• The official Compose setup is suitable for learning, not production.

Suggested Pace

Allow two to four hours per lesson. Before moving on, complete the lab, exercises, and checklist for the lesson. Finally, implement the capstone project in Lesson 10; it will provide a bridge to MLOps pipelines.

Official References

• Airflow documentation
• Airflow 3 public interface
• CLI reference
• Task SDK

The stable documentation can change. These course examples were written for 3.3.1; review the release notes before upgrading.