Initial working commit

Author: jksinghpro

.idea/checkstyle-idea.xml

@@ -0,0 +1,86 @@
+# VERSION 1.9.0
+# AUTHOR: Jaskaran singh
+# DESCRIPTION: Basic Airflow container
+# BUILD: docker build --rm -t jksingh/docker-airflow .
+
+FROM python:3.6-slim
+LABEL maintainer="JKsingh_"
+
+# Never prompts the user for choices on installation/configuration of packages
+ENV DEBIAN_FRONTEND noninteractive
+ENV TERM linux
+
+# Airflow
+ARG AIRFLOW_VERSION=1.9.0
+ARG AIRFLOW_HOME=/usr/local/airflow
+
+# Define en_US.
+ENV LANGUAGE en_US.UTF-8
+ENV LANG en_US.UTF-8
+ENV LC_ALL en_US.UTF-8
+ENV LC_CTYPE en_US.UTF-8
+ENV LC_MESSAGES en_US.UTF-8
+
+RUN set -ex \
+    && buildDeps=' \
+        python3-dev \
+        libkrb5-dev \
+        libsasl2-dev \
+        libssl-dev \
+        libffi-dev \
+        build-essential \
+        libblas-dev \
+        liblapack-dev \
+        libpq-dev \
+        git \
+    ' \
+    && apt-get update -yqq \
+    && apt-get upgrade -yqq \
+    && apt-get install -yqq --no-install-recommends \
+        $buildDeps \
+        python3-pip \
+        python3-requests \
+        mysql-client \
+        mysql-server \
+        default-libmysqlclient-dev \
+        apt-utils \
+        curl \
+        rsync \
+        netcat \
+        locales \
+    && sed -i 's/^# en_US.UTF-8 UTF-8$/en_US.UTF-8 UTF-8/g' /etc/locale.gen \
+    && locale-gen \
+    && update-locale LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 \
+    && useradd -ms /bin/bash -d ${AIRFLOW_HOME} airflow \
+    && pip install -U pip setuptools wheel \
+    && pip install Cython \
+    && pip install pytz \
+    && pip install pyOpenSSL \
+    && pip install ndg-httpsclient \
+    && pip install pyasn1 \
+    && pip install apache-airflow[crypto,celery,postgres,hive,jdbc,mysql]==$AIRFLOW_VERSION \
+    && pip install celery[redis]==4.1.1 \
+    && apt-get purge --auto-remove -yqq $buildDeps \
+    && apt-get autoremove -yqq --purge \
+    && apt-get clean \
+    && rm -rf \
+        /var/lib/apt/lists/* \
+        /tmp/* \
+        /var/tmp/* \
+        /usr/share/man \
+        /usr/share/doc \
+        /usr/share/doc-base
+
+COPY script/entrypoint.sh /entrypoint.sh
+COPY config/airflow.cfg ${AIRFLOW_HOME}/airflow.cfg
+
+RUN chmod +x /entrypoint.sh
+
+RUN chown -R airflow: ${AIRFLOW_HOME}
+
+EXPOSE 8080 5555 8793
+
+USER airflow
+WORKDIR ${AIRFLOW_HOME}
+ENTRYPOINT ["/entrypoint.sh"]
+CMD ["webserver"] # set default arg for entrypoint

Dockerfile

@@ -0,0 +1,13 @@
+Copyright [2018] Jaskaran singh
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

LICENSE

@@ -0,0 +1,54 @@
+# Docker-airflow with MySql as backend
+
+This repository contains **Dockerfile** of [apache-airflow](https://github.com/apache/incubator-airflow) .
+
+## Informations
+
+* Based on Python (3.6-slim) official Image [python:3.6-slim](https://hub.docker.com/_/python/) and uses the official [MySql](https://hub.docker.com/_/mysql/) as backend and [Redis](https://hub.docker.com/_/redis/) as queue
+* Install [Docker](https://www.docker.com/)
+* Install [Docker Compose](https://docs.docker.com/compose/install/)
+* Following the Airflow release from [Python Package Index](https://pypi.python.org/pypi/apache-airflow)
+
+
+## Build
+
+For Building the docker container for airflow you first need to build the docker image using the dockerfile added in the repository.For adding [Extra Packages](https://airflow.incubator.apache.org/installation.html#extra-package) edit the docker file and then build.
+
+    docker build --rm -t jksingh/docker-airflow .
+
+Don't forget to update the airflow images in the docker-compose files to jksingh/docker-airflow:latest.
+
+## Usage
+
+By default, docker-airflow runs Airflow with **SequentialExecutor** :
+
+    docker run -d -p 8080:8080 jksingh/docker-airflow webserver
+
+If you want to run another executor, use the other docker-compose files provided in this repository.
+
+For **LocalExecutor** :
+
+    docker-compose -f docker-compose-mysql-local.yml up -d
+
+For **CeleryExecutor** :
+
+    docker-compose -f docker-compose-mysql-celery.yml up -d
+
+NB : If you want to have DAGs example loaded (default=False), you've to set the following environment variable :
+
+`LOAD_EX=n`
+
+    docker run -d -p 8080:8080 -e LOAD_EX=y jksingh/docker-airflow
+
+
+## UI Links
+
+- Airflow: [localhost:8080](http://localhost:8080/)
+- Flower: [localhost:5555](http://localhost:5555/)
+
+
+# Issues
+
+- Issue Tracker: https://github.com/jksinghpro/docker-airflow/issues
+
+