Skip to content

A simple docker-compose ready repository for using Apache Airflow.

License

Notifications You must be signed in to change notification settings

frankcash/airflow_compose

Folders and files

NameName
Last commit message
Last commit date
Oct 22, 2019
Aug 31, 2021
Oct 14, 2021
Aug 31, 2021
Mar 26, 2019
Mar 26, 2019
Mar 27, 2019
Oct 14, 2021
Oct 21, 2019
Oct 29, 2019
Aug 31, 2021
Apr 24, 2019
Mar 26, 2019
Oct 14, 2021

Repository files navigation

airflow_compose

A project for quickly spinning up Airflow in docker-compose.

Airflow is a tool for maintaining ETLs and workflows. If you do not know a lot about it please check out the official docs.

Running with Docker Compose

docker-compose up --build

Running Locally

Getting dependencies installed

cd /path/to/my/airflow_compose/
virtualenv -p `which python3` venv
source venv/bin/activate

From inside the virtualenv:

export SLUGIFY_USES_TEXT_UNIDECODE=yes
pip install -r requirements.txt
airflow initdb

Running the web server

cd /path/to/my/airflow/workspace
source venv/bin/activate

export AIRFLOW_HOME=`pwd`/airflow_home
airflow webserver

Running the scheduler

cd /path/to/my/airflow/workspace
export AIRFLOW_HOME=`pwd`/airflow_home

source venv/bin/activate
airflow scheduler

Adding database

Go to the configuration tab underneath admin to add a database connection.

Resources for Learning Apache Airflow

Technical Reads

Podcasts