Airflow tutorial

This is the code for Apache Airflow Tutorials playlist by Tuan Vu on Youtube

Contents

Part	Title	Git Tag
1	Introduction to Apache Airflow (blog post)	v0.1
2	Set up airflow environment with docker (blog post)	v0.2
3	Set up airflow environment using Google Cloud Composer (blog post)	N/A
4	Writing your first pipeline (blog post)	N/A
5	Airflow concept (blog post)	N/A
6	Build a data pipeline using Google Cloud Bigquery (blog post)	v0.6
7	Airflow variables (blog post)	v0.7

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

• Clone this repo
• Install the prerequisites
• Run the service
• Check http://localhost:8080
• Done! 🎉

Prerequisites

• Install Docker
• Install Docker Compose
• Following the Airflow release from Python Package Index

Usage

Run the web service with docker

docker-compose up -d
# Build the image
# docker-compose up -d --build

Check http://localhost:8080/

• docker-compose logs - Displays log output
• docker-compose ps - List containers
• docker-compose down - Stop containers

Other commands

If you want to run airflow sub-commands, you can do so like this:

• docker-compose run --rm webserver airflow list_dags - List dags
• docker-compose run --rm webserver airflow test [DAG_ID] [TASK_ID] [EXECUTION_DATE] - Test specific task

If you want to run/test python script, you can do so like this:

• docker-compose run --rm webserver python /usr/local/airflow/dags/[PYTHON-FILE].py - Test python script

Connect to database

If you want to use Ad hoc query, make sure you've configured connections: Go to Admin -> Connections and Edit "postgres_default" set this values:

• Host : postgres
• Schema : airflow
• Login : airflow
• Password : airflow

Credits

• Apache Airflow
• docker-airflow