4.33 out of 5
4.33
248 reviews on Udemy

The Complete Hands-On Course to Master Apache Airflow

Learn to author, schedule and monitor data pipelines through practical examples using Apache Airflow
Instructor:
Marc Lamberti
1,234 students enrolled
English [Auto-generated]
Create plugins to add functionalities to Apache Airflow.
Using Docker with Airflow and different executors
Master core functionalities such as DAGs, Operators, Tasks, Workflows, etc
Understand and apply advanced concepts of Apache Airflow such as XCOMs, Branching and SubDAGs.
The difference between Sequential, Local and Celery Executors, how do they work and how can you use them.
Use Apache Airflow in a Big Data ecosystem with Hive, PostgreSQL, Elasticsearch etc.
Install and configure Apache Airflow
Think, answer and implement solutions using Airflow to real data processing problems

Apache Airflow is an open-sourceĀ  platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have.

In this course you are going to learn how to master Apache Airflow through theory and pratical video courses. Starting from very basic notions such as, what is Airflow and how it works, we will dive into advanced concepts such as, how to create plugins and make real dynamic pipelines.

Course Introduction

1
Important Prerequisites
2
Course Objectives
3
Course Structure
4
Who I am
5
Development Environment

First Approach to Airflow

1
Introduction
2
Why use Airflow?
3
What is Airflow?
4
How Airflow works?
5
[Practice] Installing Airflow
6
[Practice] Quick Tour of Airflow UI
7
[Practice] Quick Tour of Airflow CLI
8
Quiz Time!
9
Recap

Coding Your First Data Pipeline with Airflow

1
Introduction
2
What is a DAG?
3
[Practice] Time to code your first DAG
4
What is an Operator?

- Operators

- Sensors

- Transfer

5
[Practice] Let's use Operators
6
Operator Relationships and Bitshift Composition
7
[Practice] Adding dependencies
8
How the Scheduler Works?
9
[Practice] A Quick Play With Backfill and Catchup
10
What is a Workflow?
11
Quiz Time!
12
Recap

Databases and Executors

1
Introduction
2
Sequential Executor with SQLite
3
Local Executor with PostgreSQL
4
[Practice] Configure a DAG with Local Executor and PostgreSQL
5
Celery Executor with PostgreSQL and RabbitMQ
6
[Practice] Configure a DAG with Celery Executor, PostgreSQL and RabbitMQ
7
Quiz Time!
8
Recap

Implementing Advanced Concepts in Airflow

1
Introduction
2
Minimising Repetitive Patterns With SubDAGs
3
[Practice] Minimising a DAG with SubDAGs
4
How to Interact With External Sources Using Hooks
5
[Practice] Getting Results From PostgreSQL Using Hooks
6
How to Share Data Between Your Tasks With XCOMs
7
[Practice] Sharing Your First Messages Using XCOMs
8
How to Execute Tasks According To Criteria Using Branching
9
[Practice] Make Your First Conditional Task Using Branching
10
Control Your Tasks With SLAs
11
[Practice] Defining a SLA in a DAG
12
Quiz Time!
13
Recap

Creating Airflow Plugins with Elasticsearch and PostgreSQL

1
Introduction
2
Adding Functionalities to Apache Airflow
3
[Practice] Creating a Hook to Interact With Elasticsearch
4
[Practice] Creating a Transfer Operator PostgresqlToElasticsearch
5
[Practice] Adding a View to Apache Airflow UI
6
Quiz Time!
7
Recap

Using Apache Airflow With Docker

1
Introduction
2
Quick Reminder About Docker
3
Installing Docker on the VM
4
[Practice] Building And Running Your First Airflow Docker Image
5
[Practice] Running Airflow Using Sequential Executor With Docker
6
[Practice] Running Airflow Using Local Executor With Docker
7
[Practice] Scaling Airflow Using Celery Executor With Docker
8
Quiz Time!
9
Recap

BONUS - APPENDIX

1
[BLOG POST] How to use the DockerOperator with Templating and Apache Spark
You can view and review the lecture materials indefinitely, like an on-demand channel.
Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don't have an internet connection, some instructors also let their students download course lectures. That's up to the instructor though, so make sure you get on their good side!
4.3
4.3 out of 5
248 Ratings

Detailed Rating

Stars 5
104
Stars 4
102
Stars 3
40
Stars 2
4
Stars 1
0
5c26f841f84953b43db05b02f01d7340
30-Day Money-Back Guarantee

Includes

3 hours on-demand video
1 article
Full lifetime access
Access on mobile and TV
Certificate of Completion