4.13 out of 5
4.13
90 reviews on Udemy

Apache Airflow : Develop Data Pipelining & Workflow

Learn to author data workflows. Build, schedule and monitor Data Pipelines using Apache Airflow in Python
Instructor:
DevTechie Interactive
390 students enrolled
English [Auto-generated]
Airflow, Developing Data Pipeline, Python, Pandas
Hands on data pipeline development and comparison with other technology
Installation, Configuration of Airflow
DAG's, Creating workflow, Operators, Tasks, dependency management, Hooks, Connections
Different Executors like Local, Celery and Sequential and differences
Airflow Architecture in detail
Advanced concepts like Xcoms, Branching, Variables and DAG Chaining
Authentication and Log storage to S3
Airflow on Docker

Data engineering is a field that can be thought as a superset of business intelligence and data warehousing which brings more elements from software engineering. The reason data engineering exists today is because companies have massive treasure troves of data, but to provide value the data must be extracted. Data engineering provides the toolbox and is how we make sense of that data quickly and effectively.

When it comes to managing data collection, munging and consumption, data pipeline frameworks play a significant role and with the help of Apache Airflow, task of creating data pipeline is not only easy but its actually fun. Originated from AirBnb, Airflow soon became part of the very core of their tech stack.

The data infrastructure ecosystem has yet to show any sign of converging into something more manageable. It seems like we’re still in a huge phase of expansion where every new day bring new distributed database, new frameworks, new libraries and new teammates. As these systems get more complicated and evolve rapidly, it becomes even more important to have something like Apache Airflow that brings everything together in a sane place where every little piece of the puzzle can be orchestrated properly with sane APIs.

So in this course we will be learning as how to reach feature completeness with this amazing orchestration tool called Apache Airflow. You will not only learn to setup the environment but also learn how to create workflow pipeline with real world example so don’t wait and sign-up today and get started.

Looking forward to seeing you in this course!

Introduction

1
Course Introduction
2
Who this course is for?
3
Introducing Dags & Pipeline
4
What is Airflow?
5
Why Use Airflow?
6
Comparison between Luigi, Azkaban, Oozie and others
7
Quiz Time

Architecture

1
Airflow Architecture
2
Quiz Time

Airflow Installation

1
Installing Airflow
2
Setting environment variables for Airflow
3
Hands on: setting environment variable and starting web server
4
Hands on : setting encryption to secure connection secrets
5
Quiz Time

Airflow Configuration

1
Airflow Configuration Overview
2
Configuration options: ORM Configuration
3
Configuration Option: Maximum Active Runs Explained
4
Configuration Option: Maximum Active Runs Explained Continued
5
Configuration Options: Additional Configuration Settings
6
Quiz Time

Developing Your First Data Pipeline

1
Problem Statement
2
Hands on: Project Setup
3
Hands on: Data Retrieval from File System
4
Hands on: Merging DataFrames
5
Hands on: Aggregation Using Pandas
6
Hands on: Database Connectivity
7
Hands on: Creating Dags
8
Summary

DAG Chaining

1
Dynamic Flow: Pattern -1 and Pattern - 2 Explained
2
Dynamic Flow: Pattern - 3 Overview
3
Dynamic Flow: Pattern - 4 Discussed
4
Dynamic Flow: Exploring Pattern - 5

Airflow Components

1
Hands on: What are Airflow Hooks?
2
Hands on: Exploring Task Branching
3
Hands on: Passing and Modifying Variables
4
Hands on: Xcom in Depth

Authentication

1
Airflow behavior with Python 2 vs Python 3
2
Setting Authentication
3
Setting Authentication Demo
4
Hands On : Creating user with Python 3
5
Creating user with Python2

Airflow: Log storage to Cloud

1
Writing Logs to S3
2
Code: How to store Logs to S3
3
Demo: Logs publishing to S3

Airflow on Docker

1
Intro to setting airflow in docker
2
Installing Docker
3
Building docker container for Airflow
4
Building containers with Local and Celery Executors
5
Dockerfile Explained
6
Assignment
7
Further on Airflow Containers

REST API's

1
Test REST API Server
2
REST API Continued
3
REST API

SLA's

1
What is an SLA?
2
SLA Implementation
3
SLA Demo
4
(Optional) SLA - Implementation with Airflow explained

Airflow Command Line

1
Useful Command Lines for Airflow

BONUS

1
BONUS: Coupon code for any course authored by us.
You can view and review the lecture materials indefinitely, like an on-demand channel.
Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don't have an internet connection, some instructors also let their students download course lectures. That's up to the instructor though, so make sure you get on their good side!
4.1
4.1 out of 5
90 Ratings

Detailed Rating

Stars 5
53
Stars 4
11
Stars 3
16
Stars 2
4
Stars 1
7
1853f99d6bbbdbb455f37e1049985542
30-Day Money-Back Guarantee

Includes

3 hours on-demand video
Full lifetime access
Access on mobile and TV
Certificate of Completion