Apache Airflow Essential Training

Course details

If you already know the basics of Apache Airflow but are eager to learn more, this course is for you. Certified Google cloud architect and data engineer Janani Ravi guides you through several essential features in Apache Airflow. After a quick review of what Airflow is all about, Janani dives into working with the TaskFlow API and using it to perform a variety of operations. She shows you how to integrate PostgreSQL in Airflow, as well as how to use sensors and work with hooks in Airflow. Level up your use of Apache Airflow with this essential course.

Concepts

0. Introduction

01 - Apache Airflow essential training

1. Working with the TaskFlow API

02 - Prerequisites
03 - Quick Airflow setup overview
04 - DAG using PythonOperators
05 - DAG using TaskFlow
06 - Passing data using XCom with operators
07 - Passing data using the TaskFlow API
08 - Tasks with multiple outputs
09 - Passing multiple outputs in TaskFlow

2. Performing Operations Using TaskFlow

10 - Branching using operators
11 - Branching using the TaskFlow API
12 - Complete branching pipeline using TaskFlow
13 - Interoperating and passing data between operators and TaskFlow
14 - Performing SQL operations with TaskFlow
15 - Data transformation and storage using TaskFlow

3. Integrating PostgreSQL in Airflow

16 - Installing PostgreSQL on macOS
17 - Installing PostgreSQL on WSL
18 - Connecting to PostgreSQL
19 - Using the PostgreSQL operator
20 - Performing PostgreSQL insert operations
21 - Performing PostgreSQL join operations
22 - A complete end-to-end pipeline with PostgreSQL
23 - Configuring PostgreSQL as a metadata database and using the LocalExecutor

4. Using Sensors in Airflow

24 - Introducing the FileSensor
25 - Creating a pipeline using the FileSensor
26 - Running DAGs with a FileSensor
27 - Creating and running a pipeline with SQLSensors

5. Working with Hooks in Airflow

28 - Setting up for a PostgreSQL pipeline with hooks
29 - Creating and running a pipeline with PostgresSQL hooks
30 - Setting up access to Amazon S3 buckets
31 - Setting up a connection to Amazon S3 buckets
32 - Creating and running a pipeline with an S3 hook

Conclusion

33 - Summary and next steps

Apache Airflow Essential Training

Authors

Janani Ravi