Advanced Google Dataflow

Course details

Once you’ve mastered the basics of Google Dataflow, you may wonder what else you can do with it. This course focuses on more advanced uses for Apache Beam and Dataflow. After providing a quick overview, instructor Kishan Iyer introduces you to stream processing with Dataflow. He shows you how to publish messages to Pub/Sub, access messages published there, and set up a pipeline that works with Pub/Sub and BigQuery. Kishan also goes over how to read and monitor messages on Pub/Sub. Next, he walks you through several different windowing operations and join operations in Dataflow. Kishan covers code-free stream processing, preparing to use Dataflow SQL, executing jobs with Dataflow SQL, and creating parametrized queries with Dataflow SQL. Plus, he discusses how to launch streaming jobs with a template and verify results for a template job.

Concepts

0. Introduction

01 - Quick technology overview

1. Quick Overview of Apache Beam and Dataflow

02 - Stream processing with Apache Beam
03 - Enabling Google Cloud APIs for Dataflow apps
04 - Setting up service account credentials
05 - Creating an Apache Beam project with Maven
06 - Reading data from Google Cloud storage
07 - Printing elements to the console
08 - Running a batch processing app

2. Introduction to Stream Processing with Dataflow

09 - Transforms on streaming data
10 - Processing streaming data
11 - Performing an aggregation on streaming data
12 - Defining a window for aggregations
13 - Using a custom aggregator

3. Streaming Messages with Pub Sub

14 - Publishing messages to Pub Sub
15 - Accessing messages from Pub Sub
16 - Setting up Pub Sub and BigQuery
17 - Reading messages from Pub Sub
18 - Monitoring messages on Pub Sub

4. Windowing Operations in Dataflow

19 - Window operations
20 - Writing results to BigQuery
21 - Verifying the pipeline output
22 - Computing an average over a window
23 - Resetting the window size
24 - Implementing sliding windows
25 - Updating running jobs

5. Join Operations in Dataflow

26 - Joining bounded PCollections
27 - Implementing stream-stream joins
28 - Feeding inputs for streaming joins

6. Using the Dataflow SQL Service

29 - Code-free stream processing
30 - Preparing to use Dataflow SQL
31 - Executing jobs with Dataflow SQL
32 - Parametrized queries with Dataflow SQL

7. Using Dataflow Streaming Templates

33 - Launching streaming jobs with a template
34 - Verifying results for a template job

Conclusion

35 - Summary and next steps