Apache Flink: Batch Mode Data Engineering
1h 7mAdvanced2020-02-20
Authors
Kumaran Ponnambalam
Working with data for 20+ years
Course details
Data engineering is the foundation for enabling analytics and data science applications in the world of big data. It requires building scalable data processing pipelines and delivering them in short time frames. Apache Flink, the powerful and popular stream-processing platform, was designed to help you achieve these goals. In this course, join Kumaran Ponnambalam as he focuses on how to build batch mode data pipelines with Apache Flink. Kumaran kicks off the course by reviewing the features and architecture of Apache Flink. He then takes a deeper look at the DataSet API and explores various capabilities available for transforming, aggregating, and combining data. To wrap up the course, he presents a use case project that allows you to leverage your new skills.
Topics include:
- The architecture of Apache Flink
- Features of the DataSet API
- Using POJO classes for DataSet typing
- Working with joins in Flink
- Using MySQL with Flink
- Using broadcast variables to share and collect data
Topics include:
- The architecture of Apache Flink
- Features of the DataSet API
- Using POJO classes for DataSet typing
- Working with joins in Flink
- Using MySQL with Flink
- Using broadcast variables to share and collect data
Skills covered
FlinkApacheData EngineeringData ScienceDeep Dive (X:Y)
Concepts
0. Introduction
- 01 - Batch mode engineering
1. Apache Flink
- 02 - What is Apache Flink
- 03 - Apache Flink features
- 04 - Architecture of Apache Flink
- 05 - Flink program structure
- 06 - Flink execution flow
2. Setting Up Flink
- 07 - Installing Flink standalone
- 08 - Creating a Flink project
- 09 - Build a sample Flink program
- 10 - Running jobs on the cluster
- 11 - Using the Flink web interface
- 12 - Setting up the exercise files
3. Dataset API
- 13 - DataSet API concepts
- 14 - Reading a CSV File
- 15 - Using Map
- 16 - Using FlatMap
- 17 - Using filters
- 18 - Using aggregates
- 19 - Using Reduce
4. Advanced Capabilities
- 20 - Using POJO classes
- 21 - Join operations
- 22 - Using MySQL with Flink
- 23 - Using broadcast variables
5. Use Case Project
- 24 - Problem definition
- 25 - Computing total score
- 26 - Printing scores for physics
- 27 - Computing average scores across subjects
- 28 - Find the top student for each subject
Conclusion
- 29 - Next steps
Related courses
- Apache Flink: Real-Time Data Engineering
- Apache Flink: Exploratory Data Analytics with SQL
- Stream Processing Patterns in Apache Flink
- Introduction to Spark SQL and DataFrames
- Spark for Machine Learning & AI
- Hands-On Data Science: 1 Analyzing Employee Data with SQL
- Hands-On Data Science: 2 Sales Dashboard with Tableau
- Hands-On Data Science: 3 Sales Analysis in Python
Related learn paths
- Advance Your Data Skills in Apache Spark
- Master Data Engineering
- Explore a Career in Data Engineering
- Moving from Data Scientist to Data Analyst
- Data Engineering Foundations Professional Certificate by Astronomer
- Advance Your Data Engineering Skills
- Prepare for the Databricks Certified Data Engineer Associate Certification
- Data Engineering Hands-On Practice