High-Performance PySpark: Advanced Strategies for Optimal Data Processing

Course details

Master the art of efficient data processing with this advanced PySpark course designed for data engineers. Instructor Ameena Ansari shows you the essentials of optimizing the data cleaning process and defining schemas to streamline ingestion at scale. Explore various data formats and compression techniques to ensure seamless performance, even with massive datasets. By the end of this course, you'll have the tools and skills you need to transform and ingest high-quality data using PySpark pipelines that are both scalable and efficient.

Learning objectives
Master data cleaning techniques for handling missing values, outlier detection, data normalization, and data transformation.
Structure data schemas for improved performance and scalability.
Work with a variety of data formats such as Parquet, ORC, Avro, JSON, and CSV.
Leverage compression techniques using Gzip, Snappy, and LZO.
Minimize data shuffling and skew, optimizing joins, aggregations, and repartitioning.

Concepts

0. Introduction

01 - High-performance data engineering with PySpark

1. Introduction to High-Performance Data Cleaning in PySpark

02 - What is data cleaning
03 - Common data quality issues
04 - Challenges in data cleaning
05 - Why PySpark for data cleaning

2. Data Cleaning Techniques with PySpark

06 - Working with GitHub Codespaces
07 - Data quality in PySpark - Identifying issues and effective cleaning techniques
08 - Detecting and handling null values in PySpark
09 - Techniques to identify and eliminate inconsistent data in PySpark
10 - Splitting combined data columns in PySpark

3. Structuring Data Schemas

11 - Importance of schema design in data engineering
12 - Using PySpark for schema enforcement and validation
13 - Schema management in data lakes and warehouses

4. Data Formats and Compression Techniques

14 - Introduction to data formats - Understanding JSON and CSV
15 - Exploring JSON
16 - Exploring Avro
17 - How Avro handles serialization and deserialization
18 - Avro schema evolution - Managing changes in data structures
19 - Avro pros and cons
20 - Understanding ORC - Optimized row columnar storage
21 - ORC pros and cons
22 - Parquet - The go-to columnar format for high-performance analytics
23 - Compression algorithms in Spark - Comparing Zstd, Snappy, and LZ4

5. Managing Data Shuffling and Skew

24 - Understanding data shuffling techniques to minimize data shuffling
25 - Addressing data skew

High-Performance PySpark: Advanced Strategies for Optimal Data Processing

Authors

Ameena Ansari