Fundamentals of Data Transformation for Data Engineering

Course details

Fundamentals of Data Transformation with pandas and DuckDB SQL presents the most essential concepts and best practices in a clear and concise format that allows students to side-step the noise and complexity. While this course is wide rather than narrow, it was designed to help you understand your options for development and make an informed choice about where to drill down.

Instructor Matt Palmer covers transformation in three separate environments: data transformation in a lake, data transformation in the warehouse, and data transformation in data lake houses.

Concepts

0. Introduction

01 - Welcome to data transformation
02 - What we'll cover and what you should know

1. Setup and Beyond

03 - Codespaces and setup
04 - Why SQL Why Python Why not Spark
05 - Types of data transformation
06 - The goal of data transformation

2. Data Transformation with SQL and DuckDB

07 - DuckDB basics and query structure
08 - Wrangling unstructured data
09 - Joins and comparisons
10 - Aggregations
11 - Windows functions - A quick refresher
12 - Window functions
13 - Advanced filters
14 - Advanced joins
15 - Lambdas and UDFs
16 - Data generation
17 - SQL challenge
18 - SQL solution

3. Data Transformation with Python and pandas

19 - DataFrame basics
20 - Wrangling unstructured data
21 - Select and filter
22 - Order and aggregate
23 - Advanced filters
24 - Data generation
25 - Windows
26 - Apply
27 - pandas challenge
28 - pandas solution

Conclusion

29 - What you learned, how to practice and grow, and next steps
30 - SQL bonus challenge
31 - SQL bonus solution

Fundamentals of Data Transformation for Data Engineering

Authors

Matt Palmer