Azure Data Engineer Associate (DP-203) Cert Prep: 4 Monitor and Optimize Data Storage and Data Processing
34mIntermediate2022-12-15
Authors
Noah Gift
MLOps Expert | Solopreneur | Author | Adjunct Professor | CTO
Course details
Are you preparing for the Microsoft Azure Data Engineering (DP-203) exam, or seeking a better understanding of how to design and develop data processing? This course, the fourth in a series, can help you. Noah Gift, founder of Pragmatic A.I. Labs and a Python Software Foundation Fellow, covers how to monitor and optimize data storage and data processing. Noah first covers how to monitor data storage and data processing, showing how to: implement logging used by Azure Monitor; configure monitoring services; measure query performance; and more. He then details topics pertaining to optimizing and troubleshooting data storage and processing, including how to: rewrite user-defined functions; tune queries by using indexers and cache; handle skew in data and data spill; troubleshoot a failed Spark job and pipeline run.
Skills covered
Cloud StorageData EngineeringAzureNetwork AdministrationCloud PlatformsCert PrepNetwork and System AdministrationCloud ComputingData ScienceMicrosoft
Concepts
0. Introduction
- 01 - Course introduction
1. Monitor Data Storage and Data Processing
- 02 - Implement logging used by Azure Monitor
- 03 - Configure monitoring services
- 04 - Measure performance of data movement
- 05 - Monitor data system pipeline cluster performance
- 06 - Measure query performance
- 07 - Schedule and monitor pipeline tests
- 08 - Interpret a Spark directed acyclic graph (DAG)
2. Optimize and Troubleshoot Data Storage and Data Processing
- 09 - Rewrite user-defined functions (UDFs)
- 10 - Handle skew in data and data spill
- 11 - Tune shuffle partitions pipelines
- 12 - Optimize resource management
- 13 - Tune queries by using indexers and cache
- 14 - Troubleshoot a failed Spark job and pipeline run
Conclusion
- 15 - Summary and next steps
Related courses
- Microsoft Azure Data Engineer Associate (DP-203) Cert Prep: 4 Monitor and Optimize Data Storage and Data Processing
- Azure Data Engineer Associate (DP-203) Cert Prep: 3 Design and Implement Data Security
- Azure Data Engineer Associate (DP-203) Cert Prep: 1 Design and Implement Data Storage
- Azure Data Engineer Associate (DP-203) Cert Prep: 2 Design and Develop Data Processing
- Microsoft Azure Data Engineer Associate (DP-203) Cert Prep: 1 Design and Implement Data Storage
- Microsoft Azure Data Engineer Associate (DP-203) Cert Prep: 2 Design and Develop Data Processing
- Microsoft Azure Data Engineer Associate (DP-203) Cert Prep: 3 Design and Implement Data Security
- Microsoft Azure Data Engineer Associate (DP-203) Cert Prep by Microsoft Press
Related learn paths
- Prepare for the Azure Data Engineer Associate (DP-203) Certification
- Prepare for the Azure Data Engineer Associate (DP-203) Certification by Microsoft Press
- Getting Started as an Azure Developer
- Fundamentals to Become a Machine Learning Engineer
- Prepare for the Azure Solutions Architect Expert (AZ-305) Certification by Microsoft Press
- Prepare for the Azure Developer Associate (AZ-204) Certification
- Advancing Your Azure Developer Skills: Exploring Complex Application Development
- Prepare for the AZ-203 Developing Solutions for Microsoft Azure Exam