Data Pipeline Automation with GitHub Actions Using R and Python

2h 12mIntermediate2024-04-23

Authors

Rami Krispin

Course details

In this course, learn how to set up workflows on GitHub Actions to automate processes with both R and Python. Instructor Rami Krispin takes you through the automation process, sharing real-world examples. He shows you how to set up a data pipeline, pull metadata from a pipeline, and deploy a live dashboard with GitHub Actions and Pages. If you’re tired of spending hours running scripts manually, or slowing down your workflow by pulling data from APIs or updating dashboards, join Rami in this course to see how automation can speed up your work.

Skills covered

Version ControlStatisticsGitHubData EngineeringSoftware Development ToolsProgramming LanguagesData ScienceSoftware DevelopmentOne-Off

Concepts

0. Introduction

01 - Data pipeline automation with GitHub Actions
02 - What you should know

1. EIA API

03 - EIA API
04 - Setting an environment variable
05 - The EIA API dashboard
06 - GET request structure
07 - Querying the data via the browser
08 - Querying data with R and Python
09 - Pulling metadata from API with R
10 - Sending a simple GET request with R
11 - API limitations with R
12 - Handling a large data request with R
13 - Pulling metadata from API with Python
14 - Sending a simple GET request with Python
15 - API limitations with Python
16 - Handling a large data request with Python
17 - Challenge - Query the API
18 - Solution - Query the API with R
19 - Solution - Query the API with Python

2. Data Automation

20 - Data pipeline scope and requirements
21 - Data pipeline architecture
22 - Data refresh process
23 - ETL supporting functions
24 - Data backfilling
25 - Data refresh output
26 - Data quality checks

3. Deployment

27 - Introduction to GitHub Actions
28 - Deployment with Docker
29 - Setting GitHub Actions workflow
30 - Reviewing workflows logs
31 - Setting secrets and environment variables
32 - Advanced workflow
33 - Data pipeline deployment

4. Monitoring

34 - Data pipeline maintenance
35 - Deploying dashboard to GitHub Pages

Conclusion

36 - Next steps