Cleaning Bad Data in R

2h 4mBeginner2025-03-10

Authors

Mike Chapple

Teaching Professor at the University of Notre Dame

Course details

Data integrity is the new focal point of the data science revolution. Now that everybody is onboard with the role of data in people's lives and business, it's not an unfair question to ask, "Can you prove that your data is accurate?" In this course, you can learn how to identify and address many of the data integrity issues facing modern data scientists, using R and the tidyverse. Discover how to handle missing values and duplicated data. Find out how to convert data between different units and tackle poorly formatted text. Plus, learn how to detect outliers, address structural issues, and identify red flags that indicate potential data quality issues.

Learning objectives
Missing data
Duplicate rows and values
Converting data
Formatting data
Working with tidy data
Tidying data sets
Dealing with suspicious data

Skills covered

RStudioRStatisticsData EngineeringData AnalysisProgramming LanguagesData ScienceBusiness Analysis and StrategyBusiness Software and ToolsOpen SourceSoftware DevelopmentOne-Off

Concepts

0. Introduction

01 - Data is messy
02 - What you need to know

1. Missing Data

03 - Types of missing data
04 - Missing values
05 - Missing rows
06 - Aggregations and missing values

2. Duplicated Data

07 - Duplicated rows and values
08 - Aggregations in the data set

3. Formatting Data

09 - Converting dates
10 - Unit conversions
11 - Numbers stored as text
12 - Text improperly converted to numbers
13 - Inconsistent spellings

4. Outliers

14 - Screening for outliers
15 - Handling outliers
16 - Outliers use case
17 - Outliers in subgroups
18 - Detecting illogical values

5. Tidy Data

19 - What is tidy data
20 - Variables, observations, and values
21 - Common data problems
22 - Wide vs. long data sets
23 - Making wide data sets long
24 - Making long data sets wide

6. Red Flags

25 - Suspicious values
26 - Suspicious multiples

7. Redacting Data

27 - Identifying sensitive data
28 - Redacting sensitive data in R

Conclusion

29 - What's next