Command Line Data Analysis

58mAdvanced2023-04-27

Authors

Miki Tebeka

CEO at 353Solutions

Course details

This course is integrated with GitHub Codespaces, an instant cloud developer environment that offers all the functionality of your favorite IDE without the need for any local machine setup. With GitHub Codespaces, you can get hands-on practice from any machine, at any time—all while using a tool that you’ll likely encounter in the workplace. Check out the “Using GitHub Codespaces with this course” video to learn how to get started.

In this course, Miki Tebeka presents the information and instructions you need to perform data analysis using the many tools that the command line offers for quick data analysis and visualization. Learn how to use the command line and command line tools for analyzing and displaying data. Go over Unix shell basics, and then use tools like grep, jq, sed, awk, curl, gnuplot, and others to analyze and display data from various sources, such as CSV files, APIs, databases, and more.

Skills covered

UnixHelp Desk SkillsWeb Development ToolsIT Help DeskData AnalysisWeb DevelopmentData ScienceBusiness Analysis and StrategyBusiness Software and ToolsOpen SourceDeep Dive (X:Y)

Concepts

0. Introduction

01 - Command-line data analysis
02 - What you should know
03 - Setting up

1. The Command Line

04 - The shell
05 - Running programs
06 - Shell expansion
07 - Pipes and redirects
08 - Files and directories
09 - Installing commands

2. Data Ingestion

10 - Cat and less
11 - curl
12 - SCP
13 - gsutil and AWS CLI
14 - Databases
15 - Challenge - ETL
16 - Solution - ETL

3. Selecting Data

17 - head, tail, and shuf
18 - grep and regular expressions
19 - cut
20 - jq
21 - Databases
22 - Challenge - Station status
23 - Solution - Station status

4. Data Transformations

24 - tr
25 - sed
26 - AWK
27 - Challenge - Normalized items
28 - Solution - Normalized items

5. Calculations

29 - sort and uniq
30 - bc
31 - AWK
32 - Challenge - Quantity sold
33 - Solution - Quantity sold

6. Visualization

34 - printf
35 - gnuplot - Introduction
36 - gnuplot - Box plots
37 - gnuplot - Bar charts
38 - Challenge - Weather
39 - Solution - Weather

Conclusion

40 - Next steps