The Data Science of Experimental Design

Course details

Interested in learning how to create an online experiment that helps you better understand your business? This course can help you get up to speed. Instructor Monika Wahi shows learners without a background in experimental design how to build an A/B test for a web page, run the test, analyze the data, and make decisions based on the results of the test. Monika begins by explaining exactly what A/B testing is and under what circumstances it is useful. She then covers potential strategies for increasing conversion rates, as well as how to choose both A and B conditions for testing. Next, she explains how to define conversion rates and develop and document case definitions, conduct a baseline analysis in Excel and, based on the results of the analysis, design an A/B test. Plus, she demonstrates how to conduct a chi-square test in Excel and get a sample size estimate using G*Power.

Concepts

0. Introduction

01 - Conducting an experiment online
02 - What you should know

1. Introduction to Experimental Testing

03 - Experiment is a type of study
04 - Features of an experiment
05 - Circumstances for experimental testing
06 - When not to do an experiment
07 - Systems ready for experimental testing
08 - Comparability of experimental conditions

2. Defining Conversions

09 - Trying to increase conversions
10 - Different types of conversions
11 - Case definition of conversion
12 - Measuring a conversion
13 - Considering time period for conversions
14 - Rates versus frequencies of conversions

3. Defining Conversion Rates

15 - Identify and prioritize conversions
16 - Operationalize counting conversions
17 - Document conversion case definitions
18 - Brainstorm denominators
19 - False positives and negatives
20 - Document denominators
21 - Determine time frames

4. Baseline Descriptive Analyses

22 - Baseline time-series analyses
23 - Data handling
24 - Baseline results as a guide
25 - Thinking about increasing conversions
26 - Strategies to increase conversions
27 - Planning a campaign

5. Designing the Experiment

28 - Designing the test
29 - Testing the implementation
30 - Choosing a test statistic
31 - Choosing the chi-squared test
32 - Chi-squared test in Excel

6. Sample Size and Statistics

33 - Installing G Power
34 - Using G Power
35 - Sample size simulation
36 - Planning the timeline
37 - Stratified analysis
38 - Conditional tests

7. Analyzing and Interpreting the Data

39 - Overall analysis approach
40 - Time-series analysis
41 - Chi-squared analysis
42 - Interpretation

Conclusion and Next Steps

43 - What actions can we take
44 - Report writing