As a “thank you” to our blog, mailing list, and Twitter followers (@WinVectorLLC) we at Win-Vector LLC have decided to re-release our formerly fee-based A/B testing video course as a free (advertisement supported) video course here on Youtube.

The course emphasizes how to design A/B tests using prior “guestimates” of effect sizes (often you have these from prior campaigns, or somebody claims an effect size and it is merely your job to confirm it). It is fairly technical, and the emphasis is Bayesian- where we are trying to get an actual estimate of the distribution unknown true expected payoff rate of the various campaigns (the so-called posteriors). We show how to design and evaluate a sales campaigns for a product at two different price points.

The solution is coded in R and Nina Zumel has contributed an updated Shiny user interface demonstrating the technique (for more on Shiny, please see here). The code for the calculation methods and older shiny app are shared here.

This sort of fills out our survey of ways to think about A/B testing:

• Classic frequentist theory: emphasizes correct decision over expected returns/value.
• Bandit Formulations: great utility theory based solution to the problem.
• Dynamic programing methods: more involved tracking of utility.
• Sequential analysis: operationalizing many of the above ideas.
• Bayesian methods: correct distribution inference for individual test runs.

We have a lot more material on statistics and data science (though not on A/B testing) in our book and our paid video course Introduction to Data Science.