[Read more...]

Previously in this series: Understanding the beta distribution Understanding empirical Bayes estimation Understanding credible intervals Understanding the Bayesian approach to false discovery rates Understanding Bayesian A/B testing In this series we’ve been using the empirical Bayes method to estimate batting averages of baseball players. Empirical Bayes is useful ... [Read more...]

Previously in this series Understanding the beta distribution (using baseball statistics) Understanding empirical Bayes estimation (using baseball statistics) Understanding credible intervals (using baseball statistics) Understanding the Bayesian approach to false discovery rates (using baseball statistics) Who is a better batter: Mike Piazza or Hank Aaron? Well, Mike Piazza has a ... [Read more...]

I was horrified to learn of the existence of the monetizr package, which adds advertisements to R functions. The package goes against the entire philosophy of open source and the spirit of the R community. Luckily, I was able to construct a fix- the a... [Read more...]

I’ve had the great privilege to be a small part of the R open source community, contributing packages like broom, gganimate, fuzzyjoin, and ggfreehand. In the process I’ve become friends and colleagues with brilliant statisticians and data scientists and learned to engage with data in powerful ways. But ... [Read more...]

Yesterday a family member forwarded me a Wall Street Journal interview titled What Data Scientists Do All Day At Work. The title intrigued me immediately, partly because I find myself explaining that same topic somewhat regularly. I wasn’t disappointed in the interview: General Electric’s Dr. Narasimhan gave insightful ... [Read more...]

If you’ve read my blog, taken one of my classes, or sat next to me on an airplane, you probably know I’m a big fan of Hadley Wickham’s ggplot2 package, especially compared to base R plotting. Not everyone agrees. Among the anti-ggplot2 crowd is JHU Professor Jeff ... [Read more...]

[Read more...]

A family member recently sent me a puzzle: One hundred people are lined up with their boarding passes showing their seats on the 100-seat Plane. The first guy in line drops his pass as he enters the plane, and unable to pick it up with others behind him sits in ... [Read more...]

Previously in this series Cleaning and visualizing genomic data: a case study in tidy analysis In the last post, we examined an available genomic dataset from Brauer et al 2008 about yeast gene expression under nutrient starvation. We learned to tidy it with the dplyr and tidyr packages, and saw how ... [Read more...]

[Read more...]

Users on Stack Overflow Careers, our site for matching developers with jobs, can create customized profiles (“CVs”) to show to prospective employers. As part of these profiles, they have the option of specifying specific technologies they like or dislike. This produces an interesting and unusual opportunity for our data team ... [Read more...]

Previously in this series Understanding the beta distribution (using baseball statistics) Understanding empirical Bayes estimation (using baseball statistics) Understanding credible intervals (using baseball statistics) In my last few posts, I’ve been exploring how to perform estimation of batting averages, as a way to demonstrate empirical Bayesian methods. We’ve ... [Read more...]

Previously in this series Understanding the beta distribution (using baseball statistics) Understanding empirical Bayes estimation (using baseball statistics) In my last post, I explained the method of empirical Bayes estimation, a way to calculate useful proportions out of many pairs of success/total counts (e.g. 0/1, 3/10, 235/1000). I used the example ... [Read more...]

Which of these two proportions is higher: 4 out of 10, or 300 out of 1000? This sounds like a silly question. Obviously , which is greater than . But suppose you were a baseball recruiter, trying to decide which of two potential players is a better batter based on how many hits they get. One ... [Read more...]

Since I joined Stack Exchange as a Data Scientist in June, one of my first projects has been reconsidering the A/B testing system used to evaluate new features and changes to the site. Our current approach relies on computing a p-value to measure our confidence in a new feature. ... [Read more...]

This weekend I gave a presentation on my broom package for tidying model objects (see my introduction here) at the UP-STAT 2015 conference at SUNY Geneseo. I’m sharing the slides here, along with some highlights below. I first explained how broom fits with other tidy tools such as dplyr, tidyr ... [Read more...]

The concept of “tidy data”, as introduced by Hadley Wickham, offers a powerful framework for data manipulation, analysis, and visualization. Popular packages like dplyr, tidyr and ggplot2 take great advantage of this framework, as explored in several recent posts by others. But there’s an important step in a tidy ... [Read more...]

Almost everyone with an R package in CRAN wonders how often it’s installed and used. Two years ago RStudio kindly started offering anonymized logs of their downloads from their CRAN mirror, which allows one to graph the number of downloads over time. Much easier than downloading and processing all ... [Read more...]

There’s no end of interesting data analyses that can be performed with Stack Overflow and the Stack Exchange network of Q&A sites. Earlier this week I posted a Shiny app that visualizes the personalized prediction data from their machine learning system, Providence. I’ve also looked at whether ... [Read more...]