[Read more...]

I have written a few posts discussing descriptive analyses of evaluation of National Standards for New Zealand primary schools.The data for roughly half of the schools was made available by the media, but the full version of the dataset is … Continue reading → [Read more...]

Today Scott Chamberlain tweeted asking for a better/faster solution to building an m x n matrix with randomly assigned 0/1. He already had a working version: Now, I’m the first to acknowledge that I’ve never got the ‘apply’ family of … Continue reading → [Read more...]

[Read more...]

I’m happy to announce that I’ve started writing a new book for O’Reilly, which will focus on teaching readers how to use Multi-Armed Bandit Algorithms to build better websites. My hope is that the book can help web developers build up an intuition for the core conundrum ... [Read more...]

At MSR this week, we had two very good talks on algorithmic methods for tuning the hyperparameters of machine learning models. Selecting appropriate settings for hyperparameters is a constant problem in machine learning, which is somewhat surprising given how much expertise the machine learning community has in optimization theory. I ... [Read more...]

What follows is like a kata to strengthen your R fundamentals. The lovely stats in the wild recently posted some hott data analysis of Olympians’ ages and sexes. Because I’m annoyingly picky about graphics, I asked for his code so I could ... [Read more...]

Lets go one with the second part of learning R by doing R (you will find the first part here. As we have used vectors, matrices and loops in the first part, we will concentrate on graphics in this one. but first we will need data to plot: Sometimes you ... [Read more...]

Over the last few weeks, I’ve made a concerted effort to develop a basic suite of optimization algorithms for Julia so that Matlab programmers used to using fminunc() and R programmers used to using optim() can start to transition code over to Julia that requires access to simple optimization ... [Read more...]

Due to my work with a multitude of statistical packages in my career I may be able to evaluate a lot of them. I’ve first used Excel for my calculations as most of the normal users do. I like the idea behind a spreadsheet and the combination of data ... [Read more...]

On July 25th, I’ll be presenting at the Seattle R Meetup about implementing Bayesian nonparametrics in R. If you’re not sure what Bayesian nonparametric methods are, they’re a family of methods that allow you to fit traditional statistical models, such as mixture models or latent factor models, ... [Read more...]

Many readers of this blog will know that I’m a big fan of Bayesian methods, in large part because automated inference tools like JAGS allow modelers to focus on the types of structure they want to extract from data rather than worry about the algorithmic details of how they ... [Read more...]

I just returned from the useR! 2012 conference for developers and users of R. One of the common themes to many of the presentations was integration of R-based statistical systems with other systems, be they other programming languages, web systems, or enterprise data systems. Some highlights for me were an update ... [Read more...]

  The NATO summit is currently being held in Chicago, and, as is typical for NATO or G# summits, the streets and tweets are full of dissent.  In the spirit of my past investigations of online dissent (#jan25, #25bahman, #12fev,… Read more › [Read more...]

Let’s say you’ve used my Python script to automate the download of a hashtag or search phrase from Twitter (in a Unicode safe way, unlike within R).  Now let’s say you want to visualize the number of tweets over time.  Easy… Read more › [Read more...]

Some people will say ‘you have to learn R if you want to get a job doing statistics/data science’. I say bullshit, you have to learn statistics and learn to work in a variety of languages if you want to … Continue reading → [Read more...]

Introduction For me, Hadley Wickham‘s reshape and plyr packages are invaluable because they encapsulate omnipresent design patterns in statistical computing: reshape handles switching between the different possible representations of the same underlying data, while plyr automates what Hadley calls the Split-Apply-Combine strategy, in which you split up your data ... [Read more...]

In the Follow-Up Part 1, I explored some of the functions in the quantstrat package that allowed us to drill down trade by trade to explain the difference in performance of the two strategies. By doing this, I found that my choice of a volatility measure may not have been the ... [Read more...]

I’m a bit obsessive with words. May be I should have used learning in the title, rather than teaching code. Or perhaps remembering code. You know? Code where one actually has very clear idea of what is going on; for … Continue reading → [Read more...]

  Here’s a fun example of how you might use my data on Congressional bill length and complexity.  Imagine you want to understand the empirical distribution of Flesch-Kincaid reading level for Congressional bills and how this distribution is related to … Continue reading → [Read more...]