A few days ago, the Centers for Medicare and Medicaid Services (CMS) released some unprecedented data on the US healthcare system. The data consists of 9 million rows showing how much each doctor in the US charged Medicare, for what, and how much Medicare paid out. It doesn't quite cover everything (... [Read more...]

I’ve had a lot of different jobs over the past 4 years, and I’ve had some incredible experiences along the way. Lately, I’ve been struggling with what to do next. Or perhaps more accurately, I’ve been struggling with how to de... [Read more...]

I recently posted about visualizing the voting patterns of senators. In the post, I scraped voting data for each senator on every vote in the 113th Congress from the Senate website, and then assigned a code of 0 for a no vote on a particular issue, 1 for a yes vote, 2 for ... [Read more...]

I very seldom pay attention to politics directly, because politics have always seemed a bit circular and cyclical to me. Most of the political news that I take in ends up worming its way into the news sources that I do consume, like the excellent longform.org. Even given my ... [Read more...]

I recently posted about automatically making music. The algorithm that I made pulled out interesting sequences of music from existing songs and remixed them. While this worked reasonably well, it also didn’t have full control over the basics of the music; it wasn’t actually specifying which instruments to ... [Read more...]

I recently went to an excellent music meetup where people spoke about the intersection of music and technology. One speaker in particular talked about how music is now being generated by computer. Music has always fascinated me. It can make us feel emotions in a way few media can. Sadly, ... [Read more...]

I have been making charts with R for almost as long as I have been using R, and with good reason: R is an amazing tool for filtering and visualizing data. With R, and particularly if we use the excellent ggplot2 library, we can go from raw data to compelling ... [Read more...]

One day, while I was walking around Cambridge, I had a random thought — how do the characters on the Simpsons feel about each other? It doesn’t take long to figure out how Homer feels about Flanders (hint: he doesn’t always like him), or how Burns feels about everyone, ... [Read more...]

In a previous post, I looked at transcripts of Simpsons episodes and tried to figure out which character was speaking which line. This worked decently, but it wasn’t great. It gave us memorable scenes like this one: Homer : D'oh! A deer! A female deer. Marge : Son, you're okay! Bart : ... [Read more...]

You probably have a favorite Simpsons character. Maybe you hope to someday block out the sun, Mr. Burns style, maybe you enjoy Homer’s skill in averting meltdowns, or maybe you identify with Lisa’s struggles for acceptance. Through its ch... [Read more...]

Introduction This will serve as an introduction to natural language processing. I adapted it from slides for a recent talk at Boston Python. We will go from tokenization to feature extraction to creating a model using a machine learning algorithm. The goal is to provide a reasonable baseline on top ... [Read more...]

I just gave a talk at Boston Python about natural language processing in general, and edX ease and discern in specific. You can find the presentation source here, and the web version of it here. There is a video of it here. Nelle Varoquaux and Micha... [Read more...]

How Many Data Scientists Are There?I've seen a lot of articles lately about “Big Data” and the looming “talent gap.” This article from the Wall Street Journal is a good example. It cites a McKinsey estimate that states that we will need 1.5 million more managers and analysts who are ... [Read more...]

Introduction I recently posted about using the Wikileaks cable corpus to find word use patterns, both over time, and in secret cables vs unclassified cables. I received a lot of good suggestions for further topics to pursue with the corpus, and probably the most interesting was the idea to do ... [Read more...]

6/18: A follow-up to this post is now available here.Recent DiscoveriesWhen I was a diplomat, I was always interested in the Wikileaks cables and what could be done with them. Unfortunately, I never got a chance to look at the site in depth, due to security policies. Now that the ... [Read more...]

This is the sixth post in my series on predicting the NBA playoffs with an algorithm. After the Boston loss in their last game, the algorithm is now 5-4 in the playoffs. Hopefully it is correct tonight! Open Sourcing the CodeI have had a couple of re... [Read more...]

This is update 4 to my original post about predicting the NBA playoffs with R. With the Thunder beating the Spurs and the Heat losing to the Celtics, the algorithm went 1-1 on predictions, making it 5-3 so far.Making some improvements I have been posting for some time about incorporating ... [Read more...]

This is my third update to my original post on predicting the NBA playoffs with an algorithm. Here are updates 1 and 2. The algorithm correctly predicted a Boston win, but missed on the Spurs/Thunder game, so it is currently 4-2. Haven't had any time... [Read more...]

This is my second follow-up to my previous two posts which were about predicting NBA games with an algorithm, and my first update to the algorithm. The algorithm's record is now 3-1, as it correctly predicted Boston and Oklahoma City as winners of the... [Read more...]

Game ResultsI recently made a post about developing an algorithm to predict the NBA playoffs, and I concluded with 2 predictions. Although Miami beat the Celtics to make my algorithm 1-0 in terms of predictions, it fell to 1-1 when the Thunder beat th... [Read more...]