Even though ensembles of trees (random forests and the like) generally have better predictive power and robustness, fitting a single decision tree to data can often be very useful for: understanding the important variables in a data set exploring unusual subsegments of the data (and the explanatory variables that define ... [Read more...]

Numerical optimization is an important tool in the data scientist's toolbox. Many classical statistical problems boil down to finding the highest (or lowest) point on a multi-dimensional surface: the base R function optim provides many techniques for solving such maximum likelihood problems. Counterintuitively, numerical optimizations are easiest (though rarely actually ... [Read more...]

As Google learned, predicting the spread of influenza, even with mountains of data, is notoriously difficult. Nonetheless, bioinformatician and R user Shirin Glander has created a two-part tutorial about predicting flu deaths with R (part 2 here). The analysis is based on just 136 cases of influenza A H7N9 in China ... [Read more...]

"Big Data" generates a lot of news these days, but sometimes small data still means big computation. Indiana's Department of Workforce Development has the responsibility to forecast future employment rates in the State of Indiana. And not just the number of jobs available: the department also needs to forecast the ... [Read more...]

If you're looking to get started with data science in R, a great place to start is OnePageR by Graham Williams. (Graham is the creator of Rattle, author of Data Mining with Rattle and R, and Director of Data Science at Microsoft.) This free (CC-licensed) resource is a series of ... [Read more...]

by Ali Zaidi, Data Scientist at Microsoft In previous post we showcased the use of the sparklyr package for manipulating large datasets using a familiar dplyr syntax on top of Spark HDInsight Clusters. In this post, we will take a look at the RxSpark API for R, part of the ... [Read more...]

Len Kiefer, Deputy Chief Economist at Freddie Mac, recently posted an analysis of global housing price trends based on the international house price database (from the Dallas Fed). Using those data, Kiefer compared housing trends price increases (and in a couple of places like Spain and Ireland, decreases) across 24 countries. ... [Read more...]

I was thrilled to be invited to speak at the Monktoberfest conference, held this past October in Portland, Maine. Not only have I been a great fan of the analysis from the Redmonk team for many years, I'd heard that it was one of the most interesting and diverse tech ... [Read more...]

The R Consortium has already funded 8 projects (and 3 more just in July) proposed by the R community, and the call for proposals for yet more projects is now open. If you have an idea for a projects that would advance R or the R Community, get your submission in by ... [Read more...]

Microsoft R Server 9.0, Microsoft's R distribution with added big-data, in-database, and integration capabilities, was released today and is now available for download to MSDN subscribers. This latest release is built on Microsoft R Open 3.3.2, and adds new machine-learning capabilities, new ways to integrate R into applications, and additional big-data support ... [Read more...]

In case you missed them, here are some articles from November of particular interest to R users. Microsoft R Open 3.3.2, based on R 3.3.2, has been released for Windows, Mac and Linux. A new, free course on EdX focuses on the big-data extensions of Microsoft R Server. Using ggplot2 to create ... [Read more...]

Few people expect politicians to write every word they utter themselves; reliance on speechwriters and spokepersons is a long-established political practice. Still, it's interesting to know which statements are truly the politician's own words, and which are driven primarily by advisors or influencers. Recently, David Robinson established a way of ... [Read more...]

Microsoft R Open 3.3.2, Microsoft's enhanced distribution of open source R, is now available for download for Windows, Mac, and Linux. This update upgrades the R language engine to version 3.3.2, adds new bundled packages and updates others, and upgrades the Intel Math Kernel Libraries. The updated R 3.3.2 engine includes some performance ... [Read more...]

If you're already familiar with R, but struggling with out-of-memory or performance problems when attempting to analyze large data sets, you might want to check out this new EdX course, Analyzing Big Data with Microsoft R Server, presented by my colleague Seth Mottaghinejad. In the course, you'll learn how to ... [Read more...]

Chicago's a great city for a bike-sharing service. It's pretty flat, and there are lots of wide roads with cycle lanes. I love Divvy and use it all the time. Not so much in the winter though: it gets very cold here. Nonetheless, this heat map of Divvy riders, created ... [Read more...]

It's Thanksgiving day here in the US, so we're taking the rest of the week off to reflect on what we're thankful for. And even if you're not in the US, today is a great day to send thanks to the R Core Group for providing their dedication, time, and ... [Read more...]

A great way to learn is by doing, so if you've been thinking about how to enable R-based computations within SQL Server, a new tutorial will take you through all the steps of building an intelligent application. In a few simple steps, you'll set up all the necessary software and ... [Read more...]

The good folks at DataCamp track activity related to R packages on the RDocumentation.org Trends page. As of this writing, it tracks statistics on 11,768 packages (distributed across CRAN, BioConductor and Github) comprising over 1.7 million R functions in total. On that page, you can find current rankings on the most ... [Read more...]

As we prepare for the upcoming release of Microsoft R Open, I've been preparing the list of new and updated packages for the spotlights page. This involves scanning the CRANberries feed (with gracious thanks to Dirk Eddelbuettel) for newly-released packages and significant updates to existing ones. This is a lot ... [Read more...]

Microsoft Cognitive Services is a set of cloud-based machine-intelligence APIs that you can use to extract structured data from complex sources (unstructured text, images, video and audio), and add "AI" type features to applications. A good example is the "Seeing AI" glasses in the video below: the image descriptions, emotion ... [Read more...]