R and (Software) Relatives

O’Reilly recently published the results of a survey from attendees of the Strata Conference related to tool usage and salary.  The entire survey is available for download.  In the survey results, R was heralded as second only to SQL as a tool used by conference attendees.  An chart from the survey appeared in this post and elsewhere online.

These two technologies overlap a bit but are are highly complementary.  SQL can be used to quickly extract data from relational databases and filter, order and summarize data.  SQL queries can be executed with R itself or in another language to produce a CSV file that can be imported into R.  R can do additional filtering, ordering and summarizing, be used for more sophisticated analysis, reshaping of data and presentation in a final form.  

As part of a in-progress R screencast, I wanted to speculate a bit about the most common “clusters” of technologies that are popular among R users (at least the Strata Conference respondents).  Although the raw data from the survey is not available, the graph cited in the survey results includes enough information to do a bit of additional analysis.  I reconstructed the original graph as a starting point with the intention of splitting out the data and non-data roles into facetted bar charts.  This would make usage reported among non-data respondents a bit clearer.  

So the first step was to replicate the original plot with a few cosmetic and editorial updates – no “Respodents” appear in the new version.  This involved the use of reshape2 and ggplot2.

With these available, I created the data frame by combining a few vectors containing the data of interest.

At this point, the results match up with what appeared in the chart from the O’Reilly report.  The numbers represent percent of respondents that use the given tool.

The data is easier to deal with if reshaped using melt.

The resulting data frame:

Convert data into required numeric type

Create the original chart

Now do the facetted example

Those in the non-data role appear to be largely coming from a more traditional software development/programming background.  The top tool in use after SQL is Java, followed by Python and JavaScript.  Hadoop is closely related as a java-based framework. Excel is used more than Excel, which suggests a fascinating opportunity for R.  Spreadsheets are and will remain useful, but anyone involved in data munging and analysis can benefit from R.   As has been oft-trumpeted, scripted R programs are far more controlled and disciplined than clicking around in a spreadsheet. They promote reproducible, less error-prone results. Ruby ranks a bit higher than among the non-data users and SAS/SPSS usage is minimal which also fits with a programmer audience.

To get a closer look at “non-Data” role.

There are a number of tools conspicuously lacking in the survey.

It also be interesting to see related data about respondents that undoubtedly impact the results (mathematical proficiency, design abilities, typical data stores / database types accessed, typical audience for summarized data).

As I have been reviewing literature and educational resources on R, I am developing a stronger opinion that R, though a remarkable functional and powerful programming language, has not been presented well to a programming audience.  Most introductions to R are more palatable to statisticians and others who have data analysis to complete but are not strongly aligned with programmer culture and expectations.  The fact that so many R packages are in essence full-fledged DSLs has further complicated R’s presentation.  As I mentioned in my previous post, Hadley’s new book and RStudio are significant in-roads that highlight R in a more programmer-friendly way.  And the involvement of programmers at the Strata Conference and similar events will increase its visibility and accessibility as well.