Getting your web application and R(Apache) to talk to each other

Here’s the situation. Web applications, built using a framework (e.g. Rails, Django) are great for fetching data from a database and rendering it. They’re not so great for crunching and charting the data. Conversely, R is great for crunching and charting, but doesn’t make for a great web application.

The idea then, is to let each do what it does best and enable the passing of data between them. There isn’t a whole lot of literature on this topic, but there are a couple of guides:

• In these seminar slides (PDF), Jeroen Ooms describes how data can be passed between a web browser and R. Briefly, he uses client-side javascript to format the data as a JSON string. Server-side R (RApache) then parses the POST variable using fromJSON() (in the rjson package), formats the results of an R function as JSON using toJSON() and sends them back to the browser.
• Slide 46 of this presentation by Mike Driscoll of Dataspora illustrates a different approach, where a Django-based web application sends data to RApache in CSV format.

As a first step in understanding all of this, we can build a small demo application using Rails (version 2.3.5), which serves both JSON and CSV. We’ll see if we can get that into R, then see if R can return results back to Rails, via RApache. Baby steps, so we’ll avoid the AJAX stuff for now and just use Rails rendering methods to serve JSON from a controller.

1. Generate the Rails application
First, generate the skeleton. I’m calling the application “rapache” and scaffolding for a simple model, named Value, containing 2 integer fields named “x” and “y”:

rails rapache
cd rapache
rake db:create
./script/generate scaffold Value x:integer y:integer
rake db:migrate

Next, edit config/environment.rb to contain these lines, required for Rails to render CSV:

config.gem "fastercsv"
config.gem "comma"

Install those gems if you don’t have them and optionally, unpack them into your application tree. I like to freeze the Rails gems too:

rake gems:install
rake gems:unpack:dependencies
rake rails:freeze:gems

We also need to edit models/values.rb to provide the comma method:

class Value < ActiveRecord::Base
  comma do |f|
    f.x
    f.y
  end
end

So far, so good. Now, we could seed the database with some test data but for just a few values, it’s as easy to use the forms generated by the scaffolding and enter some X and Y values. I made my Y = X, fired up ./script/server and then opened up “http://localhost:3000/values” in the browser, where I saw a view like that in the image, above-right in this blog post.

Generating the JSON and CSV end-points couldn’t be easier. Just edit the index method in controllers/values_controller.rb to look like this:

def index
  @values = Value.all
  respond_to do |format|
    format.html # index.html.erb
    format.xml  { render :xml => @values }
    format.json { render :json => @values }
    format.csv  { render :csv => @values }
  end
end

That gives us JSON in the browser when we go to “http://localhost:3000/values.json” and a CSV file when we go to “http://localhost:3000/values.csv”. Depending on your browser setup, the browser may download the CSV, offer to download it or offer to open it in a spreadsheet application.

2. Experiments in R
With the Rails development environment still running, I opened an R console. First, fetching the data from the Rails application via JSON:

library(rjson)
data.json <- fromJSON(readLines(url("http://localhost:3000/values.json")))
# Warning message:
# In readLines(url("http://localhost:3000/values.json")) :
# incomplete final line found on 'http://localhost:3000/values.json'
length(data.json)
[1] 6
data.json[[1]]$value$x
[1] 0
data.json[[1]]$value$y
[1] 0

Not too bad. The warning message arises because readLines() can’t detect an end of line character; the message could be hidden using suppressWarnings(). We could use either rjson or RJSONIO – both of them have the fromJSON() method, but neither can read directly from a URL connection, hence the requirement for readLines(url(…)).
R read the JSON into a list of lists. That could be transformed into a dataframe, but it’s a little unwieldy. Let’s have a look at the CSV approach:

data.csv <- read.csv("http://localhost:3000/values.csv", header = T)
dim(data.csv)
# [1] 6 2
data.csv
#   X Y
# 1 0 0
# 2 1 1
# 3 2 2
# 4 3 3
# 5 4 4
# 6 5 5

Perfect – read.csv() can read the Rails-rendered CSV directly into a data frame.

3. Calling R from within Rails and displaying the results
The last step is to call R and retrieve results from within the Rails application. For this, you’ll need to have installed RApache. That’s beyond the scope of this post – it’s not difficult, under Ubuntu at least. I assume that R scripts will be served from /var/www/R, which is configured using the Apache configuration file /etc/apache2/conf.d/rapache.conf:

<Location "/R">
  ROutputErrors
  SetHandler r-script
  RHandler sys.source
  REvalOnStartup "library(lattice)"  # not used here
</Location>

What we should really do is write a Rails controller method to retrieve the appropriate data and then write a generic R function to accept CSV from any URL and return the results. However, for demonstration purposes, I’m just going to write a test R script, /var/www/R/plot.R, to plot a scatterplot of the X versus Y data and then wrap the result inside a Rails image_tag. Here’s the R:

setContentType("image/png")
d <- read.csv("http://localhost:3000/values.csv", header = T)
t <- tempfile()
png(t,type="cairo")
plot(d, main = "Y = X!")
dev.off()
sendBin(readBin(t,'raw',n=file.info(t)$size))
unlink(t)
DONE

And here’s the Rails – I just added this to the bottom of views/values/index.html.erb:

<div style="position:absolute; top:50px; left:400px;">
  <%= image_tag "http://localhost/R/plot.R" %>
</div>

The result – see image, right.

That covers the basics of serving JSON or CSV from Rails to RApache and sending a result back to Rails. Obviously, a real application would require more model or controller methods, views, R functions, error checks and prettier views, perhaps with a dash of AJAX thrown in. You’d probably also store your R scripts in the Rails directory tree and serve them from there. On the whole though, I’d say it’s surprisingly easy to create a RESTful API using Rails and use it to communicate with R.
Filed under: programming, R, rails, research diary, ruby, statistics