Gunicorn, Web Servers, and Concurrency

First and foremost: R is single-threaded. This is one of the most common criticisms I hear about R running in production settings, especially in the Python vs. R for production discussions I’ve been in. Of course, those discussions tend to ignore that Python also runs single-threaded, but I digress.

R running single-threaded and not managing concurrency particularly well isn’t a problem in and of itself. Other languages (Python, Ruby, etc.) that are very often used in production systems of all sizes have the same issue. The problem with R in particular is that unlike Python, which has Gunicorn, Uvicorn, and other web server implementations, and Ruby, which has Puma and similar, R has no widely-used web server to help it run concurrently. In practice, this means that if you, for instance, were to run a FastAPI service in production, you’d generally have a “leader” that delegates “work” (processing requests) to workers. Gunicorn or Uvicorn would handle this for you. This would mean that your service can handle as many concurrent requests as you have workers without being blocked.

As I mentioned, R has no equivalent web server implementation, which, in combination with running single-threaded, means that a Plumber service really can only handle one request at a time before getting blocked. In my view, this makes running high-load production services in R a non-starter, as concurrency and throughput are the ultimate source of lots of scalability problems in APIs. Yes, Plumber does indeed integrate with future and promises to allow for some async behavior, but my view is that it’s hard to make an argument that async Plumber is a viable substitute for a genuinely concurrent web server.

But let’s put aside the “non-starter” bit for a second, and let’s imagine that you, like me, want to try everything in your power to get R working in production. The following sections will cover other issues we’ve run into, and a number of workarounds we attempted, to varying degrees of success.

Types and Type Conversion

In my opinion, one of the biggest issues with R is the type system. R is dynamically typed, and primitive types are generally represented as length-one vectors. That’s why these two variables are of the same type:

class(1)

[1] "numeric"

class(c(1, 2))

[1] "numeric"

This is a big problem. What happens when we try to serialize the number 1 to JSON?

jsonlite::toJSON(1)

[1] 

It returns [1] – as in: A length-one list, where the one element is the number one. Of course, you can set auto_unbox = TRUE, but that has other issues:

jsonlite::toJSON(1, auto_unbox = TRUE)

1 

This is fine, but the problem with auto_unbox = TRUE is that if you have a return type that is genuinely a list, it could sometimes return a list, and sometimes return a single number, depending on the length of the thing being returned:

get_my_fake_endpoint <- function(x) {
  jsonlite::toJSON(x + 1, auto_unbox = TRUE)
}

get_my_fake_endpoint(1)

2 

get_my_fake_endpoint(c(1, 2))

[2,3] 

In these two examples, I’ve gotten two different response types depending on the length of the input: One was a list, the other was an integer. This means that, without explicit handling of this edge case, your client has no guarantee of the type of the response it’s going to get from the server, which will inevitably be a source of errors on the client side.

In every other programming language that I’m aware of being used in production environments, this is not the case. For instance:

import json
import sys

x = 1
y = [1, 2]

print(type(x))

<class 'int'>

print(type(y))

<class 'list'>

json.dump(x, sys.stdout)

1

json.dump(y, sys.stdout)

[1, 2]

In Python, the number 1 is an integer type. The list [1, 2] is a list type. And the JSON library reflects that. No need for unboxing.

But there’s more! R (and Plumber) also do not enforce types of parameters to your API, as opposed to FastAPI, for instance, which does via the use of pydantic. That means that if you have a Plumber route that takes an integer parameter n and someone calls your route with ?n=foobar, you won’t know about that until the rest of your code runs, at which point you might get an error about n being non-numeric.

Here’s an example:

library(plumber)

pr() %>%
  pr_get(
    "/types",
    function(n) {
      n * 2
    }
  ) %>%
  pr_run()

Obviously, n is indented to be a number. You can even define it as such in an annotation like this:

#* @param n:int 

But R won’t enforce that type declaration at runtime, which means you need to explicitly handle all of the possible cases where someone provides a value for n that is not of type int. For instance, if you call that service and provide n=foobar, you’d see the following in your logs (and the client would get back an unhelpful HTTP 500 error):

<simpleError in n * 2: non-numeric argument to binary operator>

If you do the equivalent in FastAPI, you’d have vastly different results:

from fastapi import FastAPI

app = FastAPI()

@app.get("/types")
async def types(n: int) -> int:
  return n * 2

Running that API and making the following call returns a very nice error:

curl "http://127.0.0.1:8000/types?n=foobar" | jq

{
  "detail": [
    {
      "loc": [
        "query",
        "n"
      ],
      "msg": "value is not a valid integer",
      "type": "type_error.integer"
    }
  ]
}

I didn’t need to do any type checking. All I did was supply a type annotation, just like I could in Plumber, and FastAPI, via pydantic, did all the lifting for me. I provided foobar, which is not a valid integer, and I get a helpful error back saying that the value I provided for n is not a valid integer. FastAPI also returns an HTTP 422 error (the error code is configurable), which tells the client that they did something wrong, as opposed to the 500 that Plumber returns, indicating that something went wrong on the server side.

Clients and Testing

Another issue with Plumber is that it doesn’t integrate nicely with any testing framework, at least that I’m aware of. In FastAPI, and every other web framework that I’m familiar with, there’s a built-in notion of a test client, which lets you “call” your endpoints as if you were an external client. In Plumber, we’ve needed to hack similar behavior together using testthat by spinning up the API in a background process, and then running a test suite against the local instance of the API we spun up, and then spinning down. This has worked fine, but it’s clunky and much harder to maintain than a genuine, out-of-the-box way to do testing that really should ship with the web framework. I’ve heard of callthat, but I’ve never actually tried it for solving this problem.

Performance

When I’ve defended R in that past, I’ve also heard a common complaint about it’s speed. There are very often arguments that R is slow, full-stop. And that’s not true, or at least mostly not true. Especially relative to Python, you can write basically equally performant code in R as you can in numpy or similar. But some things in R are slow. For instance, let’s serialize some JSON:

library(jsonlite)

iris <- read.csv("fastapi-example/iris.csv")

result <- microbenchmark::microbenchmark(
  tojson = {toJSON(iris)},
  unit = "ms", 
  times = 1000
)

paste("Mean runtime:", round(summary(result)$mean, 4), "milliseconds")

[1] "Mean runtime: 0.7482 milliseconds"

Now, let’s try the same in Python:

from timeit import timeit
import pandas as pd

iris = pd.read_csv("fastapi-example/iris.csv")

N = 1000

print(
  "Mean runtime:", 
  round(1000 * timeit('iris.to_json(orient = "records")', globals = locals(), number = N) / N, 4), 
  "milliseconds"
)

Mean runtime: 0.1166 milliseconds

In this particular case, Python’s JSON serialization runs 6-7x faster than R’s. And if you’re thinking “that’s only one millisecond, though!” you’d be right. But the general principle is important even if the magnitude of the issue in this particular case is not.

JSON serialization is the kind of thing that you’re going to need to do if you’re building an API, and you generally want it to be as fast as possible to limit overhead. It also takes longer and longer as the JSON itself is more complicated. So while in this particular case we’re talking about microseconds of difference, the underlying issue is clear: Plumber uses jsonlite to serialize JSON under the hood, and jsonlite is nowhere near as fast as roughly equivalent Python JSON serialization for identical data and the same result.

The takeaway here is that while it may be true that vectorized R code to create a feature for a model or low-level BLAS or LAPACK code that R calls to perform matrix multiplication should be equally performant to the equivalent Python, R can sometimes have overhead, like in JSON serialization, that becomes apparent as the size and complexity of both the body of the request as well as the response body scale up. There are certainly other examples of the same overhead. When we moved a Plumber service to FastAPI with no changes to the logic itself, we got about a 5x speedup in how long it took to process requests. And just to reiterate: That 5x speedup had nothing to do with changes to logic, models, or anything tangible about the code. All of that was exactly the same.

Integration with Tooling

Another issue with R for production is that very often in production services, we want to do some things on the following (in no particular order and not exhaustive) list:

1. Serve predictions from one version of a model to some users and another version of a model to other users.
2. Have reports and alerts of when errors happen in our API.
3. Use a database - such as a feature store - for retrieving features to run through our models.

There are tools for doing all of these things. The first would be a feature flagging tool like LaunchDarkly. The second would be an error monitoring tool like Sentry. And the last would be a feature store like Feast which might use something like Redis under the hood.

Python supports all of these tools. All of them have APIs in Python, and are easily integrated into a FastAPI, Flask, or Django service. R has no bindings for any of them, meaning that if you wanted to run R in a system where you use feature flags, for instance, you’d need to roll your own flagging logic or find a more obscure tool that supports R. That’s fine for some teams, but writing feature flagging logic isn’t generally a good use of a data scientist’s time. And especially not when there are a whole variety of great tools that you can grab off the shelf and slot into a Python service seamlessly.

This issue expands beyond just production tooling, too. For instance, there are a number of MLOps tools for all parts of the machine learning process, such as Weights & Biases for experiment tracking and model versioning, Evidently for monitoring, Bento for model serving, and so on, that all only have Python bindings. That’s not to say that there are no tools that support R – some, like MLFlow certainly do – but the set of tools that support R is a strict, and small, subset of the ones that support Python. I’m also aware of the great work that the Tidymodels team is doing on Vetiver, Pins, and related packages in the Tidymodels MLOps suite, but the reality is that these tools are far behind the state of the art (but are catching up!).

Load Balancing on PaaS Tools

The first “fix” for R’s concurrency issue we tried was the most expensive one: Buying our way out of the problem. We horizontally scaled our R service up from a a single instance to having multiple instances of the service behind a load balancer, which bought us a significant amount of runway. It’s expensive, but this could be a reasonably good solution to R’s concurrency issues for most teams. However, there’s only so far that you can scale horizontally before needing to address the underlying issues. For instance, if you have a model that takes 250ms to run predictions through and return to the client, on average you can process four requests per second per instance of your API. But since R is running single-threaded, you probably can really only run about one or two requests per second before you start to get concerned about requests backing up. If one request takes one second, now you could have four more requests in the queue waiting to be processed, and so on.

Horizontal scaling fixes this issue to some extent, but the magnitude of the problem scales linearly as the amount of throughput to the service increases. So ultimately, it’s inevitable that you’ll need to either address the underlying problem of the performance of the service, or spend potentially exorbitant amounts of money to buy your way out of the problem.

NGINX As A Substitute

We also tried to get around R’s lack of an ASGI server like Uvicorn by sitting our Plumber API behind an NGINX load balancer. The technical nuts and bolts were a little involved, so I’ll just summarize the highlights here. We used a very simple NGINX conf template:

## nginx.conf

events {}

http {
  upstream api {

      ## IMPORTANT: Keep ports + workers here
      ## in sync with ports + workers declared
      ## in scripts/run.sh

      server localhost:8000;
      server localhost:8001;
      server localhost:8002;
      server localhost:8003;
  }

  server {
    listen $PORT;
    server_name localhost;
    location / {
      proxy_pass http://api;
    }
  }
}

Then, we’d boot the API as follows:

## run.sh

# !/bin/bash

## IMPORTANT: Keep ports + workers here
## in sync with ports + workers declared
## in nginx.conf

Rscript -e "plumber::options_plumber(port = 8000); source('app.R')"&
Rscript -e "plumber::options_plumber(port = 8001); source('app.R')"&
Rscript -e "plumber::options_plumber(port = 8002); source('app.R')"&
Rscript -e "plumber::options_plumber(port = 8003); source('app.R')"&

sed -i -e 's/$PORT/'"$PORT"'/g' /etc/nginx/nginx.conf && nginx -g 'daemon off;'

The basic premise was that we’d boot four instances of our Plumber API as background processes, and then let NGINX load balance between them.

This worked well in theory (and also in practice, to an extent) until we ran into an odd problem: At certain levels of load, the “workers” started to die and not reboot, which resulted in cascading failures. Essentially, one worker would go down, resulting in more load on the other workers, until a second worker went down, causing the service to spiral and eventually crash. You can see this happening in the load test below.

At relatively low levels of traffic (~25 RPS) the service starts to have issues. Those issues snowball, until eventually every request begins failing. Note that 25 requests per second sounds like a lot, but that load is distributed across four workers, meaning each worker is attempting to handle 5-6 requests per second.

For some services, burning down under this amount of load is fine. But it was disconcerting for us, especially since the amount of actual work that the endpoint we were hitting in our load test was doing was just extracting a pre-computed item from a list. The vast majority of the overhead in these API calls was JSON serialization, type checking, and network latency, but the R service itself was only taking about 10ms to process the request (read: extract the necessary element) once the endpoint got the request.

The problem for us was that if the service is burning down in this case where we’re doing virtually no lifting at all after getting 25 or so requests per second, what happens when the processing time for a request jumps up from 10ms to 250ms for creating features, making predictions, and so on? In that world, I’d expect that even behind NGINX, our service could probably only safely process about 5 requests per second before it starts getting dicey and us needing to start thinking again about horizontally scaling to more instances, and that wasn’t nearly enough headroom to be comfortable with in a production system.