Complete list of CRAN packages

Initially, my thought was to scrape the package information directly from CRAN, the Comprehensive R Archive Network. It is the central repository for R packages. CRAN describes itself as “a network of ftp and web servers around the world that store identical, up-to-date, versions of code and documentation for R.” If you’re installing an R package in the standard way then it is provided by one of the CRAN mirrors. (The install.packages function takes a repos argument that you can set to any of the mirror or to the central “http://cran.r-project.org”.)

CRAN provides full lists of all available package by name and by date of publication. The latter page in particular has a nice html table with all package names, titles, and dates. This would be easy to scrape. If you want to get an intro to webscraping with the rvest package then check out a previous blogpost of mine.

However, the R community had once again made my task much easier. As I was pondering a respectful and responsible scraping strategy, I came across this post on Scraping Responsibly with R by Steven Mortimer, who was working on scraping CRAN downloads. In it, he quoted a tweet by Maëlle Salmon recommending to use tools::CRAN_package_db instead as a gentler approach.

This tool is indeed very fast and powerful. It provides a lot of columns. For the sake of a simple dataset, I’m only selecting a subset of those features here. Feel free to explore the full range.

df <- tools::CRAN_package_db() %>% 
  as_tibble() %>% 
  janitor::clean_names() %>% 
  select(package, version, depends, imports, license, needs_compilation, author, bug_reports, url, date_published = published, description, title) %>% 
  mutate(needs_compilation = needs_compilation == "yes")

The columns I picked include names, versions, dates, and information about dependencies, authors, descriptions, and web links. Here are the first 50 rows:

df %>% 
  head(50) %>% 
  gt() %>% 
  tab_header(
    title = md("**A full list of R packages on CRAN derived via tools::CRAN_package_db**")
    ) %>% 
  opt_row_striping() %>% 
  tab_options(container.height = px(600))

Now you could take this data, aggregate the date_published by month, and plot the growth of the R ecosystem for yourself. For instance like this:

df %>% 
  mutate(date = floor_date(ymd(date_published), unit = "month")) %>% 
  filter(!is.na(date)) %>% 
  count(date) %>% 
  arrange(date) %>% 
  mutate(cumul = cumsum(n)) %>% 
  ggplot(aes(date, cumul)) +
  geom_line(col = "blue") +
  theme_minimal() +
  labs(x = "Date", y = "Cumlative count", title = "Cumulative count of CRAN package by date of latest version")

But this doesn’t really show you the true historical growth, does it? The x-axis range and my plot title are already telling you what’s going on here. The date_published that CRAN_package_db gives us (and that the CRAN website lists) corresponds to the last published version of the package. See for instance the entry for dplyr:

df %>% 
  filter(package == "dplyr") %>% 
  select(package, version, date_published) %>% 
  gt()

The cornerstone of the tidyverse was first published a little while before April 2022. Version 1.0.9 is the most recent one, at the time of writing.

Naturally, that means that in this table those packages with frequent updates will be weighted heavier towards more recent dates. Which is perfectly fine if you’re only interested in the most recent package. But if you, like me, want to see how the R ecosystem grew over time, then you need the historical dates of the first published versions. This is where our next package comes in.

Package history

I found the packageRank package through googling “R package history”. Its documentation on github is detailed, and it performs very well for me. In the packageHistory function you give it a package name and it does the rest. Let’s find out more about the release history of our favourite dplyr package:

df_hist_dplyr <- packageRank::packageHistory(package = "dplyr", check.package = TRUE) %>% 
    as_tibble() %>% 
    janitor::clean_names()
df_hist_dplyr %>% 
  gt() %>% 
  tab_header(
    title = md("**The release history of the dplyr package**")
    ) %>% 
  opt_row_striping() %>% 
  tab_options(container.height = px(400))

First released in January 2014. It’s been an interesting journey for the tidyverse since then. I started using the tidy packages first in 2017, and I would find it hard to go back to base R now.

With data like this for a single R package, you could for instance investigate the yearly frequency of releases over time:

df_hist_dplyr %>% 
  mutate(year = floor_date(date, unit = "year")) %>% 
  count(year) %>% 
  ggplot(aes(year, n)) +
  geom_col(fill = "purple") +
  scale_x_date() +
  theme_hc() +
  labs(x = "", y = "", title = "Number of releases of 'dplyr' per year - in July 2022")

Interesting pattern there from 2014 to 2016. Since then the number of releases have been pretty consistent. At the time of writing, we are still in the middle of 2022.

(As a side note, you’ll see that I’ve decided not to use an x-axis label here. I feel like a year axis is often self-explanatory; and I’ve used a descriptive title to prevent misinterpretation. Let me know if you disagree.)

To get the history for all the entries in our complete list of CRAN packages, we can then simply loop through the package names. The loop takes about an hour, but you don’t have to run it yourself. This is what I created the Kaggle dataset for. You can download “cran_package_history.csv” and start working with it immediately.

df_hist <- read_csv("../../static/files/cran_package_history.csv", col_types = cols())

Here are the first 50 rows:

df_hist %>%
  head(50) %>% 
  gt() %>% 
  tab_header(
    title = md("**The first rows of the cran_package_history.csv table**")
    ) %>% 
  opt_row_striping() %>% 
  tab_options(container.height = px(400))

Now we can filter the initial release date for each package and visualise the number of CRAN packages created over time:

df_hist %>% 
  group_by(package) %>% 
  slice_min(order_by = date, n = 1) %>% 
  ungroup() %>% 
  mutate(month = floor_date(date, unit = "month")) %>% 
  count(month) %>% 
  arrange(month) %>% 
  mutate(cumul = cumsum(n)) %>% 
  ggplot(aes(month, cumul)) +
  geom_line(col = "purple") +
  theme_minimal() +
  labs(x = "Date", y = "Cumlative count", title = "Cumulative count of CRAN package by date of first release")

Still an impressive growth, and now we give proper emphasis to the early history of CRAN; reaching back all the way to before the year 2000. There are many more angles and visuals that this dataset will allow you to explore.

Notes and suggestions:

• I will keep updating this dataset on a monthly basis. After the initial collection of all the version histories, from now on I only need to update the histories for those packages that have a new version released. This should simplify the process significantly.

• Some ideas for EDA and analysis: how long did packages take from their first release to version 1.0? What type of packages were most frequent in different years? Who are the most productive authors? Can you predict the growth toward 2025?

• All of this analysis can be done directly on the Kaggle platform! On the dataset page, on the top right, you will see a button called “New Notebook”. Click that to get an interactive editor in R or Python and start exploring immediately.

Have fun!