Recently I needed to count lines of code for a project at work work (this is an expression of the person honored in this post), and happened to discover that Bob Rudis had started an R package wrapping the Perl CLOC script. Of course! He has packages for a lot of things! And he’s always ready to help: after I asked him a question about the package, and made a pull request to renew its wrapped CLOC script, he made it all pretty and ready-to-go!

He himself defined his Stack Overflow Driven-Development (SODD) workflow in a blog post: someone will ask him a question on Stack Overflow, and he’ll write a long answer eventually becoming a package, that will or will not make it to CRAN… Which is the motivation of this blog post. How can I output a list of all packages Bob has on GitHub?

The magic of GitHub API V4, ghql and jq/jqr

I’ve recently started using GitHub API V4 instead of V3, that you might have seen on this blog in my post about random seeds and my post about initial commit messages. I’ll probably introduce my workflow more formally and extensively elsewhere but here are a few appetizers:

• Find out why GitHub is using GraphQL. I was particularly receptive to the idea of getting just the data I needed from the API.

• rOpenSci has a very good general purpose GraphQL R client, ghql by Scott Chamberlain, that includes R6 classes of course (being a package of Scott’s) and whose doc examples are for… GitHub API V4!

• One can build queries in GitHub V4 explorer that has auto-completion and the docs on the right so it made, and still makes, things easier for me as a GraphQL n00b.

• One can wrangle JSON using jq via the rOpenSci jqr package which is a real gem. Carl Boettiger wrote about it here, I trusted him and gave it a try and my mind was blown. Now I do not find the syntax intuitive but guess what, there’s an explorer for that too! So I could writeClipboard JSON output from the API and use the explorer to write code.

The results of my recent discoveries, and my using the API for work work stuff, can be found in my very nascent/work-in-progress gs package. By the way if you’re after something similar i.e. high-level, but for GitHub API V3, check ghapi out.

Using my own GitHub package

I first retrieved all GitHub repositories of Bob’s. My gs package iterates for you thanks to a piece of code by Scott Chamberlain.

You’ll note some things could be made easier if the output of my own functions was a bit different so you can imagine that yes, I’ll break my own package from time to time, but at least this gives a taste of what can be done.

You can read the code of my gs::get_repos function on GitHub.

library("magrittr")
repos <- gs::get_repos("hrbrmstr")
repos <- tidyr::separate(repos, name, into = c("owner", "repo"),
                         remove = FALSE, sep = "/")
# remove forks
repos <- dplyr::filter(repos, !is_fork)

I found 274 repos.

head(repos) %>%
  knitr::kable()

Now as you see I’m nearly done since my own function returns the GitHub description of the repo (I had forgotten about that when starting to write this post!) and since Bob seems to often write a description for his repos which we should all do; but I only want packages. Fear not, I already have a recipe for this as well, is_package_repo! It looks whether a repo contains man, R, DESCRIPTION and NAMESPACE. Now I’m pretty sure the code below could be hugely simplified, but I’m in a hurry.

is_pkg <- function(df){
  message(df$name)
  tibble::tibble(owner = df$owner,
                 repo = df$repo,
                 is_pkg = gs::is_package_repo(owner, repo))
}

are_pkgs <- split(repos, repos$name) %>%
  purrr::map_df(is_pkg)

are_pkgs <- dplyr::left_join(repos, are_pkgs, by = c("owner", "repo"))
hrbrpkgs <- dplyr::filter(are_pkgs, is_pkg)

I found 170 packages whose whole table I’ll put at the very end of this post. I can’t wait to browse it!

The end?

What can you do with this list? Explore it to find the package(s) you didn’t know you needed, or the package(s) you’d like to contribute to, because I can recommend contributing to Bob’s packages, and he encourages people to!

I’ve got a GH full of diverse #rstats ? repos that all need tweaks/enhancements ranging from docs to ƒ()s. I’ll gladly help anyone — esp new PRers — contribute.

— hrbr'Challa (@hrbrmstr) 13 de gener de 2018

And as promised here is the table!

library("magrittr")
hrbrpkgs %>%
  # thanks https://stackoverflow.com/questions/30901027/convert-a-column-of-text-urls-into-active-hyperlinks-in-shiny
  dplyr::mutate(link = glue::glue("https://github.com/hrbrmstr/{repo}"),
                link = glue::glue("<a href='{link}'>{link}</a>")) %>%
  dplyr::mutate(created_at = lubridate::as_date(created_at),
                updated_at = lubridate::as_date(updated_at)) %>%
  dplyr::rename(package = repo) %>%
  dplyr::select(package, description, created_at, updated_at, link) %>%
  dplyr::arrange(package) %>%
  knitr::kable()