Using purrr with dplyr

purrr was finally released on CRAN last week. This package is focused on working with lists (and data frames by the same token). However it is not a DSL for lists in the way dplyr is a DSL for data frames. It aims at creating a "better standard lib" focused on functional programming. Purrr should feel like R programming and bring out the elegance of the language. That said, purrr can be a nice companion to your dplyr pipelines especially when you need to apply a function to many columns. In this post I show how purrr's functional tools can be applied to a dplyr workflow.

dplyr provides mutate_each() and summarise_each() for the purpose of mapping functions but I find that they are not as easy to use as the rest of the interface. This is mostly because there is no easy way to map a function to parts of your data frame. It's all columns or nothing. Also, they introduce a custom notation for lambda functions that can be a bit cumbersome. These are two areas where purrr shines in comparison. And since the interface has been designed with pipes in mind, purrr's functions integrate dplyr pipelines quite well.

Mapping to columns conditionally

One of my favourite functions in purrr is map_if(). It accepts a predicate function or a logical vector that specifies which columns should be mapped with a function. This makes it easy to apply a function conditionally, as in the following snippet where we transform all factors to a character vector:

library("purrr")
library("dplyr")
data(diamonds, package = "ggplot2")

diamonds %>% map_if(is.factor, as.character) %>% str()

#> Classes 'tbl_df', 'tbl' and 'data.frame':    53940 obs. of  10 variables:
#>  $ carat  : num  0.23 0.21 0.23 0.29 0.31 0.24 0.24 0.26 0.22 0.23 ...
#>  $ cut    : chr  "Ideal" "Premium" "Good" "Premium" ...
#>  $ color  : chr  "E" "E" "E" "I" ...
#>  $ clarity: chr  "SI2" "SI1" "VS1" "VS2" ...
#>  $ depth  : num  61.5 59.8 56.9 62.4 63.3 62.8 62.3 61.9 65.1 59.4 ...
#>  $ table  : num  55 61 65 58 58 57 57 55 61 61 ...
#>  $ price  : int  326 326 327 334 335 336 336 337 337 338 ...
#>  $ x      : num  3.95 3.89 4.05 4.2 4.34 3.94 3.95 4.07 3.87 4 ...
#>  $ y      : num  3.98 3.84 4.07 4.23 4.35 3.96 3.98 4.11 3.78 4.05 ...
#>  $ z      : num  2.43 2.31 2.31 2.63 2.75 2.48 2.47 2.53 2.49 2.39 ...

Mapping to specific columns

While cleaning a dataset, it is common to apply the same transformation to many variables. For example, reversing a scale or shifting it to zero. Instead of writing a long mutate() call with those transformations, I prefer to do it in one go.

This can be done with map_at() which takes a vector of column positions or column names. For example, let's assume you have written two functions reverse_scale() and shift_to_zero() that should be applied to specific variables. You record those variables in character vectors just before starting the dplyr/purrr pipeline, and then add the relevant map_at() calls.

to_reverse_vars <- c(
  "cyl", "am", "vs",
  "gear", "carb"
)
to_zero_vars <- c(
  "cyl", "gear", "carb"
)

mtcars %>%
  select(-disp) %>%
  map_at(to_reverse_vars, reverse_scale) %>%
  map_at(to_zero_vars, shift_to_zero)

Expanding one column to many with lmap()

lmap()'s story starts with the mysterious tweet and the gist that show up when you google "hadley monads". While I'm not sure I really understand how it is monadic, lmap() is quite useful to extend a data frame without having to deal with binds, merges or having to define new column names.

Let's say you have a numeric variable that you want to discretise for data exploration or modelling (for example, to use as pivot in a ggplot facetting). There are several ways to cut a vector into pieces. Ideally, the cutpoints should be derived from theory, but it's often not possible or too time consuming to do so. In this case, I like to create different categorisations and check if the results are consistent (and investigate when they are not). Let's define two cutting functions, one that tries to create categories with equal sample sizes while the other just uses equal ranges to determine cutpoints.

cut_equal_sizes <- function(x, n = 3, ...) {
  ggplot2::cut_number(x, n, ...)
}

cut_equal_ranges <- function(x, n = 3, ...) {
  cut(x, n, include.lowest = TRUE, ...)
}

It'd be nice to "grow" the data frame at specific numeric columns in such a way that that two news discretised variables appear just next to them with appropriate column names. lmap() is adapted to this because instead of applying a function to the vectors contained in a data frame, it applies it to subsets of size 1 of that data frame. This has several advantages:

• You get the name of the vector as an attribute of the enclosing data frame.

• The usual mapping tools work on columns, so when you return a list or a data frame of vectors, they'll try to stick these inside a list-column, which is not what we want in this case. By comparison, lmap() gives a data frame to a function and expects a data frame in return and has no problem dealing with it when it has more than one column.

Let's write a function to be mapped in such a way. This function doesn't work with vectors but with vectors enclosed in a data frame. It takes and returns a data frame.

cut_categories <- function(x, n = 3) {
  # Record the name of the enclosed vector
  name <- names(x)

  # Create the new columns
  x$cat_n <- cut_equal_sizes(x[[1]], n)
  x$cat_r <- cut_equal_ranges(x[[1]], n)

  # Adjusting the names of the new columns
  names(x)[2:3] <- paste0(name, "_", n, names(x)[2:3])

  x
}

Then we just add a lmap() call to our data cleaning pipeline:

to_discretise_vars <- c(
  "mpg", "disp", "drat",
  "wt", "qsec"
)

mtcars %>% lmap_at(to_discretise_vars, cut_categories) %>% str()

#> Classes 'tbl_df', 'tbl' and 'data.frame':    32 obs. of  21 variables:
#>  $ mpg        : num  21 21 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 ...
#>  $ mpg_3cat_n : Factor w/ 3 levels "[10.4,16.7]",..: 2 2 3 2 2 2 1 3 3 2 ...
#>  $ mpg_3cat_r : Factor w/ 3 levels "[10.4,18.2]",..: 2 2 2 2 2 1 1 2 2 2 ...
#>  $ cyl        : num  6 6 4 6 8 6 8 4 4 6 ...
#>  $ disp       : num  160 160 108 258 360 ...
#>  $ disp_3cat_n: Factor w/ 3 levels "[71.1,146]","(146,293]",..: 2 2 1 2 3 2 3 2 1 2 ...
#>  $ disp_3cat_r: Factor w/ 3 levels "[70.7,205]","(205,338]",..: 1 1 1 2 3 2 3 1 1 1 ...
#>  $ hp         : num  110 110 93 110 175 105 245 62 95 123 ...
#>  $ drat       : num  3.9 3.9 3.85 3.08 3.15 2.76 3.21 3.69 3.92 3.92 ...
#>  $ drat_3cat_n: Factor w/ 3 levels "[2.76,3.17]",..: 2 2 2 1 1 1 2 2 3 3 ...
#>  $ drat_3cat_r: Factor w/ 3 levels "[2.76,3.48]",..: 2 2 2 1 1 1 1 2 2 2 ...
#>  $ wt         : num  2.62 2.88 2.32 3.21 3.44 ...
#>  $ wt_3cat_n  : Factor w/ 3 levels "[1.51,2.81]",..: 1 2 1 2 2 2 3 2 2 2 ...
#>  $ wt_3cat_r  : Factor w/ 3 levels "[1.51,2.82]",..: 1 2 1 2 2 2 2 2 2 2 ...
#>  $ qsec       : num  16.5 17 18.6 19.4 17 ...
#>  $ qsec_3cat_n: Factor w/ 3 levels "[14.5,17]","(17,18.6]",..: 1 1 3 3 1 3 1 3 3 2 ...
#>  $ qsec_3cat_r: Factor w/ 3 levels "[14.5,17.3]",..: 1 1 2 2 1 3 1 2 3 2 ...
#>  $ vs         : num  0 0 1 1 0 1 0 1 1 1 ...
#>  $ am         : num  1 1 1 0 0 0 0 0 0 0 ...
#>  $ gear       : num  4 4 4 3 3 3 3 4 4 4 ...
#>  $ carb       : num  4 4 1 1 2 1 4 2 2 4 ...

The data frame comes out of the pipeline with the new discretised variables nicely arranged and named.

Mapping a function within groups

purrr is also able to deal with dplyr groupings. The groups can be defined with either dplyr::by_group() or purrr::slice_rows(). To apply a function to all columns within groups, just combine a mapping function with the by_slice() adverb:

mtcars %>%
  slice_rows("cyl") %>%
  by_slice(map, ~ .x / sum(.x))