This journey started almost exactly a year ago, but it’s finally been sufficiently worked through and merged! Yay, I’ve officially contributed to the tidyverse (minor as it may be).

It began with a tweet, recalling a surprise I encountered that day during some routine data processing

Source of today's mild heart-attack: I have categories W, X_Y, and Z in some data. Intending to keep only the second two:

data %>% filter(g %in% c(“X Y”, “Z”)

Did you spot that I used a space instead of an underscore? I sure as heck didn't, and filtered excessively to just Z.— Jonathan Carroll (@carroll_jono) March 6, 2018

For those of you not so comfortable with pipes and dplyr, I was trying to subset a data.frame ‘data‘ (with a column g having values "W", "X_Y" and "Z") to only those rows for which the column g had the value "X_Y" or "Z" (not the actual values, of course, but that’s the idea). Without dplyr this might simply be

data[data$g %in% c("X Y", "Z"), ]

To make that more concrete, let’s actually show it in action

data <- data.frame(a = 1:5, g = c("X_Y", "W", "Z", "Z", "W"))
data
#>   a   g
#> 1 1 X_Y
#> 2 2   W
#> 3 3   Z
#> 4 4   Z
#> 5 5   W

data %>% 
   filter(g %in% c("X Y", "Z"))
#>   a g
#> 1 3 Z
#> 2 4 Z

filter isn’t at fault here — the same issue would arise with [ — I have mis-specified the values I wish to match, so I am returned only the matching values. %in% is also performing its job – it returns a logical vector; the result of comparing the values in the column g to the vector c("X Y", "Z"). Both of these functions are behaving as they should, but the logic of what I was trying to achieve (subset to only these values) was lost.

Now, in some instances, that is exactly the behaviour you want — subset this vector to any of these values… where those values may not be present in the vector to begin with

data %>% 
   filter(values %in% all_known_values)

The problem, for me, is that there isn’t a way to say “all of these should be there”. The lack of matching happens silently. If you make a typo, you don’t get that level, and you aren’t told that it’s been skipped

simpsons_characters %>% 
   filter(first_name %in% c("Homer", "Marge", "Bert", "Lisa", "Maggie")

Technically this is a double-post because I also want to sidenote this with something I am amazed I have not known about yet (I was approximately today years old when I learned about this)… I’ve used regexmatching for a while, and have been surprised at how well I’ve been able to make it work occasionally. I’m familiar with counting patterns ((A){2} to match two occurrences of A) and ranges of counts ((A){2,4} to match between two and four occurrences of A) but I was not aware that you can specify a number of mistakes that can be included to still make a match… 

grep("Bart", c("Bart", "Bort", "Brat"), value = TRUE)
#> [1] "Bart"

grep("(Bart){~1}", c("Bart", "Bort", "Brat"), value = TRUE)
#> [1] "Bart" "Bort"

(“Are you matching to me?”… “No, my regex also matches to ‘Bort’”)

data %>% 
   filter(fct_match(g, c("X Y", "Z")))
#> Error in filter_impl(.data, quo): Evaluation error: Levels not present in factor: "X Y".

data %>% 
   filter(fct_match(g, c("X_Y", "Z")))
#>   a   g
#> 1 1 X_Y
#> 2 3   Z
#> 3 4   Z

data %>%
   pull(g) %>%
   (function(x) !fct_match(x, c("X_Y", "Z")))
#> [1] FALSE  TRUE FALSE FALSE  TRUE

fct_exclude <- function(f, lvls, ...) !fct_match(f, lvls, ...)

data %>%
   pull(g) %>%
   fct_exclude(c("X_Y", "Z"))
#> [1] FALSE  TRUE FALSE FALSE  TRUE