After using R for a few years people come to realize how inefficient loops are (in all languages), even after pre-allocating memory. Then you begin to use the apply functions and revel at their speed. However, once you have datasets > 10-15 million the apply family can begin to creep along (relative to the amount of logic in your function). Luckily, most datasets I work with are between 1 – 5 million records and do not encounter performance issues too often. However, when we do run into large datasets we have our parallel packages.

Demo Machine

While writing this demo from my home laptop I was not seeing performance gains in parallel, because I was running out of RAM to maximize the efficiency of parallel processing. Therefore, I decided to checkout Google’s Cloud and was quite happy with the aesthetics and performance of their platform. Now, I do not work for Google, but it is worth checking out their trial.

custom (8 vCPUs, 32 GB memory)

For Loop

Most programmers know to try and avoid loops whenever possible because of they are extremely resource intensive. However, with languages such as R, the majority of the time spent is manipulating the dataset. Oftentimes this includes cleaning and feature engineering which requires user-defined functions. If you have less than 10,000 observations and the function is not overly complex then for loops are fine. The example below processes 150 observations in three hundredth of a second.

IrisClassifier  <- function(df,i){
  if(df[1][i,] >= 5.8 & df[4][i,] >= 1.8){
    return('virginica')
  }else if(df[1][i,] <= 5.7 & df[4][i,] <= 1){
    return('setosa')
  }else{
    return('versicolor')
  }
}
iris$Check <- ''
system.time(
  for(i in 1:nrow(iris)){
    iris$Check[i] <- IrisClassifier(iris,i)
  }
)

Vectorized Operations

How does the apply family speed up our functions? It converts a scalar (single item) process to a vector (array) process. Think of a mill having to cut lumber piece-by-piece or having one large saw and applying it to all of the lumber at once.

IrisClassifier  <- function(col1,col2){
  if(col1 >= 5.8 & col2 >= 1.8){
    return('virginica')
  }else if(col1 <= 5.7 & col2 <= 1){
    return('setosa')
  }else{
    return('versicolor')
  }
}

system.time(iris$Check <- mapply(psuedoClassifier,iris[,1],iris[,4]))

Loop vs. mapply

Required Libraries:

library(doSNOW)
library(foreach)
library(data.table)
library(sqldf)
library(ggplot2)

The Dataset

For this example we will be using a simple dataset starting with 1,000,000 million observations and increment by 1,000,000 to 30,000,000 million observations. The entire script is located at the end of this example.

df <-   df <- data.frame(a = round(rnorm(values[i],mean = 18,sd = 10),0),
                   b = round(rnorm(values[i],mean = 10,sd = 11),0),
                   c = round(rnorm(values[i],mean = 400,sd = 44),0),
                   d = round(rnorm(values[i],mean = 100,sd = 33),0),
                   e = round(rnorm(values[i],mean = 11,sd = 3),0),
                   f = round(rnorm(values[i],mean = 223,sd = 1000),0)
  )

Test Function

For this example we will use the mapply to show how to apply a function with multiple parameters. I find most articles online only demonstrating how to apply with one parameter so I used this as an excuse to demonstrate applying a function with multiple parameters.

unspirited.func <- function(a,b,c,d,e,f){
  if(a > 20 & b  300 & c  30 & log(c)> 6) {
    return('Fail - A')
  }else if(b > 38 & log(c)> 6) {
    return('Fail- B')
  }else if(d > 175 & log(c)> 6) {
    return('Fail - D')
  }else if(e > 18 & log(c)> 6) {
    return('Fail - E')
  }else if(f > 1989 & log(c)> 6) {
    return('Fail - F')
  }else{
    return('Unsure')
  }  
}

Comparing For Loop to Vectorized Operations

Mapply vs. Parallel Mapply

Parallel Operations
Luckily most current laptops, pcs, and cloud machines all offer multiple cores of processing power. Therefore, how does parallel processing work? Essentially, R starts up n number of instances and sends subsets of the original data to be processed in those instances using its own processing core, and then finally returning the results back together.

Considerations

• Required Setup time
• Parallel will end up using more RAM
• Shared server resources

Final Note

After hearing Max Kuhn speak a few years ago he said always check to see if someone has already written the function you are coding. After, writing this post I did stumble upon some builtin in parallel apply functions. However, I have not researched exactly how these operate.

The Entire Script

###############################################################################################################################
# Name             :  Parallel Process Example 
# Date             :  2016-09-03 
# Author           :  Christopher M
# Dept             :  Business Analytics 
# Purpose          :  ~
###############################################################################################################################
# ver    user        date(YYYYMMDD)        change  
# 1.0     ~          20160903              initial
############################################################################################################################### 

library(parallel)
library(ggplot2)

## Display number of CPU cores on host
detectCores(all.tests = FALSE, logical = TRUE)

## Example function for processing
unspirited.func <- function(a,b,c,d,e,f){
  
  if(a > 20 & b  300 & c  30 & log(c)> 6) {
    return('Fail - A')
  }else if(b > 38 & log(c)> 6) {
    return('Fail- B')
  }else if(d > 175 & log(c)> 6) {
    return('Fail - D')
  }else if(e > 18 & log(c)> 6) {
    return('Fail - E')
  }else if(f > 1989 & log(c)> 6) {
    return('Fail - F')
  }else{
    return('Unsure')
  } 
  
}

## kfold function to split data for parallel processing
kfold <- function(df,k){
  require(sqldf)
  if( length(which(colnames(df)=='folds'))>0){
    stop(print('Rename column(s) named folds'))
    break
    
  }
  #randomize the data
  df<-sqldf("SELECT * FROM df order by random()")
  
  #Create equally size folds
  df$folds <- cut(seq(1,nrow(df)),breaks=k,labels=FALSE)
  
  return(df)
  
}

## function for parallel processing
parallelApply <- function(df,cols,func,cores){
  
  require(doSNOW)
  require(foreach)
  require(data.table)
  cl <- makeCluster(cores, type="SOCK")
  registerDoSNOW(cl)
  
  ## Create ID to compute in parallel
  df <- kfold(df,cores)
  
  ## Process data in parallel
  system.time(NewDataFrame <- foreach(i = 1:cores) %dopar% {
    tmpdata <- subset(df, folds == i)
    mapply(func,tmpdata[,cols[1]],tmpdata[,cols[2]],tmpdata[,cols[3]],tmpdata[,cols[4]],tmpdata[,cols[5]],tmpdata[,cols[6]])
    
  })
  
  ## Bind data back into dataframe
  NewDataFrame <- rbindlist(Map(as.data.frame, NewDataFrame))
  colnames(NewDataFrame) <- c('ParRow')
  df <- cbind(df,NewDataFrame)
  stopCluster(cl)
  return(df)
  
}

## values for dataframe length
values <- seq(1e6,30e6,1e6)

capture <- data.frame()

for(i in 1:length(values)){
  print(paste("processing:",values[i],"rows. There are",length(values)-i,"iterations left to complete"))
  
  df <- data.frame(a = round(rnorm(values[i],mean = 18,sd = 10),0),
                   b = round(rnorm(values[i],mean = 10,sd = 11),0),
                   c = round(rnorm(values[i],mean = 400,sd = 44),0),
                   d = round(rnorm(values[i],mean = 100,sd = 33),0),
                   e = round(rnorm(values[i],mean = 11,sd = 3),0),
                   f = round(rnorm(values[i],mean = 223,sd = 1000),0)
  )
  ##______________________________________________________________________________
  ## Apply
  
  performance <- system.time(df$Check <- mapply(unspirited.func,df[,1],df[,2],df[,3],df[,4],df[,5],df[,6]))
  capture <- rbind(capture,data.frame(Type = 'mapply',Rows = values[i],elapsed = performance[3],PercentDifference = 1))
  df$Check <- NULL
  print('finished')
  print(capture)
  ##______________________________________________________________________________
  ## Parallel Apply
  base <- as.numeric(subset(capture,Rows == values[i] & Type == 'mapply')[3])  
  for(x in 2:6){
    print(paste('running on',x,'cores'))
    performance <- system.time(df <- parallelApply(df,c(1,2,3,4,5,6),unspirited.func,x))
    capture <- rbind(capture,data.frame(Type = paste('Parallel-',x,sep=''),Rows = values[i],elapsed = performance[3],
                                        PercentDifference = abs(1-(performance[3]/base))))
    df$ParRow <- NULL
    df$folds <- NULL
  }
  gc()
  print(capture)
}
options(scipen=999)

## Example plots
ggplot(subset(capture,Type != 'Loop'),aes(Rows,elapsed,color = Type))+
  geom_line( aes(linetype=Type))+
  geom_point(size=3 )+
  ylab("elapsed (seconds)") +
  geom_text(data=subset(capture,Rows ==max(capture$Rows)),
            aes(label=Type))+
  ggtitle("Mapply  ~ Parallel")


tmp <- subset(capture,Type != 'mapply')
tmp$PercentDifference <- tmp$PercentDifference + 1
mapply <- subset(capture,Type == 'mapply')
capture_ <- rbind(tmp,mapply)

ggplot(subset(capture_,Type != 'Loop'),aes(Rows,PercentDifference,color = Type))+
  geom_line( aes(linetype=Type))+
  geom_point(size=3 )+
  ylab("Performance Increase") +
  ggtitle("Mapply  ~ Parallel") +
  scale_x_continuous(breaks = seq(5e6,30e6,5e6))+
  scale_y_continuous(breaks = seq(0,3,.1))

1. R Markdown: How to number and reference tables
2. RDBL – manipulate data in-database with R code only
3. R Markdown: How to format tables and figures in .docx files
4. R Markdown: How to insert page breaks in a MS Word document
5. Working on Data-Warehouse (SQL) with R