Testing Different Methods for Merging a set of Files into a Dataframe

I previously posted a method I used for merging a set of files into a dataframe. It wasn’t long before I had some very helpful comments from the R-bloggers community suggesting better methods to achieve my goal. In this post, I compare the different methods and see which is the most efficient (i.e., fastest).

The Methods

My original method is outlined in my post. In the comments, you can see two further methods suggested. One by sayan involves the use of the do.call function and lapply. A second by dan involves the use of plyr‘s ldply function. Check out the comments for the full discussion.

I will therefore compare three methods:

• My original method
• sayan‘s lapply method
• dan’s plyr method

Testing

The R Code

# lapply method
lap = replicate(N, system.time(
  full_data<- do.call(
  "rbind",lapply(file_list, 
  FUN=function(files){read.table(files,
  header=TRUE, sep="\t")})))[3])
lap

# plyr method
ply =
replicate(N, system.time(
  dataset <- ldply(file_list, read.table, header=TRUE, sep="\t")
)[3])
ply

# original method
orig= 
replicate(N, system.time(
for (file in file_list){
  # if the merged dataset doesn't exist, create it
  if (!exists("dataset")){
    dataset <- read.table(file, header=TRUE, sep="\t")
  }
  # if the merged dataset does exist, append to it
  if (exists("dataset")){
    temp_dataset <-read.table(file, header=TRUE, sep="\t")
    dataset<-rbind(dataset, temp_dataset)
    rm(temp_dataset)
  }
}
)[3])
orig