Project management for scalable data analysis

Downloading the template

To download this project template, we will need to set up git on your machine first. I will have much more to say about git and GitHub in a later post in the series that will directly concern the process of versioning and sharing reproducible research, but for now we’ll only cover what is needed to get our project template from GitHub. If you are on a Mac, git is most likely already on your machine. To verify that this is the case you can open up the ‘Terminal.app’ on your machine and type this command at the prompt:

which git

If git is already installed, then a path to this software, similar to the one below, will be returned.

/usr/local/bin/git

If a path is not returned, or you are on a PC, then you will need to download and install the software from the git homepage. Follow this link to the git downloads page.

Figure 1: Git downloads page.

Download the installer for you operating system and run this installer with the default installation recommendations.

Now that you have git on your machine, let’s set some global options that will personalize your software while since we are already working at the command line. If you are on a Mac or Linux, open the ‘Terminal’. If you are on a Windows machine, navigate to the programs menu and open ‘git bash’.

At the terminal, enter the following commands –replacing ‘Your Name’ and ‘your@email.com’ with your information.

git config --global user.name 'Your Name'
git config --global user.email 'your@email.com'

Next, let’s download the project, ‘recipes-project_template’ from my personal GitHub repository. To do this, you will want to first decide where on your machine you will like to store this project. If you are following the Recipe series, I recommend that you create a new directory called Recipes/ somewhere convenient on your hard disk. You can then use this directory to house this and other upcoming projects associated with posts in this series.

To create this directory, I’ll use the mkdir command, or ‘make directory’. The ~ is a shortcut operator for the current users home directory. On my Mac the full path would be /Users/francojc/Documents/Recipes/.

mkdir ~/Documents/Recipes/

Once we have created the main directory Recipes/ to house the repository, we need to navigate to that directory using cd, or ‘change directory’.

cd ~/Documents/Recipes/

Verify that your current working directory is correct by entering pwd, or ‘path to working directory’ to get the current directory’s path.

pwd

The result should print the path to your Recipes/ directory.

Now we are ready to use git to copy the remote project from my GitHub repository recipes-project_template to our current directory. Enter the following command at the terminal prompt.

git clone https://github.com/francojc/recipes-project_template.git

You should get some information about the download, or clone, that looks something similar to the output below.

Cloning into 'recipes-project_template'...
remote: Counting objects: 48, done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 48 (delta 21), reused 38 (delta 15), pack-reused 0
Unpacking objects: 100% (48/48), done.

Now cd into the recipes-project_template/ directory that you just cloned into your Recipes/ directory.

You can now inspect the new directory, subdirectories, and files that now reside in the recipes-project_template/ directory with either at the command line, or with your operating systems file explorer. At the command line you can use the ls, or ‘list structure’ command like so:

ls

You should see the following output.

README.md   code        figures     log
_pipeline.R data        functions   report

We can now leave the Terminal, or git bash, and return to RStudio. We are now ready to link an R project to our cloned project.

Creating an R Project within RStudio

As we have seen, RStudio provides a of host tools for facilitating work with R. A feature that makes working with the sometimes numerous data files, scripts, and other resources more manageable is the ‘R Project’ tool. In a nutshell, this RStudio tool allows us to select a directory where our project files live and effectively group these files and the work we do as a unit. At a basic level it simply helps manage individual projects more easily. As we move on to other posts in this series, and particularly when we discuss creating reproducible research, we will see that this feature will really prove its worth. For now, let’s make the project template an R project and turn to focus on the file and directory structure as it relates to doing efficient and reproducible data analysis in R.

To link our template to an R Project, start up R and select the ‘New Project…’ dialogue from the RStudio toolbar menu. You will be presented with various options as seen in Figure 2.

Figure 2: Options for creating an R Project in RStudio.

The first option is for starting a project from scratch. The last option is for cloning a project from a versioning repository like GitHub. With versioning software, like git, on our machine, we can clone and create an R Project in one step. We will make use of this option in future posts now that we have a basic understand of git and GitHub. For now, however, we want to link the project template we cloned manually using the command line, so select ‘Existing Directory’ from this menu.

Next navigate to the directory which we cloned either typing the path to the directory, or more conveniently using the ‘Browse’ button. Once we have selected the directory and create the R Project, RStudio will open a new session with our directories and files listed in the Files pane.

Figure 3: View of the project template as an R Project.

You will notice that RStudio has created a file named recipes-project_template.Rproj. From now on you can navigate to this file using your operating system’s file explorer and open it to return working on this project. Your workspace settings, history, environment variables, etc. will be restored to the last time you were working on the project –more on this later.

Directories

This template includes directories for data (data), code (code), and communicating findings (report). These directories are core to your project and where the heavy lifting takes place. The data and report directories have important subdirectories that separate key stages in your analysis. data/original is where the data in its raw form will be stored and data/derived is where any changes you make to the data for the particular analysis are stored. This distinction is an important one; to safeguard our analysis and to ensure that our analysis is reproducible we do not want to make changes to the original data that are not reflected in the project itself. The subdirectories of report separate the potential formats that we may use to communicate insight generated from this analysis.

Before moving on to discuss the files included in the template, let’s discuss the other three supporting directories: figures, log, and functions. You will most likely generate figures in the course of the analysis. Grouping them together in the figures directory enables us to quickly reference them visually and also include them in any one or all of the reports that may be generated. The log directory is a convenient and easily identifiable place to document aspects of your analysis that will may not picture in your reports. Finally, a directory for housing custom functions you may write to facilitate particular stages of the analysis is provided, functions. We will soon see how powerful and indispensable custom functions are but for now just let me say that keeping them organized in a separate directory will enhance the legibility of your code and help you take full advantage of their power.

Files

This template also includes various file templates that are associated with the tasks typically performed in a data analysis project. The R scripts in the code/ directory are script templates to carry out the sub-tasks of our three main project steps: organize data; get the original data (acquire_data.R), clean and prepare key features of the data (curate_data.R), manipulate the data creating the needed variables and structure for the data analysis (transform_data.R), data analysis; visualize and perform statistical analyses on the data (analyze_data.R), and communicating findings; report results in appropriate formats (generate_reports.R).

Each of these R scripts has a common structure which is outlined using code commenting. Take a look at the structure of the acquire_data.R script, copied below:

# ABOUT -----------------------------------------------------------

# Description: <The aim of this script>
# Usage: <How to run this script: what input it requires and output produced>
# Author: <Your name>
# Date: <current date>

# SETUP -----------------------------------------------------------

# Script-specific options or packages

# RUN -------------------------------------------------------------

# Steps involved in acquiring and organizing the original data

# LOG -------------------------------------------------------------

# Any descriptives that will be helpful to understand the results of this
# script and how it contributes to the aims of the project

# CLEAN UP --------------------------------------------------------

# Remove all current environment variables
rm(list = ls())

The template shown here leverages code commenting to separate the script into meaningful tasks. The ABOUT section is where you will provide an overview of the purpose of the script in your project, how to use it, who created it, and when it was created. The SETUP section provides a space to load any required packages, source any required custom functions, and configure various other options. RUN is where the bulk of your code will be entered. As it has been stated various times, commenting is a vital part of sound coding practices. It is often helpful not only to comment the particular line of code, which we do by adding the # symbol to the immediate right of the code and then describe the task, but also to group coding sub-tasks in this section. RStudio provides a tool to create comment sections. You can use this tool by selecting ‘Code > Insert Section…’ or the keyboard shortcut shift + command + R (Mac) or shift + ctrl + R (PC). Either approach will trigger a dialogue box to enter the name of the section. Once you have entered the name it will then appear in the section listing, as seen in Figure 4.

Figure 4: View of the section listing in RStudio.

You can use this listing to skip from section to section which can be very helpful as your script becomes more complicated with subsequent code.

The last two sections LOG and CLEANUP are good-housekeeping sections. LOG is where you can divert any meta-information about the analysis to the log/ directory. This is an important step to take at this point as the last section, CLEANUP, is where you will remove any of the objects your script has created with the rm(list = ls() command.

Although removing objects created is not strictly required, it has two key benefits: it will free up any memory that these objects claim from our R session and it helps keep each script as modular as possible. Freeing up memory after an object is no longer needed is good practice as memory handling in R is not one of the language’s strong points. Striving for modularity speaks more to reproducibility and analysis workflow. If a subsequent step relies on an object generated by a previous script that is held in memory, we must run the previous script in the workflow each time before the next. As the number of tasks increases and as these tasks become more processing intensive, it will lead to an unnecessary loss of computing resources and time. To avoid this scenario, each script in our analysis should only require input that is read from an session-external source; that is from a resource online or from the hard disk. This means that if an object created in a script will be needed a some point in our analysis, it should be written to disk –preferably in a plain-text version.¹

The last of the three main steps in a data analysis project, ‘Reporting results’, is associated with the file generate_reports.R in the code/ directory which is tied to the various files in the report/ directory. These later files have the extension .Rmd, not .R. This distinction reflects the fact that these files are a special type of R script: an RMarkdown script. RMarkdown is a variant of the popular markup language Markdown. RMarkdown goes beyond standard Markdown documents in that they allow for the intermingling of code, prose, and graphics to dynamically create reports in PDF document format (report/article/report.Rmd), presentation slides (report/slides/presentation.Rmd), and interactive web pages (report/web/webpage.Rmd). Data and figures generated in by the R scripts in your analysis can be included in these documents along with citations and a corresponding bibliography sourced from a Bibtex file report/bibliography.bib.² Together these features a provide powerful tool belt for creating publication quality reports. The generate_reports.R file simply runs the commands to render these files in their specific formats.

I have provided rough outlines for each of these RMarkdown output formats. We will explore the details of creating reports later in the series. But for now, I encourage you to browse the RMarkdown gallery and explore the documentation to get a sense of what RMarkdown can do.

The last two files in this template are the _pipeline.R script and the README.md document. README.md is the file used to describe the project in general terms including the purpose of the project, data sources, analysis methods, and other relevant information to help clarify how to reproduce the research with these project files. The README file may or may not include the .md extension. If it does, as in the example in this template, you will have access to the Markdown syntax options to provide word processing style formatting, if needed. If you end up storing you project on a code repository site, such as GitHub, this file will be rendered as a web document and be used as the introduction to your project.

The _pipeline.R script is the master script for your analysis. It is a standard R script and includes the same internal commenting sections as the other .R scripts in the the code/ directory (i.e. ABOUT, SETUP, RUN, LOG, and CLEANUP). This script, however, allows you to run the entire project from data to report in sequential order. In the RUN section you will find sub-steps which call the source() function on each of our processing scripts. Since each script representing a step in our analysis is modular, only the required input is read and output generated for each script. As logging step-specific information is taken care of in each particular script, the LOG section in the _pipeline.R script will most typically only include a call to the sessionInfo() function which reports details on the operating system and the packages and the versions of the packages used in the analysis. This information is vital for reproducing research as it documents the specific conditions that successfully generated the analysis.

R Project sessions

Once the files and directories are linked to an R Project your workspace settings, command history, and objects can be saved at any point and restored to continue working on the analysis. To see this in action, let’s do a little work with this project template. Let’s run the _pipeline.R file. There isn’t much to our analysis at this point, as it is just boilerplate material for the most part, but it will serve to highlight how to work with an R Project session. To run this file, open it from the Files pane. It will appear in the Editor pane where we can now use the keyboard shortcut option + command + R (Mac) or alt + ctrl + R (PC) to run the entire master script.

Once you have run the _pipeline.R script, some new files will appear in your directory structure, seen below.

├── README.md
├── _pipeline.R
├── code/
│   ├── acquire_data.R
│   ├── analyze_data.R
│   ├── curate_data.R
│   ├── generate_reports.R
│   └── transform_data.R
├── data/
│   ├── derived
│   └── original
├── figures/
├── functions/
├── log/
│   └── session_info.txt
├── recipes-project_template.Rproj
└── report/
    ├── article.Rmd
    ├── article.html
    ├── bibliography.bib
    ├── slides.Rmd
    ├── slides.html
    ├── web.Rmd
    └── web.html

The new files appear in the log/ and report/ directories. The session_info.txt file is our log of the session information. The article.html, slides.html, and web.html files are the rendered versions of the RMarkdown templates. If you browse to the History tab in the Environment pane you will see that we have one line in our history –the code that ran the _pipeline.R file.

Let’s quit our R session now by closing RStudio. When prompted, choose ‘Save’ from the ‘Quit R session’ dialogue box. Now reopen our R project by either starting RStudio then choosing ‘File > Recent Projects’ in the RStudio toolbar or by navigating to the recipes-project_template.Rproj file with your operating system’s file explorer and double-clicking it.

Choosing to save our project before closing RStudio has the effect of taking a snapshot of the current workspace. The files that were open on closing the session are returned to the workspace. Any variables we had in memory and the command history are also returned. The details of this snapshot are stored in the files you will now find at the root of our project directory: .RData and .Rhistory.

Figure 5: View of the R project snapshot files .RData and .Rhistory.

If you choose not to save the workspace when quitting RStudio, these files will not be generated, if they do not already exist, or they will not be overwritten by the current session if they already exist.