How to Scrape and Store Strava Data Using R

Set Up Targets

The whole data pipeline is implemented with the help of the targets package. You can learn more about the package and its functionalities here.

In order to reproduce the analysis, perform the following steps:

• Clone the repository: https://github.com/duju211/pin_strava
• Install the packages listed in the ‘libraries.R’ file
• Run the target pipeline by executing targets::tar_make() command
• Follow the instructions printed in the console

OAuth Dance from R

The Strava API requires an ‘OAuth dance’, described below.

< size="4">1. Create an OAuth Strava app< >
To get access to your Strava data from R, you must first create a Strava API. The steps are documented on the Strava Developer site. While creating the app, you’ll have to give it a name. In my case, I named it r_api.

After you have created your personal API, you can find your Client ID and Client Secret variables in the Strava API settings. Save the Client ID as STRAVA_KEY and the Client Secret as STRAVA_SECRET in your R environment.¹

STRAVA_KEY=<Client ID>
STRAVA_SECRET=<Client Secret>

Then, you can run the function define_strava_app shown below.

define_strava_app <- function() {
  oauth_app(
    appname = "r_api",
    key = Sys.getenv("STRAVA_KEY"),
    secret = Sys.getenv("STRAVA_SECRET")
  )
}

< size="4">2. Define an endpoint< >

Define an endpoint called my_endpoint using the function define_strava_endpoint.

The authorize parameter describes the authorization url and the access argument exchanges the authenticated token.

define_strava_endpoint <- function() {
  oauth_endpoint(request = NULL,
                 authorize = "https://www.strava.com/oauth/authorize",
                 access = "https://www.strava.com/oauth/token")
}

< size="4">3. The final authentication step< > Before you can execute the following steps, you have to authenticate the API in the web browser.

define_strava_sig <- function(endpoint, app) {
  oauth2.0_token(
    endpoint,
    app,
    scope = "activity:read_all,activity:read,profile:read_all",
    type = NULL,
    use_oob = FALSE,
    as_header = FALSE,
    use_basic_auth = FALSE,
    cache = FALSE
  )
}

The information in my_sig can now be used to access Strava data. Set the cue_mode of the target to ‘always’ so that the following API calls are always executed with an up-to-date authorization token.

Access Activities

You are now authenticated and can directly access your Strava data.

< size="4">1. Load all activities< >
Load a table that gives an overview of all the activities from the data. Because the total number of activities is unknown, use a while loop. It will break the execution of the loop if there are no more activities to read.

read_all_activities <- function(sig) {
  activities_url <- parse_url("https://www.strava.com/api/v3/athlete/activities")

  act_vec <- vector(mode = "list")
  df_act <- tibble::tibble(init = "init")
  i <- 1L

  while (nrow(df_act) != 0) {
    r <- activities_url %>%
      modify_url(query = list(
        access_token = sig$credentials$access_token[[1]],
        page = i
      )) %>%
      GET()

    df_act <- content(r, as = "text") %>%
      fromJSON(flatten = TRUE) %>%
      as_tibble()
    if (nrow(df_act) != 0)
      act_vec[[i]] <- df_act
    i <- i + 1L
  }

  df_activities <- act_vec %>%
    bind_rows() %>%
    mutate(start_date = ymd_hms(start_date))
}

The resulting data frame consists of one row per activity:

## # A tibble: 605 x 60
##    resource_state name  distance moving_time elapsed_time total_elevation~ type 
##             <int> <chr>    <dbl>       <int>        <int>            <dbl> <chr>
##  1              2 "Hes~   31153.        4699         5267            450   Ride 
##  2              2 "Bam~    5888.        2421         2869            102.  Run  
##  3              2 "Lin~   33208.        4909         6071            430   Ride 
##  4              2 "Mon~   74154.       10721        12500            641   Ride 
##  5              2 "Cha~   34380         5001         5388            464.  Ride 
##  6              2 "Mor~    5518.        2345         2563             49.1 Run  
##  7              2 "Bin~   10022.        3681         6447            131   Run  
##  8              2 "Tru~   47179.        8416        10102            898   Ride 
##  9              2 "Sho~   32580.        5646         6027            329.  Ride 
## 10              2 "Mit~   33862.        5293         6958            372   Ride 
## # ... with 595 more rows, and 53 more variables: workout_type <int>, id <dbl>,
## #   external_id <chr>, upload_id <dbl>, start_date <dttm>,
## #   start_date_local <chr>, timezone <chr>, utc_offset <dbl>,
## #   start_latlng <list>, end_latlng <list>, location_city <lgl>,
## #   location_state <lgl>, location_country <chr>, start_latitude <dbl>,
## #   start_longitude <dbl>, achievement_count <int>, kudos_count <int>,
## #   comment_count <int>, athlete_count <int>, photo_count <int>, ...

< size="4">2. Preprocess activities< >
Make sure that all ID columns have a character format and improve the column names.

pre_process_act <- function(df_act_raw, athlete_id) {
  df_act <- df_act_raw %>%
    mutate(across(contains("id"), as.character),
           `athlete.id` = athlete_id)
}

< size="4">3. Extract activity IDs< >
Use dplyr::pull() to extract all activity IDs.

Read Measurements

< size="4">1. Read the ‘stream’ data from Strava< >
A ‘stream’ is a nested list (JSON format) with all available measurements of the corresponding activity.

To get the
available variables and turn the result into a data frame, define a helper function read_activity_stream. This function takes an ID of an activity and an authentication token, which you created earlier.

The target is defined with dynamic branching which maps over all activity IDs. Define the cue mode as never to make sure that every target runs exactly once.

read_activity_stream <- function(id, sig) {
  act_url <-
    parse_url(stringr::str_glue("https://www.strava.com/api/v3/activities/{id}/streams"))
  access_token <- sig$credentials$access_token[[1]]

  r <- modify_url(act_url,
                  query = list(
                    access_token = access_token,
                    keys = str_glue(
                      "distance,time,latlng,altitude,velocity_smooth,cadence,watts,
                      temp,moving,grade_smooth"
                    )
                  )) %>%
    GET()

  stop_for_status(r)

  fromJSON(content(r, as = "text"), flatten = TRUE) %>%
    as_tibble() %>%
    mutate(id = id)
}

< size="4">2. Bind the single targets into one data frame< >
You can do this using dplyr::bind_rows().

The data now is represented by one row per measurement series:

## # A tibble: 4,821 x 6
##    type            data              series_type original_size resolution id    
##    <chr>           <list>            <chr>               <int> <chr>      <chr> 
##  1 moving          <lgl [4,706]>     distance             4706 high       62186~
##  2 latlng          <dbl [4,706 x 2]> distance             4706 high       62186~
##  3 velocity_smooth <dbl [4,706]>     distance             4706 high       62186~
##  4 grade_smooth    <dbl [4,706]>     distance             4706 high       62186~
##  5 distance        <dbl [4,706]>     distance             4706 high       62186~
##  6 altitude        <dbl [4,706]>     distance             4706 high       62186~
##  7 heartrate       <int [4,706]>     distance             4706 high       62186~
##  8 time            <int [4,706]>     distance             4706 high       62186~
##  9 moving          <lgl [301]>       distance              301 high       62138~
## 10 latlng          <dbl [301 x 2]>   distance              301 high       62138~
## # ... with 4,811 more rows

< size="4">3. Turn the data into a wide format< >

meas_wide <- function(df_meas) {
  pivot_wider(df_meas, names_from = type, values_from = data)
}

In this format, every activity is one row again:

## # A tibble: 605 x 14
##    series_type original_size resolution id         moving latlng velocity_smooth
##    <chr>               <int> <chr>      <chr>      <list> <list> <list>         
##  1 distance             4706 high       6218628649 <lgl ~ <dbl ~ <dbl [4,706]>  
##  2 distance              301 high       6213800583 <lgl ~ <dbl ~ <dbl [301]>    
##  3 distance             4905 high       6179655557 <lgl ~ <dbl ~ <dbl [4,905]>  
##  4 distance            10640 high       6160486739 <lgl ~ <dbl ~ <dbl [10,640]> 
##  5 distance             4969 high       6153936896 <lgl ~ <dbl ~ <dbl [4,969]>  
##  6 distance             2073 high       6115020306 <lgl ~ <dbl ~ <dbl [2,073]>  
##  7 distance             1158 high       6097842884 <lgl ~ <dbl ~ <dbl [1,158]>  
##  8 distance             8387 high       6091990268 <lgl ~ <dbl ~ <dbl [8,387]>  
##  9 distance             5587 high       6073551706 <lgl ~ <dbl ~ <dbl [5,587]>  
## 10 distance             5281 high       6057232328 <lgl ~ <dbl ~ <dbl [5,281]>  
## # ... with 595 more rows, and 7 more variables: grade_smooth <list>,
## #   distance <list>, altitude <list>, heartrate <list>, time <list>,
## #   cadence <list>, watts <list>

< size="4">4. Preprocess and unnest the data< >

The column latlng needs special attention, because it contains latitude and longitude information. Separate the two measurements before unnesting all list columns.

meas_pro <- function(df_meas_wide) {
  df_meas_wide %>%
    mutate(
      lat = map_if(
        .x = latlng,
        .p = ~ !is.null(.x),
        .f = ~ .x[, 1]
      ),
      lng = map_if(
        .x = latlng,
        .p = ~ !is.null(.x),
        .f = ~ .x[, 2]
      )
    ) %>%
    select(-c(latlng, original_size, resolution, series_type)) %>%
    unnest(where(is_list))
}

After this step, every row is one point in time and every column is a measurement at this point in time (if there was any activity at that moment).

## # A tibble: 2,176,926 x 12
##    id      moving velocity_smooth grade_smooth distance altitude heartrate  time
##    <chr>   <lgl>            <dbl>        <dbl>    <dbl>    <dbl>     <dbl> <dbl>
##  1 621862~ FALSE             0             1.8      0       527        149     0
##  2 621862~ TRUE              0             1.2      5       527.       150     1
##  3 621862~ TRUE              0             0.9     10.9     527.       150     2
##  4 621862~ TRUE              5.68          0.8     17       527.       150     3
##  5 621862~ TRUE              5.81          0.8     23.3     527.       150     4
##  6 621862~ TRUE              5.88          0.8     29.4     527.       150     5
##  7 621862~ TRUE              6.13          0.8     35.6     527.       151     6
##  8 621862~ TRUE              6.15          0       41.6     527.       150     7
##  9 621862~ TRUE              6.14          0       47.8     527.       150     8
## 10 621862~ TRUE              6.13          0.8     53.9     527.       150     9
## # ... with 2,176,916 more rows, and 4 more variables: cadence <dbl>,
## #   watts <dbl>, lat <dbl>, lng <dbl>

Create Visualisation

Visualize the final data by displaying the geospatial information in the data. Every facet is one activity. Keep the rest of the plot as minimal as possible.

vis_meas <- function(df_meas_pro) {
  df_meas_pro %>%
    filter(!is.na(lat)) %>%
    ggplot(aes(x = lng, y = lat)) +
    geom_path() +
    facet_wrap( ~ id, scales = "free") +
    theme(
      axis.line = element_blank(),
      axis.text.x = element_blank(),
      axis.text.y = element_blank(),
      axis.ticks = element_blank(),
      axis.title.x = element_blank(),
      axis.title.y = element_blank(),
      legend.position = "bottom",
      panel.background = element_blank(),
      panel.border = element_blank(),
      panel.grid.major = element_blank(),
      panel.grid.minor = element_blank(),
      plot.background = element_blank(),
      strip.text = element_blank()
    )
}

And there it is: All your Strava data in a few tidy data frames and a nice-looking plot. Future updates to the data shouldn’t take too long, because only measurements from new activities will be downloaded. With all your Strava data up to date, there are a lot of appealing possibilities for further data analyses of your fitness data.

Note from the Editor: Julian’s post neatly breaks down complex tasks, walking readers through the steps as well as rationale of his decisions. His use of the targets package demonstrates how an organized workflow enables replicability and ease. In addition, Julian showcases how the R programming language can fulfill a vision sparked by one’s passions. It is an inspiring example of how we can use R to create something that is informative, beautiful, and personal.

1. You can edit your R environment by running usethis::edit_r_environ(), saving the keys, and then restarting R.↩︎