Prerequisites

Before getting access, you have to fulfill several formal prerequisites to be eligible for application:

• You are either a master’s student, a doctoral candidate, a post-doc, a faculty member, or a research-focused employee at an academic institution or university.
• You have a clearly defined research objective, and you have specific plans for how you intend to use, analyze, and share Twitter data from your research.
• You will use this access for non-commercial purposes.⁴

Furthermore, you need a Twitter account which is also used to log in to the Twitter Developer Platform after a successful application. This portal lets you configure your API projects, keep an eye on your monthly tweet cap⁵, and more. A more detailed explanation of prerequisites can be found on the Twitter API academic research track Track website.

Application

The whole process can be initiated by clicking Apply on the official Twitter API academic research track Website. You’ll be asked to log in with your personal Twitter account.

Figure 1: Twitter Application Steps for academic research track API access.

The figure above visualizes the steps you have to complete before your application can finally be submitted for Twitter’s internal review:

1. Basic Info: such as phone number verification and country selection
2. Academic Profile: such as link to an official profile (department website or similar) and academic role
3. Project Details: such as information about findings, description of the project itself, and how the API should be used there (e.g. methodologies and how the outcomes will be shared)
4. Review: provides an overview of the previous steps
5. Terms: developer agreement and policy

Before starting, it is recommended to carefully read which kind of career levels, projects and data behaviors are not allowed to use the API and thus have a high chance of receiving a refusal for their application. To give an example, if you plan to share the content of tweets publicly, you most probably won’t get access to the API as this would violate the Twitter rules. Again, more detailed information about this can be found on the Twitter API academic research track and Developer Terms information guides.

Step one requests generic information about your Twitter account while in step two you have to provide information about your academic profile. This includes a link to a publicly available record on an official department website and information regarding the academic institution you are working in. The third step is the most sophisticated one: your research project. It asks for short paragraphs about the project in general, what and how Twitter data is used there, and how the outcome of your work is shared with the public. The last two steps, review and terms, do not require any user-specific input but provide an overview of all filled-in information as well as the chance to read the developer agreement and policy.

After submitting your application

After submitting your application, you receive a decision via the e-mail address connected with your Twitter account (usually) within a few days. However, according to Twitter, this process can take up to two weeks.

You application may be rejected for two common reasons: First, you do violate the policy at one point according to the information given, or second, you do not meet the requirements (as described above). Further explanations what can be the next steps after a rejection can be found in the Developer Account Support FAQ.

As of writing this blog post (May 2022), submitting a reapplication for access using the same account is not possible.

Postman-as-a-playground

Postman is an easily accessible application to try out different queries, tokens, and more. Without any programming knowledge, you get the API results immediately. Here you can find an official tutorial to use Postman with the Twitter API.

However, there are several reasons why Postman cannot replace a package and programming code.

• Building flexible queries (e.g., a list of users to retrieve tweets from)
• Handling large responses which come split up during pagination
• Handle rate limit restrictions
• Transforming responses into manageable data structures (e.g., dataframe and comma-separated values)

All of these tasks can be handled by a suitable package in your favorite programming language.

Which package should I choose?

It has to be noted that there are dozens of packages out there but only some of them already integrated the academic research track of the Twitter API. A selection of packages is listed below:

• academictwitteR (R): The package offers customizable fucntions for all common v2 API endpoints. Additionally, it smoothly guides the developer through all critical steps (e.g. authentication or data processing) of the API interaction.
• RTwitterV2 (R): Although RTwitterV2 as of now has less API endpoints included than academictwitteR it still is a valuable alternative which covers all basic functionalities.
• rtweet (R): rtweet does not support the academic research track yet, however it offers much basic functionality by using the previous API version. A dedicated Methods Bites blog post introducing rtweet in detail can be found here.
• searchtweets-v2 (Python): This is the official package developed and maintained by Twitter available for Python. The library offers flexible functions which even handle very specialized requests but one has to dive deeper into the technical aspects of the API.
• tweepy (Python): tweepy is the most common package for Python and backed up by a large developer community. As a bonus, it includes many examples of how to use the various features offered by the package.

Which package you pick should depend on your preferred programming language as well as whether the feature list of a package fits your research purpose.

academictwitteR: a code walkthrough using R

In this blog post academictwitteR (Barrie and Ho 2021) (available for R) is used to demonstrate a simple scenario of retrieving tweets from German members of the parliament. The name academictwitteR is derived by the Twitter API academic research track for which it is developed for.

We will start by first loading all the needed R packages for the walkthrough:

## Save package names as a vector of strings
pkgs <- c("dplyr", "academictwitteR", "quanteda", "purrr")

## Install uninstalled packages
lapply(pkgs[!(pkgs %in% installed.packages())], install.packages)

## Load all packages to library and adjust options
lapply(pkgs, library, character.only = TRUE)

After loading the packages, we need to share our API Bearer Token with academictwitteR. The following code will guide you through the process to store the key in an R-specific environment file (.Renviron) which we introduced earlier in this blog post:

academictwitteR::set_bearer()
## Instructions:
## ℹ 1. Add line: TWITTER_BEARER=YOURTOKENHERE to .Renviron 
##      on new line, replacing YOURTOKENHERE with  actual bearer token
## ℹ 2. Restart R

After restarting R, everything is initialized and we can load a table of Twitter user IDs from German MPs into R:

german_mps <- read.csv("data/MP_de_twitter_uid.csv",
                       colClasses=c("user_id"="character"))
head(german_mps)
##              user_id                   name party
## 1           44608858       Marc Henrichmann   CDU
## 2 819914159915667456      Stephan Pilsinger   CSU
## 3         1391875208   Markus Alexander Uhl   CDU
## 4          569832889 Sigmar Hartmut Gabriel   SPD
## ...

To prevent replication issues with your work, it is recommended to use the Twitter user ID (e.g. 819914159915667456) instead of the user handle (e.g. @StephPilsinger) as the user handle can be changed by the user over time. This would result in not being able anymore to recrawl tweets of these users. In case you only have access to the handle, there is a v2 API endpoint to receive a user object from a handle: /2/users/by/username/:username

Databases and lists of Twitter users can be retrieved from the following sources:

• The Twitter Parliamentarian Database (Vliet, Törnberg, and Uitermark 2020)
• Public Twitter lists (e.g. https://twitter.com/i/lists/912241909002833921)⁶
• legislatoR R Package (Göbel and Munzert 2021)

Afterward, we are ready to crawl our first tweets using a simple wrapper function (get_tweets_from_user()) asking for a single user_id. get_all_tweets(), which is called inside this function is the heart of our code. It manages the generation of queries for the API, working with rate limits as well as storing the data in JSON-files (which can be transformed later).

In case you look for specific content, tweet types, or even topics, you can add another parameter to the package function: query. It allows you to narrow down your search by using specific strings. To give an example, one could look for English retweets containing the keywords putin or selenskij having a geo-location attached. This can be achieved by simply assigning the following string to the query parameter:

(putin OR selenskyj) -is:retweet lang:en has:geo

Beyond that, there exist many more parameters to individualize the crawling method. All of them are documented in the official academictwitteR CRAN documentation of the package. However, in this tutorial I only restrict my search to a Twitter user ID as well as a start and end date for the tweets we are interested in:

# function to retrieve tweets in a specific time period of a single user
# (list of user IDs would be possible but one should keep
# the max. query string of 1024 characters in mind)

get_tweets_from_user <- function(user_id) {
  # Another option is to add "query" parameter
  academictwitteR::get_all_tweets(
    users = user_id,
    start_tweets = "2021-01-01T00:00:00Z",
    end_tweets = "2021-09-30T00:00:00Z",
    data_path = "data/raw/",
    n = 100)
}

The function is then called for each user_id in the dataframe by using walk() from the purrr package (the purrr package allows you to work with functions and vectors):

purrr::walk(german_mps[["user_id"]], get_tweets_from_user)

To import the tweets into a workable format, call bind_tweets() from academictwitteR. It consolidates all available files in the given data_path and organizes them into the requested format (in our case tidy). In addition, only a relevant fraction of columns is selected in the code below by using select() from the dplyr-package.

# concatenate all retrieved tweets into one dataframe and select which columns
# should be kept
# Another option: set parameter "user" to TRUE to retrieve user information
tweets_df <- academictwitteR::bind_tweets(data_path = "data/raw/",
                                          output_format = "tidy") %>%
  dplyr::select(
    tweet_id,
    text,
    author_id,
    user_username,
    created_at,
    sourcetweet_type,
    sourcetweet_text,
    lang
  )

Finally, I store the tweets in a single .csv-file:

write.csv(tweets_df, "data/raw/tweets_german_mp.csv", row.names = FALSE)

Congratulations! You successfully applied to the academic research track, got admitted, and crawled a selection of tweets using the R package academictwitteR.