Studying Politics on and with Wikipedia

Application 1: Using Pageviews to Measure Public Attention

Pageviews measure the aggregate number of clicks for a given Wikipedia article. Data on pageviews can be collected from different sources. First, this interactive tool provides summary data which allows users to compare various search items’ popularity in a specified period. Secondly, Wikimedia Downloads, a collection of archived Wikimedia wikis, offers pageviews data through August 2016 as well as data using a new pageviews definition from May 2015 onward.

The code chunk below demonstrates how to collect and graphically display pageviews data using the pageviews package. We use the command article_pageviews(), where the argument project = "en.wikipedia" specifies that we want to collect pageviews of article = "Donald Trump" from the English Wikipedia. We can only restrict our query to a given language edition; it is not possible to limit queries to pageviews from a specific country. We also specify the argument user_type = "user", which ensures that we exclude pageviews generated by bots and spiders. Finally, start and end define the period on which we want to collect pageviews data: July 2015 to May 2017. We proceed analogously for article = "Hillary Clinton".

< summary> Code: Pageviews Data Collection

# get pageviews
trump_views <-
  article_pageviews(
    project = "en.wikipedia",
    article = "Donald Trump",
    user_type = "user",
    start = "2015070100",
    end = "2017050100"
  )
head(trump_views)

clinton_views <-
  article_pageviews(
    project = "en.wikipedia",
    article = "Hillary Clinton",
    user_type = "user",
    start = "2015070100",
    end = "2017050100"
  )

This query allows us to retrieve the pageviews for both Trump’s and Clinton’s Wikipedia articles by date. We can then plot the frequencies of pageviews over time to identify trends in search behaviour. As we can see, the data indicate that Trump attracted considerably more attention than Clinton throughout the 2016 election campaign.

< summary> Code: Plotting Pageviews

# Plot pageviews
plot(ymd(trump_views$date), trump_views$views, col = "red", type = "l", xlab="Time", ylab="Pageviews")
lines(ymd(clinton_views$date), clinton_views$views, col = "blue")
legend("topleft", legend=c("Trump","Clinton"), cex=.8,col=c("red","blue"), lty=1) 

Application 2: Using Article Links to Create a Network Graph of German MPs

The WikipediR package is a wrapper for the MediaWiki API that can be used to retrieve page contents as well as metadata for articles and categories, e.g. information about users or page edit histories. The functionality of the package includes:

• page_content(): Retrieve current article versions (HTML and wikitext as possible output formats)
• revision_content(): Retrieve older versions of the article; this also includes metadata about the revision history
• page_links(): Retrieve outgoing links from the page’s content (which Wikipedia articles does the page link to?)
• page_backlinks(): Retrieve incoming links (which Wikipedia articles link to the page?)
• page_external_links(): Retrieve outgoing links to external sites
• page_info(): Page metadata
• categories_in_page(): What categories is a given page in?
• pages_in_category(): What pages are in a given category?

For our application, we use the page_links() function to extract mutual referrals between the articles on members of the 2017-2021 German Bundestag. We can then use this information to create a network graph of current German MPs. First, we use the legislatoR package to retrieve a list of all German MPs of the 2017-2021 German Bundestag, including information on their page IDs and page titles in the German Wikipedia. Using this information, we then extract all page_links() in every MP’s Wikipedia articles. The third step identifies the subset of links for every MP that link to the Wikipedia article of another current MP.

This allows us to finally plot an interactive network using the forceNetwork() command from the networkD3 package. We can save the interactive network graph as an HTML widget, which is included below.

< summary> Code: Creating an Interactive Network Graph Based on Article Links

## step 1: get info about legislators
dat <- semi_join(
  x = get_core(legislature = "deu"),
  y = filter(get_political(legislature = "deu"), session == 19),
  by = "pageid"
)

## step 2: get page links (max 500 links)
if (!file.exists("studying-politics-wikipedia/data/wikipediR/mdb_links_list.RData")) {
  links_list <- list()
  for (i in 1:nrow(dat)) {
    links <-
      page_links(
        "de",
        "wikipedia",
        page = dat$wikititle[i],
        clean_response = TRUE,
        limit = 500,
        namespaces = 0
      )
    links_list[[i]] <- lapply(links[[1]]$links, "[", 2) %>% unlist
  }
  save(links_list, file = "studying-politics-wikipedia/data/wikipediR/mdb_links_list.RData")
} else{
  load("studying-politics-wikipedia/data/wikipediR/mdb_links_list.RData")
}

## step 3: identify links between MPs
# loop preparation
connections <- data.frame(from = NULL, to = NULL)
# loop
for (i in seq_along(dat$wikititle)) {
  links_in_pslinks <-
    seq_along(dat$wikititle)[str_replace_all(dat$wikititle, "_", " ") %in%
                               links_list[[i]]]
  links_in_pslinks <- links_in_pslinks[links_in_pslinks != i]
  connections <-
    rbind(connections,
          data.frame(
            from = rep(i - 1, length(links_in_pslinks)), # -1 for zero-indexing
            to = links_in_pslinks - 1 # here too
            )
          )
}

# results
names(connections) <- c("from", "to")

# make symmetrical
connections <- rbind(connections,
                     data.frame(from = connections$to,
                                to = connections$from))
connections <- connections[!duplicated(connections), ]


## step 4: visualize connections
connections$value <- 1
nodesDF <- data.frame(name = dat$name, group = 1)

network_out <-
  forceNetwork(
    Links = connections,
    Nodes = nodesDF,
    Source = "from",
    Target = "to",
    Value = "value",
    NodeID = "name",
    Group = "group",
    zoom = TRUE,
    opacityNoHover = 3,
    height = 360,
    width = 636
  )

Using the underlying connections data set, we can also identify which members of the German parliament share the most nodes with others. Perhaps unsurprisingly, we see the German chancellor Angela Merkel on top of the list, followed by a list of current and former federal ministers and (deputy) party leaders.

< summary> Code: Top 10 MPs by Connections Counts

nodesDF$id <- as.numeric(rownames(nodesDF)) - 1
connections_df <-
  merge(connections,
        nodesDF,
        by.x = "to",
        by.y = "id",
        all = TRUE)
to_count_df <- count(connections_df, name)
arrange(to_count_df, desc(n))

## # A tibble: 712 x 2
##    name                     n
##    <fct>                <int>
##  1 Angela Merkel           59
##  2 Andrea Nahles           40
##  3 Heiko Maas              38
##  4 Katarina Barley         38
##  5 Peter Altmaier          38
##  6 Wolfgang Schäuble       38
##  7 Wolfgang Kubicki        37
##  8 Hans-Peter Friedrich    34
##  9 Hermann Gröhe           34
## 10 Ursula von der Leyen    33
## # ... with 702 more rows

Application 3: Using Clickstream Data to Analyze Referral Patterns

Wikipedia articles usually “provide links designed to guide the user to related pages with additional information”. This allows us to collect clickstream data. Clickstreams yield information on the incoming and outgoing traffic of articles. They capture the articles that refer users to a given article as well as the links within a given article that users click to navigate to other articles. Clickstream data are inherently dyadic: Observations represent referral patterns for article-pairs (previous site → current site). Thus, our quantity of interest is the cumulated number of times this pattern was observed in a given period of time.

Clickstream data are offered as monthly aggregate counts for the major Wikipedia language editions. To obtain the data, we first have to download the raw clickstream data from this page, where they are offered as compressed files. After extracting the files, we can load them into R.

In the example below, we focus on two party groups of the 8th (2014-2019) European Parliament: the euroskeptic EFDD (Europe of Freedom and Direct Democracy) and the far right ENF (Europe of Nations and Freedom). In particular, we are interested in clickstreams between the two party groups, between the party groups and their member parties, and between the individual member parties.

Toward this end, we download clickstream data from the English Wikipedia for May 2019, the month of the 2019 European Parliament elections. We identify 19 articles of interest and store them in the object articles. Having retrieved and extracted the clickstream data from May 2019, we import the TSV file into R using read.table(). Lastly, we subset the data to observations that involve referrals between all available article-pairs of the 19 articles.

< summary> Code: Collecting and Processing Clickstream Data

# retrieve article titles of interest
enf <- "Europe_of_Nations_and_Freedom"
efdd <- "Europe_of_Freedom_and_Direct_Democracy"

enf_parties <- c(
  "Freedom_Party_of_Austria",
  "Vlaams_Belang",
  "National_Rally_(France)",
  "The_Blue_Party_(Germany)",
  "Lega_Nord",
  "Party_for_Freedom",
  "Congress_of_the_New_Right"
)

efdd_parties <- c(
  "Svobodní",
  "The_Patriots_(France)",
  "Debout_la_France",
  "Alternative_for_Germany",
  "Five_Star_Movement",
  "Order_and_Justice",
  "Liberty_(Poland)",
  "Brexit_Party",
  "Social_Democratic_Party_(UK,_1990–present)",
  "Libertarian_Party_(UK)"
)

articles <- c(enf, efdd, enf_parties, efdd_parties)

# import raw clickstream data
cs <-
  read.table(
    "clickstream-enwiki-2019-05.tsv",
    header = FALSE,
    col.names = c("prev", "curr", "type", "n"),
    fill = TRUE,
    stringsAsFactors = FALSE
  )
cs$n <- as.integer(cs$n)

# subset
cs <- subset(cs, prev %in% articles & curr %in% articles)

Next, we aim to analyze aggregate referral patterns. We first assign both previous (prev) and current (curr) articles to one of four categories: Articles on the EFDD and ENF parliamentary groups (one article each), articles on ENF member parties (7 articles), and articles on EFDD member parties (10 articles). We then summarize the data to obtain aggregate referral counts between all category pairs. Lastly, we display these in an interactive Sankey diagram using the plotly package.

< summary> Code: Analyzing and Plotting Clickstream Data

# assign categories
cs <- cs %>%
  mutate(
    curr_cat = ifelse(
      curr == enf,
      "ENF Group",
      ifelse(
        curr == efdd,
        "EFDD Group",
        ifelse(curr %in% enf_parties, "ENF Parties",
               "EFDD Parties")
      )
    ),
    prev_cat = ifelse(
      prev == enf,
      "ENF Group",
      ifelse(
        prev == efdd,
        "EFDD Group",
        ifelse(prev %in% enf_parties, "ENF Parties",
               "EFDD Parties")
      )
    )
  )

# summarize data
cs_sum <-  cs %>%
  group_by(curr_cat, prev_cat) %>%
  summarize(n = sum(n)) %>%
  arrange(prev_cat)

# Sankey diagram using plotly
labels <- c(unique(cs_sum$prev_cat), unique(cs_sum$curr_cat))
colors <- ifelse(grepl("EFDD", labels), "#24B9B9", "#2B3856")
sankey_plot <- plot_ly(
  type = "sankey",
  orientation = "h",
  
  node = list(
    label = labels,
    color = colors,
    pad = 15,
    thickness = 15,
    line = list(color = "black",
                width = 0.5)
  ),
  
  link = list(
    source = as.numeric(as.factor(cs_sum$prev_cat)) - 1L,
    target = as.numeric(as.factor(cs_sum$curr_cat)) + 3L,
    value =  cs_sum$n
  ),
  
  height = 340,
  width = 600
) %>%
  layout( = list(size = 10))

The diagram shows that in our data, clickstream dyads involving the articles on the EFDD and ENF parliamentary groups are much more numerous than dyads involving the member parties. Much of this can be attributed to clickstreams between the two party groups, EFDD ↔ ENF. Whereas clickstreams between members of the same parliamentary group are also fairly frequent, clickstreams between the member of one group to a member of the respective other group are rare.

Moving beyond clickstreams between the four categories, we can also visualize the full network structure of all individual articles in our data. The code below starts with some preparatory data management and then uses the igraph package to create the network and to customize its graphical display.

In the final section of the code, we use the intergraph, ggnetwork and plotly packages to produce an interactive HTML5-compatible figure for this blog post. On your own machine, you may skip this section and simply use plot.igraph() on cs_net without transforming the object to a ggplot friendly format.

< summary> Code: Interactive Network Graph

# construct edges
cs_edge <-
  cs %>%
  group_by(prev, curr, prev_cat, curr_cat) %>%
  dplyr::summarise(weight = sum(n)) %>%
  arrange(curr)

# get list of unique articles to construct as nodes
cs_node <- 
  gather(cs_edge,
         `prev`,
         `curr`,
         key = "where",
         value = "article") %>%
  ungroup() %>%
  select(article) %>%
  distinct(article)
names(cs_node) <- c("node")
cs_node$category <-
  ifelse(cs_node$node == enf,
         "ENF Group",
         ifelse(
           cs_node$node == efdd,
           "EFDD Group",
           ifelse(cs_node$node %in% enf_parties, "ENF Parties",
                  "EFDD Parties")
         )
  )

# generate graph
set.seed(3)
cs_net <-
  graph.data.frame(cs_edge,
                   vertices = cs_node,
                   directed = F)
cs_net <-
  igraph::simplify(cs_net, remove.multiple = T, remove.loops = T)

# generate colors based on category
V(cs_net)$color <- 
  ifelse(grepl("EFDD", V(cs_net)$category), "#24B9B9", "#2B3856")

# compute node degrees (#links) and use that to set node size
deg <- igraph::degree(cs_net, mode = "all")
V(cs_net)$size <- deg / 10

# set labels
V(cs_net)$label <- NA
V(cs_net)$label.cex = 0.5
V(cs_net)$label = ifelse(igraph::degree(cs_net) > 5, V(cs_net)$label, NA)
cs_hc_labels <- as.vector(cs_node$node)

# set edge width based on weight
E(cs_net)$width <- log(E(cs_net)$weight) / 5
E(cs_net)$edge.color <- "gray80"

# transform the network to a ggplot friendly format
# (required to generate interactive graph embedded in blog post)
gg_cs_net <-
  ggnetwork(
    cs_net,
    layout = "fruchtermanreingold",
    weights = "weight",
    niter = 50000,
    arrow.gap = 0
  )

cs_plot <- ggplot(gg_cs_net, aes(x = x, y = y, xend = xend, yend = yend)) +
  geom_edges(aes(color = edge.color), size = 0.4, alpha = 0.25) +
  geom_nodes(aes(color = color, size = size)) +
  geom_nodetext(aes(color = color, label = vertex.names, cex = 0.6)) +
  guides(size=FALSE) +
  theme_blank() +
  theme(legend.position = "none")

In the graph above, node diameters indicate the relative weight (total counts) of each article; node colors indicate whether an articles belongs to the EFDD or ENF. We see that members of the same party group tend to share more clickstreams. The Alternative for Germany (AfD), however, shares many connections with members of the ENF. This makes sense when we consider that the AfD has sought closer cooperation with numerous ENF member parties since 2016 with whom it eventually formed the new far right EP group, Identity and Democracy, in June 2019.

Lastly, a word of caution: One should keep in mind that clickstream counts heaviliy depend on how prominently (if at all) outgoing links are placed in a given Wikipedia article. Furthermore, raw counts from an isolated subset of clickstreams (as in the examples above) give no information on the relative importance of a given referral pattern relative to all outgoing referrals of a given article. Users should thus ensure that they use clickstream data in a way that adequately addresses their substantive inquiries.

Application 1: Social Media Adoption Rates

To retrieve legislatoR data, we first load the entire core data set of a given national parliament using the get_core() command. In the example below, we focus on the German Bundestag. We immediately right_join() the core data set with the political component using get_political(). We then filter() the data to retain the legislative session of interest (here, the most recent session of the Bundestag, 2017-2021).

In the next step, we left_join() this data set with the social component using get_social(). This gives us full information on the social media accounts of all MPs of the 2017-2021 Bundestag. Whenever MPs do not have an account, this is stored as missing information (NA). Using this information, we can calculate the social media adoption rates in the German parliament.

< summary> Code: Retrieving legislatoR Data and Calculating Social Media Adoption Rates

## Get social media adoption rates
# get data: Germany
dat_ger <- right_join(
  x = get_core(legislature = "deu"),
  y = filter(
    get_political(legislature = "deu"),
    as.numeric(session) == max(as.numeric(session))
  ),
  by = "pageid"
)
dat_ger <- left_join(x = dat_ger,
                     y = get_social(legislature = "deu"),
                     by = "wikidataid")
dat_ger$legislature <- "Germany"
dat_ger_sum <- dat_ger %>%
  dplyr::summarize(
    twitter = mean(not(is.na(twitter)), na.rm = TRUE),
    facebook = mean(not(is.na(facebook)), na.rm = TRUE),
    website = mean(not(is.na(website)), na.rm = TRUE),
    session_start = ymd(first(session_start)),
    session_end = ymd(first(session_end)),
    legislature = first(legislature)
  )

##     twitter  facebook  website session_start session_end legislature
## 1 0.7591036 0.6918768 0.640056    2017-10-24  2021-10-24     Germany

Given the inclusion of political variables in our data set, we could think of numerous feasible extensions. For instance, we could look at social media adoption rates by party. Alternatively, we could use the constituency identifiers to add external data on the rurality/urbanity of German electoral districts and analyze whether politicians competing in urban districts are more likely to maintain social media profiles.

Application 2: Public Attention to Members of the German Bundestag

In the second application, we use pageviews to identify peaks in public attention for MPs over time. This is particularly interesting in the context of politically significant events. For instance, we may want to know about public attention to parliamentarians following scandals, around elections, or during election campaigns. The code below illustrates this logic by averaging daily pageviews across all Wikipedia articles on members of the German Bundestag between July 2015 and December 2017.

< summary> Code: Plotting Average Daily Pageviews for German MPs

## Visualize average pageviews data of German MPs
# get data
ger_traffic <- right_join(
  x = get_traffic(legislature = "deu"),
  y = filter(
    get_political(legislature = "deu"),
    session_end >= as.Date("2015-07-01")
  ),
  by = "pageid"
)
ger_traffic <- left_join(x = ger_traffic,
                         y = get_core(legislature = "deu"),
                         by = "pageid")
ger_traffic <-
  dplyr::select(ger_traffic, pageid, date, traffic, session, party, name)

# aggregate data
ger_traffic$date <- ymd(ger_traffic$date)
ger_traffic_date <- group_by(ger_traffic, date)
ger_traffic_legislators <- group_by(ger_traffic, pageid)
ger_traffic_sum <-
  summarize(ger_traffic_date, mean = mean(traffic, na.rm = TRUE))
ger_traffic_sum <- mutate(
  ger_traffic_sum,
  mean_l1 = lag(mean, 1),
  mean_f1 = lead(mean, 1),
  peak = (mean >= 1.8 * mean_l1 &
            mean > 180)
)

# identify peaks
ger_traffic_peaks <- filter(ger_traffic_sum, peak == TRUE)
ger_traffic_peaks_df <-
  filter(ger_traffic, date %in% ger_traffic_peaks$date)
ger_traffic_peaks_group <-
  group_by(ger_traffic_peaks_df, date) %>% 
  dplyr::arrange(desc(traffic)) %>% 
  filter(row_number() == 1)
ger_traffic_peaks_group <- arrange(ger_traffic_peaks_group, date)
events_vec <-
  c(
    "deceased",
    "drug affair",
    "bullying affair",
    "candidacy for presidency",
    "deceased",
    "???",
    "chancellorchip announcement",
    "elected president",
    "???",
    "policy success",
    "TV debate",
    "general election",
    "threat to resign",
    "elected speaker of parliament"
  )

# plot
par(oma = c(0, 0, 0, 0))
par(mar = c(0, 4, 0, .5))
par(yaxs = "i", xaxs = "i", bty = "n")
layout(matrix(c(1, 1, 3, 2, 2, 3), 2, 3, byrow = TRUE),
       heights = c(1, 2, 3),
       widths = c(5, 5, 1.8))
# names labels
plot(
  ymd(ger_traffic_sum$date),
  rep(0, length(ger_traffic_sum$date)),
  xlim = c(ymd("2015-07-01"), ymd("2018-01-01")),
  xaxt = "n",
  ylim = c(0, 8),
  yaxt = "n",
  xlab = "",
  ylab = "",
  cex = 0
)
text(
  ger_traffic_peaks_group$date,
  0,
  ger_traffic_peaks_group$name,
  cex = .75,
  srt = 90,
  adj = c(0, 0)
)
# pageviews time series
par(mar = c(2, 4, 0, .5))
plot(
  ymd(ger_traffic_sum$date),
  ger_traffic_sum$mean,
  type = "l",
  ylim = c(0, 1.25 * max(ger_traffic_sum$mean)),
  xlim = c(ymd("2015-07-01"), ymd("2018-01-01")),
  xaxt = "n",
  yaxt = "n",
  xlab = "",
  ylab = "mean(pageviews)",
  col = "white"
)
abline(h = seq(0, 1.5 * max(ger_traffic_sum$mean), 250), col = "lightgrey")
lines(ymd(ger_traffic_sum$date), ger_traffic_sum$mean, lwd = .5)
dates <- seq(ymd("2015-07-01"), ymd("2018-01-01"), by = 1)
axis(1, dates[day(dates) == 1 &
                month(dates) %in% c(1, 4, 7, 10)], labels = FALSE)
axis(1,
     dates[day(dates) == 1 &
             month(dates) %in% c(1)],
     lwd = 0,
     lwd.ticks = 3,
     labels = FALSE)
axis(1,
     dates[day(dates) == 1 &
             month(dates) %in% c(7)],
     labels = as.character(year(dates[day(dates) == 15 &
                                        month(dates) %in% c(7)])),
     tick = F,
     lwd = 0)
axis(2, seq(0, 1.5 * max(ger_traffic_sum$mean), 250), las = 2)
# events labels in time series
for (i in seq_along(events_vec)) {
  text(ger_traffic_peaks_group$date[i],
       ger_traffic_sum$mean[ger_traffic_sum$peak == TRUE][i] + 80,
       i,
       cex = .8)
  points(
    ger_traffic_peaks_group$date[i],
    ger_traffic_sum$mean[ger_traffic_sum$peak == TRUE][i] + 80,
    pch = 1,
    cex = 2.2
  )
}
# election date
# events labels explained
par(mar = c(0, 0, 0, 0))
plot(
  0,
  0,
  xlim = c(0, 5),
  ylim = c(0, 10),
  xaxt = "n",
  yaxt = "n",
  xlab = "",
  ylab = "",
  cex = 0
)
positions <-
  data.frame(
    events_xpos = 0.45,
    events_ypos = seq(6.5, (6.5 - .5 * length(events_vec)),-.5),
    text_xpos = .5
  )
text(0, 7, "Events", pos = 4, cex = .75)
for (i in seq_along(events_vec)) {
  text(positions$events_xpos[i], positions$events_ypos[i], i, cex = .8)
  points(positions$events_xpos[i],
         positions$events_ypos[i],
         pch = 1,
         cex = 2.2)
  text(
    positions$text_xpos[i],
    positions$events_ypos[i],
    events_vec[i],
    pos = 4,
    cex = .75
  )
}

The plot shows 14 notable spikes in daily pageviews. We identify which politician’s article generated the most traffic during each of these events. Furthermore, we add a legend that lists salient political events that likely caused these spikes in attention to MPs Wikipedia entries. For example, spike 14 marks the day on which Wolfgang Schäuble (CDU) was elected speaker of the parliament.