Pre-processing with udpipe

In prior text-mining posts, I used tidytext to handle tokenization, however, in this analysis I will leverage the udpipe package. The udpipe is a R wrapper around the C++ library of the same name that uses a pre-trained language models to easily tokenize, tag, lemmatize or perform dependency parsing on text in any language. The “ud” in udpipe stands for Universal Dependencies which is a “framework for consistent annotation of grammar”.

In order to prepare the data for the model there needs to be some light pre-processing as udpipe expects the data to have a doc_id and a text field.

#Columns need to be doc_id and text for the model
cleaned <- iglite %>% 
  mutate(doc_id = row_number(),
         text = str_to_lower(reviews),
         text = str_replace_all(text, "'", ""))

To annotate our data with udpipe I’ll call the udpipe() function with my data and the language of the model to use. This function is will download the appropriate language model, in this case English, and then annotate the data.

annotated_reviews    <- udpipe(cleaned, "english")

To show what the udpipe model did to the data we can look at the first review before the annotations:

and after the annotations:

annotated_reviews %>% filter(doc_id == 1) %>% head(3) %>% knitr::kable()

We now get a ton of metadata including indicators for the sentence, we can the token (token) and its lemma (lemma) (note that consumes becomes consume), parts of speech (upos), and dependency relationships (deprel) and more.

Now that we’ve tokenized the data we can start using it to analyze the reviews.

Biterm Modeling

The first analysis task will be biterm modeling using the BTM package. The Biterm Topic Model model was developed by Yan et. al as a means to determining the topics that occur in short-texts such as Tweets (or in this case Google Play Reviews). Its meant to provide an improvement to traditional topics modeling in uses cases such as this. My understanding of the difference between traditional topic modeling and biterm topic model is that in the former, the model learns word co-occurrence within documents, while with the later, the model learns word co-occurrences within a window across the entire set of documents. In this context a “biterm” consists of two words co-occurring in the same context, for example, in the same short text window. This analysis is modeled after the one from bnosac.

In the BTM model we can explicitly tell the model which word co-occurrences we care about vs. letting it run on everything. This enables us to only care about certain parts of speech, words of certain lengths, and non-stop words. For this analysis we will consider a co-occurrence window of 3 while removing stopwords, removing words with less than 3 characters, and only keeping nouns, adjectives, verbs, and adverbs.

#Define a Dictionary of BiTerms
library(data.table)
library(stopwords)
biterms <- as.data.table(annotated_reviews)
biterms <- biterms[, cooccurrence(x = lemma,
                                  relevant = upos %in% c("NOUN", "ADJ", "VERB") & 
                                             nchar(lemma) > 2 & !lemma %in% stopwords("en"),
                                  skipgram = 3),
                   by = list(doc_id)]

The biterm data set we’ve constructed looks like:

biterms %>% head(5) %>% knitr::kable()

This states that in the first review, the word pair (like, post) occurs within a 3 word window twice in the document.

Now we can actually construct the biterm model. For simplicity, I’m setting it to train 9 topics. The background = T setting makes the 1st topic a background topic that reflects to empirical word distribution to filter out common words (which is why k = 10):

set.seed(123456)

train_data <- annotated_reviews %>% 
  filter(
    upos %in% c("NOUN", "ADJ", "VERB"),
    !lemma %in% stopwords::stopwords("en"),
     nchar(lemma) > 2
  ) %>%
  select(doc_id, lemma)

btm_model     <- BTM(train_data, biterms = biterms, k = 10, iter = 2000, background = TRUE)

Now that we’ve constructed topics, there needs to be a good way to visualize those topics. Fortunately the textplot package handles this nicely:

library(textplot)
library(ggraph)
set.seed(123456)

plot(btm_model, top_n = 10,
     title = "BTM model of IGLite Reviews",
     labels = c("",
                "Reels",
                "Likes the App",
                "Takes Too Long",
                "Can't Upload",
                "Dark Mode", 
                "Bugs", 
                "Feature Requests",
                "Uses Less Resources", 
                "Instagram Lite"))

From this chart we can see that there’s a lot of people mentioning bugs and other problems, specifically around upload. People talking about how IG Lite consumes less space and data, people wanting new features such as a music sticker option in stories, and a LOT of people wanting Dark Mode. And there are people who like it and think its a good app.

Sentiment Analysis withe Dependency Parsing

In many sentiment analyses a dictionary method is used to assign positive sentiment and negative sentiment and then some sort of aggregation occurs to determine whether a document is “happy” or “sad” or whatever other type of emotion. But what gets left on the table is “Why” there is positive or negative sentiment. In this case, we can see that people gave IG Lite bad ratings or complained about issues, but without looking through every review, it tough to know why.

This next piece is based on a bnosac blog post and will leverage the dependency output from udpipe to see what words are connected to the words with negative sentiment.

To first determine words with negative sentiment I will need an external dictionaries to identify:

• Positive vs. Negative words - the base positive vs. negative scoring
• Amplifying and Deamplifying words - words like ‘very’ which make an emotion more intense or ‘barely’ which make an emotion less intense.
• Negators - words like ‘not’ which would flip the sentiment

For these lists I will get the data used in the sentometrics package:

load(url("https://github.com/SentometricsResearch/sentometrics/blob/master/data-raw/FEEL_eng_tr.rda?raw=true"))
load(url("https://github.com/SentometricsResearch/sentometrics/blob/master/data-raw/valence-raw/valShifters.rda?raw=true"))

and break them up into separate vectors of words:

polarity_terms <- FEEL_eng_tr %>% transmute(term = x, polarity = y)
polarity_negators <- valShifters$valence_en %>% filter(t==1) %>% pull(x) %>% str_replace_all("'","")
polarity_amplifiers <- valShifters$valence_en %>% filter(t==2) %>% pull(x) %>% str_replace_all("'","")
polarity_deamplifiers <- valShifters$valence_en %>% filter(t==3) %>% pull(x) %>% str_replace_all("'","")

Finally, I can use udpipe’s txt_sentiment function to use these lists to score my annotated data.

sentiments <- txt_sentiment(annotated_reviews, term = "lemma", 
                            polarity_terms = polarity_terms,
                            polarity_negators = polarity_negators, 
                            polarity_amplifiers = polarity_amplifiers,
                            polarity_deamplifiers = polarity_deamplifiers)
sentiments <- sentiments$data

In addition to the initial annotations there are now columns for polarity (just the positive / negative based on the term) and sentiment_polarity which incorporates the additional information.

Now that there are sentiments I’m going to want to find the words that those negative terms modify using cbind_dependencies().

reasons <- sentiments %>%
  #Attached Parent Words to Data
  cbind_dependencies() %>%
  #Filter Columns
  select(doc_id, lemma, token, upos, polarity, sentiment_polarity, token_parent, lemma_parent, upos_parent, dep_rel) %>%
  #Keep Only Terms with Negative Sentiment
  filter(sentiment_polarity < 0)

The revised data now looks like:

head(reasons) %>% knitr::kable()

A quick look at the data calls out a problem that exists with all dictionary based approaches which is that there is a context that the analyst knows that a dictionary cannot. For example, the term above “less data” is taken to be a negative because having “less data” would be bad… except in the context of Instagram Lite using “less data” would actually be good.

To get a better understanding of why we’re seeing negative sentiment I will construct a network graph between the negative term and the thing they are modifying and looking for the common phrases.

# Keep only dependency relationships that are adjectival modifiers 
# (terms that modify a noun / pronoun)
reasons <- filter(reasons, dep_rel %in% "amod")

# Count Number of occurrences
word_cooccurences <- reasons %>% 
  count(lemma, lemma_parent, name = "cooc", sort = T) 

# Create the Nodes as either the term in the dictionary or a word linked 
#to the term in the dictionary
vertices <- bind_rows(
  data_frame(key = unique(reasons$lemma)) %>% 
    mutate(in_dictionary = if_else(key %in% polarity_terms$term, 
                                   "in_dictionary", 
                                   "linked-to")),
  data_frame(key = unique(setdiff(reasons$lemma_parent, reasons$lemma))) %>% 
    mutate(in_dictionary = "linked-to")
  )

library(ggraph)
library(igraph)

# Keep Top 20 Words CoOccurances
cooc <- head(word_cooccurences, 20)
set.seed(123456789)

cooc %>%  
  graph_from_data_frame(vertices = filter(vertices, 
                                          key %in% c(cooc$lemma, 
                                                     cooc$lemma_parent))) %>%
  ggraph(layout = "fr") +
  geom_edge_link0(aes(edge_alpha = cooc, edge_width = cooc)) +
  geom_node_point(aes(color = in_dictionary), size = 5) +
  geom_node_text(aes(label = name), vjust = 1.8, col = "darkgreen") +
  scale_color_viridis_d(option = "C", begin = .2, end = .8) + 
  ggtitle("Which words are linked to the negative terms") +
  theme_void()

In the network we see the “less data” as the strongest co-occurrence even thought it (and many other words in this group) are not strictly negative words. Some of these connections make sense to be negative like “slow speed” or “useless app” which seems unquestionably bad. But some of these don’t make sense to me like “full screen” being bad. Although looking at a few of the sample reviews that say full screen they are usually in reference to full screen modes not working. So while it does appear that the sentiment model is capturing that “full screen” is discussed as a negative thing, the graph view above does not make that clear.

So dependency parsing for sentiment analysis seems like a cool idea but is a bit “your mileage may vary”.

Word Clouds on Keywords

The last text analysis technique for this post will probably be the most well known… wordclouds. It will show what are the most common words in our data set and can be used to understand the set of reviews at a quick glance. But rather than relying on most common words, I’ll use the textrank package to extract relevant keywords text where keywords are defined as combinations of words following each other. To try to get the most relevant set of keywords, I will be limiting to nouns, adjective, and verbs and will create a wordcloud of the top 30.

textrank_keywords(annotated_reviews$lemma,
                  relevant = annotated_reviews$upos %in% c('NOUN', 'ADJ', 'VERB')) %>% 
  .$keywords %>% filter(ngram > 1 & freq > 1, !str_detect(keyword, 'be')) %>%
  slice_max(freq, n = 50) %>% 
  with(wordcloud(keyword, freq, max.words = 50, colors = brewer.pal(10, 'Dark2')))

So what are people saying about IG Lite…. that they want dark mode, they want music stickers and that its a good app.