Ten Thousand Tweets
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
I started learning the statistical programming language R this past summer, and discovering Hadley Wickham’s data visualization package ggplot2
has been a joy and a revelation. When I think back to how I made all the plots for my astronomy dissertation in the early 2000s (COUGH SUPERMONGO COUGH), I feel a bit in awe of what ggplot2
can do and how easy and, might I even say, delightful it is to use. I recently passed the 10,000 tweet mark at my personal Twitter account, so in this blog post I am going to use ggplot2
to analyze my Twitter archive. You can download your own Twitter archive by following these directions. When I planned this project, I was mentally prepared to parse JSON and such, but it turns out that when you download your own Twitter archive, one of the files is a lovely, neat CSV file with every tweet you’ve ever tweeted. Handy!
First, I’ll load the libraries that I need to make my plots and read in the data from the CSV file.
The timestamp on each tweet is a string at this point, so let’s use a function from the lubridate
package to convert the timestamp to a date-time object. The timestamps are recorded in UTC, so to make more interpretable plots, we need to convert to a different time zone. I joined Twitter in 2008 when I lived in Connecticut, and in the intervening years I lived in Texas, and now in Utah. Not to mention all the times I traveled elsewhere! I don’t have geotagging turned on for my tweets because that freaks me out somewhat, so I don’t have any information for individual tweets to assign time zones to them on a case-by-case basis. For a rough estimate, let’s just assign all tweets to the Central Time Zone, using another function from lubridate
. If we needed to do a better job on this, we could divide my tweets up by when I moved between various time zones; this still wouldn’t be perfect because of tweeting while traveling.
Now for some graphs!
Tweets by Year, Month, and Day
First let’s make a basic histogram showing the distribution of my tweets over time.
This is obvious to more experienced users, but it took me a while to figure out that when writing out code for a plot over multiple lines, the +
needs to go on the end of the previous line, not at the beginning of the new line. Use an aesthetic with fill = ..count..
in the call to geom_histogram
to make the fill color of each histogram bin reflect that bin’s height. You can see I hit my own personal peak Twitter sometime in late 2012.
Perhaps we’d like to see how many tweets I’ve tweeted each year. Use the year()
function from lubridate
on the x-axis of the histogram and specify the breaks of the histogram in the call to geom_histogram
.
What does my tweeting pattern look like over the days of the week? We can take a similar approach and use the wday()
function from lubridate
and specify the breaks for the histogram.
I’ve seen other analyses showing that people tweet less on the weekends. Also, what a big difference between Wednesday and Tuesday/Thursday! I want to do a statistical test and see if these are real effects. The appropriate test is a single sample chi-squared test (or goodness of fit test); this test will take the distribution of tweets by weekday and see if the distribution we have is consistent with a given hypothesis, to within random sampling error. For example, could we have gotten this distribution of tweets just by chance, or do I really tweet less on the weekends? This kind of test can be done with any kind of expected frequencies, but first let’s compare against the hypothesis of equal expected frequencies, i.e. the hypothesis that I tweet at the same rate on all days and I got this distribution of tweets just by chance and random sampling.
The chi-squared test indicates that the distribution of my tweets is highly unlikely to be uniform; we can reject the null hypothesis (the hypothesis that I tweet at the same rate on all days) with a high degree of confidence. Can the distribution in tweets be explained only as a difference between weekday and weekend behavior? It looks like Monday through Thursday are higher than Friday through Sunday.
The values for Monday through Thursday are 1.283 higher than the other days, on average, close to 5/4. Let’s see if the chi-squared test says that my pattern of tweets is consistent with tweeting 1.25 times as often (5/4) on Monday through Thursday as on Friday through Sunday.
The p-value here is still very low, so we can reject this simple hypothesis too. I would not have guessed that there was such a strong association between the days of the week and my tweeting patterns.
For the months of the year, there is yet another convenient function in lubridate
, the month()
function, to put into the histogram.
This is a pretty interesting annual pattern; I would not have predicted all of this before looking at this histogram. The decrease in August and September makes sense as this is generally a very busy time of year both in the various jobs I’ve had since 2008 and for my family. The decrease going into spring? I’m not really sure. Let’s do the chi-squared test again, just to make sure these are not due to random sampling.
Again, the chi-squared test indicates that the distribution of tweets by month is highly unlikely to be uniform and we can reject the null hypothesis (the hypothesis that I tweet the same amount in each month) with a high degree of confidence.
Tweets by Time of Day: Time to Put the Phone Down?
To see what time of day I tweet, we need to take the timestamp date-time objects and strip out all the year, month, etc. information and leave only the time information. After fiddling about with this for quite a while, I got this to work using the trunc
function in base R. This next line of code was the most difficult part of this analysis for me.
I also found that for my earliest tweets, there was no meaningful time information recorded. A good number of my earliest tweets have real dates recorded but 00:00:00 UTC for the time. Let’s find these tweets and set the new timeonly
column to NA
for them. How many of my tweets don’t have real time information?
It turns out it’s about 21.7%, so for my tweets vs. time graph, we will actually have not-quite-10,000 tweets. To make the graph, let’s convert the new timeonly
column to a date-time object and make a histogram. We can use a scaling function from the scales
library to scale and label the x-axis.
So there is when I tweet. I see mid-morning and mid-afternoon peaks, and a major peak after my kids go to bed. I was actually surprised to see any overnight tweets at all; I am far from an night owl. What might be going on there?
What are these tweets? I am never awake in the middle of the night!
4am here. 3yo has been awake since 1am, crying, coughing, demanding chocolate milk, waking up croupy baby when I finally get him to sleep.
— Julia Silge (@juliasilge) March 3, 2012
Oh…
2am vomit! (Not me.) Motherhood is so GLAMOROUS sometimes...
— Julia Silge (@juliasilge) May 26, 2013
Hmmmm…
So it turns out that I have occasionally been up in the middle of the night. Not all these overnight tweets were related to small children and illness. Some were from times when I was traveling west to the Pacific Time Zone and were not actually as late as they appeared because of the way I assigned the time zone. And some were just from when I stayed up a bit late, of course! The tweets that came in peaks that you can see in the histogram were more likely to be episodes of sick little ones. I am so glad that my kids are older now and we are mostly past this stage of parenthood! You can see how the number of late night tweets (never large to start with) has decreased over the years I have been on Twitter.
HASHTAG RETWEET ME, BRO
Let’s look at my usage of hashtags, retweets, and replies.
I can use regex and grep
to find all the hashtags in my tweets. Let’s see what fractions of my tweets have hashtags and look at that visually.
As I would have guessed, I am not a big user of hashtags; only 1.7% of my tweet have hashtags. Maybe #rstats and/or the process of trying to spread the word of my new professional blog will change my mind, but it is not very “me”. In fact, most of the tweets that do have hashtags and fall in the right bin in this graph are actually retweets of other people’s tweets with whatever hashtags they used.
Speaking of retweets, one of the columns in the CSV file from Twitter codes whether the tweet is a retweet or not, so it is not tough to see how many tweets are retweeted vs. original content.
Only 4.8% of my tweets are retweets of others. The scale_x_discrete
function is a nice, useful thing for relabeling the levels of a categorical variable for plotting. Before I applied this function, the labels just said “FALSE” and “TRUE”.
I am not a huge user of hashtags or retweets; now let’s look at my replying habits. There is another column in the CSV file that codes whether the tweet is in reply to another tweet.
So 26.1% of my tweets are in reply to another person’s tweets, quite a bit higher than the other categories we’ve looked at. I am a fairly social user of the social media, it appears.
Let’s put these categories together and see if there have been changes in my patterns of tweeting over time. I’ll make a new column that codes the “type” of the tweet: regular, RT, or reply. My approach here is to first assign all the tweets to the regular category, then go through and overwrite this type with RT or reply if appropriate. I want this to be a factor variable, and lastly I reorder the factor variable in the order I want.
Let’s look at how my habits in original tweeting, retweeting, and replying have changed over time.
Hmmmm, that is good, but it might be better to see this as a proportion of the total.
By changing to position = "fill"
in a call to geom_bar
we can see how the proportion of these categories has changed. This is not entirely a great measure here, because Twitter did not keep track of which tweets were retweeted or replied before sometime in 2010. Undoubtedly, some of my tweets in 2008 and 2009 were RTs and replies, but Twitter did not keep track of this officially for each tweet. Looking at the more recent years when these data were tracked, my proportion of retweets, replies, and original tweets looks mostly stable. This does mean that the estimates in the bar graphs above for reply and RT are biased low; estimating from the graph, those numbers may be up to twice as high.
No More Characters For You
Lastly, let’s look at the distribution of the number of characters in my tweets. I can use one of the functions from the apply
family of functions to count up the characters in each tweet in my archive.
Let’s see what the distribution of character count looks like for my tweets.
So that is fairly interesting. I am more likely to use many or most of those precious 140 characters allowed on Twitter than to go for brevity. BUT WAIT! What are those tweets with MORE than 140 characters?!? What magic have I worked? I can find these tweets by indexing the data frame using tweets[(tweets$charsintweet > 140),]
. After looking through them, it turns out that they are all tweets that use special characters and certain special punctuation and the like. The CSV file that I downloaded from Twitter uses special extra punctuation to make sure it is clear what is going on in these situations. Here is an example tweet that has a bunch of these special characters:
This means that my histogram above is biased high a little bit; the number of characters per tweet is actually slightly lower than shown there. If I needed to, I could use regex and count each &
only once and so forth. Or I could use the number of tweets that have greater than 140 characters to correct for this effect, or the number of tweets that contain special characters at a given character count, or some similar strategy, if it was important that I know the number of characters per tweet to this level of precision. I am happy with this current estimate of my character usage for my purposes right now.
The tweets with really low numbers of characters made me a bit suspicious too, but it turns out they are all real and normal tweets. Many were just very short replies to other people, but sometimes it turns out you don’t need many characters to say what you want to say.
Naps are awesome.
— Julia Silge (@juliasilge) November 11, 2008
So true.
UGH, CANDY.
— Julia Silge (@juliasilge) November 1, 2014
Amen.
The End
So there are some examples of how to make histograms and bar charts using ggplot2
, and do a bit of related analysis. The R Markdown file used to make this blog post is available here. I am happy to hear feedback and suggestions as I am very much still learning about how to use R!
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.