[This article was first published on Analysis with Programming, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
It’s been two weeks for search and rescue operations on Malaysia Airlines Flight MH370, after it vanished from the radar on March 8, 2014. And wherever they are, we hope and pray for them.
In this post, we are going to do text data mining on Twitter tweets containing #PrayForMH370 from March 8, to March 20, 2014 using Twitter API. First, we need to have an authentication on Twitter API, to obtain the data. In the proceeding tutorial, the idea and codes for Twitter authentication were based from Julianhi’s amazing blog, and I am going to replicate his code to save a copy of it.
Go to https://dev.twitter.com/, and sign in with your twitter account. When you are done, click on your profile picture on the top right corner like the one below.
Click on My applications, then Create New App. Fill in what’s necessary (i.e. Name, Description and Website).
Julianhi recommends http://test.de/ for valid Website, and name it anything you want (same case for description). To finish this, check Yes, I agree on Developer Rules of the Road, and click on Create your Twitter application. You will have something like this,
Keep this open on your browser, and go to your R script. Copy and paste the following code,
Replace the apiKey and apiSecret with the one on the API Keys tab of your Twitter app (see above photo — refer back to your browser), then run the above code. If you encounter error that says Error: Unauthorized, make sure to remove the spaces (if any) both at the end of your apiKey or apiSecret between the quotations, i.e.
This happens especially when you do copy and paste (I’m guilty of that). But if there is no error, twitCred$handshake will return a link. Copy and paste this link to your browser, then authorize the app,
After clicking Authorize app, take note of the PIN and enter it to the R Console; finally, run the last function (line 24 above). Now that we have an access, we can extract data then. For this post, we will make a word cloud for #PrayforMH370 and #MH370 (bonus) on the said inclusive dates. Here it is, run the following:
The above code for creating word cloud is originally from Mining twitter with R site, and below is the output of that,
Just ignore this, as this simply mean that there are only 599 tweets in total that use #PrayforMH370 during the inclusive dates. Now, let’s investigate the above word cloud. I know very little about Bahasa Melayu (Malay Language), but I can tell the words in high scale are likely to be Malay prepositions, or in computing term, Malay stop words. This is because we exclude Malay stop words (only ‘prayformh370’, ‘prayformh’ and stopwords(‘english’) — English stop words) in the code as it is not yet supported by the stopwords function. Hence, not a good word cloud for tweets exploratory. So I look for a list of Malay stop words on the web, and led me to this site. Since this is what we are looking for, thus we import the list using the XML package.
Supplying the previous code with the Malay stopwords, thus we have
And why is it in square? and not circle? Well that is due to the small screen I have and large number of dataset (1000 data points). So, from the two plots we have (#PrayForMH370 and #MH370), we can say subjectively, that tweets under #MH370 were mostly, maybe, retweets of media tweets since there are words like cnn, malaysia, with link like httpcodjczaqtm. There is also name of the CNN anchor andersoncooper. Another sobrevolando (spanish for flying over, Google knows it well). We can also see the word probably, since there are many guesses as to what happen when this news came out. And of course, most Twitter users from Malaysia during the inclusive dates (March 8 to 20), were expressing their prayers on the lost plane. That’s what I see, what about your exploration?
 |
| Photo from VENUS – Wall of Hope & Prayers for MH370 |
In this post, we are going to do text data mining on Twitter tweets containing #PrayForMH370 from March 8, to March 20, 2014 using Twitter API. First, we need to have an authentication on Twitter API, to obtain the data. In the proceeding tutorial, the idea and codes for Twitter authentication were based from Julianhi’s amazing blog, and I am going to replicate his code to save a copy of it.
Go to https://dev.twitter.com/, and sign in with your twitter account. When you are done, click on your profile picture on the top right corner like the one below.
Click on My applications, then Create New App. Fill in what’s necessary (i.e. Name, Description and Website).
Julianhi recommends http://test.de/ for valid Website, and name it anything you want (same case for description). To finish this, check Yes, I agree on Developer Rules of the Road, and click on Create your Twitter application. You will have something like this,
Keep this open on your browser, and go to your R script. Copy and paste the following code,
Replace the apiKey and apiSecret with the one on the API Keys tab of your Twitter app (see above photo — refer back to your browser), then run the above code. If you encounter error that says Error: Unauthorized, make sure to remove the spaces (if any) both at the end of your apiKey or apiSecret between the quotations, i.e.
This happens especially when you do copy and paste (I’m guilty of that). But if there is no error, twitCred$handshake will return a link. Copy and paste this link to your browser, then authorize the app,
After clicking Authorize app, take note of the PIN and enter it to the R Console; finally, run the last function (line 24 above). Now that we have an access, we can extract data then. For this post, we will make a word cloud for #PrayforMH370 and #MH370 (bonus) on the said inclusive dates. Here it is, run the following:
The above code for creating word cloud is originally from Mining twitter with R site, and below is the output of that,
Just ignore this, as this simply mean that there are only 599 tweets in total that use #PrayforMH370 during the inclusive dates. Now, let’s investigate the above word cloud. I know very little about Bahasa Melayu (Malay Language), but I can tell the words in high scale are likely to be Malay prepositions, or in computing term, Malay stop words. This is because we exclude Malay stop words (only ‘prayformh370’, ‘prayformh’ and stopwords(‘english’) — English stop words) in the code as it is not yet supported by the stopwords function. Hence, not a good word cloud for tweets exploratory. So I look for a list of Malay stop words on the web, and led me to this site. Since this is what we are looking for, thus we import the list using the XML package.
Supplying the previous code with the Malay stopwords, thus we have
And why is it in square? and not circle? Well that is due to the small screen I have and large number of dataset (1000 data points). So, from the two plots we have (#PrayForMH370 and #MH370), we can say subjectively, that tweets under #MH370 were mostly, maybe, retweets of media tweets since there are words like cnn, malaysia, with link like httpcodjczaqtm. There is also name of the CNN anchor andersoncooper. Another sobrevolando (spanish for flying over, Google knows it well). We can also see the word probably, since there are many guesses as to what happen when this news came out. And of course, most Twitter users from Malaysia during the inclusive dates (March 8 to 20), were expressing their prayers on the lost plane. That’s what I see, what about your exploration?
To leave a comment for the author, please follow the link and comment on their blog: Analysis with Programming.
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.