Another year, another web scraping course. Taught through SSRMC at the University of Cambridge. Below are slides from all three sessions.In the course I tried to achieve the following:- Show how to connect R to resources online- Use loops and functions... [Read more...]

What is the best way to read data in JSON format into R? Though really common for almost all modern online applications, JSON is not every R user's best friend. After seeing the slides for my Web Scraping course, in which I somewhat arbitrarily veered between using the packages rjson ... [Read more...]

APIs present researchers with a diverse set of data sources through a standardised access mechanism: send a pasted together HTTP request, receive JSON or XML in return. Today we tap into a range of APIs to get comfortable sending queries and processing... [Read more...]

The latest slides from web scraping through R: Web scraping for the humanities and social sciencesSlides from the first session hereSlides from the second session hereThis week we look in greater detail at scaling up digital data-collection: coercing s... [Read more...]

Slides from the second web scraping through R session: Web scraping for the humanities and social sciencesIn which we make sure we are comfortable with functions, before looking at XPath queries to download data from newspaper articles. Examples includ... [Read more...]

Slides from the first session of my course about web scraping through R: Web scraping for the humanities and social sciencesIncludes an introduction to the paste function, working with URLs, functions and loops. Putting it all together we fetch data in... [Read more...]

Here I show how to convert tabulated data into a json format that can be used in d3 graphics. The motivation for this was an attempt at getting an overview of topic models (link). Illustrations like the one to the right are very attractive; my motivati... [Read more...]

How exactly should we visualise topic models to get an overview of how topics relate to each other? This post is a brief lit review of that debate - I realise the subject matter is sooo last year. I also present my chosen solution to the dilemma: I use dendrograms ... [Read more...]

This post is a collection of scripts I've found useful for integrating a SQL database into more complex applications. SQL allows quickish access to largish repositories of text (I wrote about this at some length here), and are a good starting point for... [Read more...]

In this post I show how a combination of MALLET, Python, and data.table means we can analyse quite Big data in R, even though R itself buckles when confronted by textual data.  Topic modelling is great fun. Using topic modelling I have been able to separate articles about the ... [Read more...]

R and Python have different strengths. There's little you can do in R you absolutely can't do in Python and vice versa, but there's a lot of stuff that's really annoying in one and nice and simple in the other. I'm sure simulations can be run in R, but it ... [Read more...]

[This article was first published on Quantifying Memory, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here) Want to share your [Read more...]

Spotting international conflict is very easy with the GDELT data set, combined with ggplot and R. The simple gif above shows snapshots of Russian/Soviet activity from January 1980 and January 2000. I think it also illustrates how Russia nowadays looks more to the east and the South than during the Cold ... [Read more...]

[This article was first published on Quantifying Memory, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here) Want to share your [Read more...]

Add references and a style sheet Below I briefly outline why Pandoc is an essential part of my research workflow, and demonstrate how to seamlessly integrate it with a bibliographic system and code written in R to produce high quality word or pdf documents. I also include all the functions ... [Read more...]

Wordclouds such as Wordle are pretty rubbish, so I thought I'd try to make a better one, one that actually produces (statistically) meaningful results. I was so happy with the outcome I decided to make it interactive, so go on, have a play!Compare any two files texts (turns out ... [Read more...]

In this post I outline how count data may be modelled using a negative binomial distribution in order to more accurately present trends in time series count data than using linear methods. I also show how to use ANOVA to identify the point at which one model gains explanatory power, ... [Read more...]

Modelling memory In the text below I present two models I've made to quantify and visualise the diverging trajectories of memory and news events, and conclude that linear regression may be used to test which model best describes the story. First, though, I contextualise this with an illustration from the ... [Read more...]

Working with Russian characters can be mind-numbingly frustrating. This is true for R, as for other applications, so below I've written out the my top five tricks for making Russian inputs work in R; i believe they should be transferable to most other languages. Having forced any number of programs ... [Read more...]

[Read more...]