R for SEO Part 9: Web Scraping With R & Rvest

Hello, and welcome back. We’re (finally) in the home stretch of our R for SEO series with part nine, where we’re talking about scraping the web using R, particularly using the rvest package.

Today, we’re going to discuss the rvest package, look at the different scraping methods available, pull data from multiple pages and then look at how we can do so without bringing our target sites down. This is going to be fairly important for our final piece, so it’s worth paying attention today.

As always, this is a long piece, so feel free to use the table of contents below and do please sign up for my email list, where you’ll get updates of fresh content for free.

Leave this field empty if you’re human:

The Rvest Package

The rvest package for R – another Hadley Wickham creation – is the most commonly used web scraping package for the R language, and it’s easy to see why. It brings a lot of the Tidyverse’s tidy data outputs and notation functionality to what can be a complex element of data analysis and SEO. I’m a fan.

It’s not included in the Tidyverse package, so you’ll need to install it separately. You can do that like so:

install.packages("rvest")

library(rvest)

This will install the rvest package and get it initialised. It’s also worth installing the Tidyverse as always.

library(tidyverse)

Now we have rvest installed, we can get scraping. But how do we find what to scrape from a page? That’s where we start needing to understand a bit about how scraping works and how to identify our data points.

XPath & CSS Selectors

XPath and CSS selectors are the most widely used ways of identifying elements on a page, and are both crucial to understand when it comes to scraping the web.

Personally, since I’m a little bit more old-school, I tend to default to XPath, but that’s not to say CSS selectors aren’t brilliant – they certainly help you write much more efficient code.

Let’s look at the differences between the two.

What Is XPath?

XPath – or XML Path Language – is a syntax used for selecting nodes in a document. Think of it like a way to pinpoint specific parts of an XML tree structure, like finding particular branches or leaves. XPath has been around a long time and is very powerful, and rvest has great support for it.

There are a number of ways that you can find the relevant XPath query that you’ll need to scrape your chosen elements from your pages, and we’ll cover that very shortly.

What Are CSS Selectors?

CSS selectors are the visual language of the web, used to style and structure web pages, but they can do much more than just create pretty sites. In R, you can leverage CSS selectors to extract the almost any data you need from web pages.

Again, rvest has a lot of native support for CSS selectors, and they tend to be a lot more efficient to write queries with, as well as needing less debugging. Both methods are completely valid and, truthfully, while I tend to default to XPath, CSS selectors are finding their way into my work a lot more due to having to write less code. But what are the key differences?

XPath Vs CSS Selectors

Using Vs is a bit of a misnomer here – it’s not a fight, because they’re both worth using, but it’s worth understanding a little bit more about the differences between the two and when to use them.

In general terms – there are always exceptions – CSS selectors are:

• Easier & more efficient to write: A CSS selector is much shorter than an XPath query, and usually more intuitive, especially if you’ve been doing a lot of technical SEO work over your career
• Faster to run: Especially if you’re using browser-native CSS, which can make large-scale scraping projects much quicker
• Primarily designed for HTML pages: They’re not great at navigating up the DOM tree, and they’re not always so good for specific text elements

Conversely, XPath tends to be:

• More powerful: You can do very complex queries with XPath, which can allow you to navigate the DOM tree in both directions, as well as scraping elements from parts of the site that don’t have CSS attached – more on that shortly
• More complex: Due to the power, an XPath query will generally be longer and more complex to write and maintain – but don’t let that put you off

So now we know the two key methods we’re going to be using to identify the elements we’ll be scraping today, let’s talk about how to find them.

Finding CSS Selectors & XPath Queries

Obviously, it’s all well and good saying you want to use R to scrape a certain part of a page, such as the H1 or image alt text, but how do you actually find the elements you need? How do you identify the CSS selector without spending ages digging through the code and how do you go about putting an XPath query together?

Fortunately, there are a few very quick and easy ways to do that using Chrome. Let’s take a look at a couple of them.

The SelectorGadget Chrome Extension

SelectorGadget is a handy, free Chrome extension that I use a lot. It’s very quick and easy to use and can help you find your CSS selector or XPath at the click of a button. It’s not perfect – few things are – but I find it hits more than it misses.

Install it into your Chrome browser and then go to the page you want to scrape an element from. Let’s look at my last post in this series.

Now let’s say we want to scrape the article title.

Click the SelectorGadget extension and hover over the article title like so:

You’ll see that it’s highlighted the selector.

If you look into the SelectorGadget bar and click the title, it’ll put the following in there:

And there we go, we have our CSS selector.

Told you it was easy.

Now if you click on the XPath button to the right of the extension, like so:

You’ll get a popup with the XPath query you need.

Again, these aren’t always perfect and sometimes won’t work the way they should, but they’ll give you a good starting point.

But what if, for some reason, you can’t install the extension, or you find it doesn’t work the way it should on certain sites? Fortunately, there’s another option that’s built right into Chrome and most other browsers.

Finding CSS Selectors & XPath Queries With Chrome Developer Tools

I’m sure if you’ve been working in SEO for a while, you’ve become very familiar with Chrome or other browsers’ Developer Tools window. I don’t think I go a day without it, but did you know that you can also use it to find the CSS Selectors or create an XPath query based on a specific element? You probably did, but let’s talk through it anyway.

Personally, I tend to use this more when I’m trying to build a scraper for an area that either doesn’t have CSS to it, such as something in the head, or when something stops SelectorGadget working effectively, but it’s definitely worth learning how to use it to help you as you get more familiar with scraping in R.

Go back to the page we discussed in the last section. Now highlight a part of the article title and right-click. Now click “Inspect”, like so:

This will bring up the familiar “Elements” window:

Now if we right click on the element we care about – our article title, in this case, you’ll see the following dialogue. If you hover over “Copy”, it’ll pop out with a few very handy options:

You can copy your CSS selector or your XPath query directly from here, ready to paste in your code. I told you it was handy!

Again, it’s also not perfect, but between developer tools and SelectorGadget, I can pretty much always find the right element notation for whatever I’m trying to scrape.

I’m conscious that we’re getting scarily close to 1,500 words and we haven’t written a single line of actual code yet! Let’s fix that by using R to scrape the article title with the CSS selector.

Scraping Article Titles With R & CSS Selectors

One of the things that is always worth remembering about using CSS selectors is that they will typically differ between websites, due to the fact that CSS styling tends to be individual to the site in question. That’s why tools like SelectorGadget or Chrome Dev Tools are so useful. Let’s write our first scraper using R’s rvest package to scrape the article title from my last post.

First, create an object of the URL to that post, like so:

scrapeURL <- "https://www.ben-johnston.co.uk/r-for-seo-part-8-apply-methods-in-r/"

Now we want to find our CSS selector. Do that with SelectorGadget, and our selector will be as follows:

.entry-title

Alright, we’re ready. Let’s create a very simple scraping call:

articleTitle <- read_html(scrapeURL) %>% html_element(".entry-title") %>% 
  html_text()

Run that in your console, now type articleTitle and you should see the following output:

Easy, right? Let’s take a look at how it works.

Our First Scraper Command

Rvest was created by the same brain behind the tidyverse (you may have guessed that I’m a fan of Mr Wickham’s work by now), and you can see certain similarities in other commands that we’ve written throughout this series.

As always, let’s break it down:

• articleTitle <-: We’re giving our object name the incredibly inventive name of “articleTitle”
• read_html(scrapeURL): Now we’re calling the read_html() function from rvest to download the html of our article that we put into scrapeURL
• %>%: We’re using the tidyverse’s pipe parameter to create multiple commands in one – you’ll have seen me use this throughout this series, but it’s always good to remind ourselves
• html_element(“.entry-title”): We’re telling rvest that we want to look specifically for the element (our CSS selector in this case) “.entry-title”
• %>% html_text(): Finally, we’re chaining in the html_text() command to only show us the text from the element we’ve selected

And there you have it – we’ve built our very first scraper. Now let’s look at how we can use XPath from R’s rvest package to scrape different elements.

Scraping Meta Titles With R & XPath

Now let’s look at how we can use XPath with R and the rvest package to pull the meta title from the same page. This is a nice, simple command, but hopefully it’ll give you an idea of how the power of XPath can be added to your web scraping work.

As before, we’re going to use my last post as the target page, so our scrapeURL object is still valid.

Since we’re going for the meta title in this part rather than something visible on the page, SelectorGadget isn’t going to help us. Fortunately, we know how to use Chrome Dev Tools to find this.

Go to your target URL and right click anywhere on the page to bring up the elements window. Now find and expand the element.

We want to focus on the