GET Hackernews Front Page Results using REST API in R

Whenever we talk about Data Collection, We usually think about Web Scraping. What’s often forgotten is that a lot of websites / web apps usually offer API to access their data in the right way. This video tutorial explains how you can use httr package to use GET requests (REST API Calls) to collected data from Hacker News, a very popular website for Tech News. The objective of this post it that it can outline how to simple use httr’s GET() to start making REST API calls and also to parse the response object and extract desired data.

Youtube: https://www.youtube.com/watch?v=zlYNx-Tnw3w

Code

library(httr)
library(jsonlite)

url <- "https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty"

response <- httr::GET(url)

top500 <- unlist(content(response))

top20 <- top500[1:20]

titles <- c()
scores <- c()
urls <- c()

for(one in top20){
  story_url <- paste0("https://hacker-news.firebaseio.com/v0/item/",one,".json?print=pretty")
  #print(story_url)
  story_response <- httr::GET(story_url)
  
  titles <- c(titles,content(story_response)$title)
  scores <- c(scores,content(story_response)$score)
  urls <- c(urls,content(story_response)$url)
  
}

hn_top20 <- as.data.frame(cbind(titles,urls,scores))

Related Posts:

• Sentiment Analysis in R with {sentimentr} that handles Negation (Valence Shifters)

• How to scrape Zomato Restaurants Data in R using rvest