How to scrape open data and tweet it using a bot with R

You’ve got your R script producing a beautiful data visualisation and you want to regularly publish it to your readers. But it’s a slog having to set aside time every day to update the data and tweet out your data visualisation. You need a scraper and an automatic tweet bot.

This Interhacktives how-to will provide basic R code for downloading data from an open data source, feeding it into your data visualisation and then automatically tweeting it.

Getting started

The prerequisites for this guide is enough R knowledge to make a basic data visualisation and a Twitter account. Some guides on using R to make graphs are:

How to create BBC style graphics with R

R Graph Gallery

We’ll be using the following R libraries.

library(httr)
library(jsonlite)
library(httpuv)
library(rtweet)

And whichever libraries you use for producing your data visualisation. For me that’s

library(ggplot2)

Finding the data

First, we need to identify the location of the open data we want to graph. We’re going to use the open data API for coronavirus data provided by the UK government. You need to make a note of the URL of the data object/query you want to scrape. For us it’s the following URL https://api.coronavirus.data.gov.uk/v1/data?filters=areaType=nation;areaName=england&structure={%22date%22:%22date%22,%22newCases%22:%22newCasesByPublishDate%22}.

Different open data sources will have different APIs and you should be able to work out the correct URL for the data you want from their documentation. Once you have the correct URL the process for scraping the data in R is similar.

Using the httr and jsonlite libraries we will request the data in JSON format from the open data server and store it as an object in R.

url <- paste0("https://api.coronavirus.data.gov.uk/v1/data?filters=areaType=nation;areaName=england&structure={%22date%22:%22date%22,%22newCases%22:%22newCasesByPublishDate%22}")
page <- GET(url)
cases <- fromJSON(url)
cases <- cases$data

We now have a data frame containing the number of positive coronavirus cases by date.

Plotting the graph

I’m just plotting a simple time series but you should make your data visualisation pretty and customise it to suit your data.

ggplot(cases, aes(x=date, y=newCases, group=1))+geom_line()

We need to save our graph as an image to include in our tweet. You can do this manually or by using

ggsave(“data_for_tweet.png”)

Now we are ready to send our tweet.

Posting the tweet

Unfortunately to use the write access parts of twitter’s API we need to create an app on Twitter’s developer panel.

Once you’ve created your app you’ll be given a set of keys. Make a note of the API key and API secret. These are a bit like a username and password that will allow your R code to interact with Twitter. If you forget them don’t worry as you can go into the settings and create new keys.

You need to edit your app’s permissions to read and write because you will be writing data to Twitter’s API, i.e. posting a tweet.

These keys will allow our R script to communicate with Twitter’s API. Once this has been set up once we are ready to send tweets from R.

You need to save the keys as variables along with the name of your app and then create a Twitter token to authenticate R’s access to Twitter.

name <- “YOUR APP NAME”

consumer_key <- “YOUR API KEY”

consumer_secret <- “YOUR API SECRET”

access_token <- “YOUR ACCESS TOKEN”

access_secret <- “YOUR ACCESS SECRET”

twitter_token <-

create_token(

app = name,

consumer_key = consumer_key,

consumer_secret = consumer_secret,

access_token = access_token,

access_secret = access_secret

)

To check it works check your twitter_token object contains information.

You can now send a tweet.

We’ll use two arguments of the function post_tweet(). Status and media. Status is the text of the tweet and media is the path to our data visualisation that we want to upload as a picture in the tweet.

post_tweet(status = "Tweet sent from R.", media="data_for_tweet.png")

We can include variables in our tweet using R’s paste function as normal.

todays_cases <- cases$newCases[length(cases)]
post_tweet(status = paste0("Today there were ", todays_cases," newly reported cases."), media="data_for_tweet.png")

We’ve now created an R script that scrapes data from an open data source, turns it into a graph and then automatically posts it on Twitter. If you want to take this further you could automate the whole process by setting your R script to automatically run by scheduling it as a task on your computer.

Full code

library(httr)
library(jsonlite)
library(httpuv)
library(rtweet)
library(ggplot2)

url <-
  paste0(
    "https://api.coronavirus.data.gov.uk/v1/data?filters=areaType=nation;areaName=england&structure={%22date%22:%22date%22,%22newCases%22:%22newCasesByPublishDate%22}"
  )
page <- GET(url)
cases <- fromJSON(url)
cases <- cases$data

ggplot(cases, aes(x = date, y = newCases, group = 1)) + geom_line()

ggsave("data_for_tweet.png")

name <- "YOUR APP NAME"

consumer_key <- "YOUR API KEY"

consumer_secret <-
  "YOUR API SECRET"

access_token <- "YOUR ACCESS TOKEN"

access_secret <- "YOUR ACCESS SECRET"

twitter_token <-
  
  create_token(
    app = name,
    
    consumer_key = consumer_key,
    
    consumer_secret = consumer_secret,
    
    access_token = access_token,
    
    access_secret = access_secret
    
  )

post_tweet(status = "Tweet sent from R.", media = "data_for_tweet.png")

todays_cases <- cases$newCases[length(cases)]

post_tweet(
  status = paste0("Today there were ", todays_cases, " newly reported cases."),
  media = "data_for_tweet.png"
)

Tags: bot, r, Twitter