Have you ever wondered what goes into making a Coronavirus data tracker in data journalism?
This week, we talked to Michael Simmons (@Simmons__) about how he built his Edinburgh Coronavirus Data website, how automating processes makes for a better social life, and learning by doing.
Michael is an Interactive Journalism postgraduate student at City University, with a background in statistics.
Hi Michael, thanks for talking to us! We want to get the inside scoop on your Coronavirus tracker. Firstly, how did it get started?
The first chart I did was the daily case numbers for Edinburgh. From the first lockdown I was really addicted to seeing what the numbers were in the area, and every day in briefings or on the BBC website they were publishing the Lothian [health board area] case number. It was annoying because you don’t organise your life in a health board area kind of sense.
I was looking on the government website to try and find more localised statistics for the Edinburgh Council area. I thought if I was interested in that, other people might be interested in more specific local data too.
The tier chart is essentially complete plagiarism because the FT had done this really nice graph showing case rates by tier for different local areas in England and Wales. I looked around to see if there was a Scottish one, and I couldn’t find one – so that came about from wanting to work out how they did that.
And then the vaccination chart was purely… by that point, everything was so negative, you’re just putting out rising cases. As soon as there were vaccine numbers, I was like, right, I’m putting these out there just to get people up a bit.
Finding the right data
When the tier system was in place, your chart comparing case rates to tiers put into context the justification (or not) behind different decisions around local lockdowns. So having that for an unreported area is pretty useful.
It was a big debate up here at the time. There was a long while where Edinburgh was being kept in Tier 3 when places higher than it [for covid case numbers] were going down into Tier 2. So it is interesting to see. You would get hopes of going down a tier and just looking at the graph it looked justified, and then for political or whatever reasons it never happened. So people were finding that kind of information useful.
Was it difficult to get that council level data? Where are you getting it from?
I knew it existed, because I had seen other people had other council’s area data, but it was quite hard to find. Eventually, Public Health Scotland made this dashboard in Tableau. So I was typing it in initially, and I thought, well, this must be in a spreadsheet somewhere.
After rooting around in the documentation I found the link to an open data site, Scottish Government Open Data. In there, there’s a spreadsheet with the daily case reports for each day. I decided to do case numbers by reported date rather than by the actual date that people have the test. It means you don’t have to adjust for the fact there’s a lag.
One particular challenge was, annoyingly, they don’t give that historic reported number in a downloadable form. When I initially had no data, I had to kind of just go around and look at news articles from each day to get the number.
Automation
Do you have a live update with the data? And is it an automated process to bring it onto the website and Twitter?
For the first week I was just doing it all manually. This was back when things were open. Say I was out I’d be sat in the pub, I’d be getting my laptop out like ‘gonna have to do this’ because people were starting to find it useful. But it was not sustainable to spend 10-20 minutes every day going on the website and updating the graph. So I decided to learn how to scrape data, and how to automate out a tweet.
The data visualisation itself I was already doing in R, and that was just being driven by a CSV that had the numbers. You can set it up to run your whole script and I have that set on my computer just to run and tweet it out.
But I don’t actually do that, I still sit there and just press the Go button. Some days there’s a slight error in the data which they correct, or it comes out late. I’m a bit paranoid so even though it’s cut work time down, I’m still double checking to make sure it’s all accurate. Apart from that, it’s all automated from downloading the data, updating the graph and then pushing it to Twitter.
Data visualisations
Leaving you plenty of time to enjoy lockdown! And how did you create your data visualisations, what decisions did you make around those?
It’s all being done in R and then it specifically uses the ggplot2 package, anyone who knows R will be using that really to make their visualisations. I also use a package called ggplotly. That’s for an interactive version of the graph on my website. That’s a really nice package, you can build a visualisation with ggplot, and then this will spit it out into an interactive version with like one line of code, and then you can tidy it up a bit.
In terms of what’s on the graph, I put the case numbers on and then the next thing I added to it was the rolling average. There were some days where there’s a crazy discrepancy in the numbers. That might be by random chance that there were a lot of positive tests those days, or there was maybe a backlog over the weekend or something. It meant that you could put it out like ‘oh wow in Edinburgh there are 100 cases today’, which at the time was quite high, and people would react to that being like the second wave.
It’s important to have that rolling average because it flattens all that out, and allows you to see the trend. I post the number of deaths in the tweet, but not on the graph. For the local area, they were frankly quite low. It just wouldn’t have worked. I think you could have too much on that one image.
More recently, when they started making it available, I’ve done the test percentage as well, because that can show you how the cases have gone up or down depending on that day’s testing number.
Getting into data journalism
Now for our quickfire questions. How did you become aware of data journalism and what draws you to it?
After my undergraduate I went straight into the civil service as a statistician, so I was using similar kinds of skills but for governmental research. But through that I started reading a lot more to do with data science statistics, and obviously through that you see a lot of data journalism work. Looking at that, I thought, this is similar to what I’m doing in my current career, but is like, way more interesting. There’s a whole world of stuff that can be investigated. So that kind of got me interested in data journalism and made me start reading different writers.
Top data journalists to look out for
Do you have any top Twitter follows that you could recommend?
John Burn-Murdoch is one of the best in data journalism. As well as the depth of information, there’s such a variety of different statistics. It’s also always presented really nicely and that’s something I’m particularly interested in because I’m not yet that good at making things look nice.
A couple maybe more niche Twitter follows is, in Scotland, Lesley-Anne Kelly, the data editor for The Courier and Press and Journal. She doesn’t tweet so much but she’ll tweet out her articles and it’s really interesting because Scotland has just not got the same level of coverage for data journalism and she’s a leading person in that. It’s not just the statistics you’d expect, she did a really interesting article, because we’ve had so many of these daily briefings up here, an analysis of the briefings, who was attending, what were common questions and stuff like that.
I’ve also noticed during this time there’s been a lot more amateur people or people trying to get into journalism, like myself. For anyone like me who’s addicted to looking at the figures every day, there’s Ganesh Ranganathan. I don’t think he’s a journalist but he tweets really in depth daily stats for the UK wide. He’ll also put up interesting ONS stuff or any other studies that come out, so a really good follow!
And is there anything that you’ve read, watched, or listened to on those subjects that you thought were good?
It’s a similar three for much data journalism The FT, The Economist and New Statesman. I also quite like The Spectator who are just kind of getting into doing data stuff. So that’s kind of what I read every day. The Guardian has also got a really nice UK map where you can see areas increasing and decreasing on their daily tracker page.
Useful resources for beginner data journalists
Are there any data sources that you think are underutilised by data journalists that we could use more of?
I think it’s quite often interesting when ONS, or in Scotland NRS, puts out their headline, like jobs statistics. If you go and look at their press release, the headlines say, ‘there’s been x vaccination numbers’. But when you go back to the actual data, you often find a little niche statistic that’s not being used.
It’s not so much hidden data sources, but going deeply into them. If you go on any of the UK dashboards on covid, it’s quite often interesting. If you look at the documentation about them, it will have links to the data that’s driving the dashboards. And quite often, there will be data in there that they’ve not actually included in the dashboard.
Any other resources or advice for just data journalism in general?
As ever, Google is your friend. For people that want to get into this stuff, it’s learning something that can maybe set you apart from the crowd. Everyone is going to be using Datawrapper, for example, or Flourish, [data visualisation] websites. If you can learn something like R, or D3, I think that’s really useful.
One thing I have to say about learning is that it’s best to pick a project that you want to do that you don’t necessarily have the skills to do, and then acquire the skills specifically for that project.
In the example we’re talking about, the only thing I knew how to do from my R knowledge was how to make the graph. To do the automation, and the Twitter bot, it’s just a case of going on Google and being like ‘how to do this in R’ and and then invariably, I’ll find answers on Stack Overflow, which has tonnes of in-depth stuff on software that’s useful for data journalism.
The way I learn anyway, is you find code that’s been used for a project, that’s slightly similar to what you want to do. Then, you just take on from that, and adapt it to your project.
Once you’ve done that you acquire that knowledge as you go. Learning by doing is the key. That’s my top tip.