King's College London

Alongside advice on how to self-isolate in style and comfort, and recriminations against those people foolish enough to go outside and/or socialise, a large portion of my (quite nerdy) social media feeds has been taken up by people, mainly with good intentions, suggesting that the current global outbreak of Covid-19 might create a “cool natural experiment”, to allow researchers (mostly social scientists) to answer hitherto unanswerable questions about some phenomenon or another.

Maybe we could learn about the effects of home schooling? Or virtual education? Or of changed routines? Reductions in food waste? Effects of delaying non-essential surgery? Reduced class sizes? Increased caseloads for social workers? Noisier measures of exam results? More state intervention in the labour market? You name it, we’ll be able to research it in a few years’ time when administrative data from this period comes to light.

No. No we won’t. Stop it.

Natural experiments require an exogenous shock – something potentially large, and fairly random – which has an impact on something and which isn’t part of the usual causal chain for that thing.

Let’s take the example of home-schooling. People can home-school for lots of reasons – because they want their child to get a unique educational experience, because of where they live, because they don’t trust the education system, or because they want to abuse their children without them coming into contact with professionals. The things that lead people to home-school are also associated with educational outcomes. So, if home-schooled children get especially good or especially poor grades, we can’t say this was because they were home-schooled – it could have been caused by the other factors. In a natural experiment, we’d want some event, like school closures, to force lots of people to home-school that otherwise wouldn’t, for a reason beyond their control. This would potentially let us find out whether home-schooling increased or decreased attainment. Some people are arguing that Covid-19 is one of these events.

They’re wrong.

There are all kinds of reasons why they’re wrong, but the most widely applicable is that we need a natural experiment to not affect the outcome – in this case, educational attainment – through any other means. This is clearly not the case here. More people are home-schooling, but they’re also more likely to be sick, as are their parents, in a substantial way at some point during the period. Normally people who home-school have time to prepare, and don’t start midway through a year, and they’re normally allowed to go outside – to museums, galleries, and other educationally beneficial places. Very often, their parents won’t be worrying about money, or elderly relatives, or trying to work their normal job alongside their educational duties. And so on.

I’ve picked one example, but you could do the same for the other suggestions given – if you happen to have some free time on your hands in the next few weeks...

Dr Michael Sanders is a Reader in Public Policy in the Policy Institute at King’s College London. He is also Chief Executive of What Works for Children’s Social Care, and Academic Lead for the Centre for Transforming Access and Student Outcomes.