When people are trying to plan for what happens next in the Covid-19 pandemic, estimating what kind of mortality rate they could be facing matters a lot. Predict too low a number, and the danger of Covid-19 may be comparatively trivialized: we probably won't do enough to prevent harm to ourselves and each other, and we'll be under-prepared for what follows – we won't have enough of everything we need to cope. That combination could drive the real life mortality rate avoidably high.

Too high a number, though, and we might over-react: we could suffer from the downsides of prevention measures that at some point are no longer worthwhile. On top of that, over-estimation could set off the classic boy-who-cried-wolf problem: and again, there will be people – maybe many of them – who won't do enough to prevent harm or prepare enough for the rocky road ahead.

One thing's certain, though, as we negotiate all these numbers: you will meet too much false precision! The best we can hope for is a range that's very likely.

And that brings us to the reason for this sequel to my post this week on John Ioannidis' controversial preprint inferring the infection fatality rate (IFR) of Covid-19. That's the proportion of people who get infected with SARS-Cov-2 that die. That post itself was a sequel of sorts to my post in March on his estimation of the coronavirus' case fatality rate (CFR) – the proportion of people known to have the virus who die. He had argued, among other things, that we were totally over-reacting: the CFR, he said, could be 0.3%, with a population infection rate of 1%. In that theoretical scenario, he projected there would be 10,000 deaths in the US. As I'm writing just over 2 months later, the US is barreling towards a heartbreaking 100,000 reported deaths in this early stage of the pandemic.

Why the sequel? Well, I was working on a post this morning about preprints in this pandemic, after thinking about the effects of this particular one. And then a tweet flew past – I can't remember from whom – talking about an analysis of the IFR that I hadn't seen before. And since I had been so publicly critical of Ioannidis' effort, I thought I had better look at it, as well as one I had linked to in my post.

These 3 preprints were posted between 18 and 19 May. They're quite different, but one is very much not like the others:

• Gideon Meyerowitz-Katz and Lea Merone – this one is a systematic review and meta-analysis of studies publishing IFRs: the 18 May one is an updated version 2 of a review published earlier in May;
• Richard Grewelle and Giulio De Leo – this one is the odd one out: it's from 18 May, too, and it reports on a method for calculating an estimated IFR globally based on number of tests, infected people, and Covid-19 deaths that countries report; and
• John Ioannidis – this one from 19 May is a review of seroprevalence studies (blood tests for signs a person has been infected), whether or not they included an estimation of IFR (if they didn't, the author did it, using additional data).

Here are the IFRs they estimate:

Those differences might not seem large, but they make a big difference when they affect millions of people. For perspective, when the IFR for New York City was estimated to be 0.85%, around 15,000 people had died of confirmed or probable Covid-19 – and that was 0.18% of the population of the whole city.

The highest estimated IFR of these 3 preprints is the one by Grewelle. That paper is the only one of the 3 that reports making all their data available online (but I didn't check that). It is a paper describing a novel method of calculating an IFR. The authors point out that basing global estimates of the IFR based on studies done in better-resourced countries with better access to health care isn't going to reflect the world's experience. I don't have the expertise to assess their proposed method on the fly. However, the result they have calculated has more uncertainty around it than the others, as it's based on the national tallies at Worldometers. The data emerging for infections, testing, and deaths in real time is riddled with all sorts of problems.

The Ioannidis preprint has the lowest estimate by far. However, the biases in the study may be pushing the estimates down. I argue in my post about it that it has a biased sample of studies, the search strategy is inadequate (and inadequately reported), the data methods used may be biasing towards lower IFRs, and there are data errors.

Ioannidis excludes full reports on government and institutional websites. The day after the cut-off for this study, the biggest seroprevalence study yet was posted online by the Spanish government. It dwarfs the pool of data in Ioannidis' study, and its IFR was just over 1%. Critically, too, Ioannidis compares the Covid-19 IFR he infers with a figure he gives for influenza IFRs in bad years. But he provides no citation to substantiate the high IFRs he attributes to influenza: they are similar to the CDC's modeling estimates for symptomatic case fatality rates (CFRs), and they are not based on seroprevalence studies. So he didn't compare like with like.

Which brings us around to the analysis, by Meyerowitz-Katz. This is a systematic review and meta-analysis, and it has a lot of problems, too. (He has indicated though that he welcomes feedback, which is great, and he's working on an update.) 

As with Ioannidis' review, the search strategy for this review is both too limited, and poorly reported. They searched only PubMed and a single preprint server for papers and preprints. Not searching more preprint servers would explain why the study I quoted above from New York City may have been missed, for example. The number of records identified by the database searches before any screening isn't reported, and the potentially eligible studies that were excluded after screening aren't reported either. (That makes it impossible to assess the application of the inclusion criteria.)

Among the 16 included studies there were "5 studies that attempted to establish IFR estimates from serosurvey results or limited known populations with very high rates of infection", which is a critically important bit of information, but these 5 aren’t identified.

In the first group of meta-analyses (Figure 1), they break the 16 included studies into 3 categories of "study design" (n) that make no sense, because they aren't mutually exclusive and they aren't all study designs: entirely modeled estimates (6), observational studies (5), and preprints (4 in the text, but 5 in the data). Preprints aren't a study design: they are a mode of publication. The division into the 3 categories also seems to be wrong. The highly controversial Santa Clara seroprevalence preprint is listed as an observational study (correct), but it’s not in the preprint category (incorrect).

There is a bigger underlying problem, though. By pooling modeling studies with each other, and then with studies that provide the data for at least some of the modeling studies, some of the primary data is effectively getting counted more than once. Some datasets are therefore getting to influence the final pooled estimate more than others because of this – especially if/when they fueled more than one of the modeling estimates. For example, the study of the Diamond Princess cruise ship's passengers and crew are a free-standing observational study, but also contribute to the CEBM assessment. I didn't try to untangle this further, but I think it's a fair bet that the Wuhan data appears in multiple studies. (And just to complicate it further, the CEBM assessment is not a journal article, preprint, or government report, and so actually doesn't meet the study's inclusion criteria.)

I would be much happier if just the observational data were pooled (regardless of whether or not it is reported in a journal article), and the modeling estimates were shown but not pooled.

Where does this leave us? I don't think any of these preprints provides a reliable and usable estimate of Covid-19's IFR so far. Improved treatment of Covid-19 might start to improve outcomes as experience and treatment options grow. But unless that makes a substantial difference, I fear very low estimates of IFR are overly optimistic.

The CDC released the scenarios it's using for current Covid-19 projections on 20 May. Of the 5 scenarios, the one they report as their "best estimate", is a symptomatic CFR (not IFR) of 0.4%, with a reproduction number of 2.5. That projected scenario also assumes 35% of all infected people will have no symptoms, so the IFR is considerably lower. Even at that rate, it's frightening. In a bad year, the CDC estimate of symptomatic CFR for influenza is less than 0.2%, but a lot of people have immunity. Only a very small percentage of people in the US have now acquired immunity to Covid-19 by infection, and somewhere around 100,000 have died.

Update 12 July 2020: The Ioannidis and Meyerowitz-Katz preprints were updated – both remain problematic (and the Meyerowitz-Katz now has an additional problem: changing inclusion criteria to match new studies found, rather than applying the inclusion criteria to new studies). The CDC updated their models, basing their new IFR on the Meyerowitz-Katz preprint: instead of a CFR of 0.4%, they now use an IFR of 0.65%.

Hilda Bastian

First posted 23 May 2020

Disclosures: I am 59 years old. A close family member who is one of the people I care about most in the world is young and immuno-suppressed, and 2 of the others in the same category are high risk for other reasons. I have written about Covid-19 at WIRED, and at my own blog at PLOS Blogs, Absolutely Maybe. I wrote a very critical post in rebuttal of Ioannidis' STAT News essay on Covid-19 here on my personal website on 18 March 2020, with a postscript on his estimation and discussion of the disease's case fatality rate on 21 March.