This is a data appendix for a piece I wrote at The Atlantic, challenging recent claims that 5 or 6 Covid vaccines have been shown to be 100% efficacious at preventing hospitalization for Covid-19 or death from the disease. The point of this analysis is not to suggest that these vaccines don't provide great protection against these serious Covid-19 outcomes: they do. It's to show why we can't put exact numbers on how high that protection is.

There are usually multiple sources of information for these phase 3 trials, including reports from the vaccine developers themselves (e.g. in medical journals), analyses by drug regulators (e.g. FDA), and protocols for the trials (pre-specified detailed study plans). All the data you need to pull together an accurate picture is generally not in a single source. There is a bibliography of the data sources I used at the end of the post: where available, I used the efficacy rates and data in the regulators' reports. And because this recent narrative was fuelled by some viral tweets with data tables full of 0s or 100%s, links and copies of the originals (where not subject to copyright) follow the bibliography.

I have split the data into 2 tables. The first includes vaccine efficacy against symptomatic Covid-19 and for severe Covid-19. It aims to provide a perspective on the data we can be most confident about, which is symptomatic Covid-19.

For all the trials, symptomatic Covid-19 was a primary efficacy outcome. That means the trials were designed to be able to answer this question, including being large enough. That's why it is the efficacy result with the least uncertainty around it.

For all but one of the vaccines, vaccine efficacy against severe Covid-19 was a secondary outcome. For one, moderate and severe weren't separated, but as there were 0 events in the vaccinated group, it didn't matter for this exercise. Trials are not necessarily designed to provide definitive answers on secondary outcomes, and may not be able to.

For primary and secondary outcomes, the data should be reported. There's more than one statistical method for calculating vaccine efficacy, so the method specific to that trial should be reported too.

The second table is for vaccine efficacy against hospitalization from Covid-19, and death caused by Covid-19. These were sometimes secondary efficacy outcomes, but sometimes not, so I have included columns to show which did not have those outcomes. Like deaths from Covid-19, hospitalizations for Covid-19 are generally reported as serious adverse events, even if they are not efficacy outcomes. However, we can't be sure of how hospitalizations were handled if we have no access to the trial protocol, or it's not explicitly addressed in a clinical trial report. 

The tables include:

• "Vax vs control": those columns include the numbers of events (e.g. hospitalization) in the vaccinated group versus the placebo or other control group.
• Vaccine efficacy where it is calculated for those numbers, and 95% CI, which is a statistical measure of the range of uncertainty around that efficacy. A range that is only 5 or 10 percentage points apart indicates that we can be fairly certain the efficacy would be roughly similar in another similar group of people – but when there's a very wide difference in that range, the degree of uncertainty is very high, too.
• I have included "n.a." (not applicable) where a vaccine efficacy was not calculated or could not be. When there are 0 events in a group, the particular statistical method used in that trial may not be suitable for calculating the 95% CI. Where the reports for that trial do not include a vaccine efficacy rate, I did not attempt to calculate one.
• I didn't include events for disease and hospitalization that happened before the timepoint designated in that trial for being sure a person couldn't have contracted Covid-19 before being vaccinated, or being sure that immunity was likely to have been reached: that is, I used the data in the group that corresponds to the primary efficacy analysis, because that makes the data consistent for each vaccine across its rows. However as two vaccinated people had died before their second dose of vaccine in one of the trials, I noted deaths that occurred earlier. For other items that are additional to the main efficacy group, that is noted.

That last point in particular explains several of the mistakes I've seen circulating about these vaccines. Here's an example for this important context to this data.

As for all the trials, the one for the BNT-Pfizer vaccine had a pre-set date which would count as enough time elapsing to be sure of 2 things: (a) that any Covid-19 illnesses weren't caused by SARS-CoV-2 infection acquired before they got the vaccine; and (b) that there had been time for immunity to develop from the vaccine. That time point was more than 7 days after the second dose for this vaccine. Those are the conditions for which the vaccine's efficacy against illness is 95%: people getting ill with Covid-19 after that date. (It was calculated from 170 sick people: 8 in the vaccine group versus 162 in the placebo group.)

Under those same conditions, the efficacy for this vaccine against severe Covid-19 was 66% – 1 person with severe illness in the vaccine group, 3 in the placebo group. (See Table 11 in the FDA report.) Those 4 people are among the 170 sick people we just saw.

But they also reported all the illnesses and severe illnesses that happened before that cut-off date, too. For example, in Table 12, they report everyone who got severely ill from the day after their first vaccine dose – including the 4 people we just saw. There was still only that same 1 person with severe illness in the vaccine group, but there were 9 in the placebo group. So the efficacy rate for that measure was higher: it was 89%.

However, that 89% efficacy rate against severe illness is no longer only counting the 170 sick people in the primary efficacy analysis. This means we have expanded the group to have 325 people in it, and some of the vaccinated people in that group may barely even have had their first shot when they got sick before the second. The primary efficacy rate for that expanded group – against all symptomatic disease – was 82% (Table 13).

You have to keep using the same group for each piece of data you choose to present, or you've got a biased group of numbers. One wrong pair of numbers would make the vaccine look less efficacious than it is – 82% against disease, and 66% against severe disease; and the other wrong pair would make the vaccine look more efficacious than it is – 95% against disease and 89% against severe disease. (This doesn't have to be deliberate: it's easy to make a mistake and grab data from the wrong table.)

You get a fairer picture of the vaccine when you stick with the pre-specified cut-off dates for the primary efficacy analysis and always make sure each analysis you pull from the data is for exactly the same group. If you move the goal post back to the day after the first dose, for example, then there are 2 "vaccinated" people who died from Covid-19 in this group of trials – both of whom were judged to have already been infected with SARS-CoV-2 when they got vaccinated (both in the Sputnik V trial). 

The order of vaccines in the table is the order in which their first phase 3 trial results were made public. I have included the pooled results for the UK and Brazil trials of the Oxford-AstraZeneca, and for Novavax, only the phase 3 trial in the UK.

Notes:

1. One vaccinated person was hospitalized with possibly severe Covid-19 2 months after vaccination was not included by Moderna, but highlighted by the FDA and EMA (European drug regulator). The EMA reported Moderna had not yet assessed this person's case. No efficacy rate including this person's illness was calculated. The number of people with severe illness in the control group could change, too, when assessments are completed.

2. Sputnik V's secondary outcome for severity included moderate and severe: 20 in total. What proportion were severe was not reported.

3. This vaccine efficacy is the one determined by the EMA, based on 2 standard doses of the vaccine.

4. Novavax has only reported some top-line interim results in a press release so far.

5. This vaccine efficacy is the overall efficacy: efficacy against the original strain of the coronavirus was 96%. It is the UK phase 3 trial only.

6. This vaccine efficacy is one that was calculated to harmonize with the definitions of symptomatic Covid-19 used by the FDA for the 2 mRNA vaccines (BNT-Pfizer and Moderna). It is an international efficacy rate, and it is a secondary outome. The primary efficacy outcome for this trial was very similar: 66%. The primary efficacy rate was also reported broken into regions: USA 72%; SA 64%; Latin America 61%.

Notes: 

1. 2 hospitalized vaccinated people may have had Covid-19, although they tested negative: these could have been false negative test results. Possible false negative test results in hospitalized people in the placebo group were not reported (that is, number unknown).
2. One vaccinated person was hospitalized with possibly severe Covid-19 2 months after vaccination was not included by Moderna, but highlighted by the FDA and EMA (European drug regulator). The EMA reported Moderna had not yet assessed this person's case.
3. Novavax has only reported some top-line results in a press release so far.
4. The secondary efficacy outcome for this trial that corresponds most closely to hospitalization in others is called "Requiring medical intervention". That does not include non-hospitalized people requiring medical intervention: it is defined as "hospitalization, ICU admission, mechanical ventilation, and/or ECMO". (ICU = intensive care unit; mechanical ventilation; ECMO = extracorporeal membrane oxygenation.)

If you spot any errors in the data in this post or problems in the descriptions, you can let me know via Twitter or send it as a comment at my PLOS Blog, where I also discuss this data. Comments there are moderated, so if you don't want the comment made public, just let me know.

If you want to read more about the spectrum of vaccine efficacy rates, Natalie Dean and Zachary Madewell have an excellent discussion of this at BMJ Blogs – highly recommended! (I have raised a couple of things to perhaps keep in mind with that post in a tweet.) 

Hilda Bastian
 

Originally posted, 7 March 2021

Disclosures: I write about Covid vaccines at my PLOS Blog (Absolutely Maybe), WIRED, and The Atlantic, and I am an occasional contributor to BMJ Blogs. My interest in Covid-19 vaccine trials is as a person worried about the virus, as one of my sons is immunocompromised: I have no financial or professional interest in the vaccines. I was contracted to an institute of the NIH from 2011 to 2018 (NCBI at the National Library of Medicine – not the one working on the Moderna vaccine, NIAID). More about me.

The cartoon is my own (CC BY-NC-ND license).

Updated soon after posting: The table by Monica Gandhi was removed after notification by Gandhi that it is copyrighted. Link to revised table by Eric Topol added.

Update March 9: Footnote 4 added to the second table explaining the secondary efficacy outcome chosen for hospitalization – with thanks to @MyrneStol for alerting me to the need for the footnote.

Data sources

BNT-Pfizer: FDA, phase 3 trial protocol, New England Journal of Medicine.

Moderna: FDA, EMA, phase 3 trial protocol, New England Journal of Medicine.

Sputnik V: Lancet. Phase 3 trial not published.

Oxford-AstraZeneca: EMA, phase 3 trial protocol, Lancet.

Novavax: Press release (and slides), phase 3 trial protocol.

Johnson & Johnson: FDA, phase 3 trial protocol.

The 3 tables

Note: The substance of my critique and upcoming op-ed were discussed with the authors in the days before publication. 

In chronological order:

Ashish Jha 

Monica Gandhi (Copy of table removed by request of the author. See also what I presume to be version 2, and version 3.)

Eric Topol and revised version without the 100% column here.