To be honest, I was hoping to avoid spending time on this. But claims by Peter Doshi in a January 4 BMJ op-ed headlined Pfizer and Moderna's "95% effective" vaccines seem to be embedding themselves into Covid vaccine skepticism lore. It's still topping BMJ Opinions "most read" list as I'm writing over 2 weeks later, and it's been reproduced in full on at least one of the major anti-vaccine websites (the one founded by Robert Kennedy Jr – but I don't want to link to it to show you). People keep asking me my opinion about it. So here it is. (Disclosure up front: Doshi and I were on opposite sides on the issue of evidence about HPV vaccines, too.)
Argument 1:
A rough estimate would be...a relative risk reduction of 19%.
Even after removing cases occurring within 7 days of vaccination...vaccine efficacy remains low: 29%.
This claim is about Tozinameran (the BNT/Pfizer vaccine), and he's arguing the data supports the skepticism he publicly expressed about this trial before its results were known. The trial's primary efficacy outcome was for people who were Covid-free until at least 7 days after the second injection, which is when they estimated immunity from the vaccine would have had a chance to kick in. The efficacy rate against symptomatic and laboratory-confirmed Covid-19 at that point was 95%. For everyone who was vaccinated in the trial, including those who didn't get a second injection, it was 82%. (Check out my explainer about guarding against being misled about vaccine efficacy rates at WIRED.)
So how does Doshi arrive at the claim that the real efficacy rate is actually only 19% or 29% – either of which, if real, would be, as he says, "far below the 50% effectiveness threshold for authorization set by regulators"?
The trial's primary estimate is based on 8 sick people in the vaccine group versus 162 getting an injection with placebo. To be counted, you had to have symptoms plus confirmation by a PCR test that you were infected with the virus. Doshi zeroes in on another group of people, labeled as having suspected Covid-19 without testing positive for the coronavirus.
That was 1,594 people in the vaccine group versus 1,816 with placebo. So he adds them to 8 versus 162 confirmed Covid illnesses, creating a category which he refers to as "a disease" called "covid-19 symptoms, with or without a positive PCR test result". That's the basis of his back-of-the-envelope calculations of alternative efficacy in the 19% to 29% range. Doshi argues it's valid to believe this means the vaccine's efficacy is so dramatically lower than 95% for 2 reasons:
- If the PCR test results for "many or most" of these people were false-negatives, that would drag efficacy down; and
- Cause doesn't matter, he says. If people with "suspected covid-19" had the same outcomes as those who also tested positive, then his category "may be a more clinically meaningful endpoint".
Firstly, is it possible that there were enough false-negative PCR test results to substantially drag down efficacy? Doshi doesn't cite any evidence to underpin his claim that this could have happened. So let's look at some.
The evidence base about the rate of false negatives in PCR tests for Covid-19 is fairly weak, for various reasons, but a systematic review of studies on this question estimated that in non-Chinese countries, it was around 6%, with a 95% confidence interval of 4% to 9% (a gauge of the uncertainty around the estimate). Even with a 90% confidence interval, the highest end of the range was 17%. So false negatives from a single PCR test aren't rare – but, even if the process of diagnosing Covid-19 was that simple in the trial (and it wasn't), false negatives would be unlikely to slash the estimate of efficacy as much as Doshi hypothesizes.
If you get a PCR test before the virus has proliferated in your system, it wouldn't be able to detect that you were infected even if you were. But by the time someone has symptoms, they have enough of the viral material the PCR test can detect from a nasopharyngeal swab, and that peaks by the end of the first week of symptoms. PCR tests were planned in that time window in the trial. That wasn't all that was done. Some context is important before we get to that, though.
Diagnosing Covid-19 was critical in these trials, not only because it was the measure of efficacy. There's always been a concern that vaccination could cause vaccine associated enhanced respiratory disease (VAERD). So Covid-19, and its severity, are also adverse events in Covid vaccine trials. Because it's a major safety issue, the trialists have to put a lot of effort into reassuring regulators that they've assessed this thoroughly.
Doshi uses the term "suspected covid-19" without explaining what's behind that. Here's what it meant in the trial (as detailed in the trial's protocol). At every visit, trial participants were reminded that they should contact the trial personnel if they experience any of a list of symptoms, most of which usually won't be caused by Covid-19. Unless it's in the first 7 days after an injection and the trial personnel are confident it sounds like an expected vaccine reaction, when a person reports they have a symptom or symptoms, they are now in the "suspected Covid-19" category. That triggers a visit, either in person or via telehealth. They get swabbed for the PCR test, possibly more than once to be sure, and blood is taken to test for antigens and antibodies. Three medically qualified people assess all the information for each person – and they are blinded to whether or not the person got the vaccine. This is a pretty rigorous system for diagnosing Covid-19. And the chances of false negatives for symptomatic Covid-19 with this process are lower than in those studies of nothing but a single PCR test.
People at the FDA combed through the data for people listed as suspected Covid-19. Their conclusion? "[T]hese data do not raise a concern".
What of the "if" part of Doshi's claim on this – if their outcomes were the same as the people with confirmed Covid-19? Well, if enough of these people to drag the efficacy rate down anywhere near 29% had the same level of illness, then dozens of them would have ended up in hospital. That would have been pretty dramatic and obvious – and, of course, the clinicians at those hospitals would have been running further tests. In fact, the FDA reports just 2 of those people got sick enough to go into hospital.
What about Doshi's second point: that it doesn't even matter if it's a respiratory illness that isn't caused by the novel coronavirus? Let's just swipe left on that. It's absurd to demand that a Covid vaccine be just as effective against all non-Covid-19 respiratory disease as it is against the novel coronavirus. "But why should etiology matter?", he wrote. Of course the cause matters: it's not non-SARS-CoV-2 infection that's causing a society-crippling pandemic! We need Covid vaccine to help us stop Covid-caused devastation, not all respiratory disease caused by bacteria and other viruses.
By the way, even leaving aside the fact that this category is not "a" disease, the calculation adds bad statistics to the mix of errors of medicine and logic on which he's built his claims. He's taken only the confirmed events from the subset of people in the 95% efficacy group, and added the people in the suspected category for the whole trial. The whole trial had 50 versus 275 confirmed Covid-19 events, not 8 versus 162. That biased calculation pushes his alternative efficacies down a couple of percentage points. You would think some fundamental expert quality control would go into such a serious thing as alternative vaccine efficacy claims flying under the BMJ banner, wouldn't you?
Argument 2:
Let’s put this in perspective. First, a relative risk reduction is being reported, not absolute risk reduction, which appears to be less than 1%.
Swipe left on this argument, too. What Doshi's done here isn't putting the vaccine in perspective. This relative versus absolute risk issue is a common manipulative device – used both by people who want to magnify or trivialize benefits. In this case, it's the latter. The trial was powered around a risk of getting ill with Covid-19 of 1.3% in a short period of time. So of course in absolute terms, any reduction of that risk would be small. But that doesn't mean our lifetime risk of getting Covid-19 is small in an unvaccinated community where the virus can't be held at bay forever.
We need to know the relative risk reduction of the vaccines because we know we need to minimize the impacts of this disease – as of this week, 1 in 1,000 Americans have died of it, another 2 or 3 may be dying every minute, and it's far from over – but those risks vary. Take me, for example. I live in the country, in Victoria, Australia. The total people from my area who have tested positive for Covid-19 in the whole pandemic so far is 11 – and the last of those was months ago. My risk of getting infected in my part of the country is basically 0. So no matter how effective a vaccine was, the absolute risk reduction I would have from it at this rate would be 0. However, if I got on a plane, flew into a hot zone where every third person was infected, and pitched in on a Covid ward without any PPE, the absolute risk reduction I could gain from a highly effective vaccine would shoot up. It's all relative.
But there's another issue here. And that's the public health and societal one. Even though my personal risk of infection is miniscule, I am affected by the pandemic. Everyone around me is – locally and globally. Our societies need community risk to be way, way down everywhere. Even if I personally have little to gain from getting vaccinated, I have a huge stake in contributing to the community gaining enough immunity to function freely and fully again. We are all affected by pandemic restrictions and their socioeconomic impacts, and vaccination can reduce those harms: Covid-19 doesn't just cause a ghastly respiratory disease that causes deaths and long-haul consequences for perhaps 10% of the people who survive symptomatic disease. Meanwhile, there are many people who can't get vaccinated for medical reasons, so we need pretty much everyone who can get vaccinated, to do it. Encouraging healthy people to think of Covid vaccination solely in terms of the short-term absolute risk reduction of disease for themselves alone is irresponsible.
Argument 3:
371 individuals excluded from the efficacy analysis for “important protocol deviations on or prior to 7 days after Dose 2.”
In the Tozinameran trial data, 371 people are listed as having unspecified protocol deviations, and were among the group excluded from the primary efficacy analysis. Doshi says this is "concerning", that the "exclusions are difficult to even spot", and we need to know more about them.
I don't see any reason for concern about this in a trial of over 40,000 people. And not just because it's so few people. Why? For 2 reasons Doshi doesn't point out to readers. Firstly, they're not a unique group of 371 people: there could be more than one reason people were excluded, and usually it was because they didn't have a second injection, or didn't get it in the defined interval for the trial – that was 1,550 people. I'd be surprised if a substantial proportion of the 371 weren't among those 1,550, but even so, I'm not concerned for a second reason: there is an efficacy analysis that includes them. And no, it's not difficult to spot!
There are 2 top level efficacy analyses in the data: the first, 95%, is called evaluable efficacy. The other is called all-available efficacy, that excludes only 99 people who didn't get any vaccine at all, and 1 for "no ICD" (which is for missing data, I presume). Out of 43,651 people randomized, those 100 really don't put a dent in it. The all-available efficacy is the 82% figure I quoted right at the top.
Remember this image that circulated with enormous fanfare when the results came out? This is the all-available efficacy population, and those 371 people are included in it. (Source: FDA.) We know what happened to them. There is no smoking gun here, and it sure isn't hidden!
Argument 4:
[W]orry about unofficial unblinding.
Both participants and investigators making judgment calls about Covid-19 were blinded in both the Tozinameran and Moderna vaccine trials. Doshi suggests that blinding may not have been very effective. Because of vaccine reactions, there was a good chance people could guess if an individual was in the vaccine group. How much uncertainty could that put around decisions around suspecting, then diagnosing, Covid-19? And it's the Tozinameran trial he's particularly pointing his finger at here, because of there being a couple of hundred more "suspected Covid-19"s in the vaccine group.
How could this play out? For example, if participants were deciding whether or not they were in the vaccine group based on adverse reactions, then the majority of people could have thought they were in the placebo group in the Tozinameran trial. They could therefore, be potentially very concerned about signs of Covid-19 – with more of them actually in the placebo group. That could have increased apparent vaccine efficacy. Given how common headaches and so on are in daily life, quite a few in the placebo group probably guessed (incorrectly) they had active vaccine. The vaccine isn't 100% protective, so if lots of people guessed they had the vaccine, and really relaxed on Covid prevention measures, that could have reduced apparent vaccine efficacy. Those are the kinds of uncertainties that guessing can introduce.
How much could these sorts of influences swing vaccine efficacy? We can't really know. Given that you would hope people in these trials were very concerned about not getting Covid-19, and how hard it is to forget about Covid-19 during this pandemic, you would hope they would all be on the lookout for signs of disease. The evidence about whether blinding participants and outcome assessors has an effect on clinical trial results is decidedly mixed, and inconclusive. That's not surprising, is it?, when there could even be effects that more or less cancel each other out. Bottom line, though, I don't think there is evidence this could be a major factor.
Argument 5:
[U]blinding and primary event adjudication committees...what criteria did they employ, and why...was such a committee even necessary? It’s also important to understand who was on these committees.
This section was odd. The committees were necessary for just the reasons Doshi raises: there were inevitably subjective decisions here, and so multiple blinded adjudicators assessed the information. You can see that from the criteria detailed in the protocol. Take determining whether it was severe illness for example: most of the criteria were objective, but there was also this "Significant acute renal, hepatic, or neurologic dysfunction". So, plenty of potential for different judgment calls there. And as already mentioned, there isn't strong evidence that blinding makes a difference.
Doshi, though, seems to think we should all be suspicious of who was on the Tozinameran committee in this context:
"While Moderna has named its four-member adjudication committee – all university-affiliated physicians – Pfizer’s protocol says three Pfizer employees did the work. Yes, Pfizer staff members."
I think this gets to a critical underlying issue about Doshi's bias and the buttons he's pushing on, intentionally or not. Considering Big Pharma untrustworthy plays a foundational role in conspiracist thinking, not just about drugs and in anti-vaccine literature and lobbying, but in general. The mentions of data being buried or hard to find push those buttons, too: they are keeping the truth from you. Of course, if they actually were, then it wouldn't be a theory. But making it free to view on the FDA website and at medical journals is pretty transparent, even if the documents are large and dense.
I don't accept that "university-affiliated" necessarily means less bias in an outcome assessor, or for other members of the trial teams. The Moderna vaccine was a partnership with the NIH, after all, and that could introduce bias too. Very rigorous methods are meant to minimize the influence that individuals' biases can have. In this particular instance, I think there are actually more signs of bias in the reporting of the Moderna trial than the Pfizer one for Tozinameran. Why?
The trial for Tozinameran reports there was 1 person with severe Covid-19 in the vaccine group. That person was classified as having severe illness even though they never needed medical care: most of us wouldn't consider what they had "severe" at all. But they were classified as severe because of the blood oxygenation level at one visit.
The Moderna group report 0 people with severe Covid-19. But the FDA analysis for their trial reports this:
One participant in the mRNA-1273 group, a participant >65 years of age who had risk factors for severe COVID-19, was hospitalized due to oxygen saturation of 88% on room air 2 months after receiving the second dose of vaccine. There was a verbal report of a positive SARS-CoV-2 RTPCR test 3 days prior to hospitalization; however, NP swab collected during hospitalization was negative for SARS-CoV-2. Due to absence of a confirmed RT-PCR result at the time of data snapshot, this case was not referred for adjudication and not captured. The pre-hospitalization RT-PCR result was later reported to be positive from an external CLIA-certified laboratory and may represent a severe COVID-19 case with hospitalization in the vaccine group.
Well, that was some diplomatic wording from the FDA. The FDA didn't conclude there were any possible false-negatives of concern for the Tozinameran trial, but this seems pretty clear.
Secondly, BioNTech/Pfizer determined more of the serious adverse events they adjudicated to be possibly or definitely vaccine-related than the FDA did; Moderna determined fewer to be possibly or definitely vaccine-related. From the start on, there's been less hype in public and in published materials from the BNT/Pfizer group than from Moderna. And the trials run by Oxford University for their vaccine have raised far more, and more serious concerns, than Doshi raises here. So he may want me to sell me on the idea of being more suspicious of the Tozinameran trial because its assessors weren't from a university, but I'm not buying it. I haven't seen evidence to justify a concern about this specific trial.
To me the bottom line here is this: these are both impressive clinical trials, with impressive results. I think the Pfizer trial is particularly scrupulously reported, but with the serious scrutiny they're getting from the FDA and the EMA, we're getting extensive and reliable information about both. Yes, it would be great if the trials had been powered to give us highly certain answers on severe Covid-19 and other key questions. But it's an emergency. And we couldn't afford to wait for months longer to start using high-efficacy vaccines. Perfect information is wonderful, but it comes at a cost, and the cost of still not having any vaccines at all right now would have been horrendous.
Argument 6:
Vaccine efficacy in people who already had covid?
There were some people in the mRNA vaccine trials who had previously been infected with SARS-CoV-2. And though the numbers are small, the vaccines seemed to protect them from re-infection. Doshi's concerns:
But with only around four to 31 reinfections documented globally, how, in trials of tens of thousands, with median follow-up of two months, could there be nine confirmed covid-19 cases among those with SARS-CoV-2 infection at baseline? Is this representative of meaningful vaccine efficacy, as CDC seems to have endorsed? Or could it be something else, like prevention of covid-19 symptoms, possibly by the vaccine or by the use of medicines which suppress symptoms, and nothing to do with reinfection?
It goes to the level of distrust Doshi has in the trials, doesn't it?, that he'd dismiss incidence rates in well-run trials in favor of .... checks links .... a report from October that isn't a systematic review, and a re-infection tracker from a news agency. But I think all this shows is that he's not well-informed about the issue of re-infection from a virus scientists are still in the early days of learning about.
Let's go with the Tozinameran trial: it was bigger, international, and there were more apparent re-infections. The proportion of people in the trial with signs of prior infection was 3%, and the incidence of confirmed, symptomatic Covid-19 a week after the second injection was similar in the placebo group, whether or not there were signs the people had been infected previously. This doesn't seem wild. For example, a recent study from a Beijing hospital concluded that 2% of the 273 people admitted there with Covid-19 were re-infections. There's a lot we don't know yet about the risk of re-infection: how long naturally acquired immunity lasts, how much immunity very mild infections provide, and whether having been infected protects you from a different variant to the one you had – so many questions. That's not a reason to distrust the vaccine trials, though.
At the outset, Doshi says he will "outline new concerns about the trustworthiness and meaningfulness of the reported efficacy results". That one was the last based on the trials' results. I don't think he comes close to establishing concerns about trustworthiness or meaningfulness, but he's clearly been successful at convincing many people. It brings us to Doshi's core argument: that we need more data (which is almost always true), and the data for these trials should be open sooner than at the end of the trials.
I, too, am firmly committed to more data transparency in trials generally, and to radical transparency for the Covid vaccine trials. But the data from these 2 trials have had an extraordinary amount of scrutiny, and a lot of data has been released. The US FDA re-analyzed the raw data, and reportedly had 150 people on staff doing it on those 2 trials. From their first report, on Tozinameran, it looks like the EMA (European Medicines Agency) did substantial re-analysis too. Then there are the committee members and people from the NIH who pored through it all as well, and the peer reviewers at a medical journal. That's a good enough start for decision making, especially where communities are in a dire situation and can't afford to sit on highly efficacious vaccines without using them for months more waiting for perfect information, while people, healthcare services, and societal social and economic fabric are stressed to or past breaking point.
I strongly support re-analysis of trials, and I frequently criticize the adequacy of trials, their analyses, and their reporting myself. I know many analyses and reports are downright dodgy – but that doesn't mean they all are, or that every criticism should bring trustworthiness into question. As we move down the path of more re-analyses of trial data, I think we have to become far more discriminating about criticisms and re-analyses. The incentives that could influence people in a company producing a vaccine are obvious. But publications that get a lot of attention, citations, and then more grant money are a critical part of academic currency and career survival – and attention-grabbing claims are in the interests of many medical journals, too. What that encourages is not always in the public interest either.
Hilda Bastian
19 January 2021
Disclosures: I write about Covid vaccines at my PLOS Blog, Absolutely Maybe, and at WIRED. However posts here on my personal website are unpaid. I am an occasional (unpaid) contributor to BMJ Opinions, but haven't done so since the pandemic started, and haven't approached them about rebutting Doshi's post (although I might). I'm a (paid) adviser to a BMJ publication, The Drugs and Therapeutics Bulletin. My interest in Covid-19 vaccine trials is as a person worried about the virus, as one of my sons is immunocompromised: I have no financial or professional interest in the vaccines. Doshi and I have agreed and disagreed on important issues over the years, but I don't think I've ever criticized his work or views publicly. However, we were on opposing sides of a long-running debate about HPV vaccines, and I actively disputed claims from some of his colleagues (Peter Gøtzsche, Carl Heneghan, and Tom Jefferson): that may at some point have included criticism of commentary he co-authored on that vaccine. I worked for an institute of the NIH from 2011 to 2018 (NCBI at NLM), but not the one working on the Moderna vaccine (NIAID). More about me.
The cartoons are my own (CC BY-NC-ND license).