It started with shock and disappointment. A damning critique of the 2018 Cochrane review of the HPV vaccine was published in BMJ Evidence-Based Medicine, and a news report in the BMJ with this alarming headline: HPV vaccine safety: Cochrane launches urgent investigation into review after criticisms. How could Cochrane have gotten such an important review so catastrophically wrong?

Well, turns out they didn't. Once I dug into it, I decided it was a manufactured controversy - a hatchet job by people with such strong intellectual and ideological biases, that they didn't even pick up that some of their most implausible allegations were based on their own analytical and investigative slip-ups. You can see how I came to that conclusion in this "forensic" post:

The HPV Vaccine: A Critique of a Critique of a Meta-Analysis.

The conversation that followed that post made me realize there were so many misconceptions out there about when we would know if the vaccine was working out the way the trials predicted - and so many blanks in my own knowledge on these things. So I went digging again, coming up very optimistic about the potential for the vaccine, with a second post:

The HPV Vaccine Should be Preventing Cervical Cancer: Can We Tell Whether It Actually Is?

That post tackles the basics you need if the debate turns to the argument - "But it's never been shown to prevent cervical cancer!"

Meanwhile, there's that saga with the Cochrane review and Cochrane's investigation of the BMJ EBM critique. So I started this post to keep the threads together.

Cochrane's response to the critique was published on 3 September (UK time). As my post predicted, there was very little efficacy data - only 5 eligible trials, with only 1 including the primary outcome (CIN2+). Incorporating the data wouldn't change the main conclusion. I added a coda to my original post.

Previously on 3 September, I tweeted to the BMJ EBM, noting serious error in what they published and asking if they would correct, and on the 4th, to its editor-in-chief. (There was no reply.)

I also submitted a comment on the Cochrane review. (In case that's behind a paywall, a copy is below this post.)

I'll also be discussing and analyzing this more as this situation evolves. But in the meantime, a quick summary of its escalation since I posted this (on 4 September):

  • On 5 September, the Nordic Cochrane Centre tweeted that the Cochrane response "does not address our most serious concerns", and they would be responding.
  • On 12 September, the editors of BMJ EBM, publishers of the Copenhagen critique, published  a response.
  • On 15 September, a link to a statement circulated on Twitter - it's an open letter dated 14 September from Peter Gøtzsche on Nordic Cochrane Centre letterhead. Gøtzsche is one of the authors of what I've been calling the critique. He reports that he was expelled from the Cochrane Collaboration and that he and 4 others resigned from the organization's board in protest. It's a declaration of civil war, weaving a wide range of internal Cochrane political issues and controversies into a particular narrative, and calling for "dissolution of the present board, new elections and a broad-based participatory debate about the future strategy and governance of the organization".
  • A statement from Cochrane's co-chairs dated 15 September was posted on the organization's website. It specifies what changes occurred to the board, and reported that the situation with Gøtzsche is "ongoing".
  • The 4 board members who resigned issued a statement of their own on 15 September too.

That's a lot to unpack, and we'll get to it. Cochrane's annual international conference (called the Colloquium) starts in Edinburgh on 16 September - and that's no coincidence. There's a lead-up to the Colloquium of Cochrane sector and governance meetings, and it is a traditional time for factional disputes and lobbying, with the occasional major schism. The organization is a registered charity - here's its constitution/articles of association.

At the AGM on 17 September local time, the Cochrane board issued this detailed statement. And around the same time, BMJ EBM published the Copenhagen group's response to the Cochrane editors' response. Here's my take on developments:

Boilover: The Cochrane HPV Vaccine Fire Isn't Really About the Evidence - but it's Critical to Science

I'm not going to try to keep on top of all the posting and documents, but I will post some.

  • On 19 September, one of the Nordic Cochrane Centre co-authors of the Copenhagen critique posted his take.
  • The Cochrane board's statement had included a statement criticizing the statement made by the resigning board members. On the 18th, the Canadian Cochrane Center director (who was not a board member), Holger Schünemann, called on Twitter for them to delete that criticism. In my blog post on the 18th, I had singled this out as an example of the type of common situation addressed by Sandman in his book on crises and outrage management: it was an unnecessary defensive and provocative action, and taking Schünemann's advice was an opportunity to demonstrate responsiveness. On 19 September, the Cochrane Collaboration removed the sentence.
  • On 20 September Europe/America time (I saw it on the 21st in Australian time), someone messaged me that the link in my blog post to the resigning board member's statement was broken. I have asked if the statement has been withdrawn. If it has been as a similarly conciliatory gesture, that would be heartening.
  • On 20 September, the BMJ published a report on Gøtzsche saying the board's process was "flawed" and came after a "show trial". They published selected excerpts Gøtzsche gave them of the lawyer's report to the board, and discussed events from some board meetings. I re-read the Cochrane Board's statement (original and revised) and the resigning board's statement carefully. I can see nothing in them that is contradicted by Gøtzsche's statement. For example, he cites a part with advice about historical incidents. The board statement, however, referred to the lawyer's finding in the current situation, where Gøtzsche made a serious allegation about a senior member of Cochrane's staff.

Gøtzsche was still both a member of the Cochrane Collaboration and its board at the time he publicly released his statement and gave this material and these anecdotes to the BMJ for publication. Whether or not you think he damaged the Cochrane Collaboration's reputation before all this, I think few would disagree that he is doing that now. 

Hilda Bastian

Originally posted on 4 September 2018. Last updated 21 September 2018.

(For background: I was "first generation Cochrane Collaboration", part of the group Iain Chalmers assembled to found an international organization out of the roots of the Cochrane Centre he had inspired into life recently in Oxford. I served on the board of the international organization from its founding in 1993 to 2001. I'm no longer a member of the organization. A disclosure of interests at the foot of each of my posts on this matter at Absolutely Maybe.)


3 updates on 16 September 2018 (the first in Australian time, 15 September European time)

17 September 2018: Deleted a paragraph on 2 comments awaiting moderation at the Absolutely Maybe blog, after I released the comments and replied to them.

2 updates on 19 September 2018 (Australian time): Added link to new blog post, then a link to Tom Jefferson's post.

On 21 September, I added bullet points for events from 19 to September.



 Text of my comment at Cochrane:

Support for the rigor of this review; concern about flaws and bias in recent critiques

This review has attracted a critical comment here by Riva and colleagues [1], and another in BMJ Evidence-Based Medicine by Jørgensen and colleagues [2]. Neither establish that there are significant flaws in the Cochrane HPV vaccine review by Arbyn and colleagues; both critiques have fundamental flaws, and especially in the case of Jørgensen et al, significant bias. I have posted a detailed critique of the Jørgensen piece at my blog [3].

The Riva comment calls on Cochrane to change the review’s title by deleting reference to cervical cancer, because the trials were designed to detect changes in precursors, but not cervical cancer: “No RCTs ever assessed the impact of HPV vaccines on cervical cancer incidence and its associated mortality”.

Firstly, Cochrane reviews should address key questions, whether or not there are trials that can provide data on them. Secondly, cervical cancer was an outcome in trials, even though they were not powered, or initially long enough, to detect a difference. And thirdly, long-term follow-up studies for trials may indeed provide useful data on cervical cancer. For example, such a study is perhaps the first to find a statistically significant difference in HPV-related cancer [4].

Riva go on to say that any conclusion based on cancer lesions is “purely speculative”. Moderate to severe cervical lesions are a well-founded surrogate for cervical cancer. For example, a modeling study in the UK found that the incidence of invasive cervical cancer after CIN2/CIN3 was up to 15.4% [5].

Observational studies in vaccinated communities are showing similar reductions to that in the Cochrane review – see for example the systematic review by Garland and colleagues [6] – and for countries with extensive national vaccination and screening programs, such as Australia and the UK, data on whether the drop in severe lesions in successive cohorts are followed by concomitant drops in cancer and associated mortality will be available in the next few years (see my second related post [7]).

Indeed, despite concerns expressed by Riva and others that the cervical cancer rate might not budge because other HPV strains cause further disease, in some ways the reverse might be happening: the drop may even be higher because of herd immunity in communities with high vaccination levels, particularly as vaccination is no longer limited to girls and vaccines against a broader group of HPV strains are now available.

Both the Riva and Jørgensen comments take issue with the Arbyn et al’s assessment of risk of bias. Those judgments are very subjective, and disagreement on this score isn’t really surprising. What is striking to me, though, is the amount of bias about the HPV vaccine evident in the comments about, as Riva for example say, “the supposed benefits of these vaccines”. Both groups criticize the review because of the conflicts of interests of the authors of the first version of this review’s protocol – despite the review reporting that Cochrane’s action in this regard, including a new protocol after the exit of conflicted reviewers. A bigger issue to me appears to be the intellectual and ideological conflicts of these critics.


[1] Riva C, Tinari S, Spinosa J-P (2018). Concerns over methodological flaws in the assessment of vaccine efficacy. Feedback section for Arbyn et al (2018) doi: doi/10.1002/14651858.CD009069.pub3 https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD009069.pub3/detailed-comment/en?messageId=154255807

[2] Jørgensen L, Gøtzsche PC, Jefferson T (2018). The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias. BMJ Evidence-Based Medicine Published Online First: 27 July 2018. doi:10.1136/bmjebm-2018-111012 https://ebm.bmj.com/content/early/2018/07/27/bmjebm-2018-111012

[3] Bastian H (2018). The HPV vaccine: a critique of a critique of a meta-analysis. Absolutely Maybe at PLOS Blogs: https://blogs.plos.org/absolutely-maybe/2018/08/25/the-hpv-vaccine-a-critique-of-a-critique-of-a-meta-analysis/

[4] Luostarinen T et al (2018). Vaccination protects against invasive HPV-associated cancers. International Journal of Cancer 142(10): 2186-2187. https://onlinelibrary.wiley.com/doi/abs/10.1002/ijc.31231 Registered at ClinicalTrials.gov in 2011, NCT01393470.

[5] Raffle AE et al (2003). Outcomes of screening to prevent cancer: analysis of cumulative incidence of cervical abnormalities and modelling of cases and deaths prevented. BMJ 326:901 https://www.bmj.com/content/326/7395/901.short

[6] Garland SM et al (2016). Impact and effectiveness of the quadrivalent Human Papillomavirus vaccine: a systematic review of 10 years of real-world experience. Clinical Infectious Diseases 63(4): 519-527. https://academic.oup.com/cid/article/63/4/519/2566619

[7] Bastian H (2018). The HPV vaccine should be preventing cervical cancer: can we tell whether it actually is? Absolutely Maybe at PLOS Blogs https://blogs.plos.org/absolutely-maybe/2018/08/31/the-hpv-vaccine-should-be-preventing-cervical-cancer-can-we-tell-whether-it-actually-is/