Refuting Peter Doshi's claims doubting "trustworthiness & meaningfulness" of COVID vaccine results

Peter Doshi has written a piece in the opinion section for the British Medical Journal (BMJ), for which he is an associate editor, questioning the "trustworthiness and meaningfulness" of the published study results for the Pfizer/BioNTech vaccine. In it, he implies a lack of transparency in the analyses and suggesting the vaccine efficacy would be more like 19%-29% based on something he argues may be a "more meaningful clinical endpoint" than the primary endpoint corresponding to the 95% efficacy estimate reported in the FDA report and New England Journal of Medicine paper. His report implies that the pharmaceutical companies are misrepresenting the results in a way that exaggerates their efficacy and safety, and that this misrepresentation has not been caught by the FDA or NEJM peer review.

His opinion piece is the most highly read at BMJ, a highly respected medical journal, and is also being highly cited by anti-vaccine groups all over the internet. If his claims are accurate, then this could be a great service to science to raise legitimate questions and correct the public's perception of the demonstrated efficacy of the Pfizer and Moderna vaccines. But if his claims are not accurate, this piece does a great disservice to the scientific community and world as we struggle to get control of this pandemic.

The Pfizer/BioNTech phase 3 study appears to be well-designed, being a double-blind randomized placebo-controlled trial with clearly pre-specified dose, schedule, and endpoint, and its results appear to be better than anyone dared hope, with the 95% efficacy far exceed the 50% threshold for FDA approval and an adverse event profile that appeared positive, with transient fever, fatigue, headache, and arm soreness among the few that differ from placebo. While there are questions remaining about durability of immunity, degree of protection against asymptomatic disease or transmission, and long term safety endpoints, these results surpassed expectations and appeared extremely promising. Is it possible that these results are not what they appear? Let's investigate Doshi's claims.

FDA report reveals that not all COVID-19 cases counted in primary efficacy analysis?

The crux of his argument is based on a section in the FDA report entitled "Suspected COVID-19 Cases" that he portrays as evidence that the vast majority of true COVID-19 cases were omitted from the primary analysis to inflate efficacy estimates, and that if all true COVID-19 cases were included the efficacy would be much lower. He presents this as some sort of "gotcha" moment, but looking at the documents, it is clear he is either misunderstanding or misrepresenting the trial and its results.

First, let's see this section in the FDA report upon which he builds his claims, which consists of two paragraphs at the top of page 42. The first paragraph starts with the sentence "As specified in the protocol, suspected cases of symptomatic COVID-19 that were not PCR- confirmed were not recorded as adverse events unless they met regulatory criteria for seriousness," and then goes into the details of the two suspected cases in the trial that met the regulatory criteria for seriousness.

The second paragraph goes on to discuss the total number of "suspected cases of symptomatic COVID-19", and since it is the springboard for Doshi's argument, here I will copy the entire paragraph:

"Among 3410 total cases of suspected but unconfirmed COVID-19 in the overall study population, 1594 occurred in the vaccine group vs. 1816 in the placebo group. Suspected COVID-19 cases that occurred within 7 days after any vaccination were 409 in the vaccine group vs. 287 in the placebo group. It is possible that the imbalance in suspected COVID-19 cases occurring in the 7 days postvaccination represents vaccine reactogenicity with symptoms that overlap with those of COVID-19. Overall though, these data do not raise a concern that protocol-specified reporting of suspected, but unconfirmed COVID-19 cases could have masked clinically significant adverse events that would not have otherwise been detected."

Doshi argues that these 3410 cases of "suspected but unconfirmed COVID-19" should have been included in the primary efficacy analysis, suggesting a better choice for the efficacy end point may have been a comparison of the total confirmed + suspected COVID-19 cases between the vaccine and placebo arms, which would have led to much lower efficacy estimates.

The 95% efficacy measure came from the fact that of all the confirmed symptomatic COVID-19 cases in the study, 162 were in the placebo arm and just 8 in the vaccine arm, suggesting (162-8)/162 = 95% of the cases that would have occurred sans vaccine were prevented by the vaccine. He argues that these "suspected but unconfirmed" cases should also be included, which would have 1594+8=1602 cases in the vaccine group and 1816+162=1978 in the placebo arm, which would result in an efficacy of (1978-1602)/1978=19% efficacy, far below the FDA bar, or if leaving out the 409 in the vaccine group and 287 in placebo group that were within 1 week of the shot and might be side effects of the vaccine, then it would be 1602-409=1193 cases in vaccine arm and 1978-287=1691 cases in the placebo arm, for an efficacy of (1691-1193)/1691=29% efficacy, still below the FDA bar.

He puts these forward as more valid efficacy measures and then mentions "Pfizer’s 92-page report didn’t mention the 3410 “suspected covid-19” cases. Nor did its publication in the New England Journal of Medicine. Nor did any of the reports on Moderna’s vaccine", implying these numbers were purposefully left out of all reports perhaps with malicious intent, presenting this as a "gotcha" result and implying the exceptionally strong 95% efficacy results for Pfizer (and Moderna) are a mirage created by suppressing legitimate COVID-19 cases counts.

Who are these "suspected but unconfirmed COVID-19 cases," and why should they be included in efficacy analyses?

If you look through the protocol, which was posted in its entirety months ago, you see no mention of a category of COVID-19 cases that are "suspected but unconfirmed." The only time "suspected" is used in the protocol is in the context of determining when a "potential COVID-illness visit" should be scheduled to give a SARS-CoV-2 PCR test based on reported symptoms. Since it was deemed infeasible to perform weekly PCR COVID-19 tests for all >30k subjects to identify asymptomatic disease, this trial (as well as the Moderna one) focused on demonstrating efficacy vs. symptomatic disease. As described in Section 8.13, page 88/137 of the protocol, they did this by instructing subjects to immediately contact the site for a SARS-CoV-2 PCR test if they experienced any of the following symptoms anew that COULD indicate potential COVID-19: fever, cough, shortness of breath, chills, muscle pain, sore throat, loss of taste/smell, diarrhea, and vomiting. The protocol states that anyone reporting one of these symptoms (except within 7 days of receiving shot) is a suspected COVID-19 patient who should immediately arrange a potential COVID-19 illness visit to receive a PCR test to determine if they are infected with the SARS-CoV-2 virus or not. Those who receive a positive PCR test are "confirmed COVID-19 cases" that are included in the primary efficacy analysis, and those whose PCR test is negative (or if done multiple times, repeatedly negative) are "suspected but unconfirmed COVID-19 cases."

Thus, it is clear that "suspected but unconfirmed COVID-19 cases" are all of those who reported one of those flu-like symptoms but whose SARS-CoV-2 PCR test came back negative.

The only reason why it might be appropriate to include these in the efficacy endpoint are if one believes that most or all of them are false negative test results, which Doshi implies is a possibility given his statement "If many or most of these suspected cases were in people who had a false negative PCR test result, this would dramatically decrease vaccine efficacy." However, it is completely implausible that many or most of these 3410 were COVID-19 cases with false negative SARS-CoV-2 PCR tests, given that for this to be the case it would have to be true that:

• 3410/~30,000 = ~11.4% of study participants were infected with symptomatic COVID-19 within ~2 months during a time in which the confirmed case rate was ~0.5%/month in USA.

• Since "suspected case" implies symptoms were reported, this would not even include asymptomatic cases, which are estimated to be 50-80% of all cases, suggesting that the true infection rate in the population of trial subjects would then be ~20-50%, which is clearly implausible.

• the false negative rate would have to be FAR beyond any documented or reasonably posited levels. The nominal false negative rate for the PCR test mentioned in the Pfizer protocol is 2% (see table 3 of FDA report for this PCR test), which would suggest that 3/3410 of these suspected cases were in fact false negative COVID-19 cases, and the updated efficacy would be (174-10)/174 x 100 =93.9% if we pessimistically assumed the vaccine efficacy was 0% for all false negative cases. Even if we assumed the FNR was 10x higher than reported, say 20%, this would mean we expected 43/3410 of these suspected cases to in fact be false negative COVID-19 cases, which would correspond to an updated efficacy of (183-29)/183 x 100 =84.2% if were pessimistically assumed the vaccine efficacy was 0% for all false negative cases. Even assuming a high 20% FNR, this would only result in ~1% of the 3410 being false negatives, obviously far short of "many or most".

• The high false negative rates that would have to be assumed to propose "many or most of the 3410" are false negatives would imply that essentially everyone in the USA has already been infected with SARS-CoV-2 given the >24 million confirmed cases in the USA for which the individual received a PCR test and a positive test result was obtained.

• To expect "many or most" of these to be false positive COVID-19 cases also implies that there were very few cases of common cold, seasonal allergies, or food-based nausea or diarrhea in this cohort of ~30,000 patients for the study duration that may have produced any of those common flu-like symptoms outside of COVID-19.

It is obvious that to suggest that "many or most" of these are false negative COVID-19 cases is patently absurd.

He seems to acknowledge the possibility that at least some of these might not be false-negative COVID-19 patients, saying "some or many of the suspected covid-19 cases may be due to a different causative agent", yet still argues that they be considered in analyses because "Why should etiology matter?" The argument he makes on this basis is even more questionable, since it implies that the efficacy of a SARS-CoV-2 vaccine should be evaluated on its ability to prevent all flu-like viral illnesses and their symptoms, which is not a reasonable expectation.

Thus, the crux of Doshi's article claiming that these "suspected but unconfirmed cases" should have been included in the efficacy analysis is based on specious arguments and demonstrates a misunderstanding or misrepresentation of the details of the trial and their data.

He also raises some other concerns about 371 subjects left out for protocol violations, NSAIDs use, and speculates that the NSAIDs use or off-protocol serology antibody tests could have led to unblinding. These are minor concerns in his article relative to the purported suppression of "suspected but unconfirmed cases," and he does not detail how these standard practices would invalidate the strong efficacy results in the published paper and FDA reports, so I do not further address them here.

He finishes with a call for sharing the raw, patient level data for the trial, and chastises companies for not doing so yet. I can appreciate a call for transparency and data sharing, but he does not make it clear what exact data he is seeking, what analyses he is proposing, or what he expects such analyses to reveal. As all biomedical researchers understand, it is difficult and often impossible to share raw, patient-level data because of privacy laws and the possibility of ascertaining the identity of trial participants, and these trials have already been marked with unprecedented transparency with full confidential protocols publicly available since trial inception, the public posting of full FDA reports containing detailed summaries of all primary and secondary efficacy and safety endpoints even before the meeting, and the prompt publication of results in a leading peer-reviewed journal.

I am in favor of full transparency and open scientific critique, and appreciate genuine attempts to challenge weaknesses in clinical trial or regulatory procedures, or raise questions about important factors that may have been missed in published reports. However, it is important for someone, especially someone in a position of influence like associate editor of a major medical journal, to get their facts right before actively discrediting published studies, and especially when the net effect will obviously be sowing distrust for a vaccine during a global pandemic.

I find the substance of Doshi's criticisms to be sorely lacking, and think this opinion piece does far more harm than good in our societal efforts to understand the true efficacy and safety of these vaccines and utilize them as appropriate to combat the pandemic.

Peter Doshi is an Associate Professor of Pharmaceutical Health Policy Research at the University of Maryland School of Pharmacy. He earned an A.B. in anthropology from Brown University, an A.M. in East Asian Studies from Harvard University and Ph.D. in history, anthropology, and science, technology and society from MIT, and his research involves critical assessment of the pharmaceutical industry and FDA procedures, including arguing for greater transparency and data sharing. He is an outspoken critic of Influenza vaccines, and has written numerous commentaries throughout the COVID-19 pandemic critical of the vaccine study designs and results, and their emergency use authorization by the FDA.