Some serious questions about the AstraZeneca vaccine trial results.

(Note: I have updated this post on 11/24 based on additional relevant trial information, but the conclusion from the original blog post remain: there are numerous questions raised about this report that need to be addressed before I am confident in these results).

AstraZeneca just released interim results from the placebo-controlled, randomized trial of the vaccine they co-developed with scientists at Oxford University. Here is the AP story on the press release and some commentary from Nature, and the Oxford University press release.

Many of the reports emphasize how this vaccine, based on inserting genetic material for the spike protein into a viral vector from a chimpanzee adenovirus, is cheaper and easier to distribute than the two mRNA-based vaccines, with the AstraZeneca priced at $2.50/dose as opposed to the BioNTech/Pfizer at $20/dose or Moderna at $15-$20/dose, and the AstraZeneca Vaccine not requiring the extreme cold temperatures for transport and storage that the mRNA vaccines require. These are two important advantages of the AstraZeneca vaccine for feasible distribution, especially to developing nations, which is clearly a primary focus based on the AZ press release emphasizing WHO emergency use approval and distribution to developing nations is a priority.

However, before this vaccine is widely distributed all around the world, its efficacy and safety need to be confirmed. The press release suggests the vaccine has been shown to be efficacious, but there are numerous questions about these results and details about the underlying trials that need to be clarified before we can understand whether we can indeed be confident in its efficacy.

The press release states that full data will be presented in a peer reviewed publication, but until that time all we have are summary statistics of efficacy and number of participants. The results state that the overall efficacy of the virus was 70%, but that the efficacy was 90% if the regimen was given as a "half dose" followed by a "full dose" after the month, and it was 62% if the "full dose" was given both time points. It was reported that the study involves a total of 11,363 participants, 2741 in the part of the study including the reduced "half dose" and 8622 in the part of the study including the "full dose".

From these, I have reverse engineered what the study results might be based on an assumption that each part has both vaccine and control patients at equal randomization, and provide the corresponding 95% confidence intervals on efficacy based on these assumptions to give us an idea of the precision of the results.

Group: Participants Total Cases Vaccine Placebo Efficacy/95% CI

"Full dose" 8622 99 27 72 62% (41%, 77%)

"Half dose" 2741 32 3 29 90% (67%, 98%)

Total 11,363 131 30 101 70% (55%, 81%)

So we see that the half-dose looks very promising, with efficacy of 90% getting close to what was found for the mRNA vaccines, but given the relatively small sample size, there is considerable uncertainty with 95% confidence interval spanning from 67% to 98%,

But looking at the full dose, the efficacy was only 62% and the 95% confidence interval only spans from 41% to 77%. This 62% efficacy is above the USA FDA's required 50% threshold, but is much lower than what was seen in the mRNA vaccine studies and for the half-dose in this study. They also report a combined efficacy of 70% aggregating information across both doses, but as I detail below, there are many reasons why it is likely not appropriate to simply aggregate this information together in this way.

This discrepancy in the results between lower and higher dose is puzzling and counterintuitive. Since these results were reported, scientists have posited that somehow the lower dose might better prime the immune system, avoid an immune reaction against the viral vector used to deliver the vaccine, or might better mimic true viral exposure. It is not clear whether these have any scientific substantiation or are just speculative propositions. However, drawing any conclusions about the superiority of the "half dose" is clearly premature based on the fact that these doses were from completely different studies and thus these differences might not be fully explained by dosage differences.

The AZ press release states that the full dose data is from the Brazilian study (COV003) involving 10,300 participants in Brazil. While the protocol is not publicly available, here is the summary of the protocol from on clinicaltrials.gov.

The AZ press release also states that the half dose data is from the UK study (COV002) involving 12,390 participants. Again, the protocol is not publicly available, but here is a summary of the protocol on clinicaltrials.gov.

Given that these two doses are from completely different studies, they are not strictly comparable, and it is questionable whether they should be aggregated into a common analysis. The fact that these studies were done on different continents makes them even more incomparable, with completely different study populations, cultures, settings, and even climate. Given these differences, it is possible that the difference in performance is not simply due to the different doses, but in other differences between the Brazilian (COV002) and UK (COV003) trials, either in the study populations or trial details.

Additionally, it is not clear that the case definitions match between the two trials. If you look at the clinicaltrials.gov summary of each of these trials, you'll see that both define the primary endpoint as symptomatic COVID-19 confirmed by PCR tests, suggesting that they both only did PCR tests if subjects reported symptoms, similar to the designs of the two mRNA vaccine trials. However, the AZ press release states that for the UK study, "In addition, weekly swabbing (and PCR tests) are done for detection of infection and assessment of vaccine efficacy against infection," while the clinicaltrials.gov description of the protocol doesn't mention any weekly swabbing or any PCR tests absent reported symptoms. Also, the Oxford press release for these results states that "Early indication that vaccine could reduce virus transmission from an observed reduction in asymptomatic infections." If the primary endpoint for the protocol based on clinicaltrials.gov was used, then there would be no way to measure asymptomatic infections, but this could be measured had weekly PCR tests been obtained for all participants. If these weekly swabs were indeed done, then the UK study would be using a completely different measure of case, including asymptomatic cases, and this would make the UK and Brazilian studies even more significantly non-comparable. This also raises questions about whether these weekly swabs were done off protocol, or are the summary details of the protocol on clinicaltrials.gov not accurate?

There are a couple of other key questions I have that raise some concerns in this study:

• Unlike the mRNA vaccine studies that were double-blinded meaning neither participants and clinicians knew who received vaccine or placebo, both the UK and Brazilian AZ studies were only single-blinded, meaning that the participant did not know whether they got vaccine or control (control was not placebo, but rather a meningitis vaccine), but the clinical staff did. One of the key benefits of double-blinding is that it ensures that clinical treatment is not differentially given to participants on one arm or the other either from conscious or unconscious bias, and the fact that the clinical staff knew who was vaccinated means that we cannot be 100% sure that this knowledge did not affect their decisions or measurements.

• The UK study that reportedly used the "half dose" actually was supposed to use the same "full dose" as the Brazilian study. The protocol apparently intended to use the same "full dose" as the Brazilian study but according to this report in Reuters, the half dose was mistakenly given at the first time point, and rather than adjust they just chose to keep that half dose going throughout the trial. This is described as serendipitous, and maybe it was, but it raises questions about the results given the protocol details were not followed, and at a minimum this type of deviation from protocol will raise considerable regulatory concerns.

• While the press release makes it clear that the 2741 participants who received the half dose followed by the full dose came from the UK study, it is not clear on whether these were all of the participants in the study or only a subset. This trial, which commenced in May, was to have a total of 12,390 participants, would likely have accrued far more than 2741 by now. Is it possible that this was only a subset of the trial, and that others were on the trial but given a different dosing regimen? The AZ press release is vague on the dosage, stating that “one or two intramuscular doses of a half dose (~2.5 x10^10 viral particles) or full dose (~5x10^10 viral particles) of AZD1222”. Does this imply that some of the trial participants got one dose and others two, and that there are some that got two-half-doses and some two-full-doses and some one half and one full? The description on clinicaltrials.gov describes numerous different study groups, mostly different ages, but mentions some groups getting single dose and others two doses, and some of the groups varied in terms of what the first and second dosages are. This is clearly not simply a randomized trial with a single dosing regimen and control, but something more complex. So that raises the question of whether the 2741 is only a subset, how did the other subsets/dosages do, and why did they not report those results? I am sure this will be explained in the peer reviewed publication (it will have to be), but this incompleteness and vagueness makes me wonder what we really know about efficacy based on these reported results.

In summary, I really hope that this AstraZeneca Vaccine can be proven safe and effective, and be approved as soon as possible for deployment around the world. Certainly, the low cost and easy of transport and storage make it much more viable for international use, especially in developing nations. However, there are lots of unanswered questions about these trials, far more than for the mRNA vaccine trials by Moderna and BioNTech/Pfizer that are relatively simple, transparently described, and for which the full protocols are available online. I am hopeful that many of these questions will be addressed by the peer reviewed publication, but many of these questions make me uneasy with the results to say the least.

It seems as if their conclusion is that the "half-dose" is more efficacious, and that AZ are trying to revise the USA protocol and other protocols to incorporate this does. I welcome this change, as certainly more data is required before I am comfortable with the conclusion that the half-dose is indeed more efficacious than the full dose, given that the two doses were from two completely different studies in different continents and different seasons, and apparently one of them counting asymptomatic cases and the other counting only symptomatic cases. The protocol design needs to be transparently described, and any deviations from protocol explained and justified. And honestly, with all of these apparent protocol changes and irregularities, it is probably best to consider this study a nice Phase IIb study, and design a fresh Phase III study that prospectively indicates the half-dose followed by the full dose, and defines both the symptomatic case and asymptomatic case endpoints in a rigorous way and see if they can reproduce the results seen here.

I am still hopeful the AZ vaccine can be part of the solution, but also hope that these crucial questions are answered before this vaccine is approved for emergency use and widely distributed in developing nations all over the world.

BTW here is a commentary by Wired from 11/25 that raises many of these same concerns, and has additional detailed questions that I touch on in my last bullet point above — indicating they have numerous doses and subgroups in the UK trial and are only reporting a select subset while completely omitting the others. The more I see the more I distrust any of these results until I see detailed explanations, and even then likely will still be skeptical until I see full results from a clean trial.

BTW, once the published Lancet paper came out with more details about these results, I updated my impressions in another blog post, in which I reach the same conclusion albeit with even more questions raised by the additional data -- more data is needed before we can conclude that efficacy has been shown for the Oxford/AstraZeneca vaccine.