Watch Israel if you want to see early validation of efficacy and safety of SARS-CoV-2 vaccines

The phase 3 studies for Moderna and Pfizer/BioNTech SARS-CoV-2 vaccines yielded outstanding results -->94% efficacy in preventing symptomatic infections and promising safety data with the key differences between placebo and vaccine groups involving transient reactogenic side effects such as soreness and fever. These positive results have led to emergency approval and widespread dispersement of these vaccines all over the world, with over 101 million vaccine doses distributed across 64 countries, and over 26 million Americans already vaccinated as of February 1st.

Unanswered questions after phase 3 studies and need for post-approval monitoring

While extremely promising, there are numerous unanswered questions remaining after the phase 3 studies.

• Is the >94% efficacy in preventing symptomatic infections observed in the study realized when distributed at the full population level?

• How well does the vaccine prevent asymptomatic infections?

• How well does it prevent transmission to others?

• How well does it prevent severe disease leading to hospitalizations, ICU stays, and deaths?

• How much protection is afforded by the first dose of the vaccine during the time period while waiting for a second dose? This may be important since logistical and supply challenges may delay the time period between first and second dose for many.

• How long does the efficacy from vaccination last, and how does this compare with the natural immunity experienced after recovery from severe, moderate, mild or asymptomatic disease?

• How well does the vaccine work with newly emerging dominant variants including the B.1.1.7 from the UK, the B.1.135 from South Africa, and the P.1 variant from Brazil, given these are characterized by mutations to some regions of the spike protein that are targeted by the vaccines.

While the phase 3 trials and other laboratory studies provide some clues into the these questions, they yield no definitive answer to them.

In terms of safety, while the phase 3 data are promising, they don't provide complete information given the limited scope of the studies in terms of number of vaccinated (~35,000 between the two vaccines), the study population (not as many very old and frail or immunocompromised individuals or individuals with autoimmune disease), and the length of time (follow up is still limited to months and so may not have information on longer term complications). The studies aren't well powered to detect serious adverse events with prevalence <1/10,000 or so, or serious adverse events that occur in specific subsets of individuals not well represented in the trials.

As a result, continued safety monitoring of the vaccines is crucial. In the USA, the CDC's VAERs is a passive reporting system that can quickly identify patterns and specific adverse events after vaccination. This system was able to quickly identify rare but potentially serious anaphylactic reactions to the vaccine, perhaps produced by the polyethylene glycol (PEG2000) used to protect in the mRNA in the vaccine, which led to adapted procedures to ensure all vaccinated individuals are monitored for 15 minutes with a epi pen nearby just in case. However, as I highlighted in a previous blog post, VAERs has serious limitations. Given that anyone can upload to the system, the data are unverified and there is no assessment for which reported adverse events the vaccine is causative or contributing factor, this system is not sufficient to establish vaccine-induced adverse events. This is explicitly acknowledged on the VAERs site, and other active monitoring systems are required to establish causation with the vaccines and better characterize their prevalence, including the Vaccine Safety Datalink, the Clinical Immunization Safety Assessment, and large national databases of electronic health records and insurance claims data, such as the FDA's Sentinel and BEST systems that include data from >100 million Americans. However, these systems are not public so its work and results are only known by their reports, and the public has no way to know how exactly these systems are working.

It is clear that continued monitoring of the efficacy and safety of the vaccine is crucial to provide answer to these unresolved questions to build public confidence in the vaccines, overcoming vaccine hesitancy, as well as to identify and acknowledge any safety or efficacy limitations that exist so that they can be taken into account in vaccine distribution strategies and eventually overcome.

What we can learn from Israel

Israel provides the world with a unique opportunity to provide validation of the trials' efficacy and safety results and begin to answer some of the important unanswered questions.

Their government has organized and is implementing the fastest universal vaccination effort in the world by far. They have already fully vaccinated >21% of their population, >70% of the over 60 population, and given at least first dose to >36%, and they appear to have the supplies and systems in place to achieve full vaccination of all Israeli citizens this spring. All Israeli citizens are required to register with a recognized health care provider, and their robust electronic medical record system will provide detailed data for all participants. The Israeli government signed a deal with Pfizer that obtained early access to sufficient doses for population-wide vaccination in exchange for access to these full data.

Although there are political/ethical controversies related to their dispersement strategy, the fact that results are for a specific vaccine (the Pfizer/BioNTech vaccine), and the potential privacy issues in sharing data on this scale, these data will be exceptionally useful in validating the phase 3 results and starting to address some of these other questions.

The Israeli data are constantly updated on the ministry's data dashboard. These data will provide detailed information about post-vaccine adverse events, efficacy against symptomatic and asymptomatic infections, reduction in hospitalizations and deaths, and also to evaluate efficacy in the interim period between first and second doses.

As the vaccination proceeds, it is possible to construct case-control studies, with cases being vaccinated individuals and controls being non-vaccinated individuals matched for demographic variables. While not as rigorous as a placebo-controlled randomized studies, these types of studies can overcome the limitations of observational systems including the VAERs for which there is no reference point for adverse events, infections, hospitalizations or deaths based on the unvaccinated counterfactual against which to calibrate the results and draw rigorous conclusions about vaccine safety and efficacy.

One of the major health systems in Israel, Maccabi Health systems, reported early results suggesting the reported efficacy of the Pfizer vaccine was being realized in the Israeli population, at least in the early weeks after full vaccination.

• Using data through 1/15/21, they wrote a MedRXiv paper presenting an analysis comparing PCR-based SARS-CoV-2 infection rates from days 13-24 first dose vs. days 1-12 after first dose. Motivated by the Pfizer phase 3 results that the vaccine conferred no benefit vs. placebo, they considered days 1-12 as the control against which they computed single dose efficacy for days 13-24, and found a 51.4% relative risk reduction from 2484/435,789 (0.57%) to 614/227,407 (0.27%). From this, they suggested it was important to give the second dose to reach full protection.

• On 1/25/21, they reported just 20/128,600 (0.016%) of those fully vaccinated reported infections in the first week, whereas across all of Israel, the weekly infection rate was 0.65%, suggesting strong efficacy.

• The national health minister released country-wide numbers , and found 317/715,425 (0.04%) reported infections in the first week after full vaccination.

• Acknowledging the entire Israeli population was a poor control group, researchers at the Maccabi system subsequently constructed a control unvaccinated group matched for age and health status. Their subsequent analysis found only 31/163,000 (0.019%) in their system to test positive within the first 10 days of full protection, while a matched control group had 11x the rate in the same period of time, suggesting 92% efficacy. Although certain details are not clear (e.g. how testing was done -- only after reporting symptoms or more systematically) and these results are in the early period post-vaccination, the analysis is more rigorous, and provides some sense of validation of the high efficacy of the Pfizer vaccine in preventing symptomatic infections.

One recent article posted on Medium presents results from a very basic analysis based on pulling data from the Israel ministry dashboards. They presented two types of analyses, one compared the rate of COVID-19 cases, hospitalizations and deaths between vaccinated and unvaccinated/not fully vaccinated cohorts, and the second compared hospitalization and COVID-19 death rates between cities with early and late vaccination.

Here I will highlight their conclusions:

• Among the 668,100 people 60 and over receiving both doses, 473 (0.07%) were diagnosed positive, 38 hospitalized, 11 in serious condition, 3 in critical condition, and two dead. The cases represents an 83% decrease in diagnosed cases when compared with those not yet fully vaccinated, and significantly lower .

• Among the 325,301 people under 60 receiving both doses, 231 (0..07%) were diagnosed positive, one in serious condition, two in critical condition and zero deaths, while the non-fully vaccinated groups showed a 0.57% positive case rate, showing an 88% reduction in cases.

• They discussed an analysis done by researchers at Weizmann Institute and Tel Aviv University comparing cities with early vaccination (at least 85% of age 60 vaccinated by 1/10/21) and late vaccination (<70% of age 60 vaccinated by 1/10/21) and see a marked reduction in weekly hospitalizations in early vaccinated cities compared with late vaccinated cities, as seen the figure below..

These studies did not have the type of rigorous matching among fully vaccinated, unvaccinated, or partially vaccinated (single dose) that I would propose for the best analysis, nor to they look at all of the various types of efficacy and safety outcomes I hint at above that could provide stronger validation and further illumination of how the vaccines are working. In principle, besides validating efficacy for detecting symptomatic infections, they could shed light on efficacy of single dose, efficacy in preventing hospitalizations, ICU stays, or deaths, and could provide a comparative safety analysis with unvaccinated groups on millions instead of tens of thousands that would be better powered to detect serious adverse events that might be caused by vaccination, or if revealing no such differences provide greater confidence on vaccine safety. If using contact tracing to perform secondary transmission studies, comparing the infection rate among contacts of vaccinated and unvaccinated people, these studies could provide solid information about the degree to which the vaccines are suppressing transmission, a crucial unanswered question whose affirmative answer is critical to the prospect of vaccine-induced herd immunity protection.

These few studies are just the start -- we should see more results coming from analyses of these rich, complete Israeli data in the coming weeks -- hopefully they are done to the highest level of rigor and care to yield the best information shedding insights into the important unanswered questions we have about the safety and efficacy of the vaccines. I hope steps will be taken to make the data publicly accessible to researchers who can use state-of-the-art methods to get reliable information about the important unanswered questions.