Israeli study suggests vaccine efficacy vs. infection decreases over time -- should we be worried?

A couple weeks ago, reports in Israel came out suggesting that Pfizer vaccine efficacy waned over time, sparking discussion of the potential need for a 3rd dose booster to be given. Without details about the data and analysis, however, it was difficult to evaluate these reports. At the end of last week, the Israeli researchers released a medRXiv paper detailing these results. Here I will assess the strength of evidence and limitations of their analysis, and discuss what I think it means for our understanding of vaccine efficacy.

Summary of key points:

• This study looked at risk of breakthrough infections (after full vaccination) in a large cohort comprised with 25% of the Israeli population, checking whether the risk increased with time since vaccination suggesting waning of immune protection.

• The modeling was rigorously done to adjust for several key confounding factors including time, age, pre-existing conditions, which is important given that those vaccinated earlier tended to be older, have more pre-existing conditions, and more opportunity for viral exposure.

• Their results found substantial increase in probability of breakthrough infections for those vaccinated earlier, but did not result in a complete loss of immune protection.

• While adjusting for some factors, they did not adjust for others that might be equally important, including potential for different exposure by profession or different propensity to be tested, which are crucial factors given Israel's use of asymptomatic testing via contact tracing.

• The results are difficult to calibrate with other studies, since they did not assess protection vs. infection relative to unvaccinated individuals in this study, and did not look at vaccine protection vs. symptomatic infection, hospitalization or death as reported in other studies.

• Overall, this is a solid study suggesting vaccine protection vs. infection might wane to some degree over time, but its limitations suggest validation is necessary before changing vaccination policies, such as proposing a third shot as a booster.

Overview of study

This study was facilitated by the extensive population level medical records data in Israel. The analysis involved all vaccinated members of a Maccabi Healthcare Services (MHS), the second largest HMO with over 2.5 million members comprising 25% of all Israeli residents that represent a representative sample of the >9 million population of the country. This analysis focused on people who were fully vaccinated, receiving their second dose sometime between January and April, resulting in 1,352,444 individuals considered in this study, of which 475,281 (35.1%) were vaccinated in January, 460,500 (34.1%) in February, 371,929 (27.5%) in March, and 44,734 (3.3%) in April.

Their goal was to compare the rate of breakthrough infections, which are PCR confirmed SARS-CoV-2 cases recorded in their medical records system in people who were fully vaccinated (>14 days after second dose), based on time since vaccination to assess whether vaccine efficacy appeared to wane over time.

Of course, people vaccinated in January have had much more time for breakthrough infections than people vaccinated in April, so to make the comparison more fair they focused their analysis on breakthrough infections during a fixed time frame, between June 1 and July 27, 2021, and omitted anyone who had a breakthrough infection before June 1. This ensured that their results were not driven by time confounding.

Note that based on Israeli testing practices, this analysis was not limited to symptomatic infections, but also included asymptomatic infections. Since Israel practiced rigorous contact tracing, testing those with known contact with someone who was infected, these data include substantial numbers of asymptomatic infections. Keep this in mind as this is an implicit factor that must be considered when evaluating and interpreting these results.

Breakthrough infection rate higher for those vaccinated earlier

A total of 4220 breakthrough infections occurred during this time frame (0.312%, or 31.2 per 10,000 vaccinated), with breakthrough infection rates increasing based on time since vaccination with those vaccinated in April (17.0 per 10k) having lower rates than those vaccinated in March (23.1 per 10k), February (33.6 per 10k), or January (36.5 per 10k). The following figure from the paper presents these data:

Does this mean vaccine efficacy is waning? Not necessarily. Because this is not a randomized, controlled trial, there could be other factors explaining these results besides waning of vaccine efficacy. In particular, any differences in the characteristics of those vaccinated earlier vs. later are called confounding factors that offer potential alternative explanations, and could make it appear that vaccine efficacy was waning even if that was not actually the case.

In particular, we know that in the early months vaccination was prioritized based on age, pre-existing conditions, and exposure probability. That is, only those who were older, had pre-existing conditions making them at higher risk for poor outcomes if infected, or had high risk of viral exposure (e.g. health care workers, HCW) were offered vaccination early, and younger people without pre-existing conditions who were not HCW had to wait until later to be vaccinated. If older people or people with pre-existing conditions were less protected by vaccination, then it is possible that this, and not waning vaccine efficacy, could explain the higher infection risk in those vaccinated earlier.

Rigorous analysis adjusting for age and pre-existing conditions

Fortunately, they did not limit their analyses to the table above, but performed a rigorous analysis to adjust for these factors using matching and covariate adjustment.

For example, to get a more fair comparison of risk of infection between those in January and April vaccination cohorts, they did a 1-to-1 matching of individuals in these cohorts based on age group (18-39, 40-59, 60+), sex, city, and socioeconomic status (SES), and performed an analysis based on these matched sets. This matching factors out potential confounding from differential age group proportions over time since it ensures equal distributions of age group, sex, city and SES in each monthly vaccination cohort.

Also, they supplemented their model with covariates to adjust for potential risk factors obesity, cardiovascular conditions, diabetes, hypertension, chronic kidney disease, cancer, COPD, inflammatory bowel disease, and immunosuppression conditions. The inclusion of covariates also helps adjust for potential bias from the fact that individuals with these conditions were vaccinated sooner and might have higher probability of infection to make the comparisons more fair. Covariate adjustment is considered a legitimate but slightly less rigorous alternative to matching in adjusting for potential confounding factors. They did similar matching and adjustment for all other month to month comparisons.

Even after these adjustments, they found that those vaccinated earlier (January/February) had a 53% increased risk of breakthrough infection than those vaccinated later (March/April), and these differences were seen in all age groups. They also found that the cohort vaccinated in January had 2.25x higher risk (125% increase) of breakthrough infections than those vaccinated in April, and 1.65x higher risk (65% increase) than those vaccinated in March. All results were highly statistically significant, indicating there was strong evidence the increase was real and not spurious results from small sample sizes.

Are these adjustments enough for us to conclude that vaccine-based immune protection wanes over time?

This adjusted analysis showed that there is still evidence for reduction of immune protection vs. breakthrough infection in the cohorts vaccinated earlier vs. those vaccinated later even after reasonably accounting for differences in age and pre-existing condition status across the cohorts.

This provides reasonable evidence that vaccine protection vs. infection might wane over time, but their analysis may not have sufficiently adjusted for all the confounding factors between the vaccine cohorts. While they matched the cohorts by age, there still might be substantial differences in the age cohorts. Which young people were vaccinated in January and are these comparable to the young people vaccinated in April? And which older people were vaccinated in April, and how are these different from the older people vaccinated in January?

As mentioned above, the young people given early access to vaccination were those with vocations putting them at high risk of exposure to the virus, e.g. heath care workers (HCW), while the vaccines were made more broadly available to younger people in later months. As a result, it is expected that the January vaccinated younger individuals were HCW would have much higher exposure to the virus than April-vaccinated younger individuals who were not HCW, and this alone could cause a higher rate of detected breakthrough infections. Given that PCR testing was done based on contact tracing after exposure to another infected person, it also stands to reason that HCW would also be much more likely to be tested than other younger people, and thus be more likely to be detected as an asymptomatic breakthrough infection under the Israeli testing strategy in place. Thus, although matched by age, the younger people in the January cohort might tend to be more exposed to the virus and more likely to be tested than those in the later cohorts, which could contribute to bias masqueraded as waning immunity over time.

Also, as shown in the paper, 297,942/325,817 (91.4%) of the older (60+) individuals were vaccinated early (January/February), and all residents of this age were offered vaccination during this time period. While the age group matching adjusted for the systematic age differences over time, are the older people vaccinated in April comparable to those vaccinated in January? Those who are oldest and most frail were prioritized first, so it is likely that those 60+ people vaccinated later are younger and healthier than those vaccinated earlier, and thus might tend to have stronger immune systems and thus be better protected vs. breakthrough infection. Also, those older individuals who were vaccinated later must have refused the initial offer for vaccination. This might make them more likely to be people who are distrustful of the health care system or skeptical about the seriousness of the virus, which also might make them less likely to cooperate with contact tracing and thus less likely to submit to asymptomatic testing. These factors could also mean that, although matched by age, the older people in the April cohort might be less likely to be infected and less likely to be tested than older people in the January cohort, and also could comprise bias contributing to the impression of waning vaccine immunity over time.

Thus, it is not clear whether their matching and covariate adjustment are sufficient to remove all the key potential confounding factors, so it is still uncertain whether these results truly indicate that immune protection wanes substantially over time.

Would this reduction indicate the vaccines are no longer protective?

Suppose that these results are not driven by unadjusted confounding factors, and that indeed the level of immune protection has decreased over time since vaccination. Does this means the vaccines are no longer protective vs. infection after 6 months?

Given the small sample sizes of the April vaccinated cohort (especially for older individuals), let's consider the comparison of those vaccinated in March to January, for which the adjusted analysis found a 65% increase in risk of breakthrough infections.

How substantial is this 65% increase, and what does this indicate about the remaining level of protection vs. infection in those vaccinated in January? Unfortunately, this study did not compare these breakthrough infection rates with the corresponding infection rates from June 1 through July 27 among a matched unvaccinated cohort. This would have provided us estimates of vaccine efficacy as a function of time since vaccination, but they did not include this analysis in this paper, so we need to think in hypotheticals.

A Lancet paper reported 91.5% efficacy vs. asymptomatic disease in an analysis of infections in Israel between January 24 and April 3. If that was the efficacy for the March vaccinated cohort, then a 65% increase in breakthrough infections for those vaccinated in January would reduce this efficacy from 91.5 to 86% [{1 - (1 - 0.915) x 1.65} x 100 = 86%}. If you use the January to April reduction of 2.25-fold, the efficacy would reduce from 91.5% to 80.9%.

If the efficacy vs. asymptomatic disease in those vaccinated in March was 80%, then a 65% increase in breakthrough infections for those vaccinated in January would reduce this efficacy from 80% to 67% [{1 - (1 - .8) x 1.65} x 100=67%], and based on the more uncertain April numbers would reduce from 80% to 55%. These reductions would be concerning, but would hardly indicate a complete loss of immune protection after 6 months, and it is questionable whether it would suggest boosters are needed.

BTW, I recognize that there are press releases from Israel suggesting the Pfizer vaccine only has 64% vaccine efficacy vs. Delta for the period of June 6 to July 3, and 39% efficacy for the time period from June 20 to July 17. However, like many experts I am skeptical of these results, given their very low sample size and the fact that they haven't released the data or even a brief report describing how they arrived at these values. Given these results are outliers relative to the large, rigorous analyses of Delta variants in the UK and Canada that found 88% and 87% efficacy, respectively, vs. symptomatic disease for which the full data and analysis methods are published in available reports, I will hold off from putting much stock in them until I have a chance to see the Israeli data and evaluate their results.

Other limitations of this study

In addition to those previously mentioned, there are a number of other limitations of this study that need to be kept in mind when trying to interpret what its results mean for our understanding of vaccine protection vs. Delta. Here I list these limitations, some of which were mentioned and explained above:

• Age matching might not have been done at a fine enough level to distinguish between the oldest and most frail who were likely vaccinated the earliest and likely have the weakest immune systems and others >60yr old who are much younger and healthier and were vaccinated later.

• Their analysis did no adjustment for potential differential exposure by profession which could be a major issue given most young who were vaccinated early were health care workers or others at high risk of exposure to the virus.

• They also did not adjust for potential differential propensity to submit to asymptomatic testing, whereby people who waited until later to be vaccination by choice might also be less likely to cooperate with the contact tracing program and be asymptomatically tested, thus resulting in lower asymptomatic infection counts. Their use of a high level of asymptomatic testing makes their results strongly dependent on different testing practices among different groups of people that have the potential to bias the results.

• Since nearly all cases in June and July in Israel were Delta variant, these results might not reflect what would be experienced in other variants.

• They did not look at symptomatic disease, hospitalizations or death in this study. This makes it harder to compare with other studies that use these endpoints and that did not do asymptomatic testing.

• The number of breakthrough infections in Israel was relatively low for the time period studied, 4220, which limits the precision of estimation, especially for comparisons with the April-vaccinated cohort that only comprises just over 3% of those vaccinated.

Conclusions

Overall, this is a solid study with a remarkably data source provided by the well-coordinated Israeli electronic medical record system enabling the study to encompass >25% of Israeli residents and using rigorous modeling approaches to adjust for the key confounders including time, age, SES, and pre-existing conditions. The study provides reasonable evidence that immune protection might wane over time, but there are enough questions about unadjusted confounders, especially differential exposure and testing rates among those vaccinated early vs. late, that might remain and be strong drivers of the outcome. My own opinion is that the data are strong enough to suggest some waning of protection, but the magnitude might be less than what is observed in this study because of these confounding factors. Also, it might be that the loss of protection is limited to asymptomatic infection and not noticeable in the more clinically important symptomatic infection, hospitalization and death rates.

Even if the results suggested by this study hold, they do not indicate the vaccines no longer work, but simply that their efficacy is somewhat attenuated. A more careful characterization of the breakthrough infection risks relative to unvaccinated infection rates is needed to assess to what degree protection vs. infection wanes, and also to look at the analogous measures for symptomatic disease, hospitalization, and death. If the breakthrough infections are largely asymptomatic infections that rarely lead to severe disease and possibly transmit to others substantially less, then it might not make sense to change the distribution policies on the basis of these results. There are numerous studies measuring immune markers over time suggesting that the vaccine-induced immune protection is quite durable. For example, this study published in New England Journal of Medicine in April measured levels of spike-binding antibodies, neutralizing antibodies, and live virus neutralization in healthy adults for many months after vaccination and found these levels to be strongly retained at 6 months past vaccination, with levels still 500-5000x higher than pre-vaccination. Another paper being written by researchers at University of Pennsylvania and soon to be submitted shows strong and durable targeted memory B-cells, helper T-cells, and killer T-cells, markers more indicative of long term immune protection than antibody levels, through at least 6 months vaccination. There are numerous others, and they have consistently found durable immune markers post-vaccination. Many of these studies were pre-Delta, so we need to see to what degree these results hold for Delta variant infections, but it is hard to imagine immune protection waning so much in this time frame to render the vaccines anywhere near non-protective.

There are other considerations in assessing whether boosters should be used or not. The pandemic is an international problem and given many countries in the world are having trouble securing enough vaccine for substantial vaccination of their populations, the benefit that a third shot might provide to boost immune protection somewhat in a country that already has high levels of vaccination needs to be weighted against the benefit of helping other countries obtain primary vaccination, that could reduce international spread and development of new variants that could eventually affect Israel.

It is clear we still have a lot to learn, and need continuing analyses and assessment using rich data like these to further characterize the durability and strength of immune protection provided by vaccines and previous infection, especially for emerging variants like Delta, to intelligently inform the policymaking.