Are the vaccines really safe? What VAERs does and doesn't tell us about vaccine safety.

Many people have been genuinely excited about the positive results from the Moderna and Pfizer/Moderna phase 3 trials suggesting the vaccines were safe and effective and leading to their emergency approval by the FDA. To many, these results signal "the light at the end of the tunnel," that as we vaccinate enough of the population to herd immunity we can effectively bring this phase of the pandemic to a close and "get back to our regular lives." Incoming reports from Johnson and Johnson, Novavax, and AstraZeneca trials soon to come may lead to more authorized vaccines available for distribution.

Many others are hesitant to support vaccination, understandably concerned about

1. the unprecedented speed of the vaccine development process

2. the limited time of follow-up in the phase 3 trials means long-term safety is unknown

3. the limited size of the trials (35,000 vaccinated patients together) might not be sufficient to detect adverse events that are rare or high risk for specific subgroups not well represented in the phase 3 trials (e.g. immunocompromised).

Additionally, media reports of anecdotal cases of deaths or serious adverse events, along with many people's distrust of the pharmaceutical industry, feed these fears that the vaccines might somehow be more dangerous than the phase 3 studies suggest.

Thus, post-approval surveillance of vaccine safety is crucial. One public source of such data is

VAERS, the CDC's passive self-reporting system for post-vaccination adverse events. Individuals or health care providers can upload reports of post-vaccine adverse events into the VAERs system, which is publicly accessible for running queries for any vaccine, side effect, or subgroup. For example, if I look at Moderna and Pfizer/BioNTech SARS-CoV-2 vaccines and look at reported death events, here is what I find in the report:

This is pretty alarming! Even though this is out of >22.5 million Americans who have already received one dose, if the vaccine has "killed 287 people" then I would be very concerned. And since the VAERs system is a passive system to which individuals, medical professionals, and pharmaceutical companies voluntarily enter information, one of its limitations is that it clearly does not capture all of the serious adverse events that occur post vaccination. One study done by researchers at Harvard suggested that only 1% of adverse events are reported to VAERS (see results, page 6). Some might think then that to get a true sense of vaccine deaths, one should multiply by 100, which would be 28,700 deaths. If the vaccines caused 28,700 deaths out of 22.5 million vaccinated, that would CERTAINLY be cause for alarm and an indication that the vaccines are not safe. Is this a reasonable inference to make from VAERs data?

It is true that the system clearly does not capture all events that occurs post-vaccination. However, since the information is voluntarily entered and there is no verification process or medical evaluation, it is not possible to know whether the reported events are accurate or verified, and even if they are accurate, whether they are caused by vaccine or coincidental. This is another limitation of VAERS that is clearly disclosed on the VAERS website, and why it cannot itself provide reliable estimates of adverse event prevalence or a determination of the role of vaccines in the reported events. Here is the disclaimer clearly appearing on any VAERs report that is generated:

When unexplained serious adverse events or deaths occur within days after vaccination, it is natural for the public to infer that the vaccine was a causative or contributing factor, but this is not necessarily the case. As I tried to explain in a previous blog post, when you distribute anything to a population of tens to hundreds of millions, thousands of adverse events and even deaths will occur by random chance on the day or days following the distribution even if that thing was perfectly safe. For example, in a typical year, about 2.8 million Americans die, >7500 per day. If we gave a shot of placebo to every single American at a random time, we would expect on the order of 7500 to die on the day the placebo was given to them, and >50k within the week. Many of these would die without explanation, and had it been a vaccine many would have been reported to VAERS. Of course, deaths don't happen uniformly to all people and over time, but this toy illustration is just meant to characterize the order of magnitude of coincidental deaths expected. Thus, if trying to understand safety and risk of vaccination, it is not accurate to tally all of those events that occurred within a short time of vaccination, not even if we limit those to deaths that are difficult to explain. The same is true for any other adverse event.

This is why it is absurd to take a VAERS report like above with 287 reported deaths, and to infer that they are caused by the vaccine, and why it is beyond absurd to assume a 1% report rate and extrapolate that the vaccines have caused 28,700 deaths so far.

That is also why randomized trials are so important -- the phase 3 trials for Moderna/Pfizer collectively had 36,805 vaccine patients and 36,796 placebo patients. Based on the full safety data presented to the FDA for Moderna (table 25 on page 40) and Pfizer/BioNTech (table 14 on page 33). There were a total of 4 deaths in vaccine group, which if we extrapolate out to the USA population of 328 million, would suggest >36k deaths would occur after vaccination if we vaccinated the entire population! Thus, if this had not been a randomized trial, we would be clearly alarmed by these death rates and possibly conclude that the vaccines were killing more than 1 in 1000 vaccinated! However, even more individuals in the placebo group, 7, also died, which would extrapolate to >60k deaths if we gave placebo to the entire USA population -- clearly no one would claim that a saline injection to the arm of all Americans would cause these 60k deaths. The fact that there are more deaths in the placebo group than the vaccinated group in double-blind randomized trials demonstrates there is no evidence for any increased risk of death due to vaccination. Unlike the observational data of the VAERS, the randomized design makes causal inference possible.

Given that the rate of occurrence of serious adverse events in the randomized trial was not any greater in the vaccine group than placebo group, this provides strong evidence of safety, which is why the FDA gave emergency approval, and why if these data hold up at 6 months post-vaccination the FDA will grant full approval to these vaccines, possibly later this spring if the companies also submit their manufacturing validation reports and biologics license application.

These limitation of the VAERs makes it very difficult to interpret, and makes it a source of alarm, potentially unwarranted alarm. Even if the vaccine was as safe in the population as suggested by the trials, it is inevitable that the enormous number of coincident events would lead to a large number of VAERs reports. That is why it has to be taken with a grain of salt, and why many people who are freaking out watching the VAERs numbers increase each day are overreacting.

HOWEVER, we should NOT dismiss reports in VAERS. While the phase 3 trials suggest the vaccine does not cause an increase in deaths or serious adverse events relative to placebo, it is based on just 35k or so vaccinated people, so any serious adverse event with prevalence <1/15,000 or so is very unlikely to show up as differentially higher in vaccine than placebo group. That is, the phase 3 trials have VERY LOW statistical power for detecting rare, serious adverse events that might be caused by the vaccine. Furthermore, if there is a certain subgroup of individuals at higher risk for serious adverse events (e.g. immunocompromised individuals) that are not well represented in the phase 3 trial, then this presents another great unknown. This is the reason why the phase 3 trials are not SUFFICIENT to establish safety of the vaccines, and there is still considerable uncertainty about the level of safety of vaccines when distributed at the population level even after approval.

This is known, and is why the FDA requires, after emergency use authorization or even full approval, continued monitoring of vaccinated patients and assessment of adverse events.

This includes VAERS, which the FDA calls a "passive monitoring system", as well as other "active monitoring systems" including the Vaccine Safety Datalink, the Clinical Immunization Safety Assessment, and large national databases of electronic health records and insurance claims data, such as the FDA's Sentinel and BEST systems that include data from >100 million Americans.

The CISA and VSD systems have interdisciplinary groups that can better assess the health risks and evaluate connections between adverse events and vaccines.

The data from these active monitoring systems are reported directly from health care providers and hospitals, sometimes automatically via the electronic medical records and others actively reported after vaccination. Thus, they are verified events, with medical context to evaluate causation, and reported directly from the healthcare providers so not filtered by the pharmaceutical companies. They can provide a better assessment of the serious adverse events and determining whether they are likely or unlikely to be a causative or contributive factor.

This system has worked as intended, with early VAERs report and subsequent verification identifying the rare but potentially dangerous anaphylactic allergic reaction, seemingly to one of the buffer ingredients included to protect the mRNA, and led to quick adaptation of vaccination procedures to require 15 minute medical observation with an epi pen handy during vaccination just in case such a reaction occurs.

There are skeptics who think vaccines are inherently dangerous or strongly distrust the FDA and CDC, thinking they are "in the pockets of Pharma". These people might suggest that these active monitoring systems are not accurate because individuals within them are beholden to Pharma, systematically dismissing serious adverse events as "not vaccine related" if there is any level of uncertainty. That is, they distrust these systems.

Without inside knowledge of how these systems are being used, I can't know for sure that they are being used in a maximally effective way, the degree to which health care providers and health departments are reliably reporting potentially vaccine-related events, or that the data from these sources is carefully analyzed (the way I would do), making sure potential vaccine-caused adverse events are accurately attributed. As a biostatistician I am skeptical and always mindful of potential bias coming from political or commercial agendas, but I will also not jump to conclusions and assume out of cynicism that these active monitoring systems are not being run properly or somehow corrupt -- especially since many of them involve collaborations of many health care professionals, academic medical centers, and government agencies which contain many people whose objective for being involved is to "get it right."

Of one thing I am sure -- sole reliance on passive monitoring systems like VAERS to get an assessment of the degree and number of vaccine-caused adverse events will NOT lead to accurate assessments of vaccine safety and adverse events.

In conclusion, as stated on the website, VAERS is designed to rapidly detect unusual or unexpected patterns of adverse events, also known as "safety signals." It is not a reliable source in and of itself for determining vaccine-related adverse event rates, but it is a useful source to identify patterns and potentially vaccine-related adverse events missed by the clinical trials. Scientifically, it should be considered a source of hypothesis generation. These hypotheses must be verified by the active systems before inferring causation or prevalence, but it is important that this be done well to identify any potential serious adverse events for which the vaccines are a causative or contributing factor, and to characterize the subset of individuals susceptible to such events so they can be. protected.

With now >25 million in the USA already receiving their first vaccine dose, it is promising that there is still no clear evidence that the vaccines are causing or contributing to any serious, long term adverse events. In order to overcome vaccine hesitancy and reassure the public, the medical and regulatory community need to reassure the public that systems are in place to detect and investigate any unexpected serious adverse events that were not detected in the phase 3 studies, and the systems transparently and clearly described. And clearly, we need better education of the public in the purpose and limitations of the VAERs system so people do not misinterpret its results and incorrectly infer that the vaccines are causing dangerous adverse events and deaths that are not in fact due to vaccination.