Kirsch, et al. have published a preprint providing commentary on a JAMA paper that compared death rates after hospitalization for COVID-19 or influenza, respectively, from a large Veteran's Administration database.
The paper used propensity score weighting to balance confounders between the influenza and COVID-19 cohorts, and found that COVID-19 hospitalization led to higher death risk than Influenza hospitalization in the core of the 2022-23 influenza season (October 2022 through January 2023). Further, they found this death risk in the COVID-19 cohort was highest in the unvaccinated and lowest in the boosted individuals.
The study does not focus on or look at vaccine effectiveness vs. hospitalization, and does not even look at risk of hospitalization but conditions on a cohort of individuals, all of which have been hospitalized. Additionally, note that the propensity score weighting is done to balance the COVID-19 and influenza hospitalized cohort, not to balance vaccinated and unvaccinated, so again is not designed to assess vaccine effectiveness. It would be possible to use the large VA database to estimate vaccine effectiveness of SARS-CoV-2 vaccines and influenza vaccines against COVID-19 and influenza hospitalization, respectively, but this would require a completely different design and analytical approach.
Kirsch et al.'s claims and argument
However, in spite of this, the Kirsch et al. preprint claims that they can determine from this paper that neither the SARS-CoV-2 vaccines nor the Influenza vaccines had any effectiveness whatsoever based on Table 1 of the JAMA paper:
Their objection is based on the percent of COVID-19 and Influenza hospitalized in the various COVID-19 and Influenza vaccinated subgroups, thinking that if there was any vaccine effectiveness vs. hospitalization at all for the two vaccines that:
there should be lower % vaccinated if there was any vaccine effectiveness
the percentages for influenza and COVID-19 hospitalized should be very different.
They extract this part out of the table and present in their preprint:
The crux of their argument is that the fact that these two columns in the table are close to one another implies neither vaccine had any effectiveness vs. infection. From this table they assert:
Had the vaccines been even minimally effective, significant differences in vaccination
percentages would be expected.
They then proceed to compute what they expect this table would look like under assumptions of vaccine effectiveness (VE) of 90% for COVID-19 vaccines vs. COVID-19 hospitalization and VE=50% for Influenza vaccination vs. Influenza hospitalization, presenting the following table:
Noting that:
The actual VA data from the JAMA paper had a much higher percentage of COVID-19 hospitalized patients who were COVID-19 vaccinated (81.11% vs. 30.04% in their expected table)
The actual VA data had a much higher percentage of Influenza hospitalized patients who were Influenza vaccinated (61.88% vs. 46.89% in their expected table)
The actual VA data had similar percent COVID-19 vaccinated and percent Influenza vaccinated in both the COVID-19 hospitalized and Influenza hosptalized groups, while their expected table had them vastly difference.
they conclude:
We are not aware of any viable alternative explanation for the observed data other than that
neither vaccine provided any protection against hospitalization
They are extremely bold and confident in these conclusions:
In conclusion, our assessment indicates that neither the influenza vaccine nor the
COVID-19 vaccines provided any measurable difference in risk reduction of
hospitalization for the very diseases they were designed to protect against.
They suggest that their paper introduces a much better way to assess vaccine effectiveness than the current design and analyses approaches that they complain "rely on complex models" Since we observed minimal differences in vaccination percentages, this approach may offer a
simpler and more expeditious way to assess vaccine efficacy of two or more vaccines
without having to rely on complex models
They go further and suggest the vaccines be pulled from the market all over the world (and suggesting the "method" they introduce in this paper be an essential part of the regulatory approval process) We propose that health authorities worldwide should reconsider the decision to approve
both of these vaccines until such time as a real public health benefit can be demonstrated
from the data using the method described in this paper, along with other complementary
analytic methods. These findings reinforce recommendations to halt the global
distribution of both the influenza vaccine and the COVID-19 vaccines
Very bold claims to make on the basis of the simple fact that they cannot understand how the vaccination percentages in Table 1 in the JAMA article could occur if the vaccines had nonzero effectiveness.
Basic evaluation of the claims
First of all, fundamentally, it is not possible to estimate or make any direct inferences about vaccine effectiveness vs. hospitalized infection from a simple table summarizing percent vaccinated for COVID-19 or Influenza in COVID-19 or Influenza hospitalized cohorts.
It may be interesting to think about what a table like Table 1 in the JAMA paper would look like under various assumed states of reality, but if one is to undertake this exercise, it is important for them to be aware of all of the assumptions they are making, and for it to have any value or validity whatsoever, needs to reflect the key basic realities of the situation.
In their preprint, they do not provide any details about how they arrive at this table based on their 90%/50% VE assumptions, but from personal correspondence with the lead author, I have learned that
They assumed independence between COVID-19 and Influenza vaccination, assuming that someone getting a COVID-19 vaccine was not more likely to get an Influenza vaccine than someone not getting a COVID-19 vaccine, and vice versa. It is extremely unlikely that these are independent, as propensity to get one vaccine increases the likelihood of receiving the other
They did not consider age confounding the age dynamics of the population, including the age distribution of the VA cohort, the stark differences across age groups in terms of both (1) propensity to get COVID-19 and/or Influenza vaccines and boosters and (2) risk of hospitalization if infected. They treated the entire population as a homogeneous group with identical hospitalization risks and vaccination rates, so ignores the massive age confounding present in this data set.
There may be other assumptions they made, as well, but again are not clear because they are not described in the preprint.
First, if their preprint is to be considered a legitimate scientific publication, it needs to provide the specific details of how they get such an "expected" table, including all assumptions and calculations, especially given it is the entire crux of their argument that there is no vaccine effectiveness that is the primary goal of their paper.
However, it is clear that some of assumptions they made are grossly oversimplified and inaccurate, enough so as to make their "expected" table they compute irrelevant for the situation at hand.
In terms of independence of COVID-19 and influenza vaccination, the literature (and common sense) are very clear that those receiving influenza vaccines are more likely to have received a COVID-19 vaccine, and vice versa. For example, this study found those receiving influenza vaccines were 5.18x more likely to receive COVID-19 vaccines. This fact alone would make the percent influenza and COVID-19 vaccinated in the influenza and COVID-19 hospitalization cohorts more similar than random chance, providing some explanation for why these were much more similar in Table 1 of the JAMA paper than the simple expected Table 2 of the preprint.
Second, and even more fundamentally, the age confounding that is by far the dominant feature of this data set, and is completely ignored in the authors' Table 2 of expected vaccination percentages. This confounding plays a major role in the percent vaccinated in the hospitalized cohorts and the structure of Table 1 of the JAMA paper, and as I will show is sufficient to produce much of the structure evident in that table.
Outline
In the remainder of this article, I will compute an "expected" table of % COVID-19 and influenza vaccinated for the VA cohort in the 2022-23 flu season based on the actual VA population numbers and age distribution, and the actual age group specific rates of influenza and COVID-19 vaccination and boosting, and the actual age group specific rates of influenza and COVID-19 hospitalization in the USA under the assumption of COVID-19 vaccines have VE=60% and influenza vaccines have VE=20% vs. hospitalization.
Even though my calculation of this expected table assumes independence of influenza and COVID-19 vaccination, only roughly accounts for age confounding, and does not accounting for any other important confounders in the population including sex and comorbidities, I will show that under these assumptions one can obtain a table with features similar to Table 1 of the JAMA article, and the authors of the preprint have absolutely no basis for claiming zero vaccine effectiveness on the basis of this table.
I provide full transparency on all my assumptions, sources, and calculations.
Basic Assumptions for Calculations
I based my calculations on the actual age group specific population, vaccination, and hospitalization numbers as documented from various resources -- and provide a hyperlink from the source in each case.
I started with the actual VA population numbers, specifically taking the veteran population on 9/30/2022 for 5 year age groups and aggregating into 3 age groups: 18-49yr, 50-64yr, and 65yr+ for further consideration.
VA population
Fall 2022
18-49yr: 5,079,613
50-64yr: 4,970,230
65yr+: 8,542,614
Next, for the same age groups, I found the data on proportion of each of these age groups receiving a flu vaccine in fall 2022:
Influenza
Unvaccinated Vaccinated
18-49yr: 65% 35%
50-64yr: 50% 50%
65yr+: 30% 70%
and the proportion of each age group unvaccinated, receiving 1-2 doses, or receiving 3+ doses:
COVID-19
Unvaccinated 1-2 Doses 3+ Doses
18-49yr: 20% 62% 18%
50-64yr: 12% 45% 43%
65yr+: 6% 24% 70%
For the same age groups, I found data on the hospitalization rate (per 100k) for each age group for Influenza and for COVID-19 for the 2022-2023 flu season (10/1/2022-9/30/2023):
Hospitalization Rate (per 100k)
Influenza COVID-19
18-49yr: 44.7 89.8
50-64yr: 105.9 226.2
65yr+: 332.4 994.3
For vaccine effectiveness, for the sake of this simulation I assumed:
Vaccine Effectiveness vs. Hospitalization
Influenza COVID-19
20% 60%
Notes on vaccine effectivenss assumptions:
The preprint authors used VE=90% for COVID-19. This would be a reasonable value for 2021 after initial rollouts, but not for end of 2022 and beginning of 2023 with Omicron and with many vaccinated many months or years beforehand. None of the literature from late 2022 and early 2023 has VE vs. hospitalization anywhere near 90%.
For the sake of this simulation, we use VE=60% and have it the same for 1-2 or 3+ doses. One could propose different VE for 1-2, 3, and 4+ doses and see how results change if they'd like
The preprint authors used VE=50% for Influenza. This is close to the estimates in the literature for Influenza A (best matching the vaccine for 2022-23) that ranged from 30%-70%, but considering other influenza clades not matching the vaccine as well we assume a VE=20% here (and again, one could propose different VE and see how results change)
I assume constant relative VE across all age groups.
Calculation of Percent Vaccinated Table
On the basis of these numbers, assuming homogeneity within age groups, independence of COVID-19 and influenza vaccination status within age groups, and no additional confounders, I followed the following steps to estimate/simulate the number hospitalized for COVID-19 and Influenza in the 2022-2023 flu season:
From the assumed vaccination rates, I computed the actual number of the VA population for each of the 3 age groups (18-49, 50-64, 65+) in fall 2022 in each COVID-19 vaccination group (unvaccinated, 1-2 doses, 3+ doses) and influenza vaccination group (vaccinated or unvaccinated in fall 2022)
From the assumed VE, I computed the hospitalization rates for Influenza and for COVID-19 for each vaccination/age group, assuming the baseline hospitalization rates mentioned above, denoted by h, are for unvaccinated, and assuming the hospitalization rate is h x (1-VE/100) for the vaccinated.
I then multiply the age/vaccine-group specific VA population numbers by age/vaccine-group specific hospitalization rates for influenza and for COVID-19 to get number hospitalized for influenza and for COVID-19 for each age/vaccine group.
Taking the cohort of COVID-19 hospitalized, I compute % unvaccinated, 1-2 doses, and 3+ doses for COVID, and % unvaccinated or vaccinated for influenza.
Taking the cohort of influenza hospitalized, I compute % unvaccinated, 1-2 doses, and 3+ doses for COVID, and % unvaccinated or vaccinated for influenza.
Combining #4-#5 together gives me an "expected" table similar to table 2 of the preprint.
Results
Based on those assumptions, following are the results I obtain for the COVID-19 hospitalized alongside the Table 1 from the JAMA paper and the Table 2 from Kirsch preprint:
Notes:
Looking at the COVID-19 hospitalization results, our results assuming a straight VE=60% yields a table similar to Table 1 in the JAMA table.
We see that in spite of the fact that VE=60% by simulation, 80% of the COVID-19 hospitalized were vaccinated. This is a result of the age confounding, and a phenomenon seen during the entire pandemic. At first glance it seems impossible to have 80% of hospitalized be vaccinated under a scenario of 60% VE, but as can be seen by these calculations the age dynamics when taken into account make a higher proportion of hospitalized vaccinated than one would think.
The percent of COVID-19 hospitalized who were flu vaccinated was close to what was seen in the JAMA paper, and recall we did not account for the known high concordance of influenza and COVID-19 vaccination nor did we account for other confounders like sex and comorbidities.
The Kirsch simulation that ignores the age dynamics and assuming VE=90% obtains results wildly different from those of our simulation that are based on more reasonable VE and the age dynamics of the population.
Based on those assumptions, following are the results I obtain for the Influenze hospitalized alongside the Table 1 from the JAMA paper and the Table 2 from Kirsch preprint:
Notes:
First, note that my simulation found that the percent receiving flu vaccine (61.4%) close to the JAMA paper (61.9%). At first glance it seems impossible that 61.4% of the influenza hospitalized were influenza vaccinated when 55% of the total VA population was influenza hospitalized under an assumption of VE=20%, but this is what we obtain given the age distributions of the VA population, and is a function of the known age confounding.
Kirsch's simulation that did not account for the population age distribution or age confounding found a much lower % influenza vaccinated (46.9%) than the JAMA paper our our simulation (61.9% and 61.4%).
My simulation had a higher % boosted (65.0% vs. 54.9%) and lower % unvaccinated (7.2% vs. 18.9%) in the influenza hospitalized cohort. This suggests those who were COVID-19 vaccinated had lower influenza hospitalization rates than expected based on the assumptions of the simulation.
Conclusions
So, we see that by simply accounting for the basic age dynamics of the VA population and accounting for some of the age confounding (by stratifying hospitalization rates and vaccination rates across 18-49yr, 50-64yr, and 65yr+ age groups), a simple simulation obtained results similar to the Table 1 of the JAMA paper under a scenario of nonzero vaccine effectiveness assumptions (60% for COVID-19 and 20% for influenza vaccine). There is no basis for the argument that the structure of Table 1 in the JAMA paper impies the COVID-19 and influenza vaccines have zero effectiveness vs. cause specific hospitalization, and certainly not sufficient support for the notion that the approval for the vaccines should be reversed and vaccines removed from the market.
The assertion that the preprint introduces a new method for estimating vaccine effectiveness that is superior to existing methods for estimating vaccine effectivenss (based on "complex models") is also completely unsubstantiated. It is not even clear what their "method" is as they do not delineate any methodology.
The simulation I conducted, although simplistic, gave results much more similar to the real world data from the JAMA study than the simulation conducted by Kirsch et al. given it used actual figures from the 2022-23 flu season, and at least partially adjusted for the age confounding by computing these quantities within broad age groups 18-49yr, 50-64yr, and 65yr+. The age confounding has major effects on the vaccination proportions in hospitalized cohorts, and so must be taken into account in any simulation.
In spite of these advantages and the fact that our simulation was able to obtain results similar to the Table 1 of the JAMA paper, it also makes many simplifying (and false assumptions), including:
Independence of COVID-19 and influenza vaccination, when we know that they are highliy dependent (e.g. influenza vaccinated >5x more likely to be COVID-19 vaccinated than influenza unvaccinated).
Constant hospitalization rates and vaccination rates within age groups, when we know that there is still major variability across the ages within each age group. For example, an 18 year old is much less likely to be COVID-19 or influenza hospitalized and much less likely to be vaccinated than a 49 year old. Thus, my simulation only partially adjusts for age confounding, and the remaining confounding can strongly affect the results.
No other confounding, when we know that sex and comorbidity status and other factors are major confounders affecting both risk of hospitalization and vaccination status, as well as other residual confounding contributing to healthy vaccinee effect (especially immediately after vaccination), and these will also impact the types of tables computed.
Because these limitations exist no matter how rigorously one tries to simulate, one cannot make firm causal inference about vaccine effects by simply conducting such a simulation. It may be interesting to shed light on various aspects of the data, but is not a primary tool for estimating vaccine effectiveness.
Because of this, I would NEVER claim that COVID-19 vaccines have VE=60% or influenza vaccines have VE=20% just because this simulation gave reasonably similar results to the actual table from the VA data. This was simply chosen as an illustration to prove the point that one can obtain tables like seen in the JAMA paper even with substantial vaccine effectiveness. The best way we have to estimate vaccine effectiveness vs. rare events like COVID-19 or Influenza hospitalization from currently available data is observational studies, which must account for key known confounders as well as possible, and assess potential residual confounding and robustness to modeling assumptions through transparently described design and analysis techniques and rigorous sensitivity analyses.
This could be done with the VA cohort from the JAMA paper using appropriate design and analysis techniques, but it is ill advised to try to infer vaccine effectiveness from a study designed for a completely different purpose without the required data to directly estimate vaccine effects on hospitalization.