This week the FDA gave emergency approval for the use of convalescent plasma to treat COVID-19, making the strategy broadly available to patients without having to enter clinical trials.
This is being touted by President Trump as a game-changer and major victory of the federal government given its investment in this strategy and aggressive efforts to harvest plasma from recovered COVID-19 patients. In his press conference, Trump claimed that there was evidence that this treatment "reduced mortality by 35%" based on a recent Mayo Clinic study.
However, this decision is controversial, with many saying there is not yet sufficient evidence of its efficacy to justify this approval, including Dr. Fauci and Dr. Francis Collins the director of the NIH.
What is convalescent plasma? This idea is simple: to take blood plasma from recovered COVID-19 patients whose immune systems have successfully fought off the virus, which have high levels of effective anti-SARS-CoV-2 antibodies, and inject them into the blood stream of current patients hoping that these antibodies will boost their anti-COVID-19 immune response and lead to better outcomes. The idea is also an old one: it was first studied in diptheria in 1892, and is the "oldest trick in the book" in fighting viruses.
This is a promising idea and a good one to try for COVID-19. The key question is whether there is enough scientific and empirical evidence to provide this new emergency approval. According to a website COVID trial explorer that tracks all ongoing COVID-19 clinical trials in the world, there are currently 118 convalescent plasma trials so we have potential to get a great deal of information about whether, when, and how this strategy might work and whether it is safe.
Out of these 118 trials, 8 are listed as completed. Unfortunately, most of these studies and insubstantial: there are two Chinese studies with 6 and 24 patients, respectively, an Indian study with 29 patients, an Iranian study with 45 patients, an Italian study with 49 patients. None of these trials have any comparison with other treatments of placebo except the Iranian study which is too small to be consequential. Based on these factors, there is very limited information that could be provided by them. A 6th study is German study that involves randomization to convalescent plasma or placebo -- this type of randomized study is promising as it can provide rigorous evidence of its potential benefit -- but it is based on only 174 patients and results are not public yet.
The remaining two studies are listed as expanded access trials which means they are clinical trials with the intent to make an unproven treatment available to patients at a given hospital or group of hospitals. These are beneficial in that they will rigorously track information from the patients in terms of clinical outcomes or adverse events, but their drawback is that they do not compare against any other standard or placebo so there is little to no frame of reference to assess whether the treatment is "working" or not or how it would perform relative to other alternative courses of treatment.
One of these is at Rutgers University study that is listed as completed, but there is not information about the number of patients or any of the results.
The other is a Mayo Clinic study that is listed as "completed" in July and for which they have published a preprint paper with all of their results on August 13th. This preprint is a completed scientific paper that has been posted online by the authors but has not yet been rigorously evaluated by scientific reviewers for publication in a medical journal.
Thus, indeed this Mayo Clinic study cited by the president is apparently the only clinical trial results upon which this decision is based. Thus, I have picked apart this paper to assess whether I am convinced that it contains sufficient evidence to justify this approval.
In short ..... um, not so much. The paper does not support the president's 35% reduction in mortality claim as it does not compare convalescent plasma vs. any alternative -- what he is likely referring to is a secondary adjusted analysis on a subset of the data showing that all together it appears that individuals who received convalescent plasma with high antibody levels had 35% reduction of risk of death within 7 days compared with those who received convalescent plasma with low antibody levels, but this analysis was barely statistically significant and as I show below the major results claimed in this paper are highly debatable based on the limitations of the study.
Here I will try to provide my detailed assessment of this study and make as generally accessible as possible, but feel free to skip ahead to the conclusions if you are in a hurry or otherwise don't want to get bogged down.
Detailed assessment:
This study includes 35,322 patients who received convalescent plasma while hospitalized for COVID-19, most of them with severe disease, with 52.3% in the ICU and 27.5% on ventilators at the time of treatment. It is not a comparative study, meaning that there is no comparison with a placebo or alternative treatment but instead all patients received convalescent plasma.
You might ask, "Given the lack of comparison with any other treatment or placebo, how can we assess from this study whether this strategy works or not?" Good question. You really can't, which is why this emergency authorization seemingly based on this study is so controversial.
The president's claim of 35% reduction in death risk implies such a comparison is done but that is not the case.
Even though it is impossible to make a determination of whether this strategy is efficacious based on this study, it is very large and well run by an outstanding internationally leading hospital and medical research environment. We can learn some things from a careful analysis of the data that might shed light on how well the treatment is working, for which patients, and under what conditions. It seems their strategy is to see whether early treatment does better than late treatment, and whether plasma transfusions with higher antibody levels do better than lower antibody levels, thinking these provide some sense that it might be working.
Thus the key variables they study are (1) timing of the plasma donation and (2) level of anti-SARS-CoV-2 antibodies in the convalescent plasma. The timing is variable, with some patients transfused the plasma at hospital entry (4% at day 0) and others long after admission (with 16% after 11+ days). For their main analysis, they summarized patients into two groups: early (0-3 day) and late (4+ days). Since the plasma each patient received is essentially from a different recovered COVID-19 patient, there is considerably variability in terms of the "antibody titers", or number of antibodies in the plasma they received. They split into 3 groups, with "low", "medium" or "high" antibody levels with the cutpoints defined so that the lowest are the bottom 20%, the highest the top 20% and the medium the middle 60%.
Given the lack of any comparisons with other treatments, their primary analyses focused on two comparisons:
Comparing outcomes for patients given early vs. late transfusions with the idea that if the treatment was working then patients given earlier transfusions should do better
Comparing outcomes for patients given transfusions with low vs. medium vs. high antibody levels, with the idea that the patients given transfusions with higher antibody levels should do better if the treatment was indeed working.
They considered two key clinical outcomes: 7 day and 30 day mortality -- the proportion of patients who died within 7 days after transfusion, and within 30 day after transfusion. I will summarize their results as well as "statistical significance" which is a measure used to assess whether there is enough evidence from the study to confidently conclude those results likely hold in the general population.
The primary results they present based on these outcomes are:
Patients with early transfusions had reduced 7 day death rates (8.7%) vs. those with late transfusions (11.9%), a >35% improvement, with results highly statistically significant.
Patients with early transfusions had reduced 30 day death rates (21.6% vs. 26.7%), with results highly statistically significant.
Patients receiving plasma with high antibody levels had lower 7 day death rates (8.9%) than those receiving plasma with medium (11.6%) or low (13.7%) antibody levels, again with results being barely statistically significant, with similar results for 30 day death rates.
In the grand scheme of things, even if these results hold true they provide at best very mild evidence that the treatment may be effective, and there are serious questions about these results.
The early/late transfusion effects seem to be very strong, but given that the decision to give transfusions early or late was not randomized, it is subject to bias if there are systematic differences between the patients given early or late transfusions. Unfortunately, the paper does not present a table that lists out the demographic and clinical characteristics of these two groups -- they should have done this to demonstrate they were comparable but didn't in spite of the fact that it was their primary comparison and they did present such tables for month of study and antibody levels -- so hmmmmm. But reading between the lines these tables provide sufficient evidence that the patient groups given early and late transfusions are surely not comparable.
Table 1 of the paper splits out all of the demographic and clinical characteristics based on month of the study, April, May and June, and it shows substantial differences across months with highly statistically significant differences for nearly every factor. The most notable difference is that April had much more severe patients at risk of death, with higher proportion with severe or life-threatening COVID-19 (80% vs. 61%), ICU care prior to infusion (66% vs. 44%), and mechanical ventilation prior to infusion (50% vs. 16%) in April than in June, and also significantly higher with respiratory failure (73% vs. 58%), multiple organ failure (19% vs. 7%), and septic shock (15% vs. 6%). As is the case throughout this pandemic in the USA, the April cases had much more severe disease and higher risk of death than June cases.
Unfortunately, the proportion of patients given early transfusions varied significantly across the months of the study, with April having a much smaller proportion of early transfusions than June (25% vs. 54%). Thus, the unadjusted comparison they presented showing reduced death rates for early vs. late transfusions could be completely driven by the fact that more of the late transfusions occurred in April when more severe patients were present.
To their credit, they did a secondary analysis that uses a statistical method to adjust for these alternative factors, and presented their results in Table 2. Unfortunately, those results show that after adjustment the differences between early and late transfusion are much smaller and clearly not statistically significant. They notably do not provide p-values (statistically significance levels) for this comparison or discuss the results but it can be seen in the figure that the differences are not statistically significant. Thus, in spite of the strong unadjusted results presented as the main finding of this paper, there is little evidence to confirm that early transfusion yields better results than late transfusion.
Regarding the antibody levels of the plasma donation, they did include a table (supplementary table 2) that shows how demographic and clinical factors vary across patients transfused with plasma with low, medium and high antibody levels. These 3 groups were comparable for nearly every factor, so although they didn't randomize to different antibody levels, comparisons of death rates across antibody levels may be relatively unbiased and is certainly reasonable to consider.
That is good and means we can potentially learn a lot from this study about how antibody levels in the donated plasma affect the patient's outcome. However, the results were not very strong in the primary analysis with low levels of statistical significance so there are questions about whether those results would replicate in another study and whether they hold for the general population.
Another question about the antibody level results is raised by a subgroup analysis they performed (Figure 3) that looked at the effects of high vs. low antibody levels in various groups of patients -- defined based on whether they received mechanical ventilation or not, had whether they had no, limited, or many severe risk factors, and whether their transfusion was early or late.
Below this figure is plotted, which presents the relative risk of death high vs. low antibody levels, with <1 indicating patients transfused with high antibody levels had lower risk of death than those transfused with lower antibody levels, >1 indicating patients transfused with high antibody levels had higher risk of death than those transfused with lower antibody levels, and 1 indicating no difference. The orange dot is the estimate and the blue lines indicate 95% uncertainty bands.
The "Common" results at the bottom of the chart show an overall relative risk of 0.65, indicating that across all patients, transfusions with higher antibody levels led to a 35% reduction in the risk of death. This is promising and could provide evidence of potential efficacy of the treatment, since one might expect a placebo to do worse than or similar to the low antibody levels. This is likely the result to which the president was referring and is in some sense legitimate.
However, there are still questions about this result. First of all, the statistical significance is marginal so it is not clear whether this result holds in the general population. Also, if you look above at the results for different subgroups defined by early/late transfusion and severity group, the results are not consistent across subgroups which you would expect them to be if convalescent plasma was clearly working. For ventilated patients, it appeared that the early transfused patients did much better on high vs. low antibodies, but late transfused patients actually did slightly worse on high vs. low antibodies. The opposite is true for unventilated patients with many severe risk factors, with late transfused patients doing much better on high vs. low antibodies but with early transfused patients doing slightly worse on high vs. low antibodies. This is difficult to interpret and raises questions.
There might be some insights in there about when to best use convalescent plasma for ventilated or non-ventilated patients from this, but again the confounding of early vs. late transfusion with month and thus severity of disease makes it difficult to unravel, and it is difficult to be confident in an inference that the convalescent plasma works on the basis of these results.
Incidentally, note that while it is not apparent in the abstract or press release, all high vs. medium vs. low antibody comparisons are done on a subset of 3082 transfused patients who received only a single unit of plasma between 150ml-250ml -- a small subset of the total 35,322 in the study, with the others having received multiple units of plasma and/or smaller or larger volumes of plasma.
Conclusions:
It is unclear what firm conclusions if any can be drawn from this study. The biggest flaw is that there is no comparison with a placebo or other treatment, so there is no frame of reference to assess whether the 7 day or 30 day survival numbers indicate effectiveness or not. They tried to suggest that they demonstrated early plasma transfusion leads to better results, but my points above draw that into serious question, and also that there was a dose effect with higher levels of antibody titers leading to better outcomes than lower titers, but those results are barely statistically signfiicant and appear inconsistent across subgroups.
What seems clear to me is that this study provides little evidence to justify emergency approval of the treatment strategy. It seems safe enough, but there is little evidence of efficacy. Indeed, the authors never conclude any such thing, with their primary conclusions in the paper being "this information may be informative for treatment of COVID-19 and design of randomized clinical trials involving convalescent plasma." They suggest they think this might be useful information in designing a randomized clinical trial that would assess whether it is indeed better than the standard of care, but that they acknowledge that the study was not designed to assess efficacy so could not draw that conclusion. Certainly this study does not support the statements made publicly by FDA director Hahn suggesting that the 35% relative risk reduction means that "35 out of 100 people would have been saved if given convalescent plasma."
Indeed, there are numerous randomized clinical trials currently accruing, including one with 1500 patients, one with 1200 patients, one with 500 patients, and another one with 500 patients. These studies will provide much better evidence of the efficacy of this approach, given the randomization vs. a comparison arm.
So what is the harm in the emergency approval? It is true that safety has been well established by this study, and it may be a long time until the results from the randomized studies are available. However, if by giving the emergency approval, that could create misimpression that efficacy has been demonstrated more than it in fact has been and cause a treatment shift towards this strategy and away from others, which may be dangerous if the convalescent plasma does not end up as efficacious once the full data are in. It could also lead to a reduction in clinical trial enrollment since the emergency access mechanism is easier, compromising our ability in the future to learn whether it really works as well as expected. Also, the treatment is already available now through clinical trials, and large expanded access trials are designed for precisely purposes like this -- to make the treatment available to many patients while carefully tracking their data to detect any safety issues and get a sense about efficacy. Even though these expanded access trials like the Mayo Clinic trial have their limitations, they still pull together all of the data for analysis, while expanded emergency use may not coordinate data collection to learn from the data as efficiently.
As I highlighted in previous discussions about wartime science here and here, we are in a unique time during this pandemic when we need to decide what to do to treat patients now, and cannot wait for answers on the usual medical research timelines. We must pull together all available information now to make the best possible decision as far as we can tell for current patients, but also continue with the rigorous studies that will provide us clear answers. Premature emergency approval before efficacy has been demonstrated may short circuit that process by arbitrarily prioritizing some treatments over others, and lead to potentially ineffective treatments receiving undue attention and greater mortality and morbidity.
We need to keep political decisions out of our evaluation of medical treatment and testing procedures to be sure that we are clearly discerning all we can as the data accrue.
Comments