Clinical trials stopped early for efficacy or benefit at risk of overestimating treatment effects and distorting risk-benefit assessments: some guiding principles for the SARS-CoV-2 COVID-19 vaccine

search: note, I wrote this paper & remitted to publish while I was at the Trump administration and actually sent it internally to FDA, CDC, Hahn, Fauci et al. warning them about the vaccine trials.

Clinical trials stopped early for efficacy or benefit at risk of overestimating treatment effects and distorting risk-benefit assessments: some guiding principles for the SARS-CoV-2 COVID-19 vaccine search

Paul Alexander¹ MSc, PhD

¹McMaster University, Evidence-Based-Medicine, Health Research Methods, Evidence and Impact (HEI), Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada

Keywords: COVID-19, vaccine, clinical trial, early stopping, interim analysis

What is new?

There is unprecedented need to deliver effective therapeutic and preventative agents in the context of the global COVID-19 pandemic. This commentary emphasizes the methodological consequences of stopping clinical trials early, specifically COVID-19 vaccine trials, and particularly from seemingly beneficial results during multiple, early interim analysis of study data. This commentary reviews the evidence to declare a needed event number of at least 500 if the vaccine trial is to be stopped early for benefit. (Bold and italics added by author.) In addition, it is critical that a prespecified stopping rule explicitly outlines all facets of the stopping rule that includes not too many interim looks at the data by the DSMB. As much sample size accrual is critical and most critically, is the at least 500 events (infections) combined in both trial arms. There must be a stricter stopping rule/boundary of p<0.001 if stopping early for benefit is considered.

Commentary

In the context of the quest to produce a vaccine for SARS-CoV-2 coronavirus disease under the present pandemic emergency, it is essential to understand the methodological ramifications in randomized clinical trials (RCTs) that are stopped early for efficacy or benefit (immunogenicity), such trials also known as truncated trials (T-RCTs). Our concern surrounds whether the estimates of effect reflect the true effect of the vaccine. Importantly, we reflect on the safety which can be considered a far more critical consideration than efficacy/effectiveness. We must ensure the latter in this COVID-19 vaccine search with no room for error in this regard.

As a case in point in this rapid race to a COVID-19 vaccine, the United States Operation Warp Speed (OWS) is coordinating ongoing clinical trials development research in tandem with regulatory, manufacturing, and distribution processes to avoid delays in disseminating vaccines to the public to prevent the spread of SARS-CoV-2 coronavirus and potential COVID-19 disease (1). The public-private partnership is innovative and thus far quite favourable in driving towards an efficacious/effective vaccine (s), and involves the synchronous trialing of various vaccine platforms with a multiplicity of vaccines to deliver vaccines. While a near majority of vaccine development is based in the United States, the race for a COVID-19 vaccine is global effort, with China, Asia, Australia, and Europe also centers for rapid vaccine development(2). This commentary guidance applies to all global parties involved in vaccine development and as such, in the process of spearheading vaccine clinical trials.

Applicable to global COVID-19 pandemic vaccine rapid trial approach, are potential risks and justifiable questions on vaccine safety and efficacy, with the demand to ensure that only a safe vaccine will come to market. The public, clinical, and scientific community must be assured that no aspects of a safe and effective vaccine development have been breached before approval of a fast-tracked vaccine. The public must always ‘only’ receive therapeutic interventions (vaccine) that were based on the highest quality, most robust and trustworthy evidence that is underpinned by high certainty, precise estimates of effect. (Bold and italics added by author.) This commentary serves as guidance but more as a warning to all COVID-19 vaccine developers, of what must be in place to derive confidence by researchers and the public.

One such area that deserves acute attention surrounds the issue of stopping a trial early for benefit that has been discussed in the media, versus continuation of recruiting trial participants. No doubt it is best not to stop early when there are initial benefits shown because there is a very high risk of making the decision to stop early based on incomplete and inaccurate trial information. Enough data is likely to have not yet emerged for the estimate of effect to be definitive or as we argue, even accurate. Should we stop the trial if the early benefits are very substantial? Can the observed results possibly be too good to be true? What happens if a trial is not stopped before early indications of efficacy, and then goes on to reveal no effect or even serious harm? Then had the trial stopped early for benefit, this could have been catastrophic for future patients. (Bold and italics added by author.) Limited adverse event safety data is a real cause for concern if a trail is stopped early, and researchers must carefully consider and balance this need, and plan to ensure this safety data is clear enough and collected longer-term even when a decision is made to truncate.

Researchers must also consider the substantial ethical elephant in the room of enrolling participants who have a random chance of being assigned to the placebo group when you have earlier indications of a potentially large treatment effect (3). What should be done? Does appropriate adherence to study methodology then exclude control groups from the possibly beneficial treatment? Should we deny the control group a potentially beneficial vaccine? Is it more important to focus on the safety of prospective patients and the larger society so that they do not make treatment decisions based on inaccurate or incomplete or even dangerous information, while there is clinical equipoise? These are the issues that the data safety monitoring board (DSMBs), also known as the data monitoring committee (DMC), who will be assessing vaccine trial data, must confront.

Some RCTs are stopped early when investigators conclude that the magnitude of effects of treatment are so large and not due to random error or chance that they must stop the trial early for benefit and administer the treatment, or vaccine. Our primary concern is that if this is based on an interim analysis of the data, that this could indeed be very misleading and drive inaccurate results. This is also very different to when researchers stop a RCT for futility (4) or harm (5). It is crucial to raise the cardinal issues that must be considered if a SARS-CoV-2 coronavirus vaccine is stopped early for benefit to ensure confidence by the public and the scientific community in the effectiveness and safety of the vaccine candidate.

The following principles are what DSMBs who monitor the safety and efficacy of ongoing RCTs, the trial expert leaders, and the national medication/vaccine safety regulators must consider in their decision making:   

1) T-RCTs stopped early for benefit are at serious risk of overestimating beneficial treatment effects and can be very misleading (5-9).  Some health care leaders such as Dr. Anthony Fauci (NIH/NIAID) and Dr. Robert Redfield (Director of the CDC) have gone on record to state that as little as 100 to 150 events (infections) would be needed to know if the vaccine is effective. (Bold and italics added by author.) This is inaccurate on its own and certainly a cause for concern if this is the threshold being considered. Hughes and Pocock (7) were prescient in their clarion call on the surprisingly broad and imprecise, misleading observed treatment effects that emerge in clinical trials that stop early, and a close examination of the published evidence do indicate that T-RCTs are routinely associated with greater effect sizes and overestimation than RCTs that did not stop early(10).

What do we mean by this overestimation or inflation of the estimate of effect? The overestimation is due to random error or chance since T-RCTs can yield results that are located at the high end of the random distribution of results(11). This is referred to as a ‘random high’. More considerable and random differences from the true treatment effect can emerge early on in the trial when the sample size is small, or the number of events is smaller, these two a result of stopping early before the trial can run to its powered (based on primary outcome) sample size. If the sample size is small or the number of outcome events is negligible when the data safety monitoring board (DSMB) takes an early look at the data, then the effect estimate will need to be of greater magnitude to meet the standard, prespecified stopping rule boundaries to control Type I error rates (avoidance of a false positive) if interim analyses are planned(4). DSMBs overseeing vaccine trials for COVID-19 must be mindful of the sample size and number of events as they assess efficacy, as well as the stopping rule used when they take interim looks at the accumulating data.

2) The prespecified stopping rule is a critical consideration for the DSMBs. If the data is checked periodically and investigators make a decision to stop the trial as soon as a large magnitude of effect is observed, then this could lead to overestimation of the effect of treatment(11). The problem arises with the repeated interim analysis of the data with no formal process, as opposed to formalized a priori rules in the interim analysis process. Moreover, a priori defined stopping criteria or a more strict boundary of p<0.001 will also work when having prespecified interim analysis, to mitigate early stopping(11). Caution is urged in setting the conditions for early stopping, and it is critical to report openly and transparently if there were formal rules defined a priori before a study is indeed stopped early. Researchers must be explicit in this stopping rule. Experts agree that routine data monitoring practice demands a predefined statistical stopping rule (10).

Evidence suggests that small trials with very large effects may be due to early stopping due to repeated interim analysis of the data. Regular looks at very short intervals, e.g., every 5 patients, is very problematic, as there are likely wide swings of the point estimate. An a priori formal rule with fewer, more infrequent interim analysis (and potential adjustments for the multiple analyses), larger intervals between each analysis, a strict stopping boundary (p< 0.001), and large sample size will work to reduce the chance of overestimation  (11, 12).  At the same time, some argue that a strict stopping boundary based on p-value may increase the risk of an inflated estimate even as it guards against early stopping. Thus, to overcome this, the call is for lesser looks at the data and looks that occur much later in the trial when a sufficiently larger number of events and sample size has accrued (11, 12). All of these aspects can increase certainty (confidence) in the estimate of effect and all things considered, it is the number of events that really drives this confidence in the estimates of effect.

3) We focus on what we consider to be a key rate-limiting step in stopping early for benefit and the troubling risk of overestimating the effect and declaring benefit when there is none. Increased events will reduce the likelihood of overestimation of effect (10). The goal is to ensure that the treatment effect is as accurate as possible and events play a core role in this assessment. Experts agree that a core aspect of any prespecified stopping rule must include a sufficiently large number of outcome events. As events accrue (for COVID-19 vaccine research, ‘events’ are the infections), the risk of an inflated estimate is reduced. Credible research suggests that this number appears to be at least 500 outcome events (10) for the estimate of effect to be more likely near the truth (10). Five hundred events as a threshold are very different from the 100 to 150 or so events (infections) publicly alluded to by Dr. A Fauci and Dr. R Redfield. This raises a serious question on the optimal minimal number of events to declare efficacy/effectiveness in COVID-19 clinical trial research. (Bold and italics by author.) As an example, a systematic review and meta-analysis (employing multilevel meta-regression) by the globe’s top research methodologists looking at T-RCTs versus RCTs that were not stopped for the same research question, found that there were very large overestimates of effect when the trial was stopped early and had outcome event numbers less than 100, large overestimates when event number was 200, and still appreciable overestimates of effect when event numbers were between 200-500(10). Trials with smaller sample-sizes, and small events stopped early are a serious issue as to spurious and inaccurate results.

This weakness or fragility in the estimates of effect appears the same even when the trial has a larger sample size if the number of events is small. Sample size has less to do with the accuracy of the results and in the case of COVID-19 vaccine research, the reported large trials with 30,000 sample size has less to do with the estimate of effect than the number of events. All this has led researchers to warn of skepticism when any trial is stopped early for benefit and mainly when the outcome event number is low (<500)(10, 11).  Simulations and recent empirical evidence do reveal that T-RCTs can miscalculate and lead to large, misleading, overestimated treatment effects when there is a small number of events (<200)(13).  Therefore, DSMBs associated with COVID-19 vaccine trials such as OWS or any clinical trial globally, should be very mindful of the minimal number of events that would drive confidence in the estimates before the decision to truncate. There will be far less confidence in the vaccine trial’s estimates of effect (result) if stopped early for benefit and the number of accrued events (infections) is less than 500.

4) The most suitable choice of primary outcome/endpoint and stopping guidelines is also a vital decision consideration for the DSMB and trial leadership, and the clinical relevance should drive the primary endpoint (and secondary) decisions. Patients need information from patient-important outcomes so that they are adequately informed for their values and preferences decision making (8). The outcome information must be clear so that patients can fully weigh the benefits as well as the harms of any intervention or vaccine. This must be set a priori and not be trifled with so as to declare success. Key consideration and balancing is also needed for what is the likely sample sizes, expected event rates, and intended duration of the trial (14).

5) With a specific focus on adverse events, if a vaccine trial is stopped early for benefit, the trial sponsors and regulators must ensure that a phase IV, post-decision pharmacovigilance study is established to follow the patients longer-term to collect data on emerging adverse events. This is critical. It is also critical to ensure confidence and safety post-marketing by providing that such longer-term surveillance (safety trials) is established to monitor safety throughout the vaccine's life cycle. (Bold and italics added by author.) Having adverse events emerge before stopping for benefit or before any assessment is critical to inform whether to continue the trial or stop it for harms. This was just the case in the recent stoppage of the AstraZeneca (in collaboration with University of Oxford) COVID-19 vaccine trial for a suspected serious adverse reaction (15). It is not atypical for trials to be paused to assess how serious the adverse reaction was. This is especially important when a study is stopped before evidence of potential adverse events have sufficiently accumulated. This is also a critical issue with concomitant vaccination with a new vaccine in the environment of other recommended vaccinations e.g. the seasonal influenza vaccine etc. It is thus critical for further follow-up to ensure the safety is clarified into the longer term.

Understandably, a decision to stop early for benefit is fraught with statistical considerations, ethical issues, and practical issues, and these are the very issues that the COVID-19 vaccine experts must grapple with as they search for a safe and effective vaccine. Moreover, a significant concern is the chilling effect (freezing effect) T-RCTs have on future research (6, 13) and particularly as mentioned, the limitation in assessing the potential harmful events that could have accumulated had the trial run to its powered sample size. If a T-RCT has prespecified stopping rules and guidelines with infrequent interim analysis of the data conducted later in the study when more sample size and events have accumulated, and have met the risk of bias criteria (optimally judged at low risk of biased estimates) with the number of events being at least 500 or greater, and with a low p value as a stopping boundary, then one can be more confident that the estimate of effect reflects the true treatment effect. At least 500 events will allow much more precision and the patient's values and preferences (16, 17) would be more assured in that they are considering a treatment (or vaccine) based on more confidence in the estimates of effect. Bassler et al. (2008) insists on continuation of enrollment and follow-up for a longer period (9). (Bold and italics added by author.)

The DSMB of a vaccine trial will also have very critical decisions to make as vaccine research continues and particularly the decision to stop early if a benefit is indicated. Importantly, never before has the medical regulatory agencies been tasked with a future decision that will impact hundreds of millions if not billions of lives and potentially steer public-health policy decision-making globally in terms of an effective vaccine for COVID-19. In this, the DSMBs has a vital and ethical role in ensuring that any beneficial intervention is given to all patients once there is an indication of efficacy/effectiveness. However, as indicated, confidence in the evidence and decision must be based on a consideration of the benefits and the risks at all instances of interim looks at the data, and as discussed, several issues must be considered and balanced typically simultaneously in deciding to stop early. This issue of balancing the benefits versus risks that have accumulated at each interim look at the data is critical and complex and deserves somber reflection and DSMBs and national drug/medications regulator, must weigh this very carefully.

Additional considerations in stopping early for benefit, is the trial would entail a high risk of bias and there will be less confidence in the purported estimates of effect of potential vaccine. Indeed, under the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) framework, treatment effect estimates in outcomes associated with studies rated as high risk of bias would lead to rating down the level of certainty. Lowered certainty of a COVID-19 vaccine intervention in a prevention outcome under the GRADE framework, would imply less confidence in the treatment effect estimates, and reduced certainty that further research would not change the treatment

Rapid COVID-19 accelerated vaccine development programs in terms of the clinical development, the process development, the manufacturing, and distribution scale-up has been significantly accelerated in innovative programs such as OWS. The US and world need a safe and effective vaccine. This rapid innovation to address this pandemic emergency should be given the recognition and credit it deserves given the risk and the synchronous coordination of many processes to achieve success. The global scientific community must also be applauded for the rapid scale and immense amount of research that has transpired in response to COVID-19. In closing, this commentary is a caution that the step to stop early for benefit and stop randomization of patients to the potential of no treatment in these COVID-19 vaccine trials should not to be taken lightly. Safety must be the core consideration as we strive for a vaccine. We are trusting in the sound judgement, scientific depth, and integrity of regulators and associated experts and at no time in COVID-19 vaccine development, must safety be in question. (Bold and italics added by author)

Outlined here are the key points to consider (Box 1) in COVID-19 vaccine development. These points (Box 1) once part of these vaccine trials, will significantly allow for increased confidence in the resulting estimates of effect and decisions. Failure to consider these will significantly reduce confidence by the research community and the public in the results. Openness, transparency, and explicitness must be the watchword and the public and global scientific community must be allowed the background data to the extent possible with a consideration of even more openness given this emergency and justified questions. This will allow researchers and the public to be able to understand the decision-making that was involved in stopping early for benefit, any emergency use authorization (EUA), or adverse events that emerged. The public must always be informed by full, comprehensive, balanced reporting and decisions that match the underlying data. Informed buy-in and decision-making by the public is critical and they can only do this with full openness, transparency, and explicitness by the COVID-19 vaccine developers. Once again, assurances of safety is the critical rate-limiting step in this vaccine, and in this, using the optimal ‘minimal’ number of events of 500 as a threshold to stop early for benefit. We caution to be very careful if any COVID-19 trials are stopped early for benefit and to closely adhere to the guidance outlined in this commentary.

Box 1: Key points to consider in COVID-19 clinical trials stopped early for benefit

Key points to consider:

1)     Need for a prespecified stopping rule is critical that explicitly outlines all facets

2)     Not too many interim assessments (looks) at the data is critical

3)     Adjustments for the multiple looks at the data

4)     Interim looks late in the trial to allow for larger sample accrual size is optimal

5)     Interim looks late in the trial allowing for larger number of event accrual is optimal

6)     A need for a stricter stopping rule/boundary of p<0.001

7)     Event number at least 500 is optimal to protect for overestimation of estimate of effect

8)     Stopping early limits accruing of important adverse events data downstream

9)     Pharmaco-vigilance surveillance for adverse effects long term is critical

10)  Risk-benefit assessment at early interim looks precludes an accurate assessment of risk

11)   Continuing the trial to the powered sample size is optimal (follow-up for a further period)

12)  Consideration of the patient-important primary outcome (s)

Roles in the study and manuscript:

P Alexander: Conceptualization, writing, final manuscript.

Declaration of interest:

The author holds expertise in evidence-based medicine and guideline development. The author is a part of the GRADE Working Group.

Funding acquisition: There was no funding for this research topic.

References

1. Slaoui M, Hepburn M. Developing Safe and Effective Covid Vaccines - Operation Warp Speed's Strategy and Approach. N Engl J Med 2020.

2. Le TT, Cramer JP, Chen R, Mayhew S. Evolution of the COVID-19 vaccine development landscape. Nat Rev Drug Discov 2020; 19: 305-306.

3. Prutsky GJ, Domecq JP, Erwin PJ, Briel M, Montori VM, Akl EA, Meerpohl JJ, Bassler D, Schandelmaier S, Walter SD, Zhou Q, Coello PA, Moja L, Walter M, Thorlund K, Glasziou P, Kunz R, Ferreira-Gonzalez I, Busse J, Sun X, Kristiansen A, Kasenda B, Qasim-Agha O, Pagano G, Pardo-Hernandez H, Urrutia G, Murad MH, Guyatt G. Initiation and continuation of randomized trials after the publication of a trial stopped early for benefit asking the same study question: STOPIT-3 study design. Trials 2013; 14: 335.

4. Jitlal M, Khan I, Lee SM, Hackshaw A. Stopping clinical trials early for futility: retrospective analysis of several randomised clinical studies. British journal of cancer 2012; 107: 910-917.

5. Walter SD, Guyatt GH, Bassler D, Briel M, Ramsay T, Han HD. Randomised trials with provision for early stopping for benefit (or harm): The impact on the estimated treatment effect. Statistics in medicine 2019; 38: 2524-2543.

6. Wang H, Rosner GL, Goodman SN. Quantifying over-estimation in early stopped clinical trials and the "freezing effect" on subsequent research. Clin Trials 2016; 13: 621-631.

7. Hughes MD, Pocock SJ. Stopping rules and estimation problems in clinical trials. Statistics in medicine 1988; 7: 1231-1242.

8. Pocock S, White I. Trials stopped early: too good to be true? The Lancet 1999; 353: 943-944.

9. Bassler D, Montori VM, Briel M, Glasziou P, Guyatt G. Early stopping of randomized clinical trials for overt efficacy is problematic. J Clin Epidemiol 2008; 61: 241-246.

10. Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q, Heels-Ansdell D, Walter SD, Guyatt GH, Group S-S, Flynn DN, Elamin MB, Murad MH, Abu Elnour NO, Lampropulos JF, Sood A, Mullan RJ, Erwin PJ, Bankhead CR, Perera R, Ruiz Culebro C, You JJ, Mulla SM, Kaur J, Nerenberg KA, Schunemann H, Cook DJ, Lutz K, Ribic CM, Vale N, Malaga G, Akl EA, Ferreira-Gonzalez I, Alonso-Coello P, Urrutia G, Kunz R, Bucher HC, Nordmann AJ, Raatz H, da Silva SA, Tuche F, Strahm B, Djulbegovic B, Adhikari NK, Mills EJ, Gwadry-Sridhar F, Kirpalani H, Soares HP, Karanicolas PJ, Burns KE, Vandvik PO, Coto-Yglesias F, Chrispim PP, Ramsay T. Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA 2010; 303: 1180-1187.

11. Guyatt G RD, O Meade M, and CooK D. . Users' guides to the medical literature.: JAMA Evidence; 2015.

12. Pocock SJ. When (not) to stop a clinical trial for benefit. JAMA 2005; 294: 2228-2230.

13. Bassler D, Montori VM, Briel M, Glasziou P, Walter SD, Ramsay T, Guyatt G. Reflections on meta-analyses involving trials stopped early for benefit: is there a problem and if so, what is it? Statistical methods in medical research 2013; 22: 159-168.

14. Zannad F, Gattis Stough W, McMurray JJ, Remme WJ, Pitt B, Borer JS, Geller NL, Pocock SJ. When to stop a clinical trial early for benefit: lessons learned and future approaches. Circ Heart Fail 2012; 5: 294-302.

15. Beasley D. AstraZeneca suspends leading COVID-19 vaccine trials after a participant's illness. Reuters; 2020.

16. Zhang Y, Coello PA, Brozek J, Wiercioch W, Etxeandia-Ikobaltzeta I, Akl EA, Meerpohl JJ, Alhazzani W, Carrasco-Labra A, Morgan RL, Mustafa RA, Riva JJ, Moore A, Yepes-Nunez JJ, Cuello-Garcia C, AlRayees Z, Manja V, Falavigna M, Neumann I, Brignardello-Petersen R, Santesso N, Rochwerg B, Darzi A, Rojas MX, Adi Y, Bollig C, Waziry R, Schunemann HJ. Using patient values and preferences to inform the importance of health outcomes in practice guideline development following the GRADE approach. Health Qual Life Outcomes 2017; 15: 52.

17. Munoz-Velandia O, Guyatt G, Devji T, Zhang Y, Li SA, Alexander PE, Henao D, Gomez AM, Ruiz-Morales A. Patient Values and Preferences Regarding Continuous Subcutaneous Insulin Infusion and Artificial Pancreas in Adults with Type 1 Diabetes: A Systematic Review of Quantitative and Qualitative Data. Diabetes Technol Ther 2019; 21: 183-200.