Event Rates Often Overestimated in Cardiovascular Trials

Event rates in RCTs evaluating cardiovascular interventions and devices are frequently overestimated, according to a study published online May 8, 2015, ahead of print in the American Journal of Cardiology.

“This underreported phenomenon has [a] fundamental impact on the design of RCTs and can have an adverse impact on the statistical power of these trials to answer important questions about therapeutic strategies,” write David R. Holmes Jr, MD, of the Mayo Clinic (Rochester, MN), and colleagues.

The researchers prespecified 10 topics commonly researched in the field of cardiovascular interventions and devices, identifying the 5 latest peer-reviewed RCTs for each of these topics published as of April 14, 2014. After excluding studies with designs other than treatment vs control, populations of less than 30 patients, manuscripts not in English, or no sample size calculation or listed estimated event rates, they analyzed event rates in 27 RCTs randomizing 19,436 patients from 2000 to 2012.

Seventeen trials were designed to show superiority and 10 had a noninferiority design. Median trial duration was 2.2 years and the median dropout rate was 2.4%. All but 5 trials were at least partially funded by industry.

Pattern Is ‘Clear’

Dr. Holmes and colleagues found “clear evidence for overestimation of event rates,” they write.

For example, the primary event rate in the control group was lower than predicted in 20 trials (74.1%), with a mean relative difference of -22.9% (95% CI -33.5% to -12.2%) between observed and estimated event rates. Disparities were greatest in trials of biodegradable-polymer DES and renal artery stenting, and the gap reached more than 10% for 16 RCTs (table 1).

In an exploratory analysis, researchers identified trends toward greater overestimation of event rates in longer studies (P = .12) and those with higher dropout rates (P = .15). Moreover, predictions based on a prior RCT appeared more likely to be accurate (P = .07).

Neither study design (superiority vs noninferiority) nor time (year of enrollment initiation) affected overestimation of events. Additionally, there was no difference between industry- and publicly funded RCTs (P = .662).

Three trials—PREVAIL, PROTECT AF, and RESPECT—were event-driven rather than based on a conventional sample-size–driven design. Hence, when these trials were removed from the analysis, variations in sample size were less profound among the 24 other trials (mean relative difference -7.9%; 95% CI -17.9% to 2.2%), with the actual sample size being smaller than planned in 7 of them.

Nine trials (33.3%) resulted in a positive outcome (superiority or noninferiority), 2 noninferiority trials (7.4%) demonstrated inferiority, and 15 superiority trials (55.6%) were nonsignificant.

Among the 14 superiority trials that failed to show significance for a single primary endpoint, 8 produced inconclusive results. On a relative risk scale, only 3 of these trials (21.4%) produced truly negative conclusions based on the prespecified minimal relative risk difference.

How Does Overestimation Happen?

“By using a topic-based approach, the current report shows how a systematic overestimation of event rates may impact the body of randomized evidence in the field of cardiovascular interventions and devices,” Dr. Holmes and colleagues write.

The trend towards fewer events with longer studies “may be explained by [the] progress of overall medical care, such as concurrent improvements in hypertension management and the advent of statins in the generally long RCTs comparing renal artery stenting with medical therapy,” they suggest. “Event rates also tended to be lower when dropout rates were higher. Evidence in this field is limited, but some studies have associated patient dropout with medication noncompliance and higher mortality, thus potentially draining an RCT of outcomes.”

Because estimation of event rates tended to be more accurate when a study was based on an earlier RCT, this would seem a “reasonable” thing to do when designing future studies, the authors note, but “the literature is conflicting with regard to the accuracy of outcome estimation in RCTs and nonrandomized studies.”

Speaking to the Hawthorne effect, they say that patients might change their behavior simply because of knowing that they were being studied. “It has been argued that the Hawthorne effect was responsible for the 12-mm Hg drop in systolic blood pressure in the sham-procedure group in a recent trial investigating renal denervation for resistant hypertension,” the researchers write. “Some of these patients thought to have resistant hypertension may simply have been nonadherent to their medication and improved their habits over the course of the trial.”

Transparency Is Key

Overestimation “can have a detrimental impact on the power of an RCT,” Dr. Holmes and colleagues comment. “Investigators can take advantage of [the inverse square law] when designing a trial in the face of budgetary restrictions. Overestimation of event rates in the control group dramatically reduces the required sample size and is more likely to be accepted by an institutional review board than adjustments to the alpha or power of the trial. However, our study illustrates that this severely reduces the odds of a trial with conclusive results.”

Even though superiority and noninferiority trials “seemed to overestimate event rates to the same extent, the latter are of special concern,” they observe, noting that noninferiority trials using an absolute margin can be biased toward a positive outcome when event rates are lower than expected. Adaptive and event-driven trials are also of concern due to the potential for “prolonged trial duration and additional investment of resources,” they add.

Going forward, “[i]nvestigators should be encouraged to use more conservative event rate estimations, even if this requires a larger sample size or results in a lower power,” the authors suggest. Another option, they say, is blinded sample size recalculation during the course of the trial.

The ethical justification of underpowered trials is up for debate, they comment.

“Although some have strongly argued against it, others have noted that well-conducted underpowered trials can still provide an unbiased point estimate of the treatment effect, and results of underpowered trials may be pooled in adequately powered meta-analyses,” the authors conclude, stressing that transparency is required. “At present, it is very hard for the casual reader to identify underpowered trials, whether one condemns or accepts such practice.”

Mahmoud KD, Lennon RJ, Holmes DR Jr. Event rates in randomized clinical trials evaluating cardiovascular interventions and devices. Am J Cardiol. 2015;Epub ahead of print.

Disclosures:

• Dr. Holmes reports serving as the principal investigator of the PROTECT AF and PREVAIL trials and as a collaborator in the PARTNER trials.

Related Stories:

• Paper Highlights Pitfalls in Interpreting Randomized Clinical Trials
• ‘Absence of Evidence Is Not the Evidence of Absence’ - How Do We Translate Clinical Research Into Clinical Practice?