Meta-analysis: Bone Marrow Stem Cell Research Riddled with Discrepancies

 Download this article's Factoid (PDF & PPT for Gold Subscribers)
 

A meta-analysis of 49 different trials involving the use of autologous bone marrow stem cell therapy for heart disease has uncovered over 600 discrepancies in design, methods, and results. Published online April 29, 2014, in the British Medical Journal, the paper contends that studies with the highest number of discrepancies reported the greatest ejection fraction effect size, while those with no discrepancies reported zero impact on ejection fraction.

Researchers led by Darrel P. Francis, MD, of Imperial College London (London, United Kingdom), examined 133 published reports from 49 clinical trials that evaluated autologous bone marrow stem cell therapy in patients with heart disease.

Overall, 604 discrepancies of varying types were identified in the reports. These included:

• Studies reported by authors to contain both accepter-rejecter and randomized
• Statistical and mathematical errors
• Inclusion of patients with 0 or negative NYHA class
• Presentation of nonsignificant differences as significant
• Conflicts in protocol or follow up
• Contradictions in figures
• Initial report of large effect size followed by second report showing much smaller effect
• Changes in reported patient gender

There was a correlation between the number of discrepancies and the reported ejection fraction effect size (P = .005). Only 5 of 49 trials had no discrepancies, according to the definitions used by the study authors. All 5 reported a negative mean effect size with this average weighted by sample size. Conversely, the greater the number of discrepancies in the remaining trials, the greater the reported mean effect size of cell therapy on ejection fraction (table 1).

Table 1. Discrepancies and Effect of Cell Therapy on Ejection Fraction

A Search for Explanations

According to the study authors, they “asked for resolution of over 150 discrepancies through journals. None were resolved, although we found it triggered correspondence from lawyers.”

Although unable to explain how so many discrepancies could have made it to publication, some in prestigious journals, Dr. Francis and colleagues suggest that “authors might feel pressure for results to match expectations.” One example of this, they note, is directed editing of rounded percentages to force them to add up to 100%. “In reality, correctly rounded percentages should often not add up to 100% when there are many categories,” they write. “The effect of even a little bias can be surprisingly dramatic.”

Importantly, they say, the possibility exists that in reports with the fewest discrepancies, “the ejection fraction effect might also have been measured with least error. If so, the true effect of bone marrow stem cells on ejection fraction is zero.”

Finally, Dr. Francis and colleagues urge that stem cell studies using change in ejection fraction as an endpoint should be properly designed to resist error and to have adequate sample size to combat the effects of biological variability.

“Left ventricular ejection fraction is a mutable variable, which in some modalities is easily manipulated innocently by clinicians who have prior beliefs on what a realistic value should be for a particular patient,” they write. “Sample size planning can sometimes be erroneously omitted when clinicians are enthusiastic to ‘demonstrate the effectiveness’ of a treatment seen as exciting.”

Moving Forward

“There are a significant number of inaccuracies that make it to publication,” said Warren Sherman, MD, of Columbia University Medical Center (New York, NY), in a telephone interview with TCTMD. “I don’t think that’s shocking news to readers of scientific literature, but few groups have tried to wrap their arms around these discrepancies and categorize them,” he commented. “From my perspective, I think this paper offers up an important template for people to start considering in terms of the design and the conduct and the reporting of these studies.”

Dr. Sherman observed that studies of autologous stem cells are—by their very nature—given to considerable variability, and although many of the discrepancies found in the meta-analysis may be viewed as “small and inconsequential,” their potential impact on outcomes “concerns us all.”

That sentiment was echoed by Jason Kovacic, MD, PhD, of Mount Sinai School of Medicine (New York, NY), who told TCTMD in a telephone interview that it is not unusual for researchers to uncover minor discrepancies. Often, he added, for a variety of reasons those discrepancies do not get formally resolved. However, he said the sheer number of discrepancies found in this meta-analysis sends a message to all researchers about the importance of checking and “triple-checking” data before publication.

The finding that the number of discrepancies correlated with outcomes may apply to other areas of regenerative medicine, Dr. Sherman said. However, he noted that it is not at all clear from the meta-analysis whether there are commonalities among the discrepancies that may be helpful in future trial design.

At present, Dr. Kovacic said that while the body of literature as a whole shows at best only a marginal signal of benefit of bone marrow stem cell therapy in heart disease, “the only way forward is to conduct large, multicenter randomized clinical studies [such as] the BAMI trial where there is rigorous oversight.” That trial will enroll 3,000 post-acute myocardial infarction patients with reduced ejection fraction randomized to stem cell administration or standard care. “Until we get results from that trial we’re left in this difficult position of not knowing whether it’s efficacious or not,” he said.

Dr. Kovacic added that the need for standardization of endpoints is another important message of the paper. Dr. Sherman concurred, saying, “We’ve all come to the point in time now where ejection fraction is no longer considered a useful surrogate endpoint.” Furthermore, Dr. Sherman added that although the field of stem cell therapy is moving forward so quickly that most of the studies in the meta-analysis no longer have relevance, “the lessons are more than applicable to where we are now and how we move forward.”

 

---

Source:
Nowbar AN, Mielewczik M, Karavassilis M, et al. Discrepancies in autologous bone marrow stem cell trials and enhancement of ejection fraction (DAMASCENE): weighted regression and meta-analysis. BMJ. 2014;Epub ahead of print.

Disclosures:

• Dr. Francis reports no relevant conflicts of interest.
• Dr. Kovacic reports serving on a National Heart, Lung, and Blood Institute stem cell committee and receiving funding and speaker’s honoraria from AstraZeneca.
• Dr. Sherman reports serving as a reviewer for one of the papers included in the meta-analysis.

Related Stories:

• Meta-analysis: Bone Marrow Mononuclear Cell Therapy Fails to Show Efficacy in Acute MI
• CADUCEUS: Autologous Stem Cells Safely Reverse Some MI Damage
• Bone Marrow Stem Cells Not Effective Early or Late After AMI