Does the Atherosclerotic Cardiovascular Disease Calculator Overestimate Patient Risk? The Debate Continues

A commonly used atherosclerotic cardiovascular disease risk calculator may be overestimating actual patient risk, especially within certain racial subgroups, according to a new analysis of a cohort without diabetes.

Since the introduction of the American College of Cardiology/American Heart Association’s Pooled Cohort Risk Equation in 2013, several studies have been published both criticizing and supporting its accuracy. Most recently, a study of almost 38,000 individuals demonstrated that the tool outperforms two other proposed methods for identifying patients who should be treated with lipid-lowering therapy. In contrast, a 2014 study of European patients older than 55 years found poor calibration and moderate-to-good discrimination with the risk calculator. A popular criticism has been that the tool was based on a mostly white and outdated patient population.

But a team from Kaiser Permanente Northern California led by Jamal S. Rana, MD, and Alan S. Go, MD, is now suggesting that the calculator needs to be fine-tuned to reflect changing demographics and patient characteristics.

“We’re not saying there should be a sudden halt to using the calculator,” Rana told TCTMD. “But I think our aim is to continue the research and to keep this thing evolving . . . so that in the future, millions of Americans are not unnecessarily taking statins.”

The study was published in the May 10, 2016, issue of the Journal of the American College of Cardiology.

Widespread Overestimation

The investigators looked at 307,591 patients in the Kaiser Permanente system treated in 2008 who had 5-year follow-up data and who had not been on statins or other cholesterol-lowering drugs before or during the study period. Diabetics were also excluded. Of note, 7.2% of the population identified as black, 17.2% as Asian/Pacific Islander, and 6.1% as Hispanic.

Overall, there were 2,061 atherosclerotic cardiovascular disease events recorded over 1,515,142 person-years, with acute MI, coronary heart disease-related death, and ischemic stroke rates of 0.5%, 0.2%, and 0.02%, respectively. Additionally, event risk as predicted by the calculator was substantially higher than the observed outcomes in each of the five tiered categories of risk.

Separating patients by sex, race/ethnicity, age, and socioeconomic status also showed overestimation of risk and moderate or lower discrimination of the calculator in every patient subgroup.

Go told TCTMD that while his team “did hypothesize that there would be differential accuracy, particularly among racial ethnic groups,” the fact that the calculator overestimated risk across the board was surprising. He attributes this discrepancy to the vastly different populations used in this study versus during the initial development of the tool as well as to the fact that risk factors can now be better controlled.

On the other hand, Donald Lloyd-Jones, MD (Northwestern University, Chicago, IL), who was not part of the study but was part of the original group that developed the ACC/AHA tool, said the study has a selection bias problem. By excluding patients who had previously received a statin, for example, “you’re actually taking away the natural history that the risk score is supposed to predict,” he said in a conference call with reporters offered by the ACC. “Of course the risk score is going to overpredict. It’s supposed to predict on a representative population.”

Additionally, he also pointed to the study’s “historically low, never before seen event rate for any population in the US.” So while the study might make some valid points in terms of demographics, Lloyd-Jones said, “not all science is equal and it’s really important to understand the methods, because if you look under the hood and the engine is not so good then I wouldn’t necessarily trust the car.”

Backing him up is Michael J. Blaha, MD (Johns Hopkins Hospital, Baltimore, MD), in an accompanying editorial. He also points to the almost “implausible” event rates, adding that “we should remain skeptical of the magnitude of these findings, suspecting some missed events, or at least misclassified events.”

However, Blaha continues, “the general trends are consistent with both the epidemiological research and clinical observations.” Hard events are becoming softer—like revascularization—and atherosclerotic cardiovascular disease is perhaps now less severe than before, he reports.

Worth Recalibrating?

So what now? To date, Lloyd-Jones reported that “well over 250,000 people” have downloaded the mobile app version of the risk calculator and he estimates that about 11,000 people a day use the website. “So somebody out there is finding this useful,” he said. “I hope that they’re using it as a starting point for a conversation and not a one-and-done kind of thing. Because that’s not really the way we intended it to be used.”

The risk calculator “was an important step forward,” Go said. “We think it’s a matter of refining it and making sure that it is applicable and as accurate as possible in contemporary populations.” Specifically, he would like to see the entire tool recalibrated to specific patient subgroups.

“However, recalibration can be confusing, and such recalibrated scores are not available to clinicians at the ACC website or using the helpful [mobile] app,” writes Blaha. Rather, he agrees with Lloyd-Jones that the tool should still be used to inform patient/physician decision making, for now. “Risk score calibration remains an elusive goal, but with important implications for clinical care,” he concludes.

The tool is only a single piece of information, “certainly not the final word,” Lloyd-Jones said, adding that he expects it to evolve with the next set of guidelines. “This is kind of the art of medicine where you take this tool as a starting point, but you have to bring it to the context of your individual patient, and that’s just as important.”

Related Stories:

• ‘Negative’ Atherosclerotic Imaging Results Shift 10-Year CVD Risk Downward: MESA Analysis
• Degree of LDL Cholesterol-Lowering on High-Intensity Rosuvastatin Determines Clinical Impact
• ACC, AHA Risk-Based Approach Remains Best Method for Statin Allocation in Primary Prevention