‘Moderate’ Agreement Between AI and Heart Team for Complex CAD Decisions

For patients with complex coronary artery disease, commercially available large language models (LLMs) aren’t always in sync with multidisciplinary heart teams regarding revascularization recommendations, even when extensive clinical context is included in the prompts, according to a single-center study.

Recommendations agreed in 75% of cases considered by either the heart team or two LLMs—ChatGPT-4o (OpenAI) and Gemini 2.0 (Google)—when the artificial intelligence (AI) tools were provided with an unstructured clinical narrative input (CNI) that included information available to the heart team at the time of its discussion.

However, when the prompts included only a less detailed structured case pro forma (SCP) with 30 clinical and angiographic variables, LLMs and the heart team agreed in just 35% of cases.

The findings, published online recently in JSCAI, “underscore the need for format-aware AI integration, rigorous validation, and careful oversight before clinical deployment,” lead author Jack Bradbury, MD (University of Melbourne, Australia), and colleagues write.

Senior author Anoop Koshy, MBBS, PhD (University of Melbourne and the Royal Melbourne Hospital), told TCTMD the results were “quite surprising.”

When giving the LLMs “a very detailed clinical prompt with all the clinical narrative about the patient and detailed aspects of their anatomy, their frailty, and everything else, the agreement between the heart team and the AI was approximately 75%, which I thought was reasonable—not perfect, but reasonable,” he said. On the other hand, discordance was much greater when less clinical detail was provided in the prompt.

One of the key take-home points, Koshy said, is that “the output that we got from AI was entirely dependent on the context that we gave it.”

This is particularly important in light of the fact that many patients are already turning to LLMs for medical advice, he added.

“We all know the intelligence of AI, and the up-to-date knowledge that it has is almost second to none, but the danger is that AI can give people a very confident answer” based on an incomplete description of the problem, Koshy said. “That is probably one of the things that both physicians and patients should be aware of.”

Discordance May ID Higher-Risk Patients

Decisions about management of complex CAD are typically made through multidisciplinary heart team discussions. AI models are being increasingly deployed throughout medicine and cardiology, but it’s not known how they will perform as decision-support tools in this setting.

Koshy explained that at his center, patients with complex coronary disease are discussed at weekly heart team meetings with clinical, interventional, and imaging cardiologists; cardiac surgeons; anesthesiologists; and intensivists.

“The heart team is not perfect, but it is the best we have,” Koshy told TCTMD. In this study, he said, his team wanted to evaluate how well LLM-guided revascularization recommendations matched up with those coming out of the heart team discussion.

The retrospective study included 546 patients (mean age 66 years; 17% women) who had complex CAD and had a treatment recommendation recorded after a multidisciplinary heart team discussion between 2019 and 2024. Three-quarters had triple-vessel disease and 12% had severe left main disease. Ultimately, the management recommendation was CABG in 67%, medical therapy in 19%, and PCI in 13%; 91% followed that advice.

The researchers ran each case through ChatGPT-4o and Gemini 2.0 with two prompts differing by the level of clinical detail. Agreement between the AI tools and the heart team in terms of treatment recommendations was better when the LLMs were prompted with a more detailed, free-text CNI than with an SCP, with consistent results for ChatGPT-4o and Gemini 2.0.

Augmenting the prompts with information from European and US revascularization guidelines did not improve model performance.

When looking at results with ChatGPT-4o specifically, cases where the LLM and heart team recommendations were discordant resulted in an increased risk of MACE (MI, stroke, or all-cause mortality) over a median follow-up of 989 days compared with cases with agreement (OR 1.67; 95% CI 1.03-2.72). Additional adjustment for markers of clinical complexity and the type of treatment patients received, however, rendered this relationship nonsignificant. There was no significant association between discordance and MACE risk when using the results of Gemini 2.0.

Nonetheless, Koshy said, “I suspect that when there is disagreement, that kind of highlights an intrinsically high-risk patient population that we probably need to put a bit more thought into.”

That added attention might include “heightened review, second opinions, or risk-mitigation strategies to ensure that due diligence, patient education, and informed consent are maintained,” he and his colleagues note in their paper.

Still, there is a need for additional research before deploying AI in this way, Koshy indicated. “There’s absolutely a role for further study to establish how it integrates as a clinical decision-making tool for everyday practice, particularly in these complex patient cohorts [for which] we sometimes don’t always know what the right thing to do is.”