TCT 2022: Data Integrity in Real-World Evidence

“How do we know what we know in clinical medicine?” asks Robert Yeh, MD, MSc, associate professor of medicine at Harvard Medical School. Dr. Yeh is currently studying the question: “Can we use observational data to understand the effectiveness of Impella versus other alternative treatment modalities?” in his role as a member of the Circulatory Systems Device Advisory, a Panel for the US Food and Drug Administration (FDA).

In this insightful presentation, Dr. Yeh discusses how we analyze data and how sometimes we get the wrong answers, despite our best intentions. “Our desire to do the best for our patients and our desire to develop evidence sometimes work in opposition to one another,” he explains. “And what we’re asked every day to do as clinicians, to do the best for our patients, actually subverts the process of generating evidence, both observational evidence and randomized trial evidence.”

Dr. Yeh discusses sources of real-world evidence and the importance of identifying sources of random variation (e.g., physician preference, formulary coverage variation) and how different results can be achieved with different methods. “This is the state of our observational evidence today,” he states, highlighting two observational studies published in the same journal in the same year with opposite conclusions regarding the relationship between vascular closure devices and outcomes.

So how do we know which study is right? “There are things that can help you break the tie that we ought to be embedding more in observational data,” Dr. Yeh explains, “and you should be looking for these types of things when you read an observational study.” He describes the features that are important to integrate into a rigorous approach to analyzing observational data, such as pre-specified design with FDA input, non-inferiority hypothesis, and outcome blind analysis.

"Physicians do not choose treatments randomly," he emphasizes, "but make intentional choices for the benefit of patients. These choices, however, can make real-world comparisons using traditional methods challenging. While a combination of state-of-the-art methods and deep clinical subject matter expertise can be used to create reliable observational causal inferences, sometimes these inferences are not possible with routinely available datasets, hence the need for randomized controlled trials."

He concludes with a discussion of “the parachute trial problem,” which highlights the difficulty of conducting trials that challenge entrenched standards and how certain factors impair our ability to interpret results. He explains that this is particularly relevant for the PROTECT IV trial and that investigators need to have the courage to enroll patients they would normally treat with Impella. “We have to do this to have a successful trial,” he emphasizes. “We have to include the patients we think will benefit the most.”