What the FDA Really Thinks About Real-World Evidence

Real-world evidence is a critical issue for the FDA, but we must work together to ensure it meets the highest possible standards, says former FDA Commissioner, Robert Califf



Real-world evidence offers such a tantalizing glimpse of a new world full of possibilities that its allure could lead to “incorrect or unreliable conclusions”. This was one of the conclusions of a “consensus of FDA leadership” on real-world evidence published in the New England Journal of Medicine last December.

The rationale behind the FDA’s decision to air its views – and warnings – around real-world evidence goes to the heart of its mission. “We have a regulated clinical trial system that’s not fit for purpose,” says former FDA Commissioner, Robert M Califf, MD. “It’s way too expensive, so we’re beginning to see a slowdown in those willing to invest in therapeutics. If you look at the cost of regulated clinical trials, a very significant part is related to the view that a trial needs to done as a set-aside experiment, separate from clinical care and with a whole lot of bells and whistles that are unnecessary.”

Until now, however, there was no alternative to the gold-standard randomized controlled clinical trial. “There wasn’t much data readily available so you had to create it. Now, everyone has a EHR – at least in the US – and there are many quality registries, plus claims data are improving in terms of curation. Consequently, we’re now in a situation where research in the real world is not only offering a more generalizable solution but a much less expensive one. All in all, it’s a much better deal.”

However, in spite of the array of possibilities, there are many obstacles to overcome, he says. “We’re just not all the way there. When I was Commissioner, the view of the FDA was that this was a critical issue, which led to the decision to publish the paper in NEJM. But, if you look at the law – at the 21st Century Cures legislation and the user fee agreement we’re working on, the use of real-world evidence has been written in as a legal mandate for the next five years, so there’s no doubt in my mind that everything is pointing in the right direction.”

This may come as a relief to some who saw the FDA’s paper as an indication of reluctance on the part of the agency around this type of evidence. While, RWE can “inform therapeutic development, outcomes research, patient care, research on health care systems, quality improvement, safety surveillance, and well-controlled effectiveness studies”, wrote Califf and colleagues, “the confluence of large data sets of uncertain quality and provenance, the facile analytics tools that can be used by nonexperts, and a shortage of researchers with adequate methodologic savvy... could result in poorly conceived study and designs that generate incorrect or unreliable conclusions.”

At a time when many more drugs are being approved through accelerated pathways, it is crucial the FDA receives the best quality evidence, says Califf. “It would be a big mistake to say that you can tell whether a drug is safe and effective with inadequate evidence. Half of the drugs that come to market now get accelerated and, in those circumstances, we accept inadequate evidence because there’s an unmet clinical need and dire consequence with no other available treatment. That’s good for society but a general drop in regulation across the board would not be a smart thing to do. Over 90% of drugs that get to phase 1 don’t make it to market because they’re either ineffective or cause harm and taking that risk without adequate studies is very bad public policy.”

It will take time to untangle the pros and cons of real-world evidence, says Califf. “It’s not entirely clear that visiting a research clinic every six to eight weeks is a more reliable way of measuring what’s happening to a person in the follow up of a clinical trial. We’re now in a situation where doing the research in the real world is not only offering a more generalizable solution but also a much less expensive solution.”

Given these concerns, will the FDA issue more guidance? “The FDA has been using real-world evidence for a long time and there’s plenty of guidance around how to use it – we don’t need a change in laws to use it, as long as it’s fit for purpose. However, the whole research community and the industry wants the FDA to put out guidance,” says Califf. “While it’s hard to put guidance out when the Administration wants to get rid of regulation, hopefully, we can, as a community, convince them that guidance is not law, rather it’s a common view on how things should be done and allows for exceptions to be made.

Collaboration lies at the heart of Califf’s vision for RWE. “Nothing would be accepted without review of the agency, [but] the beauty of the FDA, as opposed to other regulatory agencies, is that it gets the primary data, so anyone with bad methods will be intercepted by that first line of defence.

“We need to think broadly; the FDA recognizes that it’s only one component of a system that includes HTA and payment for services. We need the whole community to come together and do a better job. We need transparency, internal controls and external validation, which are evolving quickly in the research world. Increasingly, in a ‘datafied’ environment with greater transparency, it will be far harder to make claims based on shabby data.”

His advice for pharmaceutical manufacturers is practical and straightforward. “Get the methods down, especially in pragmatic trials where you use randomization. Learn when observational analysis for supplemental information is useful. Meet early and often [with the FDA], and stay on the same page. Not all parts of FDA are equally adventurous; you need to have the right conversations before you get too far down the line.”

Leaning on more than four decades of experience, Califf is convinced on the transformative power of RWE. “I’ve been using RWE for 40 years but now we’re heading into an exponential growth phase, a phase of improved evidence and of translating that evidence into practice. I’m very optimistic but there are many challenges ahead; take whole genome sequencing, for example, it’s like going from one clinical test at a time to 3.2 billion at a time, and that is going to be available at a reasonable price very shortly.

“When it comes to the potential of RWE to improve outcomes, we need to get the clinical community up to speed so that they understand where interventions are useful and how to apply them. That’s going to take a little longer but I have no doubt that it’s going to be a new world.”


This article first appeared in Trends in Real-World Evidence, created for the  Data, Evidence and Access Summit (November 13-14). Pre-order the brochure here


Since you're here...
... and value our content, you should sign-up to our newsletter. Sign up here

comments powered by Disqus