In December of 2018, the FDA released its Framework for the Real-World Evidence Program, mandated by the 21st Century Cures Act. The goal of the framework was to develop guidance for the use of real-world evidence (RWE) in regulatory decision-making. Those guidance documents are due by the end of 2021, but even without their formal release, real-world data (RWD) from electronic health records (EHR) and other secondary sources can augment clinical assertions from randomized controlled trials (RCTs) for use at the point of care including encompassing more clinically diverse populations and uncovering variations in existing practice patterns. RWD is also used to speed up trial enrollment, evaluate RCT protocol feasibility, and to increase the generalizability of trial outcomes to broader and more heterogeneous patient populations.
The FDA has approved a number of new drugs and label expansions for existing drugs based on RWD. These include new drugs Bavencio for metastatic Merkel cell carcinoma, Brineura for infantile batten disease, and Yescarta for diffuse large B-cell lymphoma. Label expansions include Blincyto for B-cell precursor acute lymphoblastic leukemia and Ibrance for advanced metastatic breast cancer in males.
The use of RWD/RWE for regulatory decision-making has gained acceptance globally with initiatives for its use in Canada, England, the European Union, India, and Japan. To improve acceptance in the drug approval process by US regulators more consistently, RWD must be research-grade and trusted, and RWE must be believable.
Generating Research-Grade RWD
Common regulatory themes around the use of RWE involve the need to generate high-quality RWD. Government regulatory bodies are coming to accept the fact that principled design and analysis of RWD can answer important questions across the drug development lifecycle, but they require research-grade data to incorporate RWE into their decision-making processes. But what is research-grade data? One school of thought is it allows looking at the data with clarity.
Researchers need to evaluate the data source to ensure it meets several criteria. It conforms to published data standards, is in concordance with defined measures, and it has plausible and consistent values. The data should be derived from a representative population, contain a sufficient number of subjects, is longitudinal where required, encompasses the key study elements, is complete, and its original source and transformations are transparent.
Building Trust in RWD/RWE
While studies that encompass RWD can effectively augment RCTs by providing a broader source of evidence, RWD studies are more vulnerable to confounding and other biases. This in turn makes healthcare decision-makers reluctant to consider RWE in policy and regulatory decisions. There needs to be a higher level of trust in RWD/RWE in order for it to become an accepted and less costly component in the requests for regulatory approval and label expansion.
Building trust in RWE begins at the study design stage with the identification of data that is fit-for-purpose, particularly when using secondary data. Assessing data quality is a critical first step, and assuring the validity of the data used in the study is key.
In addition to the evaluations conducted to determine that the data is research-grade, researchers should identify if the data has been cited in peer-reviewed publications for similar studies; understand if prior evidence from analyses match conventional thought or have been replicated in other data sources; be able to clearly define the benefits and limitations of the data as related to the question(s) under study; determine the degree of missingness for key data elements, especially outcomes, and articulate the impact on the study conclusions.
Ensuring Believability of RWD/RWE
For regulators to confidently embrace RWE as part of their decision-making process, they need to believe that it is relevant, reliable, and accurate. There are a number of key factors that enhance the believability of RWE.
Already noted is ensuring the quality, completeness, reliability, validity, and fit-for-purpose nature of the RWD source selected.
Next, since the protocol for an observational study is different from an RCT, it is important to have a clearly stated hypothesis, study design, and analytic approach. The latter should assure regulators that possible selection, information, and confounding biases have been sufficiently described within the analysis plan and addressed through rigorous statistical analysis and testing. Once the protocol is finalized, any required amendments must be transparent in that enough information on what has changed from the original protocol and what has been added so anyone can replicate the study
Another key to establishing believability is designing the study to minimize bias. This can be done by using active comparator new-user designs, identifying propensity score matched cohorts, using negative control outcomes, and minimizing the inevitable residual confounding.
Ultimately, researchers can establish believability of RWD/RWE through scientific rigor when assessing both for regulatory use. This means answering questions such as:[1]
- Are outcomes/exposures/populations captured well?
- Are the coding algorithms validated?
- Is data quality adequate and missingness sufficiently minimal to allow inference?
- Are biases mitigated by design and analytic methods?
- Do sensitivity analyses reveal robustness of results?
- Have potential biases been quantified?
- Can the results be replicated?
As the demand to accelerate regulatory decision-making grows, the role of RWD and RWE will inevitably expand as well. However, for RWD/RWE to become an integral part of the process, it must be research-grade, trusted, and believable.
In the second post of the series, we will examine the important role RWD and RWE is playing in chronic disease research and the impact of the COVID-19 pandemic.
Content for this blog post was derived from the following ISPOR 2021 Presentations:
- What is Research-Grade Real-World Evidence and Why Does it Matter? Moderator: Dan Riskin, MD, MBA, Verantos, Menlo Park, CA, USA; Panelists: Tina Hernandez-Boussard, M.P.H, PhD, School of Medicine, Stanford University, Stanford, CA, USA; Nancy Dreyer, MPH, PhD, Real-World Solutions, IQVIA, Cambridge, MA, USA; Keri L. Monda, PhD, Center for Observational Research, Amgen Inc., Thousand Oaks, CA, USA
- How to Build Trust in RWE: A Group Discussion with the RWE Special Interest Group; Suzanne Belinson, PhD, MPH, Tempus, Inc.; Amy Buchanan-Hughes, MA, MSci, PGCert (Public Health), Costello Medical; Meeting Participants
- What Makes Real World Evidence (RWE) Believable for Regulatory Decisions? Is It Real, Chance or Confounding? Jacinthe Leclerc, RN, PhD Universite du Quebec a Trois-Rivieres; Cynthia Girman, DrPH, FISPE, CERobs Consulting, LLC; Magdalene Assimon, PharmD, PhD, University of North Carolina
[1] Evaluating the feasibility of electronic health records and claims data sources for specific research purposes, Ritchey, ME, Girman C., Therapeutic Innovation & Reg. Sci. 2020,
