More Data, More Studies, Fewer Insights: Why the RWE Community Has a Trust Problem

Key takeaways:

• The rapid expansion of real-world data (RWD) access and automated analytics has accelerated research output without a corresponding increase in scientific rigor, eroding trust in real-world evidence (RWE) across the field. 
• Data quality and evidence quality are not the same thing. High-quality data can still produce untrustworthy findings when the research question is poorly framed, methodology is misaligned, researchers lack training, or assumptions go untested. 
• The infrastructure, methodology, and expertise needed to produce trustworthy evidence exist today. What is needed is a community-wide commitment to applying them consistently, regardless of how quickly technology can generate an answer. 

I spent the better part of last week in Milan at the ISPE Annual Meeting, where  pharmacoepidemiology researchers, regulators, and industry experts gathered around a shared ambition: unlocking the power of data and pharmacoepidemiology to improve patient health. The conversations were energizing. The science on display was impressive. And underneath it all ran a current of tension that I think our community needs to address more directly than we typically do in conference halls. 

We are producing more publications than ever before. We are also trusting it less. 

The Infrastructure Got Ahead of the Science

Processes that once took months or even years can now be completed in days or hours. Expanded data networks, improved interoperability, automated analytics, and artificial intelligence (AI) have made it possible to generate hypotheses quickly, explore cohorts iteratively, and produce outputs at scale. Access to RWD has never been easier. Analytical tools have never been more sophisticated. 

And yet the ability to generate answers now exceeds our willingness to trust them. 

This is not a data quality problem, though that is usually where the conversation goes first. The instinct, when findings fail under scrutiny, is to blame the data. Increasingly, there is an instinct to blame the tools. Both instincts are understandable. Both instincts often miss the mark. 

The challenge is a lack of sufficient attention to a disciplined scientific process, one that is led by a well-trained researcher who develops clear research questions that can be addressed with appropriate methods and fit-for-purpose data. The community demands faster answers without demanding the scientific discipline and training to ensure those answers are trustworthy. 

Fit-for-purpose, high-quality data are necessary for trustworthy evidence. But data do not become defensible evidence unless rigorous scientific judgment is applied, and that judgment cannot be automated, accelerated, or assumed. 

What the Numbers Tell Us

The reproducibility crisis in science is well documented and not limited to any single field. A survey reported in Nature showed more than 70% of researchers were unable to reproduce another scientist’s findings, while large replication initiatives across psychology, cancer biology, and other disciplines have revealed substantial reproducibility gaps. The RWE community is not immune. 

Across the field, teams are encountering a persistent and costly problem: peer-reviewed evidence that appears sound but weakens upon review. Analyses generated from similar, or even the same, datasets yield conflicting conclusions. Findings that seem plausible fail under expert scrutiny. Internal teams struggle to reconcile competing results produced by different vendors or tools, each claiming methodological rigor. 

The ISPE community has been grappling with exactly this challenge. The 2026 Annual Meeting’s focus on bridging the gap between robust RWE generation and its translation into clinical, regulatory, and public health decision-making reflects just how much is at stake when evidence quality erodes.  

The frameworks our community has built to protect evidence quality are designed around disciplined processes: strong training, study design, implementation, reporting, and peer review. What expanded access to data and analytic infrastructure has done is allow the volume of research studies to overwhelm our ability to ensure adherence to those processes and high-quality peer review. 

When Faster Isn’t Better

Traditional research friction, including long timelines, limited access to data, and scarce statistical and programming expertise, once acted as a natural brake. Studies that lacked rigorous foundations were harder to fund, harder to complete, harder to publish, and harder to proliferate. That friction has largely disappeared. 

As a result, organizations are increasingly exposed to findings that move quickly through internal and external channels without adequate application of scientific intelligence. The cost is not always immediate. Fragile evidence often survives initial review, only to fail later when challenged by regulators, reviewers, or commercial scrutiny. By the time weaknesses surface, decisions may have already been made. 

The consequences are real. I have seen flawed studies influence decisions and result in wasted resources. I have seen the cost of that to research programs, organizations, and patients.

The Problem Is the Standard We Accept

In 2020, a high-profile report published in Lancet linked hydroxychloroquine use to increased mortality in COVID-19 patients using a large global dataset. Within weeks, serious concerns emerged: unverifiable data provenance, an inability to independently audit the dataset, and inconsistencies in reported hospital-level data. The study was ultimately retracted after data validation could not be completed. A related New England Journal of Medicine paper using the same dataset was also withdrawn. 

The lesson is not that large datasets are untrustworthy. The lesson is that large datasets cannot compensate for missing transparency, traceability, and methodological rigor. It is the responsibility of researchers to understand the data they are using, follow community guidance on research process, apply the right blend of intelligence and methodology, and report methods and findings transparently. 

What Comes Next

The infrastructure to do better exists today. The methodology exists. The expertise exists within this community, as the conversations in Milan made clear. What the RWE field needs now is a shared commitment to training those beyond our community, to critically review papers, to promote transparency in publications, and highlight the strong research studies while critiquing those that lack scientific rigor. 

That means treating data, analytic tools, and evidence quality as distinct, not interchangeable terms. And it means remembering that researchers are responsible for the research they conduct – the data and the tools can be used to support strong designs or poor designs. And it means recognizing that the goal is not to produce more answers but rather to produce more answers we can trust.

This post is adapted from TriNetX’s new eBook, The Evidence Trust Crisis: Why More Data, Faster Analytics, and Easier Access Are Accelerating Output While Eroding Trust. 

About Jeffrey Brown, PhD 

With more than 25 years of experience in research and consulting, Jeff is an internationally recognized expert in the use of RWD to support the evidentiary needs of regulatory agencies and medical product sponsors and an expert in the assessment of data quality of RWD resources.