The Data Isn’t the Problem. Your Study Design Might Be

Key takeaways:

•  When real-world evidence (RWE) fails under scrutiny, the instinct is to blame the data or the tools. In most cases, the failure occurs earlier, at the point where assumptions about what the data can support are left untested. 
• Data quality and evidence quality are not interchangeable. High-quality data processed through well-documented analytic tools can still produce misleading findings when mismatched with the question being asked. 
• Trustworthy evidence requires four elements in lockstep: the right question, the right study design and analytical method, the right data, and the right implementation by a well-trained research team. Weakness in any one degrades the whole. 

When RWE findings fail under scrutiny, the conversation almost always turns first to data quality, but that is rarely where the real problem begins. 

I have spent the better part of three decades in healthcare research. I have reviewed enough studies, and watched enough findings fail, to know that the most consequential failures in RWE generation are rarely dramatic. They are subtle, cumulative, and often unnoticed until scrutiny is applied. And in most cases, they do not originate in the data at all. They originate earlier, in the decisions made before the analysis begins. 

As the ISPOR Europe community prepares to convene, questions of evidence credibility are squarely on the agenda. Health economists, outcomes researchers, regulators, and payers will spend four days debating how RWE can more reliably shape access decisions and clinical practice. Those are the right conversations. But they will only produce durable answers if the field is also willing to confront an uncomfortable truth: the problem is not the data or the analytics. It is training and the standard of science we accept.

The Blame Game

Even well-structured data analyzed with robust and well-documented analytic tools can mislead when mismatched with the question being asked and the methods used to answer the question. Claims data reflect administrative processes rather than diagnostic truth. Electronic health records (EHRs) can capture rich clinical detail but miss care delivered outside a given system. Context determines whether data are appropriate, regardless of how well standardized the data may be. And context requires scientific judgment, not scoring. 

Generalized claims about high-quality data make this worse. They encourage a false sense of confidence that can obscure deeper evidence risks. The data cannot be held accountable for what researchers do with it. Even with perfect data, producing RWE is about having the scientific expertise to use that data appropriately. 

This is not a comfortable thing to say in a field that has invested heavily in data infrastructure and analytic capability. But the investment in those capabilities has outpaced the investment in the scientific discipline and training required to use them well. The result is a growing volume of research that generates results faster than the community can verify them. 

Where Flawed Science Actually Happens

Evidence quality problems are rarely the result of obvious error. They are the result of assumptions that seem reasonable in isolation but undermine the conclusion when taken together. 

When patients in comparator groups are not truly similar, results are misleading. A study comparing treatment outcomes across two patient populations is only valid if those populations are genuinely comparable. Sicker patients concentrated in one group will skew results in ways that survive initial review but collapse under expert scrutiny. 

Poorly considered measurement windows and index dates create a different category of problem. The windows and index date that define when patients enter a study, or when outcomes are collected, are critical to consider within the context of patient population, the question, and the data. They are scientific decisions. Get them wrong, and the findings can be misleading.  

Missing clinical information is another common failure modality. Important events and clinical information found only in unstructured clinical notes, rather than coded fields, may not be included in the analysis at all. If they are absent from the data used, they are effectively treated as if they never happened. The conclusions that follow are not wrong because the data are wrong, but they might be misleading if the analysis didn’t consider what information was observable and what information was not. 

These and other methodologic errors are often subtle. Each decision or omission could appear reasonable in isolation. Together, they can render an otherwise well-resourced study unreliable. And because the decisions are not obvious, fragile evidence often survives initial review, only to unravel later when challenged by regulators, reviewers, or commercial scrutiny. By the time weaknesses surface important decisions may have already been made. 

Four Elements That Must Stay in Lockstep

Trustworthy evidence is not the product of good data alone, or good tools alone, or good intentions. It is the product of an integrated research system in which four elements remain aligned from the outset. 

1. The Right Question

The research question and its intended use must be defined prospectively and with precision. Vague or shifting questions produce vague or shifting evidence. Regulators and payers are increasingly evaluating not just results, but the integrity of the process used to generate them, including whether the question was clearly defined before the analysis began. 

2. The Right Study Design and Analytical Method

The methodologic and statistical approach must be selected to match both the question and the intended use. This is not a decision that can be delegated to automation or resolved by defaulting to a familiar method. It requires scientific judgment about which design best controls for the biases inherent in the data being used. Target trial emulation is a wonderful tool to force researchers to think clearly about how to design a study and implement a specific research question.  

3. The Right Data

Data sources must be selected because they match the research question and the chosen methods, not because they are available or familiar. A mismatch between the question and the data source is one of the most common and costly sources of evidence failure, and it frequently goes undetected until the findings are challenged. 

4. The Right Implementation

Applying the right human intelligence and processes to ensure robust and repeatable findings is where rigorous science is either upheld or quietly abandoned. Study execution by a well-trained team of researchers must follow well-documented and locked protocols. Deviations must be documented. Assumptions must be surfaced and stress-tested throughout the process, not assumed away at the outset. 

Break alignment at any point across these four elements and evidence quality degrades, often in ways that are not immediately apparent. Evidence produced without all four is incomplete. 

Feasibility Is Not a Formality

One of the most consistent sources of evidence failure is the treatment of feasibility assessment as a preliminary formality rather than a substantive scientific step. At TriNetX, feasibility is embedded into the research workflow from the first conversation. Before any analysis begins, the research teams explicitly evaluate whether the available data can support the question being asked. 

That evaluation requires careful scrutiny of what the data capture, how they were generated, where they are incomplete, and which assumptions are likely to fail if left unexamined. Our researchers do not begin by running models or pulling cohorts. They begin by interrogating the data itself, assessing longitudinal depth, availability of exposure and outcome data, assessment of viable cohort, exposure, and outcome phenotypes, key stratification variables, cohort definitions, and consistency with clinical expectations. 

If data constraints prevent defensible inference, the answer is not to force a result or smooth over uncertainty. The answer is to say so. If you cannot explain your data, you cannot defend your evidence. That discipline is precisely what makes evidence able to withstand regulatory review, peer scrutiny, and real-world decision-making without unraveling. 

The Standard Has to Rise

The conversations that will be happening at ISPOR Europe are important ones. Shaping the future of access through science and policy requires evidence that regulators, payers, and clinicians can rely on. That will not happen by improving data infrastructure alone, though infrastructure matters. It will not happen by deploying more sophisticated analytic tools alone, though those matter too. 

It will happen when research teams consistently spend more time interrogating data than analyzing it. When consideration of the question, the design, the data, and the implementation are treated as a foundational critical process for conducting robust research.  

The infrastructure, methodology, and expertise needed to produce trustworthy evidence exist today. What is needed is a community-wide commitment to applying them consistently, regardless of how quickly technology can generate an answer. 

This post is adapted from TriNetX’s new eBook, The Evidence Trust Crisis: Why More Data, Faster Analytics, and Easier Access Are Accelerating Output While Eroding Trust. 

About Jeffrey Brown, PhD 

With more than 25 years of experience in research and consulting, Jeff is an internationally recognized expert in the use of RWD to support the evidentiary needs of regulatory agencies and medical product sponsors and an expert in the assessment of data quality of RWD resources.