Data Alone Can’t Answer Your Question: How to Extract Meaningful Insights That Drive Impact

I recently took the podium at ISPOR 2026 to share something I’ve been thinking about for the better part of three decades: how we, as a community, can use real-world data (RWD) to generate robust and actionable evidence.  It’s really easy to get access to data and to generate “answers”. It’s much harder to produce an answer you trust. And as patients, regulators, and industry leaders increasingly rely on RWD to inform their decisions, the cost of producing poor quality findings has never been higher. 

The honest truth (and the title I originally wanted for my talk) is that the data are dumb. They don’t know anything. We do. The insight, the rigor, the trust in the result: all of it comes from the researchers who understand how to interrogate the data to generate trustworthy evidence. 

That framing matters more now than ever. RWD are no longer a “should we?” question. They’re a “how do we do this well?” question. Patients are asking us to use these data to improve public health and monitor the safety of drugs and vaccines. Regulators are leaning in. And yet, the gap between getting an answer and getting an answer you trust is growing wider. Closing that gap was the focus of my session, and it’s the focus of this post. 

Start With the Question, Not the Dataset

The single most important shift I try to encourage, whether I’m talking to a data vendor, a sponsor, or a young epidemiologist, is to stop starting with the data. Start with the question. Then ask what study design and analytic method will actually answer it. Then go to the data source and ask whether it’s appropriate for the question and the intended use. 

The research question and study design govern everything. A vaccine safety study comparing Vaccine A to Vaccine B may need an entirely different dataset than a study comparing vaccinated to unvaccinated populations, even though the two sound nearly identical. There is no such thing as “good data” or “bad data”; only data that are, or are not, fit for purpose. Sometimes the data you have can’t answer the question you want to ask. That’s okay. What’s not okay is forcing a question to a dataset that isn’t fit-for-purpose and then using the result to inform decision making. 

The Discipline of Data Characterization

I sometimes call this work “data characterization.” Others call it data quality. A former FDA Commissioner once called the people who do it “data janitors,” a phrase that drew criticism at the time, but the underlying point was right. What he was saying is that the people who understand the data deeply are the most important members on the team. They’re the ones who can sit down with a methodologist and say, “yes, this question can be answered with these data,” or, more importantly, “no, it can’t, and here’s why.” 

You should be spending more time thinking about your data than running your analysis. The analysis is the easy part. Determining whether the data can actually support that analysis is the hard part. A framework a colleague and I developed organizes this thinking around conformance, completeness, and plausibility. The U.S. Food and Drug Administration (FDA), the European Medicines Agency (EMA), The Professional Society for Health Economics and Outcomes Research (ISPOR), and the International Society for Pharmacoepidemiology (ISPE) each have their own variants; the words differ, but the discipline is the same. 

Here’s the counterintuitive part: I don’t trust a database that’s too clean. Every database I’ve ever worked with has women coded with prostate cancer. Every electronic health record (EHR) will have a patient with a BMI of 30,000 or a blood pressure of 12,000 over 8,000. We know these aren’t true. But if I don’t see them, it means someone cleaned the data, and now I don’t know what else they did. I’d rather see the raw values and all the messiness of RWD and decide as a study team how to handle the potential issues. 

Matching the Question to the Outcome to the Source

Once you’ve defined the question and the method, you have to interrogate the data source against both. This is where most RWD studies succeed or fail, long before any analysis is run. When I evaluate a dataset for a study, I run through a few questions that have become almost reflexive. Is the outcome medically attended? Migraines often aren’t. Flu may or may not be. It can depend on whether the patient gets sick on a Tuesday or a Saturday. If the outcome of interest doesn’t typically generate a medical encounter, no amount of data will give you an interpretable result. 

Is the onset of the outcome acute and the diagnostic criteria clear? Heart attacks and strokes are easier because 10 clinicians looking at the same chart will agree, and the coding for these events in RWD is well understood. Guillain-Barré syndrome is at the other extreme. I’ve watched neurologists disagree about the same patient repeatedly. The harder the diagnostic criteria, the harder the study. 

Is the exposure self-administered and medically administered? An infusion in a clinician’s office leaves a clear coded documentation trail. An over-the-counter medication may leave none. These details shape whether your study is feasible before you write a single line of code. 

I also look hard at operational characteristics: sample size, data lag, refresh cadence, and the patient population represented. Too many researchers over-index on sample size. The most isn’t always the best, and a daily-refresh feed sounds appealing until you realize hyper-fresh data are also hyper-unstable. 

EHR Data Deserve Their Own Conversation

EHR data have transformed what we can study, but they also reward (and punish) discipline. I once showed an audience a slide of every “unit of measure” we found in a single platelet field at FDA Sentinel. There were many dozens of units in the data where only one unit of measure was expected. None of that is a reason to walk away from EHR data. It’s a reason to interrogate them carefully and build your study assuming the messiness exists, characterizing the issue, and designing a study that addresses any important data issues 

This is also where time-series checking becomes non-negotiable. Group your data by month and year and look at them. If you only look at aggregate counts, you’ll miss the ICD-9-to-ICD-10 mapping that inconsistently defines an outcome definition over time, or the partner who suddenly reclassified outpatient procedures as hospitalizations. Patterns that move smoothly up and to the right are reassuring. Patterns that jump or flatten unexpectedly require a phone call. 

One Variable, Many Definitions

The other reality I keep returning to is that no variable means just one thing. A “hospitalization” might be an overnight inpatient stay in one system and an ER visit in another. A diagnosis code in a problem list is different from one captured at an encounter. A medication on a medication list is different than a prescription which is different than a fill in a claims database and different than an inpatient administration. Context shapes every variable. 

Although I am a strong proponent of Linked data, we have to be cautious about how to use it well. Claims plus EHR are a powerful combination; we spent 30 years studying cholesterol without knowing cholesterol levels, and that finally changed. But linkage isn’t magic. If I have claims data from 2018 and EHR data from 2025, I have a linked database in name only. The right question isn’t “do these data link?” but “how much meaningful overlap exists, and is it enough to answer the question?” 

The Practice of Asking More Questions

If there’s one thing I hope attendees took away from my session, and one thing I’d ask of every reader here, it’s this: ask more questions. If you’re a data vendor, ask more questions of the researchers you support. If you’re a researcher, ask more questions of your data source vendor. Worry less about what you know and more about what you don’t know.  

Write your question. Use target trial emulation. Write your protocol. Write your statistical analysis plan. Then do the work. Don’t iterate on your research after you’ve seen the outcome. Be disciplined. It takes longer, but it produces evidence that decision-makers can actually trust. 

This is the moment for the RWD community to get this right. Regulators are pushing harder than they have in a decade, and when regulators move, sponsors and vendors follow. The organizations that will lead the next chapter of this field are not the ones with the most data. They’re the ones with the most discipline about how to generation trustworthy results. That’s the work in front of us. And it’s worth doing well.

About Jeffrey Brown, PhD

With more than 25 years of experience in research and consulting, Jeff is an internationally recognized expert in the use of RWD to support the evidentiary needs of regulatory agencies and medical product sponsors and an expert in the assessment of data quality of RWD resources.