A Diagnostic for Clinical AI Teams Whose Results Aren’t Matching the Promise

Key takeaways:

• Inconsistent AI results in clinical development almost always trace back to gaps in the data foundation. But most teams don’t have a structured way to identify exactly where those gaps are. 
• The four pillars of AI-ready data (comprehensiveness, quality, recency, and transparency) give you a specific framework for evaluating where your foundation is strong and where it’s creating drag on AI performance. 
• The full diagnostic, covering all four pillars plus AI capabilities, integration, and demonstrated outcomes, is available as a downloadable checklist you can bring to your next vendor conversation or internal review. 

The first two posts in this blog series established the argument and the stakes. The framework post, Why Two Clinical Teams Can Run the Same AI and Get Completely Different Results, explained why artificial intelligence (AI) performance in clinical development is determined by data quality, not algorithm sophistication. The smoking data post, Three Out of Four Patients Are Missing Smoking Data. That’s Not the Real Problem, showed what one version of a data quality failure looks like in practice, and what it costs. 

This post does something different. It turns the framework into a diagnostic. 

Most clin ops leaders know intuitively that their AI results have been inconsistent. Fewer know exactly which part of the data foundation is responsible, or what specific questions to ask to find out. The questions below, drawn from the full evaluation checklist available at the end of this post, are organized around the four pillars and designed to surface the gaps that most often get left unexamined until a study is already in trouble. 

Data Foundation and Provenance

The most fundamental question about any dataset isn’t what it contains. It’s where it came from and whether you can trace it. 

• Are your data sourced directly from healthcare organizations, or aggregated through third parties whose preparation methods aren’t visible to you? 
• Can every data element be traced back to its source and the transformations applied to it? 
• Are the provider relationships behind your data stable and long-term, or subject to disruption? 

What to watch for: If a data or AI vendor can’t answer provenance questions in detail, that opacity creates downstream risk: scientific, operational, and regulatory. Transparency isn’t optional for organizations building toward submission-grade real-world evidence (RWE). 

Comprehensiveness

Narrow data produce narrow insights. An AI model is only as broad as the population it’s been trained on. 

• Do your data capture patients across geographies, health systems, and care settings, or are they concentrated in a limited set of institutions? 
• Are the therapeutic areas and patient populations most relevant to your programs well represented? 
• Is your data network growing over time, and can that growth be directed toward your therapeutic priorities? 

What to watch for: Feasibility predictions that look plausible in a dashboard but collapse on contact with actual enrollment targets are almost always a comprehensiveness problem. The data didn’t capture enough of the relevant patient population to know what it was predicting. 

Quality

Comprehensiveness without quality produces confident outputs that are wrong. Healthcare data are inherently messy. The question is what’s been done about it. 

• Have data been rigorously normalized and harmonized across the different coding systems, documentation practices, and institutional standards that produced them? 
• Are cohort and endpoint definitions clinically validated, not just technically constructed? 
• Do the people responsible for data preparation include clinical and scientific experts, not just data engineers? 

What to watch for: AI outputs that look precise but don’t hold up under scrutiny (narrow confidence intervals, confident predictions that miss badly) usually reflect quality gaps. The model found patterns in the noise, not in the signal. The smoking data problem covered in the last post is one example of how this plays out: clinically meaningful information exists in the record but never makes it into the structured data feeding the model. 

Recency 

A dataset that was comprehensive and high-quality two years ago may no longer reflect the clinical reality your AI is trying to predict. 

• How frequently are your data refreshed? 

• When new data become available, how quickly do they become usable for analysis? 

• Do your feasibility predictions and site recommendations reflect current patient availability and site capabilities, or a snapshot from months ago? 

What to watch for: Site recommendations that don’t match actual patient availability at selected locations, and feasibility predictions that overshoot enrollment reality, are often recency problems disguised as algorithm problems. 

AI Capabilities Rooted in Data Quality

The questions above assess the data themselves. These assess whether the AI built on it is genuinely grounded in that data, or just claiming to be. 

• Can the vendor link AI performance improvements directly and specifically to data quality improvements? 
• When the AI generates a recommendation, can you interrogate where it came from and what data informed it? 
• Is conversational AI (natural language queries against patient population data) reproducible and reliable, or does it produce different answers to the same question asked different ways? 

What to watch for: Vendors that lead with model sophistication and deflect questions about data provenance. The question “what data are these trained on and how were they prepared?” should get a detailed, confident answer, not a pivot to a demo. 

If More of These Questions Produced Hesitation Than You Expected

That’s a useful finding, not a discouraging one. It means the drag on your AI performance has a specific diagnosis and, by extension, a specific fix. 

The sections above cover the questions that most often reveal the gaps. The full diagnostic goes further, with additional questions on data recency and provenance, AI capability grounding, system integration, future-readiness for autonomous AI, and demonstrated outcomes. It’s formatted as a one-page checklist you can bring to your next vendor conversation, RFP review, or internal data strategy discussion. 

Download the AI-Ready Data Checklist

A one-page diagnostic for evaluating RWD and AI partners across all four pillars.

Want the Full Framework Behind the Checklist?

The checklist surfaces the questions. The Real-World Data Advantage: Why Clinical Operations Teams Are Rethinking AI Strategy explains why each one matters, walks through the case studies behind the organizations getting this right, and breaks down what AI-ready data enables across protocol design, site selection, recruitment, and RWE. 

About Joshua Hartman 

As Senior Director, Clinical Study Feasibility & Analytics, Josh leads a team of clinical experts in utilizing real-world data to enhance clinical trial design, feasibility, and site identification. His work reflects a deep commitment to advancing public health through data-driven insights.