The AI Validation Gap: What Funding and FDA Authorization Don't Show

← Back to All Insights

Healthcare AI keeps drawing venture capital. A funding round is not evidence that an algorithm performs outside the setting it was built in, and an FDA authorization on its own isn't either.

The distance between how an AI-enabled device performed in its validation study and how it performs across other hospitals, equipment and patient groups is the part of the record a diligence read has to check directly.

The Validation Gap Explained

Here's the core problem: AI/ML models are developed and validated on data that may not match the settings where they are later used. When a model is deployed in health systems with different patient populations, data quality, equipment and workflows, its performance can degrade.

The issue has three dimensions:

• Development vs. deployment: The environment where an algorithm is developed and tested is often cleaner than routine care. Deployment introduces data distribution shift, equipment variability and workflow friction the model may not have seen in training.
• Dataset bias: Training data drawn from a single health system, or skewed toward particular demographics, can produce a model that performs well on that population and worse on others.
• Generalizability: Performance measured at one institution does not reliably carry over to others. In one published study, pneumonia-detection models trained on chest radiographs performed better on internal test data than on another hospital system's images in three of five comparisons (Zech et al., PLOS Medicine, 2018).

The Real-World Data Problem

Common Diligence Question	What the Record Should Show
What's your accuracy on your validation set?	What's your performance across several independent health systems with different equipment and workflows?
Do you have FDA clearance?	Do you have peer-reviewed publications showing real-world clinical impact and patient outcome improvement?
What's your TAM in addressable markets?	What's your actual deployment penetration? Are clinicians actually using it, or does it sit on a shelf?
What's your reimbursement strategy?	Do you have actual, executed coverage agreements, or are you relying on payer projections?
How much training data do you have?	How representative is your training data? What demographic, geographic and equipment biases exist in your dataset?

Change Control Plans: What to Read

FDA's final guidance on predetermined change control plans (PCCPs) for AI-enabled device software functions, first issued in December 2024, describes how a manufacturer can have planned modifications, and the methods to develop, validate and implement them, reviewed with the original 510(k), De Novo or PMA submission. Changes made within an authorized PCCP don't need a new marketing submission; changes outside it still do.

For a diligence read, a PCCP raises questions rather than answering them:

• What modifications does the authorized plan actually cover, and what would still need a new submission?
• How will the company validate a retrained model, and on which populations and sites?
• Does postmarket performance data exist, and does it match the premarket validation?
• Has the cost of collecting new data and retraining been budgeted?

FDA's separate draft guidance on AI-enabled device software functions (January 2025) sets out what the agency recommends a marketing submission include across the product lifecycle, including how performance was validated and how it will be monitored after marketing.

Training Data Bias: The Compounding Problem

Training datasets are often built from historical data at a single institution or a few affiliated health systems. That can build in systematic bias:

• Demographic bias: datasets can overrepresent certain racial groups, age groups or socioeconomic backgrounds
• Equipment bias: models trained on one manufacturer's imaging or diagnostic equipment may perform worse on another's
• Protocol bias: protocols differ across institutions, and a model trained on one may generalize poorly to others
• Selection bias: data from academic medical centers can differ from community hospital data

Correcting for these biases can take new data collection and retraining. Whether that work appears in the company's plan and budget is worth asking.

Reimbursement: A Separate Question from Authorization

An FDA authorization does not create a payment pathway. For AI-enabled devices, the reimbursement questions often include:

• Whether an existing code covers the service, or a new code or payment policy is needed
• What evidence payers will want before covering it, and on what timeline
• Whether the pricing model (per use, subscription or bundled) fits how the buyer is paid
• What performance commitments health system customers ask for

CMS's proposed 2027 Physician Fee Schedule also takes up payment for algorithm-based services; see our article on the proposal. A company with an authorization but no coverage or payment path is at a different stage from one with both, and a diligence read that tracks only regulatory milestones can miss the difference.

The Right Questions for AI Device Diligence

On Training Data and Bias

• What is the demographic composition of your training dataset? Is it representative of the intended real-world population?
• How have you validated performance across racial and ethnic groups?
• Have you tested your algorithm on data from equipment manufacturers other than those used in training?
• What is your strategy for addressing known biases in your training data?

On Real-World Validation

• How many independent health systems have deployed your algorithm? For how long?
• What is your actual real-world performance on diverse populations, not your validation set performance?
• Have you published peer-reviewed results on real-world clinical impact?
• What is your user adoption rate and actual utilization by clinicians?

On Reimbursement

• Do you have executed coverage agreements with major payers, or are you relying on projected CPT codes?
• What is the reimbursement rate for your specific algorithm, and how does it compare to the cost of deployment?
• Have you engaged with payers on real-world evidence requirements? What is their timeline?

What a Vantage Report Reads on an AI Device

A Vantage report assembles the public and provided record behind these questions and cites each item to its source:

• Authorization record: the FDA decision or 510(k) summary, the validation study it describes (retrospective or prospective, number of sites, sample size where reported) and any authorized change control plan
• Published evidence: peer-reviewed validation and deployment studies, including subgroup results where they exist
• Data-room materials: the company's own validation, dataset composition and postmarket performance data, when provided
• Reimbursement record: existing codes, coverage policies and payment proposals relevant to the service
• Comparable devices: how similar authorized devices were validated, starting from FDA's list of AI-enabled medical devices

Each report is reviewed by a physician and a biomedical engineer. It surfaces what the record shows and what it leaves open; it does not predict how a company will perform.