Setting Acceptance Criteria From Your SOTA Analysis

Written by HATEM RABEH, MD, MSc Ing

Your Clinical Evaluation Expert And Partner

in
S

Acceptance criteria define the measurable thresholds your device must achieve. They appear in your Clinical Evaluation Plan and are validated in your Clinical Evaluation Report. They determine whether your evidence demonstrates conformity with safety and performance requirements. Get them wrong, and your entire clinical evaluation strategy fails.

The most common mistake is setting criteria without reference to SOTA. Teams pick round numbers that feel reasonable. They copy criteria from similar products without understanding the context. They set targets based on what they hope to achieve rather than what the clinical context requires.

What Regulatory Guidance Requires

MDCG 2020-1 requires software to demonstrate valid clinical association, analytical performance, and clinical performance, with evidence tailored to device risk and intended population. MDR Annex I mandates that acceptance and performance criteria are defined up front, including reliability, accuracy, robustness, and safety under normal and foreseeable misuse.

The key phrase is “defined up front.” You cannot evaluate evidence without knowing what success looks like. And you cannot define success without understanding the clinical context. This is why SOTA analysis must precede acceptance criteria definition.

The Four-Step Criteria Method

Step 1: Identify the key performance metrics aligned with your intended use. What does success look like for this device? Sensitivity? Specificity? Time to result? Usability measures? List every metric that matters for your clinical claims.

Step 2: Benchmark against state-of-the-art competitors and published literature. What do current solutions achieve for each metric? What does the literature establish as clinically meaningful thresholds? Build a table of current performance levels.

Step 3: Set thresholds based on risk analysis and clinical requirements. Where does your device need to match current standards? Where does it need to exceed them? Where is acceptable performance lower because your device addresses a different need?

Step 4: Validate that criteria are measurable and achievable. Can you actually measure these metrics with available data? Is there a realistic path to achieving these thresholds? Criteria that cannot be measured or achieved are useless.

Real-World Examples

Strong acceptance criteria look like this:

Sensitivity at least 90% and specificity at least 85% on external datasets with 500 or more patients. AUROC at least 0.90 on benchmark data and at least 0.85 on real-world data. Performance degradation no more than 5% moving from clean to noisy data. Subgroup thresholds met across demographic categories. Latency no more than 2 seconds per case. No more than 1 serious user error per 100 operations.

Notice the specificity. Not just “high sensitivity” but a number with a dataset requirement. Not just “fast” but a measurable time threshold. Each criterion is tied to how it will be verified.

Risk-Based Scaling

CORE-MD recommends scaling evidence depth and criterion strictness according to harm risk. Higher risk requires tighter thresholds and more pre-market data. Lower risk may allow wider tolerances with post-market monitoring.

Your acceptance criteria should reflect this scaling. A diagnostic device for a life-threatening condition needs tighter sensitivity thresholds than a wellness device. A device used in emergency settings needs faster response time criteria than one used in routine screening.

Document the risk reasoning behind each threshold. Why is 90% sensitivity the right target? Because lower sensitivity in this clinical context could result in missed diagnoses with specific harm potential. This reasoning demonstrates that criteria are derived, not arbitrary.

Criteria for Different Evidence Types

Different evidence types require different criteria formats:

For clinical performance, define outcome thresholds with confidence intervals. For analytical performance, define technical specifications with tolerance ranges. For usability, define error rates and task completion metrics. For safety, define acceptable adverse event frequencies relative to clinical benefit.

Each criterion type connects to a specific evidence generation activity. Clinical performance criteria drive clinical study design. Analytical criteria drive verification testing. Usability criteria drive human factors studies.

Documentation Requirements

Your CEP should contain a claims-to-metrics mapping table showing each claim, its associated metric, target value, floor threshold, evidence pillar, GSPR reference, dataset, analysis method, and decision rule.

This table makes criteria traceable and auditable. Reviewers can see exactly what you will measure, how you will measure it, and what results you will accept. No ambiguity, no interpretation required.

In the next post, we will address the often-neglected Step 4 of SOTA analysis: documenting risks, limitations, and uncertainties honestly and constructively.

Peace,
Hatem
Your Clinical Evaluation Partner