Selection Bias: The Hidden Flaw in Your Literature Search

Written by HATEM RABEH, MD, MSc Ing

Your Clinical Evaluation Expert And Partner

in
S

This is not an isolated case. Selection bias in literature screening is one of the most frequent deficiencies I encounter in clinical evaluation work. It undermines the entire clinical evidence base, triggers major non-conformities, and delays market access.

The problem is rarely intentional. Most teams believe they are conducting objective literature reviews. But the way the search is structured, how inclusion criteria are applied, and how exclusions are justified often introduces systematic bias that becomes visible only under regulatory scrutiny.

What Selection Bias Actually Means in Clinical Evaluation

Selection bias occurs when the process of identifying and selecting literature systematically favors certain types of evidence while excluding others in ways that distort the overall picture of clinical safety and performance.

Under MDR Annex XIV Part A and MDCG 2020-5, clinical evaluation must be objective and based on a systematic review of all available clinical data. The word “systematic” is critical. It means the method must be transparent, reproducible, and free from subjective filtering that skews results toward predetermined conclusions.

Selection bias manifests in multiple ways:

• Search strategies designed too narrowly to capture only favorable outcomes
• Inclusion and exclusion criteria applied inconsistently across studies
• Exclusion rationales that reject unfavorable data on subjective grounds
• Over-reliance on manufacturer-sponsored studies while excluding independent research

The result is a clinical evaluation that appears comprehensive but actually presents a distorted view of the evidence landscape.

Where Selection Bias Enters the Process

Selection bias does not usually start during the screening phase. It starts earlier, during search strategy design.

I have seen search protocols where the inclusion criteria are written so narrowly that only positive device-specific studies could possibly qualify. The team then conducts the search, finds exactly what the criteria allow, and declares the review complete.

On paper, the process looks systematic. In practice, it was designed to produce a predetermined result.

Search Strategy Design

The first opportunity for bias is in how search terms and databases are selected. If the search strategy is built around very specific device features or intended uses, it may exclude broader clinical evidence relevant to the technology class, mechanism of action, or patient population.

For example, searching only for the proprietary device name and trademarked features will miss comparative studies, systematic reviews of the device category, and safety signals reported in broader literature.

This is a particular problem for devices with equivalence claims. If you search only for your own device, you will not capture the full evidence base for the equivalent device, which is precisely what you need to support the equivalence argument.

Inclusion and Exclusion Criteria

Criteria must be defined before the search begins, and they must be applied consistently. But I regularly see criteria that function as filters for inconvenient data.

Common examples include:

• Excluding case reports or post-market data unless they show efficacy, but including them when they describe complications
• Requiring randomized controlled trials for safety endpoints but accepting lower-quality evidence for performance claims
• Excluding studies with small sample sizes selectively, while retaining small favorable studies

The inconsistency is the problem. If a criterion is valid, it must apply to all studies regardless of their findings.

The Exclusion Log

The exclusion log is where bias becomes most visible. This document lists every study identified in the search but not included in the appraisal, along with the reason for exclusion.

Notified Bodies read this document carefully. They look for patterns.

If all excluded studies with negative findings share the same exclusion rationale, but that rationale is not applied to positive studies with the same characteristics, the bias is obvious.

I have seen exclusion logs where every study reporting adverse events was excluded for “insufficient detail,” while every study reporting successful outcomes with equally limited detail was included. The reviewer does not need to read the full studies to see the problem. The pattern in the log is enough.

Why This Happens

Most selection bias is not deliberate. It emerges from cognitive bias, time pressure, and misunderstanding of what clinical evaluation is meant to achieve.

Teams often approach literature review as a task of finding evidence to support the device. The mindset is: “We need data to demonstrate safety and performance.” So the search becomes a mission to locate that data, and anything that does not serve that mission feels like noise.

But clinical evaluation is not about supporting the device. It is about assessing the device. Those are not the same thing.

Assessment requires examining all available evidence, including evidence that raises questions, suggests risks, or indicates limitations. The goal is an accurate understanding of benefit-risk balance, not a collection of favorable studies.

When the starting assumption is that the device is safe and effective, and the literature review is conducted to confirm that assumption, selection bias is almost inevitable.

How Reviewers Detect Selection Bias

Notified Bodies and competent authority reviewers use several methods to identify selection bias during CER assessment.

First, they reconstruct the search. They take your search terms, databases, and date ranges, and they run the search themselves or evaluate whether the number of results you report is plausible.

If your search terms were highly specific and your result count is suspiciously low, they will question whether the search was designed to limit findings.

Second, they examine the flow from initial results to final appraisal. A large drop-off between identified studies and included studies is not inherently problematic, but it triggers scrutiny. They will read the exclusion log in detail.

Third, they look at the balance of findings. If every included study is favorable and every excluded study is unfavorable or neutral, the pattern suggests bias.

Fourth, they cross-reference your literature against known publications in the field. If there are widely cited studies, meta-analyses, or safety alerts relevant to your device type that do not appear in your review, they will ask why.

Finally, they assess the objectivity of your appraisal. Even if the screening process was unbiased, the critical appraisal must evaluate study quality and validity objectively. If every included study is rated as high quality regardless of actual methodological limitations, that is also a form of bias.

What Objective Literature Screening Looks Like

An objective literature review begins with a broad search strategy designed to capture all potentially relevant clinical data, not just data that supports the device.

Search terms should include the device type, clinical indications, patient populations, comparator devices, and relevant synonyms. The search should cover multiple databases, including both peer-reviewed literature and regulatory databases that capture post-market safety data.

Inclusion and exclusion criteria must be defined in advance and documented in the clinical evaluation plan or literature review protocol. These criteria should be based on objective factors like study type, population relevance, and data quality, not on whether findings are favorable.

During screening, every decision must be documented. Each excluded study should have a clear, specific rationale. Generic exclusion reasons like “not relevant” or “out of scope” are insufficient.

The screening process should be reproducible. If a different reviewer applied the same protocol, they should arrive at the same set of included studies.

Handling Unfavorable Evidence

One of the clearest tests of objectivity is how unfavorable evidence is treated.

If a study reports adverse events, complications, or performance limitations related to your device or device type, that study is relevant. It must be included and appraised.

The appraisal can evaluate the quality of the study, the applicability of the findings to your specific device, and the clinical significance of the reported issues. But the study cannot be excluded simply because the findings are negative.

This is where many clinical evaluations fail. Teams exclude negative studies during screening, which means those studies never reach the appraisal stage. The CER presents only favorable evidence, and the benefit-risk analysis is based on an incomplete picture.

When unfavorable evidence is properly included and appraised, the clinical evaluation becomes stronger, not weaker. It demonstrates that the manufacturer has considered all relevant data, assessed the risks honestly, and concluded that the benefits outweigh the risks based on a complete evidence base.

Notified Bodies trust clinical evaluations that acknowledge limitations and address risks transparently. They distrust clinical evaluations that present only positive findings.

Practical Steps to Avoid Selection Bias

Avoiding selection bias requires intentional process design and discipline during execution.

Start by developing a detailed literature review protocol before conducting the search. Document your search strategy, databases, search terms, and inclusion/exclusion criteria. Have this protocol reviewed by someone not involved in the search execution.

Use at least two independent reviewers for screening. Each reviewer applies the criteria to the identified studies independently. Disagreements are resolved through discussion and documented.

Maintain a complete exclusion log with specific rationales for each excluded study. Avoid generic reasons. If a study is excluded for methodological quality, specify the quality issue. If excluded for population mismatch, specify how the population differs.

Periodically audit your exclusion decisions. Take a random sample of excluded studies and re-evaluate whether the exclusion rationale is objective and consistently applied.

Include a section in your CER that explicitly addresses how bias was minimized. Describe the safeguards you implemented, the use of independent reviewers, and the transparent documentation of decisions.

The Regulatory Consequence

Selection bias in literature screening leads to major non-conformities under MDR Annex XIV. It undermines the clinical evaluation, which is the foundation of technical documentation.

When a Notified Body identifies selection bias, the typical response is rejection of the CER and a requirement to repeat the literature review using an objective, documented process. This can add months to the timeline and requires significant rework of the clinical evaluation and related documentation.

In some cases, selection bias raises broader concerns about the manufacturer’s quality system and approach to clinical evidence. It can trigger expanded audits and additional scrutiny of other technical files.

The reputational impact is also significant. Notified Bodies communicate with each other and with competent authorities. A manufacturer known for biased clinical evaluations will face increased skepticism in future submissions.

Final Thought

Selection bias is not about dishonesty. It is about methodology.

A biased literature review can be produced by competent, well-intentioned teams who simply did not structure the process correctly. The solution is not more effort. It is better process design and a clear understanding of what clinical evaluation is meant to achieve.

When the literature review is genuinely systematic, objective, and transparent, the clinical evaluation becomes defensible. Even if the evidence base has gaps or limitations, the assessment is credible because it is honest.

That credibility is what Notified Bodies are looking for. It is what separates clinical evaluations that pass review from those that get rejected in the first round.

And it starts with recognizing that literature screening is not about finding support for your device. It is about finding the truth about your device.

Peace,
Hatem
Clinical Evaluation Expert for Medical Devices
Follow me for more insights and practical advice.