Most Literature Reviews Fail Before They Start

Written by HATEM RABEH, MD, MSc Ing

Your Clinical Evaluation Expert And Partner

in
S

The problem is not finding clinical data. The problem is knowing what to do with it once you find it.

Most manufacturers treat literature review as a collection exercise. They search databases, download PDFs, fill tables, and count publications. They assume that volume signals rigor. It does not. Rigor comes from appraisal.

Appraisal means separating signal from noise. It means determining which data is credible, which is relevant, and how much weight it carries in your clinical evaluation. Without this, your CER becomes a library with no index.

And reviewers know the difference immediately.

What Appraisal Actually Means Under MDR

MDCG 2020-13 defines clinical data appraisal as the structured assessment of data quality, relevance, and contribution to the demonstration of safety and performance. This is not editorial commentary. This is analytical work.

Data quality refers to the study design, conduct, and reporting. You assess bias, statistical power, and adherence to recognized methods. A poorly designed study does not become useful because it supports your device.

Relevance refers to alignment with your device, indication, population, and clinical context. A study on a similar technology in a different patient group may not be relevant. Similarity is not equivalence.

Weight refers to the contribution the data makes to your clinical evaluation conclusion. High-quality, directly relevant data carries more weight. Lower-quality or indirectly relevant data may still contribute, but it cannot stand alone.

These three dimensions interact. A high-quality study with low relevance does not carry significant weight. A highly relevant study with poor quality introduces uncertainty. Appraisal requires you to assess all three, not just one.

Where Appraisal Breaks Down in Practice

The first breakdown happens during literature search. Teams search broadly, which is correct, but then they fail to filter systematically. They either include everything or they cherry-pick studies that support their device.

Neither approach survives scrutiny.

Including everything without appraisal creates a data dump. The reviewer sees 40 studies in your table and asks: why are these here? What do they contribute? How did you assess them? If you cannot answer clearly, the CER is deficient.

Cherry-picking is worse. Reviewers compare your included studies with the search results. If relevant studies are excluded without justification, it raises questions about bias. Clinical evaluation requires objectivity, not advocacy.

The second breakdown happens when manufacturers confuse citation with appraisal. They list studies, summarize findings, and move on. But summarizing is not appraising. Appraisal requires you to critically assess the study’s validity, applicability, and limitations.

MDCG 2020-6 is explicit about this. You must assess the study design, patient population, endpoints, follow-up duration, statistical methods, and risk of bias. You must explain how the study findings apply to your device and your intended use.

This takes time. It requires clinical and statistical understanding. It cannot be automated or templated.

Assessing Data Quality: The Hard Questions

Quality assessment starts with study design. Randomized controlled trials generally rank higher than case series, but design alone does not determine quality. A poorly conducted RCT is less reliable than a well-conducted cohort study.

You assess the methods. Was randomization adequate? Was blinding used? Were endpoints pre-specified? Was follow-up complete? These are not formalities. They determine whether the results are trustworthy.

Statistical analysis matters. Was the sample size justified? Were confidence intervals reported? Was multiplicity addressed? Many published studies have statistical flaws that limit their reliability. You must identify these.

Risk of bias is central. Tools like the Cochrane Risk of Bias tool or ROBINS-I exist for this reason. They guide systematic assessment across domains: selection bias, performance bias, detection bias, attrition bias, reporting bias. You apply these tools, not just mention them.

But here is what trips manufacturers up: they assess quality in isolation. They evaluate whether a study is well-designed without asking whether it answers a question relevant to their device.

Quality without relevance does not help you.

Assessing Relevance: The Clinical Logic

Relevance is where clinical judgment becomes critical. You must determine whether the study population, device characteristics, indication, and outcomes align with your device.

Population relevance includes age, disease stage, comorbidities, and treatment history. A study in young, healthy volunteers may not be relevant for elderly patients with multiple conditions. You explain why, or you do not include it.

Device relevance includes technology, design, materials, and mechanism of action. A study on a predicate device may be relevant if equivalence is demonstrated. But if the devices differ in critical characteristics, relevance decreases.

Indication and use environment matter. A study conducted in a controlled hospital setting may not reflect real-world use in primary care. You assess this and explain the implications.

Outcome relevance is often overlooked. The study may measure surrogate endpoints, but your device requires clinical outcomes. Or the study tracks short-term results, but your device is used long-term. You address these gaps explicitly.

I have seen CERs where manufacturers include studies on completely different devices because the technology category is similar. This does not work. Relevance requires alignment on multiple dimensions, not just one.

Assigning Weight: The Integration Step

Weight reflects the contribution the data makes to your clinical evaluation conclusion. High-quality, highly relevant data carries the most weight. It directly supports your safety and performance claims.

Moderate-quality or moderate-relevance data contributes but cannot stand alone. It supplements stronger evidence. You acknowledge its limitations and explain its role.

Low-quality or low-relevance data may provide context but does not support primary claims. You use it cautiously, if at all.

The mistake manufacturers make is treating all included studies as equal. They list them in a table, summarize findings, and assume the reviewer will integrate them. This does not happen.

You must integrate the data yourself. You explain how the studies collectively support your conclusions. You identify consistencies and inconsistencies. You address gaps and limitations. You build the argument.

This is synthesis, not summary. Synthesis requires clinical reasoning and regulatory understanding. It is the difference between data collection and clinical evaluation.

When I review a CER, I look for the synthesis section. If it is missing or superficial, the entire evaluation is weak. Because without synthesis, you have not evaluated. You have only listed.

The Role of Appraisal in Equivalence Claims

Appraisal becomes even more critical when you rely on equivalence to a predicate device. MDCG 2020-6 requires you to demonstrate clinical, technical, and biological equivalence. But equivalence is not claimed. It is demonstrated through data.

You appraise the clinical data on the predicate device. You assess its quality, relevance, and weight. You determine whether the data is sufficient to support claims for your device.

If the predicate data is limited or low-quality, equivalence does not transfer those limitations into strengths. You cannot leverage weak data just because the devices are equivalent.

I have reviewed equivalence-based CERs where the manufacturer claimed equivalence to a device with minimal published data. The logic was: the devices are equivalent, so we inherit the market history as evidence. But market history is not clinical data. It does not demonstrate safety and performance with the rigor required by MDR.

Appraisal forces you to confront this. It reveals whether your equivalence strategy is viable or wishful thinking.

Practical Appraisal: What This Looks Like

Effective appraisal is documented clearly. For each study, you include:

A summary of the study design, population, intervention, and outcomes. This is factual and concise.

An assessment of quality using recognized tools. You score or rate the study across bias domains. You explain the rating.

An assessment of relevance. You compare the study characteristics to your device, indication, and population. You identify alignments and differences.

An assignment of weight. You state how much this study contributes to your clinical evaluation. You justify the weight based on quality and relevance.

This structure is transparent. The reviewer sees your reasoning. They may disagree with your conclusions, but they understand your method. That is what matters.

Appraisal tables are useful, but only if they contain the right information. I have seen tables with 20 columns of metadata and zero columns for critical appraisal. This misses the point.

The columns that matter are: quality score, relevance justification, and weight assignment. These columns show that you have thought critically about the data.

What Happens When Appraisal Is Missing

Without appraisal, your CER lacks structure. The reviewer reads your literature section and asks: why did you include this? Why did you exclude that? How do these studies support your conclusion?

If you cannot answer these questions clearly, the CER is deficient. The deficiency is not about the data. It is about the analysis.

Notified Bodies are trained to assess appraisal. They look for systematic methods, transparent reasoning, and justified conclusions. If your CER lacks these, it gets rejected regardless of how many studies you included.

I have seen CERs with extensive literature reviews fail because the appraisal was weak. The manufacturer spent months collecting data but days analyzing it. This imbalance shows in the document.

Reviewers do not need volume. They need clarity. They need to see that you understand the data, its limitations, and its contribution to your clinical evaluation argument.

Appraisal provides that clarity. Without it, you are asking the reviewer to trust you. Trust is not a regulatory standard.

Moving from Data to Argument

Appraisal is the bridge between data collection and clinical evaluation conclusion. It transforms raw information into structured evidence. It turns a list of studies into a reasoned argument.

This is not optional under MDR. MDCG 2020-13 explicitly requires appraisal. Article 61(5) requires clinical evaluation to be based on sufficient clinical data. Sufficiency is determined through appraisal.

Manufacturers who skip this step are not cutting corners. They are building on sand. The structure looks solid until someone examines the foundation.

Reviewers examine the foundation.

In the next part of this series, we will address how to integrate appraised data into the state of the art analysis. Because knowing the data is one thing. Knowing where your device stands relative to alternatives is another.

Appraisal separates signal from noise. Integration separates your device from the rest.

Peace,
Hatem
Clinical Evaluation Expert for Medical Devices
Follow me for more insights and practical advice.

Literature review is a foundational CER component. Master the full CER process in our CER guide under MDR.