Why your PMCF survey returns empty noise instead of data

Written by HATEM RABEH, MD, MSc Ing

Your Clinical Evaluation Expert And Partner

in
S

The moment you write your first PMCF survey question, you decide whether your post-market surveillance will generate regulatory evidence or empty reassurance.

Most PMCF surveys fail not because they ask too few questions, but because they ask questions that cannot produce actionable clinical data. They collect responses. But responses are not the same as valid evidence.

This matters because under MDR Article 61 and the requirements detailed in MDCG 2020-7, your PMCF must generate data that confirms or challenges your benefit-risk determination. Surveys that produce unusable responses do not fulfill that obligation. They create documentation burden without regulatory value.

Let me show you what separates survey designs that pass review from those that create deficiency loops.

The structural problem most surveys carry

A typical PMCF survey for a Class IIb device starts with questions about satisfaction. Did you find the device easy to use? Would you recommend it to others? Rate your experience from 1 to 5.

These questions feel intuitive. They mirror patient satisfaction surveys used in clinical practice. But they do not address the regulatory question your PMCF must answer.

The regulatory question is not whether users are satisfied. The question is whether the clinical performance claimed in your technical documentation remains valid under real-world conditions, and whether residual risks remain acceptable when normal use introduces variability you could not fully simulate in your clinical investigation.

I see this pattern in roughly sixty percent of PMCF plans submitted for initial review. The survey exists. Questions exist. But the connection between what you ask and what you claimed in your clinical evaluation is missing.

When that connection is missing, you cannot use the responses to update your benefit-risk profile. You collect opinions, not evidence.

The chain that makes survey data regulatory-valid

Valid PMCF survey design begins before you write any question. It begins with three documents open in front of you: your clinical evaluation report, your risk management file, and your IFU.

From your CER, extract the specific performance claims. Not general statements. Specific measurable claims about what your device does, under what conditions, with what expected outcome.

From your risk management file, extract the residual risks you accepted and the assumptions you made about user behavior, clinical environment, or patient population that might not hold perfectly in post-market reality.

From your IFU, extract the instructions that depend on user interpretation or clinical judgment.

Now you have your survey scope. Every question you write must trace back to one of these three sources. If a question does not link to a performance claim, a residual risk, or an instruction requiring validation, remove it.

The question structure that generates usable responses

Most survey questions fail because they are too broad or too leading. Consider this common question: “Did the device perform as expected?”

What does “as expected” mean to the respondent? Expected based on their prior experience with similar devices? Expected based on what they read in the IFU? Expected based on what the sales representative told them?

The question cannot be answered consistently because the reference point is undefined. Responses will vary not based on device performance but based on individual interpretation of the question.

Now consider the alternative structure. You claimed in your CER that your suturing device reduces procedure time by fifteen percent compared to manual suturing in laparoscopic procedures. Your residual risk file identifies potential user error if the device is loaded incorrectly.

Your survey questions become:
– In what percentage of procedures did you complete the suturing step in less time than your typical manual suturing approach?
– How many times per procedure did you need to reload the device due to initial loading error?
– What clinical situations made you choose manual suturing instead of using the device?

These questions are specific. They reference measurable outcomes. They can detect deviations from your claimed performance. And critically, they can reveal use errors or clinical conditions you did not anticipate in your risk analysis.

The response format that preserves interpretability

Multiple choice questions feel safer. They are easier to analyze. They produce clean quantitative data. But they also constrain responses in ways that hide unexpected findings.

When you design multiple choice options, you are pre-defining what answers are possible. If reality falls outside your options, the respondent either skips the question or selects the closest approximation. Both outcomes corrupt your data.

I reviewed a PMCF survey for a continuous glucose monitoring device. One question asked: “How often do you calibrate the device?” The options were: Once daily, twice daily, three times daily, more than three times daily.

The survey showed ninety percent compliance with the IFU recommendation of twice daily calibration. But during user interviews conducted separately, we discovered that twenty percent of users were calibrating incorrectly, using capillary blood samples taken at times when glucose was rapidly changing. They were calibrating twice daily as instructed, but the calibration was clinically invalid.

The multiple choice question measured frequency but missed the quality of the action. The residual risk of incorrect calibration remained undetected until we added open-ended follow-up.

The solution is not to eliminate multiple choice questions. It is to pair each closed question with an open field that asks: “If you selected ‘other’ or if your experience differed from these options, please describe.”

This hybrid structure gives you quantitative trends while preserving the ability to detect signals you did not anticipate.

Sample size and the illusion of statistical significance

I regularly see PMCF plans that specify a target sample size of fifty responses. When I ask why fifty, the answer is usually: “It seemed reasonable.”

Sample size for a PMCF survey is not about what seems reasonable. It is about statistical power to detect a meaningful change in the parameter you are measuring.

If your claimed performance is that your device reduces surgical site infection by three percentage points compared to standard of care, and your clinical investigation showed infection rates of two percent, you need enough responses to detect if that rate increases to five percent in real-world use.

Fifty responses cannot reliably detect that difference. You need several hundred, depending on your confidence intervals and the baseline event rate.

But here is the regulatory reality. Most manufacturers cannot collect several hundred responses. The user base is too small, the response rate is too low, or the device is used infrequently.

This does not mean your PMCF survey is impossible. It means you must acknowledge the limitation explicitly in your PMCF plan and compensate with other data sources. Surveys become one input among several: complaint data, returned product analysis, literature surveillance, registry data if available.

Timing and the decay of recall accuracy

When you send a survey matters as much as what you ask. Most PMCF surveys are distributed annually or at fixed intervals that align with regulatory submission cycles, not with the user experience timeline.

If your device is used in a single acute procedure, sending a survey six months later means you are asking the respondent to recall details that have faded. If your device is used continuously over months, sending a survey after one week means you miss long-term performance drift or late-onset complications.

The timing of your survey must match the temporal characteristics of the risks and performance parameters you are monitoring. If you claimed immediate procedural benefits, survey immediately post-procedure. If you accepted residual risks that manifest over time, survey at intervals that match the risk timeline.

I worked with a manufacturer of implantable sensors. Their claimed performance included stability over ninety days. Their PMCF survey was sent at day thirty. By design, it could not detect the performance parameter they claimed. The survey schedule was built around administrative convenience, not clinical evidence needs.

We restructured to survey at day seven for immediate post-implant issues, day thirty for early-phase performance, and day ninety for the claimed long-term stability. Each survey had different questions mapped to the risks and claims relevant at that timepoint.

The validation step most plans skip

Before you deploy your PMCF survey to your full user base, test it with five users. Not a formal validation study. A structured pilot where you sit with the respondent as they answer each question.

Watch where they hesitate. Ask them to explain their answer in their own words. Identify where your question language does not match their clinical vocabulary.

I have seen survey questions completely misunderstood because the manufacturer used technical terminology from their IFU that clinicians never use in practice. The device was called a “tissue approximation system” in regulatory documents. Surgeons called it a stapler. Questions using the formal term were answered inconsistently because respondents were not sure the question applied to the device they were using.

This pilot step takes two hours. It prevents months of collecting responses that cannot be interpreted.

What to do when response rates collapse

Even a perfectly designed survey fails if no one completes it. Response rates for medical device PMCF surveys average between eight and fifteen percent. You can increase that slightly with reminders and incentives, but you will never reach universal participation.

Low response rate creates bias. Users with negative experiences are more motivated to respond. Users for whom the device works unremarkably do not respond. Your survey data skews toward problems even if overall performance is acceptable.

You cannot eliminate this bias, but you must quantify it. Compare the demographic and use characteristics of respondents versus your known user base. If ninety percent of your users are in hospital settings but eighty percent of respondents are from ambulatory clinics, your data is skewed.

Declare this limitation in your PMCF report. Explain how you weighted the interpretation of results to account for the skew. Show that you cross-referenced survey findings with complaint data and other sources that capture the non-respondent population.

Notified Bodies accept that perfect response rates are unrealistic. They do not accept pretending the bias does not exist.

The final check before distribution

Before you send your survey, perform this test. Take each question and ask: “If every respondent answers this question identically, does that confirm my device still performs as claimed, or does it just mean the question was not sensitive enough to detect deviation?”

If uniform positive responses do not actually validate your claim, the question is not regulatory-useful.

Then ask the inverse: “If twenty percent of respondents give an unexpected answer, do I know what action that triggers in my risk management process?”

If an unexpected response does not connect to a decision point in your post-market surveillance system, the question generates noise, not actionable data.

Every question you keep must pass both tests.

Moving from survey design to evidence

Survey design is not the end goal. It is a tool to generate one stream of evidence that feeds your ongoing clinical evaluation.

The quality of that evidence depends entirely on whether your questions are precise enough to validate specific claims and sensitive enough to detect deviations from your assumptions.

Most PMCF surveys fail not because they are poorly written, but because they were designed in isolation from the clinical and risk documents they are supposed to validate.

When you build traceability first, write questions that reference measurable parameters, test your survey language with real users, and acknowledge limitations transparently, your survey stops being a compliance checkbox and becomes a valid evidence source.

That is the difference reviewers look for.

Peace,
Hatem
Clinical Evaluation Expert for Medical Devices
Follow me for more insights and practical advice.