Your PRO data means nothing if the collection method is flawed

Written by HATEM RABEH, MD, MSc Ing

Your Clinical Evaluation Expert And Partner

in
S

Patient-reported outcomes have become essential in clinical evaluation, especially for devices where objective measurements tell only part of the story. Pain levels, quality of life, functional improvement—these outcomes matter to patients and often drive clinical decision-making. But here is what most manufacturers miss: the value of PRO data depends entirely on how you collect it.

I see this pattern repeatedly. Teams invest significant effort gathering patient feedback, building databases, analyzing trends. Then during Notified Body review or regulatory assessment, the data gets questioned or dismissed. Not because the outcomes were irrelevant, but because the collection method could not withstand scrutiny.

The gap between clinical intent and regulatory acceptance often lies in the fundamentals of data collection methodology.

Why PRO Data Gets Challenged

Under MDR Article 61 and Annex XIV Part A, clinical data must be based on scientifically sound methods. This applies to all forms of clinical data, including patient-reported outcomes. The regulation does not distinguish between objective measurements and subjective assessments in terms of methodological rigor.

When I evaluate PRO data in a clinical evaluation report, I look at three fundamental questions: Is the instrument valid? Is the collection process standardized? Are the results interpretable?

Most deficiencies arise because these questions were never properly addressed during study design or PMCF planning.

Here is where methodology becomes critical. PRO instruments are measurement tools just like pressure sensors or imaging devices. They must demonstrate that they measure what they claim to measure, consistently and accurately.

Validated Instruments Versus Custom Tools

The first decision point in PRO collection is instrument selection. You can use a validated instrument developed through rigorous psychometric processes, or you can develop your own tool. Each choice carries different regulatory implications.

Validated instruments—like the EQ-5D for quality of life, VAS for pain, or disease-specific questionnaires—come with established evidence of reliability, validity, and responsiveness. They have been tested across populations. Their measurement properties are documented. When you use them according to their validated protocol, the measurement method itself is not in question.

What remains in question is whether you applied the instrument correctly. Did you use the right version? Did you follow the scoring guidelines? Did you administer it at appropriate intervals? Did you account for language validation if your population differs from the original validation cohort?

I have seen manufacturers use validated instruments incorrectly—modifying questions, changing response scales, administering at non-standard time points—then claiming the instrument’s validation applies to their modified version. It does not work that way.

Custom instruments are sometimes necessary when no validated tool addresses your specific clinical question. But developing a valid custom instrument requires substantial work: item generation, content validity testing, reliability assessment, construct validity evaluation, and responsiveness studies.

Most manufacturers do not have the time, resources, or methodological expertise to properly validate a custom PRO instrument. What happens instead is they create questionnaires based on clinical judgment, use them to collect data, then struggle to defend the results during regulatory review.

If you must use a custom instrument, document at minimum: the process for item development, expert review of content validity, pilot testing results, internal consistency data, and test-retest reliability if feasible. This will not equal a full validation study, but it demonstrates methodological awareness and provides some basis for confidence in the data.

Collection Protocol and Standardization

Beyond instrument selection, the collection protocol determines whether your PRO data will be credible. Standardization is the key principle. Every aspect of data collection must be defined, documented, and consistently applied.

When should PROs be collected? The timing must be clinically justified and consistently implemented. Baseline assessment before intervention. Follow-up assessments at defined intervals. The schedule should align with expected clinical changes and allow meaningful comparison across patients.

I frequently see protocols where timing is described as “baseline and follow-up,” with follow-up ranging anywhere from three weeks to two years depending on when the patient happens to return. This creates data that cannot be pooled or compared. You are measuring different things at different recovery stages and calling it the same outcome.

How should PROs be administered? The mode of administration affects responses. Self-administered questionnaires, telephone interviews, in-person assessments—these are not equivalent. Patients respond differently when reading questions themselves versus hearing them from a healthcare provider.

The mode must be standardized across your data collection. If circumstances require mixed modes, this must be documented and the potential impact on results must be discussed. You cannot ignore it and assume equivalence.

Who administers the questionnaire matters when the mode is not self-administered. Training is essential. If multiple staff members conduct interviews or assist with questionnaires, they need standardized training on question delivery, response recording, and handling patient queries. Variability in administration introduces measurement error.

Response Rates and Missing Data

Here is a question that reveals whether your PRO collection method is robust: What is your response rate, and how do you handle missing data?

Response rates below 70-80% raise immediate concerns about bias. Who are the patients not responding? Are they the ones with poor outcomes? The ones lost to follow-up because they switched providers? The ones too satisfied to bother with questionnaires? The ones who died or experienced serious complications?

If non-responders differ systematically from responders, your PRO data is biased. And you cannot know if they differ unless you track non-responders and understand the reasons for non-response.

I review PMCF plans where the strategy is to send questionnaires and hope for responses. No follow-up protocol. No tracking of non-responders. No analysis of response patterns. When the data comes in at 45% response rate, the manufacturer presents the results as representative of all patients. This is not methodologically sound.

Missing data within questionnaires is a separate issue. If patients skip questions or entire sections, how do you score the instrument? Many validated instruments have specific rules for handling missing items. Follow those rules. If you are using a custom instrument, define your approach in advance.

Practical Implementation in PMCF

Post-market clinical follow-up is where PRO collection becomes most challenging. You are not running a controlled study. You are gathering real-world data from clinical practice. The conditions are variable. Patient populations are heterogeneous. Follow-up is opportunistic.

But the methodological principles still apply. You still need a valid instrument, a standardized protocol, and a strategy for maximizing response rates and handling missing data.

The PMCF plan must specify these elements clearly. Which PRO instrument will be used? When will it be administered? How will it be administered? Who is responsible for administration? What is the expected response rate? How will non-responders be followed up? How will the data be recorded and stored?

Then the PMCF report must demonstrate that the plan was followed. If deviations occurred, they must be documented with justification. If response rates were lower than expected, the impact on data interpretation must be discussed.

I see PMCF reports that present PRO data with no reference back to the collection method described in the plan. The reader cannot assess whether the data is credible because the methodology is not described. This creates uncertainty and invites questions during review.

Documentation for Regulatory Defense

When PRO data is included in your clinical evaluation, the documentation must allow a reviewer to assess the quality of the evidence. This means describing not just the results but the entire collection methodology.

For each PRO dataset, the clinical evaluation report should address: What instrument was used and why? What is the evidence of validity and reliability for this instrument in this population? How was the instrument administered? At what time points? By whom? What was the response rate? How was missing data handled? What are the limitations of this data?

This level of documentation allows the reviewer to evaluate the evidence quality using the framework in MDCG 2020-6. PRO data can be high-quality evidence if the methodology is sound. It can also be low-quality or unusable evidence if the methodology is weak.

The documentation determines which category your data falls into.

When PRO Data Cannot Stand Alone

Even with valid collection methods, PRO data has inherent limitations. Patient perception does not always correlate with objective clinical outcomes. Patients can feel better while underlying pathology worsens. They can report dissatisfaction despite objective improvement.

This is why PROs are typically part of a broader evidence base, not the sole evidence. They complement objective measurements, imaging, functional tests, and clinical assessments. The clinical evaluation should integrate these different data types to build a complete picture of performance and safety.

For some devices, PROs are the primary outcome of interest—devices aimed at symptom relief or quality of life improvement where objective measurements are secondary. Even in these cases, the clinical evaluation should consider objective evidence where available to validate and contextualize the patient reports.

The strength of your clinical evidence comes from multiple converging lines of evidence, each collected with appropriate methodology.

Moving Forward

If you are planning to collect PRO data—in PMCF, in clinical investigations, or from literature—the methodology must be defined before data collection begins. You cannot fix methodological flaws after the data is collected. You can only document the limitations and accept that the evidence quality is reduced.

Investing effort upfront in proper instrument selection, protocol development, and standardization will determine whether your PRO data adds value to the clinical evaluation or becomes a source of deficiencies during review.

The regulatory expectation is clear: all clinical data, including patient-reported outcomes, must be scientifically sound. The collection method is where scientific soundness is established or lost.

PRO data collected with valid methods can be powerful evidence of clinical performance from the patient perspective. PRO data collected without methodological rigor is just noise that creates more questions than answers.

Peace,
Hatem
Clinical Evaluation Expert for Medical Devices
Follow me for more insights and practical advice.