Patient-reported Outcome Measure Comparison Worksheet

Choose for a defined purpose, not for an imaginary universal ranking

A PROM can be well supported and still be the wrong choice for a particular evidence-synthesis purpose. Another can be easier to administer yet have an evidence gap that matters for the intended interpretation. The useful question is therefore not “Which questionnaire has the highest psychometric score?” It is “Which candidate is best supported for the construct, population, score, language and application we actually need?”

Current COSMIN methodology makes that distinction explicit. After the measurement properties of each PROM have been evaluated, the review can conclude that a PROM can be recommended for use when there is high-quality evidence that all relevant measurement properties are sufficient, recommend against its use in the current form when high-quality evidence shows a relevant property is insufficient, or conclude that the evidence is not yet strong enough for a firm conclusion.¹² When more than one PROM is suitable, feasibility and interpretability can legitimately help determine which is most appropriate for the particular application.²

This worksheet starts at that later decision point. It assumes that study-level evidence has already been extracted, appraised and synthesized. It does not ask you to re-enter individual ICCs, factor-loading tables or COSMIN Risk of Bias judgments. Instead, it places the summarized evidence for each candidate beside the practical conditions of the decision, keeps result direction separate from certainty, and leaves the final choice to the review team.

Start with the decision you are actually trying to make

Instrument comparison becomes unreliable when the candidates arrive before the question. A PROM designed for one construct or population cannot become suitable merely because it has an impressive validation literature elsewhere. COSMIN asks review teams to define the construct, population and context of use clearly because content validity and the interpretation of measurement evidence depend on that scope.² The same PROM family may also contain a total score, several subscales, a short form and different modified versions whose evidence is not interchangeable.

The first section of the worksheet therefore records the target construct, target population, intended application and exact score or domain required. These fields are not cosmetic metadata. They tell you what “relevant measurement property” means later in the comparison. Criterion validity, for example, may not be relevant when no gold standard exists. Internal consistency is not relevant to a formative measurement model. COSMIN explicitly notes both exceptions in its current Step 7 guidance.²

Other relevance decisions can be more context dependent. A review interested in change over time will naturally care about responsiveness and the interpretation of change. A single-language application may place different practical demands on cross-cultural evidence than a multinational programme. Those judgments should be stated rather than hidden. The worksheet provides a decision-context note so reviewers can explain why a particular property or practical condition matters for this application.

What belongs in a candidate profile?

The strongest input is the end product of a rigorous PROM systematic review, not a collection of attractive individual coefficients. COSMIN separates study quality, individual study results, summarized results and certainty for good reason.² A candidate profile should therefore record the summarized direction of evidence and the certainty attached to that conclusion. If the evidence is inconsistent or indeterminate, say so. If a property has not been evaluated, use that state rather than borrowing the symbol for an indeterminate result.

The worksheet also asks for the current Step 7 conclusion when one exists. That conclusion has a stronger methodological basis than an improvised comparison score. A PROM with high-quality evidence that all relevant measurement properties are sufficient can be recommended for use. A PROM with high-quality evidence that one relevant property is insufficient should be recommended against in its current form. Everything else remains a no-firm-conclusion situation in the current COSMIN framework.¹²

There is an important nuance for reviews in which no candidate yet reaches a firm conclusion. COSMIN allows reviewers to identify one or more PROMs that have potential for use while more evidence is developed; such a PROM should have at least very-low-certainty evidence for sufficient content validity, and the review should propose a research agenda.² “Promising, but more evidence is needed” is therefore available in this worksheet as a human documentation label. It is not presented as a fourth official COSMIN category.

Content validity is not just another box to average

Content validity asks whether the PROM content is relevant to the construct and population, sufficiently comprehensive, and comprehensible as intended. COSMIN describes it as the most important measurement property and requires its assessment for every PROM in a systematic review.²⁴

The stopping condition is specific. If there is high-quality evidence that content validity is insufficient, COSMIN states that the remaining properties need no longer be evaluated because the PROM should not be recommended.² That is different from weak or incomplete evidence. Very-low-certainty evidence suggesting insufficiency is not the same state as high-certainty evidence showing insufficiency. “No study found” is different again.

For that reason, the worksheet never merges direction and certainty into one traffic-light score. It also allows content validity to be recorded as a summary while preserving notes about relevance, comprehensiveness and comprehensibility. COSMIN recommends reporting those three aspects separately and permits a total summarized content-validity rating; when reviewers consider one aspect more important than another, COSMIN recommends not creating that total summary.²

Structural validity deserves a warning, not an invented automatic ban

COSMIN’s current sequence evaluates structural validity after content validity for reflective measures. High-quality evidence of insufficient structural validity creates a serious interpretive problem because it can become unclear what the scale score represents. The manual says reviewers may decide not to continue evaluating later properties in that situation.² That language matters. It is not identical to the content-validity stopping statement.

The worksheet therefore records structural validity in the same direction-plus-certainty format as other properties and surfaces a reviewer note rather than calculating a formal exclusion. The exact score is also visible. Evidence against a total score should not automatically contaminate a separately supported subscale, and COSMIN recommends making recommendations for subconstructs separately where appropriate.²

Unknown, indeterminate and insufficient are different decisions

The current COSMIN criteria use sufficient, insufficient and indeterminate result ratings, with inconsistent ratings appearing when results are summarized across evidence sources or studies.³ But a comparison worksheet needs another operational state: “not evaluated.” That means the review has no summarized result for the property. It is not a synonym for an indeterminate result.

This distinction prevents a common selection error. A new PROM may have excellent content-validity work but no reliability study yet. That is an evidence gap. A different PROM may have high-certainty evidence that reliability is insufficient. Those two candidates should not receive the same red cell, the same number of points lost, or the same narrative conclusion.

Evidence quantity also needs careful handling. More validation papers can increase the total information and sometimes improve precision, but study count is not a quality scale. COSMIN grades certainty using risk of bias, inconsistency, imprecision and indirectness; it does not treat publication count as a substitute for those domains.²

Feasibility can legitimately determine the final choice

Feasibility is not a measurement property, but COSMIN explicitly treats it as important when selecting a PROM for a specific application. The current manual mentions patient burden, time to administer, number of items, access to the instrument, licensing and costs, administration mode, available translations and popularity, including use in studies or recommendation in core outcome sets.²

That creates two boundaries worth preserving. First, practical convenience does not turn an insufficient measurement property into a sufficient one. Second, practical information is not trivial. When two PROMs are both defensible on measurement grounds, one may be the better choice because the required language exists, administration fits the setting, interpretability is better developed, licensing is workable or respondent burden is more acceptable.

Popularity needs equally careful wording. Historical uptake, availability of reference values, recommendation in a core outcome set and use across earlier studies can be useful for implementation or comparability. None of those facts proves validity. The worksheet therefore records established use under feasibility rather than granting a psychometric bonus.

Language availability is not the same as cross-cultural validity

A translation can make implementation possible while leaving measurement equivalence uncertain. Content comprehensibility may also differ by language; COSMIN recommends assessing comprehensibility separately for language versions where appropriate.² Cross-cultural validity or measurement invariance is a separate measurement-property question.

The worksheet therefore records language availability and cross-cultural evidence in different places. It does not claim that a translated PROM is automatically validated across cultures, but it also does not impose an unsupported blanket prohibition on any synthesis involving language versions without invariance testing. The review team should document what evidence exists, what the intended comparison requires, and what uncertainty remains.

Worked comparison: two defensible PROMs and one promising candidate

A decision record should show why the alternatives lost

A transparent comparison is not complete when it names the preferred PROM. It should show what the candidate set was, what evidence was decisive, which limitations remained, and why plausible alternatives were not preferred. PRISMA-COSMIN asks systematic reviews of outcome measurement instruments to report their recommendation methods and resulting recommendations transparently.⁷

That is why the final part of this worksheet asks for trade-offs, evidence gaps, the candidate named in the decision and a written rationale. The decision label is intentionally human-entered. “No firm selection possible” is a legitimate result. So is identifying a promising PROM with an explicit research agenda when current evidence cannot support a firm recommendation.