COSMIN Risk of Vias Checklist Navigation Guide

Find the right COSMIN appraisal pathway before you judge the study

The difficult part of COSMIN risk-of-bias assessment often begins before a reviewer reaches the first standard. A paper may call an analysis “reliability” when it actually reports measurement error, describe comparison with another questionnaire as “criterion validity” when no accepted gold standard exists, or report several measurement properties in one article. The first job is therefore to identify what was actually assessed and decide which current COSMIN pathway applies. The structured navigation record below helps document that decision without reproducing the official checklist or calculating the final methodological-quality rating.¹²

Why a navigation layer is useful

COSMIN distinguishes an article from a study. An article is the published paper. A study, for COSMIN purposes, is an assessment of a measurement property. One article can therefore contain several separate studies: structural validity and internal consistency may be evaluated in the same sample, while reliability and measurement error may be evaluated in another part of the same paper. If a measurement property is reported separately for distinct subgroups and the design or evidence differs, separate appraisal may also be needed.²

That distinction changes the way a review record should be built. A single row labelled with the article citation and one global “quality” judgment is often too coarse. The reviewer needs to know which PROM version, scale or subscale, population or subgroup, measurement property and analysis the methodological judgment refers to. COSMIN’s current manual explicitly describes measurement-property assessment as modular: the properties actually evaluated determine which of the ten Risk of Bias boxes are relevant.¹²

The problem becomes more difficult because authors do not always use COSMIN terminology. A paper’s label should therefore be treated as a clue rather than the final classification. The current manual advises reviewers to examine the design and analysis and map them to the COSMIN taxonomy. It gives examples of limits-of-agreement analyses described by authors as reliability but treated by COSMIN as measurement error, and comparisons with another PROM described as criterion validity but usually treated as hypotheses testing for construct validity when the comparator is not a genuine gold standard.²

What a navigation record should capture, and what remains with COSMIN

A reproducible navigation record captures the facts needed to reconstruct a later judgment. These include the exact PROM version, relevant scale or subscale, study population, property under investigation, measurement model where relevant, study design, statistical approach, information that could not be verified, and the reasoning of independent reviewers. It should also identify the corresponding COSMIN box so that reviewers can open the current official checklist rather than rely on memory or an old secondary summary.

The navigation record should not reproduce the individual COSMIN standards. That boundary is both methodological and practical. COSMIN updates its materials, and its current checklist is the authority for the wording and available response options. Reproducing every standard locally would create a risk of using outdated wording and could encourage reviewers to treat an unofficial copy as authoritative. The official checklist dated 27 August 2024 identifies itself as version 3 and directs users to the current systematic-review manual for application guidance.¹

The navigation record also should not judge whether an observed coefficient or model-fit statistic shows a good measurement property. COSMIN distinguishes standards, which concern study design and preferred statistical methods, from criteria, which are used later to judge whether a measurement-property result is sufficient, insufficient or indeterminate.² Mixing those two stages is one of the easiest ways to turn critical appraisal into an invalid composite judgment.

How to route the ten current COSMIN pathways

The structured record above names the ten boxes in the current COSMIN Risk of Bias Checklist without reproducing the standards inside them. The notes below explain what kind of study question normally brings each pathway into scope. They are orientation rules, not substitutes for the official checklist.¹²

Three decisions deserve extra documentation

Some routes are straightforward. Others depend on judgments that should be written down because a second reviewer may reasonably reach a different conclusion. The first is the measurement model. COSMIN distinguishes reflective instruments, where items are treated as manifestations of an underlying construct, from formative instruments, where components collectively form the construct. Structural validity and internal consistency are primarily meaningful under a reflective model. When the model is unclear, the current manual recommends retaining the evidence and evaluating those properties on the possibility that the model is reflective rather than simply declaring the studies irrelevant.²

The second difficult decision is criterion validity. For most patient-reported constructs there is no unquestioned gold standard. The current COSMIN manual gives limited examples in which a comparator may be treated as a gold standard, such as using a long version when evaluating its shortened form or using the patient-completed PROM as the reference when evaluating a proxy version. A study that compares two different questionnaires measuring a similar construct will usually belong under hypotheses testing for construct validity instead.²

The third is the line between reliability and measurement error. Both can use repeated observations, and the same article may evaluate both. Reliability concerns how well people can be distinguished from one another despite measurement error; measurement error concerns the absolute amount of error in the score itself. The 2020 COSMIN work on reliability and measurement error reinforces the need to match design and analysis to the measurement question rather than to the terminology chosen by the authors.⁵

Worked example: one paper, two different appraisal records

How to handle missing information without manufacturing certainty

Risk-of-bias appraisal often reaches a point where the paper does not report enough detail to tell whether a methodological requirement was met. That is different from evidence that the method was definitely poor. The distinction should remain visible in the working record. Use the missing-information field to state exactly what cannot be verified: the interval may be reported without evidence that participants remained stable, an ICC may be given without enough information to understand the model, or an analysis may be named without describing how groups were defined.

Where clarification could materially change the appraisal, contacting study authors can be reasonable. Record the question and response rather than silently filling the gap with an assumption. If the information remains unavailable, use the response options and instructions in the current COSMIN box. The documentation step should not decide how missing reporting changes a particular standard because that decision belongs to the official appraisal method.

Independent reviewers are part of the method

Current COSMIN guidance recommends that two reviewers assess risk of bias independently and come to consensus, consulting a third reviewer when needed. It also suggests that reviewers may benefit from practising on a few studies and discussing their ratings before the full appraisal begins.² The latter is a useful calibration step, not a fixed requirement that every review must pilot an arbitrary number of papers.

The reason for keeping two independent rationale fields is not bureaucratic. COSMIN appraisal includes judgments about study design, terminology and appropriateness of statistical methods. Earlier empirical work on the original COSMIN checklist found that agreement and reliability varied considerably across items, supporting the practical value of training, discussion and transparent consensus processes.⁶ A consensus note should therefore preserve why a judgment changed, not merely replace two different answers with a third answer.

A completed navigation record is an audit trail, not a certificate

The value of a structured navigation record lies in traceability. It makes it possible to reconstruct why a paper was routed to a particular COSMIN box, which study facts were considered, where information was missing and how two reviewers arrived at consensus. That is different from automating critical appraisal. COSMIN’s four study-quality ratings (Very good, Adequate, Doubtful and Inadequate) are determined by applying the relevant standards, with the overall box rating based on the lowest applicable standard under the “worst score counts” approach. The current manual also explains that response options were designed so that serious flaws drive the strongest downgrades; methodological strengths in one part of a study do not simply compensate for a fatal weakness elsewhere.²³

That final appraisal remains a human methodological judgment. The navigation record helps reviewers reach it in a controlled way and leaves enough information for another member of the review team to understand what was done. It also keeps the appraisal ready for the next COSMIN stages, where study quality is considered alongside the measurement-property results and later contributes to certainty assessment. Those later result-extraction and synthesis tasks are methodologically separate from the risk-of-bias navigation stage.