Indirect Treatment Comparison Appraisal Checklist and Evidence Table

By Dr. Esmaeel Saeedy Robat, Founder of MetaSyn Academy · Published meta-analyst (Nature Human Behaviour, 2026)

The main appraisal question is not “Was an indirect treatment comparison performed?” It is “Does this particular evidence structure support this particular comparison?” Two trials can share a comparator and still be poor candidates for indirect comparison because the common arm is not clinically equivalent, the populations differ on plausible treatment-effect modifiers, the endpoint or timepoint is not comparable, or the reported estimates target different treatment effects. Conversely, a population-adjusted analysis can look statistically sophisticated while remaining fragile because important variables were unavailable or overlap was weak.^1,3,7,14

This resource is built around an evidence dossier rather than a generic quality score. You first reconstruct the comparison: what treatment effect is being requested, which trials supply each leg, whether a randomized anchor exists, and which population the result is meant to inform. You then compare the common comparator, patient characteristics, treatment-effect modifiers, outcomes, estimands, effect measures and any population-adjustment diagnostics side by side. Only after that record is visible do you make a narrative credibility judgment. The tool does not calculate Bucher estimates, MAIC weights, STC predictions or ML-NMR models; it appraises whether a reported analysis has a defensible evidentiary foundation.^4,5,8

Start with the comparison claim, not the statistical method

An ITC is often introduced by naming its method—Bucher, MAIC, STC, ML-NMR—before stating the treatment effect that decision-makers actually need. That reverses the logic. The target comparison must come first. Record the population, interventions, comparator, endpoint, time horizon and treatment effect of interest. Then ask which evidence can identify that effect. The same dataset can support one estimand reasonably well and another poorly. For example, two oncology trials may both report progression-free survival, yet differ in treatment switching, assessment schedules or censoring strategies. The labels match; the target effects may not.^13,16

The first distinction is structural. An anchored comparison links treatment A and treatment B through a common comparator C. Randomization is preserved within A-versus-C and B-versus-C, so the indirect comparison uses relative effects rather than raw arm outcomes. An unanchored comparison has no randomized common comparator linking the evidence. Quantitative analysis may still be possible, but the protection provided by within-trial randomization no longer carries across the comparison, and the burden of modeling cross-trial differences becomes much larger.^1,7,12

The common comparator is an anchor only when the common treatment is genuinely comparable

The common comparator does more than connect trial diagrams. In a simple adjusted indirect comparison, it allows relative effects from separate randomized trials to be contrasted without directly comparing raw outcomes across unrelated patient groups. That logic fails if the “same” comparator does not represent the same clinical treatment strategy across trials. A placebo plus modern background therapy can be a different clinical comparator from placebo plus older supportive care. The same active drug at another dose, route, schedule or treatment line may also change the effect observed in the control arm.^1,3,7

This is a treatment-level problem, not merely a covariate problem. NICE DSU TSD 18 is explicit that population adjustment addresses differences in patient covariate distributions; it does not adjust away treatment differences such as dosing, formulation, administration, co-treatment, titration or switching. If the anchor itself changes, matching age, disease severity and prior therapy cannot restore the randomized contrast that would have existed under an equivalent comparator.^7,8

Build the evidence dossier before making a credibility judgment

Reading trial reports one at a time encourages memory-based appraisal: “the populations looked similar,” “both used placebo,” or “the same endpoint was reported.” Those impressions are difficult to audit. The dossier below forces the relevant evidence into one record. It is deliberately broader than a baseline-characteristics table because cross-trial credibility can fail at the treatment, population, outcome or analysis level. The aim is not to make every difference disappear. It is to distinguish differences that are likely irrelevant from differences that alter the treatment effect being transported across trials.^5,6,14

How to read the dossier: four failure mechanisms that should not be collapsed into one score

Three worked appraisals

The examples below are illustrative. They show why the evidence dossier should produce a reasoned statement rather than a pass/fail verdict.

Turn missing information into an explicit evidence finding

One of the most important outputs of an ITC appraisal is often not a judgment about what the trials show, but a record of what the publications do not allow you to establish. Critical-appraisal research has long shown that indirect comparisons are difficult to judge reliably when clinical and methodological details are poorly reported. A missing characteristic should therefore not be silently treated as balanced, absent or irrelevant. It is an evidence gap in the appraisal itself.^5,6

This matters most when the missing variable has a plausible causal role. Suppose disease stage is a known treatment-effect modifier, but Trial 1 reports its distribution and Trial 2 does not. The correct entry is not “similar populations.” Nor is it automatically “invalid ITC.” The defensible statement is narrower: cross-trial balance on a plausible modifier cannot be assessed from available reporting. That uncertainty may affect the credibility of an anchored analysis; in an unanchored analysis the same missing variable can be even more consequential if it also predicts absolute outcome.

The same rule applies to treatment and estimand information. If the comparator description does not make clear whether rescue therapy was permitted, record the switching/rescue strategy as insufficiently documented. If the analysis does not state whether an effect is marginal or conditional, do not infer the estimand from the software or model label. If the target trial reports only selected baseline variables, list the clinically important variables that remain unavailable. An appraisal is stronger when it preserves these unknowns instead of converting them into assumptions of equivalence.

Separate statistical precision from causal credibility

Indirect estimates are often less precise than direct randomized comparisons because uncertainty from both evidence legs contributes to the final contrast. Simulation work has shown that indirect comparisons can require substantially more evidence to achieve the precision of a direct comparison, particularly when one side of the anchor is supported by few trials.¹⁷ That loss of precision is important, but it is not the same problem as bias. A wide interval around a well-anchored estimate and a narrow interval around a confounded unanchored estimate require very different interpretations.

The checklist therefore records precision-related diagnostics without turning them into a validity grade. A small ESS, extreme MAIC weights, one trial per evidence leg or a wide confidence interval can make an estimate fragile. But a precise estimate can still be causally weak if the comparator changed, a treatment-effect modifier was omitted, or the outcome estimands are incompatible. Conversely, an imprecise estimate can arise from limited sample size even when the structural assumptions are reasonable. This distinction is particularly important in HTA, where a decision-maker needs to know both how uncertain the numerical estimate is and how credible the identification strategy is.^2,18

Completion audit: what must be visible before the ITC leaves appraisal

Before an indirect estimate is carried into an HTA dossier, economic model, guideline evidence table or manuscript conclusion, the appraisal should leave an auditable trail. The purpose is not bureaucratic completeness. Each item below prevents a different form of ambiguity from being hidden behind the final effect estimate.

The result is intentionally different from a generic “quality checklist.” A user should be able to hand the completed dossier to another reviewer and show exactly where the treatment bridge is sound, where the trials differ, which variables were adjusted, which variables were unavailable, and what part of the uncertainty remains after the statistical analysis. That traceability is the practical value of the resource.

What this resource can establish—and what it cannot

A completed dossier can show whether the comparison was defined coherently, whether a randomized anchor exists, whether the common comparator appears clinically equivalent, whether plausible treatment-effect modifiers were considered, whether outcomes and estimands align, and whether a population-adjustment analysis reports the diagnostics needed for interpretation. It can also expose missing information that would otherwise disappear inside a polished statistical result.

It cannot validate an indirect treatment comparison by itself. It does not replace statistical programming, reproduce the full conduct guidance for MAIC/STC/ML-NMR, or determine whether a model has been correctly estimated. Nor does it convert an unanchored analysis into randomized evidence. Some concerns require specialist statistical review; others require disease-area expertise because the relevance of a difference depends on how treatments work and how outcomes are generated.^7,10,14,15

Next step: explain the failure mechanism before judging the estimate

The paired methodology guide, Indirect Treatment Comparisons Explained: Common Comparators, Assumptions and Credibility, examines why naive cross-trial comparisons fail, what a randomized anchor preserves, how anchored and unanchored assumptions differ, and which kinds of mismatch can or cannot be addressed by population adjustment. For the broader decision context, see Evidence Synthesis for HTA and HEOR: Comparative Effectiveness and Decision Support.