Meta-analysis Model Selection and Reporting Checklist

The familiar question, “fixed effect or random effects?”, compresses several decisions that answer different methodological questions. A defensible synthesis separates the target of inference from the statistical model, the estimator of between-study variance, the confidence-interval method, the decision to present a prediction interval, and the software implementation. This checklist provides a structured record for those decisions before results are interpreted.

Why the model label is only one layer

Model labels are often treated as settings that can be chosen after viewing a heterogeneity statistic. That practice reverses the correct order of reasoning. The first question is what quantity the review intends to estimate. If the included studies are regarded as estimating one common underlying effect, subject only to sampling error, a common-effect model expresses that target. If the review instead treats study effects as different but related values sampled from an assumed distribution, a random-effects model targets the mean of that distribution. These are interpretations, not merely alternative formulas.^1,2

The target must be stated in relation to the actual studies and populations. A pooled mean from a random-effects model is not automatically an effect that applies to every setting. Equally, a common-effect result does not become invalid merely because observed estimates differ; sampling error produces variation even under a common-effect assumption. The question is whether the assumption and target are credible enough for the intended inference, and whether remaining diversity is addressed transparently.^1,4

Begin with the synthesis objective and estimand

An estimand specifies the quantity the analysis is designed to estimate. It includes the outcome, contrast, population, time point, effect measure and handling of intercurrent events or competing outcomes where relevant. The model decision should be connected to that estimand. For example, an analysis seeking the average intervention effect across meaningfully diverse settings has a different inferential target from an analysis estimating a common pharmacological effect under tightly standardized conditions.¹

The synthesis objective should also define the scope of generalization. A random-effects mean estimated from a small, selectively assembled set of studies does not license inference to every possible future study. The assumed distribution describes the effects represented by the synthesis under its sampling and exchangeability assumptions. Those assumptions deserve a written statement rather than an unexplained software selection.²

Common-effect assumptions

Under a common-effect interpretation, each included study estimates the same underlying effect. Differences among observed estimates are attributed to within-study sampling variation. Inverse-variance, Mantel-Haenszel and other methods may implement a common-effect calculation for different data structures. The method and the inferential interpretation should both be reported because identical numerical machinery can support different targets in other fixed-effects formulations.⁴

The assumption is strongest when the intervention, comparator, outcome definition, follow-up, design and target population are sufficiently aligned. It is not proven by a non-significant heterogeneity test. Such tests often have low power when few studies are available, while trivial differences can become detectable with large evidence bases. A common-effect plan therefore needs a substantive justification and a statement of how unexplained diversity will affect interpretation.¹

Random-effects assumptions

A conventional random-effects model assumes that underlying effects vary across studies according to a distribution, commonly taken to be normal on the analysis scale. The pooled estimate describes the mean of that distribution, and the between-study variance quantifies its estimated width. This framework can represent unexplained heterogeneity, but it does not make heterogeneity disappear and it cannot correct bias that differs across studies.²

The distributional assumption is difficult to verify, particularly with few studies. If small studies are systematically different from larger studies, random-effects weighting can give them relatively more influence than a common-effect analysis. That is not a reason to prohibit random-effects models; it is a reason to prespecify checks for asymmetry, influential studies and alternative specifications, and to interpret the mean together with heterogeneity and uncertainty.^2,3

Prose equivalent: define the estimand first; state the model and its assumptions; specify how between-study variance and its uncertainty will be estimated; specify summary and prediction intervals; record the software implementation; then preserve outputs, deviations and reporting language.

Between-study variance is an estimated parameter

Random-effects implementation requires an estimate of between-study variance, written as τ². DerSimonian-Laird, Restricted Maximum Likelihood (REML) and Paule-Mandel are among the available estimators. Their performance varies with the number and size of studies, the effect measure, the true amount of heterogeneity and other features. Reviews of the evidence do not support a universal winner across every setting.³

REML and Paule-Mandel often have attractive statistical properties, while DerSimonian-Laird remains common and can be computationally simple. The responsible reporting choice is to name the estimator, explain why it fits the analysis, and prespecify sensitivity analyses when plausible alternatives could materially affect the result. It is misleading either to leave the estimator hidden as a software default or to declare one historical method categorically forbidden.^1,3

Uncertainty around τ²

A point estimate of τ² can look more certain than it is. With a small evidence base, estimates may be unstable and can equal zero even when important uncertainty remains. Profile-likelihood or Q-profile procedures can provide intervals for the heterogeneity variance in compatible settings. Reporting such an interval can help readers see whether the apparent precision of the heterogeneity estimate is justified.³

The choice of τ² interval method should be documented separately from the point estimator. A review may use REML for the point estimate and Q-profile for an interval, depending on software support and analysis conditions. The checklist records both fields so that a result cannot be described vaguely as “random effects” without its defining implementation choices.^1,3

Summary-effect confidence intervals

The conventional Wald-type interval treats the estimated standard error of the pooled mean in a familiar large-sample way. Hartung-Knapp-Sidik-Jonkman (HKSJ) procedures alter the uncertainty calculation and can improve coverage in some random-effects settings, especially when the number of studies is limited. However, performance is not uniformly superior under every configuration, and modified procedures exist to address particular counterintuitive cases.⁵

The interval method therefore needs an explicit rationale. Record whether HKSJ is used, which implementation or modification is applied, and what will happen in edge cases such as very few studies, a near-zero heterogeneity estimate or an interval unexpectedly narrower than the conventional alternative. “HKSJ is mandatory” is too broad; “HKSJ was prespecified because these conditions and this software implementation were judged appropriate” is auditable.⁵

Prediction intervals answer a different question

A confidence interval around a random-effects mean concerns uncertainty about the location of the mean effect. A prediction interval attempts to summarize the range in which an underlying effect for a similar setting might lie, accounting for estimated between-study variation and uncertainty. A narrow confidence interval can coexist with a wide prediction interval because the mean may be estimated precisely while effects vary substantially.^1,2

Prediction intervals depend strongly on assumptions, including the distribution of effects, and can behave poorly when few studies inform τ². Their relevance also depends on whether a “new” setting is sufficiently similar to those represented. The plan should state whether a prediction interval will be reported, which method and scale will be used, the minimum information considered adequate, and how limitations will be explained. Absence of a prediction interval should likewise have a reason rather than reflecting an unchecked default.¹

Prose equivalent: connect the protocol target to a prespecified method record, implement it in validated software, preserve any deviation with its reason and approval, and report the actual model, estimator, interval methods and software rather than only the intended plan.

Heterogeneity handling and sensitivity planning

Heterogeneity statistics describe aspects of observed inconsistency; they do not choose the model. I² is the percentage of variability in effect estimates attributable to heterogeneity rather than sampling error under its definition, but its uncertainty can be large and the familiar thresholds are rough conventions. Clinical meaning depends on effect magnitude, direction, outcome and context, not on a threshold alone.¹

The plan should identify plausible influential decisions before results are known. Examples include alternative eligible effect measures, alternative τ² estimators, conventional versus HKSJ intervals, exclusion of a study with a documented data problem, and assumptions for clustered or multi-arm designs. Sensitivity analysis tests robustness to decisions; it should not become a search for a preferred answer. Detailed diagnosis of heterogeneity, subgroup analysis, meta-regression and small-study effects belongs to the separate H07 pathway.

Software is part of the method

Software names alone are insufficient because defaults and available methods change. Record the program, version, package or module, analysis command or settings, estimator, confidence level, transformations, prediction-interval method and output location. RevMan added REML, Q-profile intervals, HKSJ options and prediction intervals in January 2025, illustrating why version-level reporting matters.¹

Preserve the analysis file, console output or structured result, not only a copied forest plot. The model checklist should be reconciled against the final output. If the software substituted a default, applied a continuity correction, failed to calculate an interval or handled a zero-heterogeneity case differently from the plan, record the discrepancy and its disposition.^1,6

Transparent reporting

Readers need enough information to reproduce the analysis and understand its target. Report the effect measure and analysis scale; common-effect or random-effects interpretation; weighting method; τ² estimator and any interval; summary-effect interval method; prediction interval method and conditions; heterogeneity statistics; software and version; prespecified sensitivity analyses; and material deviations. PRISMA 2020 supplies the broader reporting context but is not a conduct or quality-assessment tool.⁶

Reporting should distinguish what was planned from what was done. A deviation is not automatically misconduct or invalidity; unforeseen data can require an adaptation. The problem is an invisible change. Record the trigger, alternative considered, decision maker, date, effect on interpretation and where both original and revised outputs are stored.⁶

Small evidence bases require explicit caution

When only a few studies are available, several quantities are weakly identified. A heterogeneity test may have little power, τ² can be estimated as zero despite substantial uncertainty, and normal approximations for a pooled mean may have poor coverage. A random-effects model does not create information that the evidence set lacks. The plan should state how small-study configurations will be recognized and which outputs will be interpreted cautiously.^1,3

Hartung-Knapp procedures, alternative τ² estimators and profile-based intervals can improve particular operating characteristics, but no single option removes all small-sample problems. Some implementations can produce counterintuitive results under special configurations. Record the exact method, any modification and the response to edge cases before results are inspected. If the evidence is too sparse for a stable prediction interval or moderator analysis, state that limitation instead of forcing an estimate.⁵

Small studies also deserve attention because their estimates can be imprecise and their relative weight may increase under random effects. That observation does not prove reporting bias or justify exclusion. It identifies a sensitivity and interpretation issue. Assessment of small-study effects and publication bias belongs to H07 and requires evidence beyond the weighting pattern.

Model choice does not decide whether pooling is appropriate

Both common-effect and random-effects calculations presuppose that the studies address a sufficiently coherent synthesis question. Statistical accommodation of variation is not a licence to combine incompatible outcomes, interventions, populations or designs. Before completing the model fields, verify the synthesis groups against the protocol and explain why their effects form a meaningful target.¹

If study effects answer different questions, the appropriate response may be separate syntheses, structured presentation without pooling, or a redesigned estimand. A mean of incompatible quantities can be mathematically computable and scientifically uninterpretable. The checklist therefore records the population of inference, exchangeability conditions and scope of generalization rather than asking only which software option will be used.^1,2

The same principle applies to dependence. Multiple outcomes, time points, intervention arms or effect estimates from one study can be correlated. Treating them as independent can overstate precision. The analysis plan must identify the statistical unit and specify an appropriate multilevel, multivariate, robust-variance or selection strategy when necessary. These choices require qualified review and cannot be inferred from the forest plot.¹

Predefine what would count as a robust conclusion

A sensitivity analysis is informative when it targets a credible decision whose uncertainty could change the scientific message. Examples include alternative defensible τ² estimators, interval procedures, data-correlation assumptions, or treatment of a study with a documented extraction problem. State the expected comparison and interpretation rule in advance. Avoid a large menu of analyses with no hierarchy.⁶

Robustness is not agreement of point estimates alone. Compare interval width, direction relative to meaningful thresholds, prediction, study influence and the wording of the conclusion. If one defensible analysis supports benefit while another includes important harm, report the disagreement and explain which assumptions differ. Do not select the more favourable specification as primary after viewing results.⁶

A conclusion can also be stable numerically but fragile scientifically. Similar pooled estimates under several models do not resolve high risk of bias, indirectness or selective reporting. The model record should connect only to the inferential claims it governs, leaving evidence certainty to the appropriate assessment process.

How to use the completed record

First, compare the checklist with the registered protocol or dated analysis plan. Second, run the prespecified analysis in validated software and preserve the complete output. Third, reconcile every method field against the output, including defaults that may not be obvious in a forest plot. Fourth, document deviations before interpreting whether they changed the result. Finally, transfer the verified method description into the review report and use the paired guide, Fixed-Effect vs Random-Effects Meta-Analysis: Assumptions, Models and Reporting, when the reasoning requires explanation.⁶

Conclusion

A model decision is defensible when its estimand, assumptions, variance estimator, interval methods, software implementation and reporting are mutually coherent. No heterogeneity threshold can supply that chain of reasoning. Run the prespecified analysis in validated software, preserve outputs and deviations, then quality-check the forest plot before publication.^1,6