Meta-analysis Heterogeneity Interpretation Worksheet

The problem: heterogeneity statistics lose meaning when separated from their analytical history

A meta-analysis may be summarized in one sentence: heterogeneity was substantial, with an I² of 78%. The number looks precise, but the statement is incomplete. It does not identify the estimand, whether pooling was scientifically meaningful, the number of independent studies versus the number of effect estimates, the between-study variance estimator, the method used to construct the mean-effect interval, the analysis scale, the handling of dependent estimates, or whether the observed variation crossed a decision-relevant threshold.^1–5

Heterogeneity statistics are conditional on the data structure and fitted model. Cochran’s Q tests compatibility with exact homogeneity under conventional assumptions; it does not measure the clinical importance or cause of variation. I² is a relative, precision-dependent transformation of Q, not an absolute measure of how far underlying effects differ. τ² is the estimated between-study variance on the squared analysis scale, whereas τ is the corresponding standard deviation on the analysis scale. Confidence intervals quantify uncertainty about a mean effect; prediction intervals make an additional, model-based statement about the dispersion of underlying effects.^1–9

The first decision is therefore scientific rather than computational: do the included studies estimate sufficiently compatible quantities for an average to be meaningful? Random-effects modelling represents unexplained variation; it does not make incompatible populations, interventions, exposures, comparators, outcomes, time points, designs, or causal contrasts compatible.¹ When the estimand is incoherent, a numerical mean may be calculable but misleading.

What the worksheet is designed to prevent

A non-significant Q test is sometimes treated as proof of homogeneity, although the test often has low power when studies are few or imprecise. A high I² is sometimes treated as a prohibition on pooling, although its importance depends on the magnitude and direction of effects, study precision, uncertainty in I², absolute dispersion, and decision thresholds. Conversely, an I² of zero is sometimes interpreted as proof that underlying effects are identical, although boundary estimation and sparse information can conceal substantial uncertainty.^1,3,4

The same caution applies to random-effects inference. The performance of τ² estimators differs across configurations, and no estimator is uniformly optimal. Normal-theory intervals can be too narrow when heterogeneity is estimated imprecisely. Hartung–Knapp-type procedures often improve coverage in relevant settings, but can be very wide with very few studies or behave unexpectedly when heterogeneity is estimated at zero; modified procedures address some, not all, of these limitations.^5–8

Prediction intervals also require disciplined wording. They are most naturally interpreted as model-based intervals for the underlying effect in a new study or setting considered exchangeable with those synthesized—not as guaranteed predictions of the noisy observed estimate from a future study. Conventional plug-in intervals rely strongly on the random-effects distribution and can have poor coverage when studies are few, study precision is highly unequal, or assumptions are misspecified.^1,8,9

From software output to a defensible research record

A completed worksheet does not validate an incorrect analysis. Its purpose is to make the reasoning auditable. The record links the outcome and estimand to the pooling decision, the pooling decision to a specified model, the model to reproducible output, and the output to a contextual interpretation. This chain makes it possible for a co-author, reviewer or future update team to determine exactly how a published claim was produced.

The form is not a scoring instrument and does not classify heterogeneity as acceptable or unacceptable. A high I² can coexist with effects that all occupy the same decision-relevant range; a low I² can coexist with clinically incompatible studies or inadequate power to detect variation. Similarly, a prediction interval crossing a null or decision threshold may be highly informative, but its interpretation remains conditional on exchangeability, the assumed distribution of underlying effects and the adequacy of the estimation method.^1,8,13

PRISMA 2020 requires reporting the results of statistical syntheses and measures of heterogeneity (Item 20b), investigations of possible causes of heterogeneity (Item 20c), and sensitivity analyses (Item 20d). The worksheet supports documentation relevant to these requirements, but completing it does not by itself establish PRISMA compliance or analytical validity.^11,12