Funnel Plot and Small-study Effects Interpretation Checklist

The problem: funnel-plot asymmetry is a signal, not a diagnosis

Funnel plots are often reported as if they directly reveal publication bias. They do not. A funnel plot displays the relationship between study effect estimates and a measure of study size or precision. An asymmetric pattern is evidence of possible small-study effects: smaller or less precise studies tend to show different effects from larger or more precise studies. Selective non-publication is one possible explanation, but so are selective non-reporting of results, greater risk of bias in smaller studies, genuine clinical or methodological differences associated with study size, sparse-data behaviour, effect-measure artefacts, influential observations and chance.^1,5,6

The opposite error is equally important. A visually symmetric plot or a non-significant asymmetry test does not establish that the evidence base is complete. Formal tests usually have low power when few studies are available and can remain uninformative when study standard errors vary little. The familiar minimum of ten studies is a rule of thumb below which tests are generally discouraged; it is not a threshold at which a test suddenly becomes reliable.^1,5,6

Method choice also matters. The original Egger regression is not a universal test. Some effect measures, notably odds ratios and standardized mean differences, can be correlated with their standard errors, creating artefactual asymmetry. Alternative tests have been proposed for particular binary-outcome settings, while diagnostic-accuracy meta-analysis requires a design-specific approach such as the Deeks effective-sample-size test. None of these methods removes the need to inspect the data, consider heterogeneity, verify assumptions and seek direct evidence about missing results.^1–4,8,9

Statistical adjustment does not solve this attribution problem. Trim-and-fill, regression extrapolation, selection models and p-value-based methods estimate what might happen under particular assumptions about selection or small-study effects. Their performance varies across data-generating mechanisms, heterogeneity levels and study counts. They should therefore be presented as sensitivity analyses under explicit assumptions, not as procedures that recover a uniquely corrected or unbiased effect.^10,11

The interpretation checklist

From a visual pattern to an auditable judgment

A completed checklist turns a funnel plot from an impression into a traceable argument. The applicability gate records whether the evidence base can support visual or formal assessment. Plot construction is documented before interpretation. The formal method is tied to the effect measure and data structure. Alternative explanations are evaluated explicitly, and direct records of unavailable studies or results are placed alongside the statistical signal rather than treated as an afterthought.

Adjustment analyses belong after this assessment, not before it. Their value lies in showing how conclusions behave under explicit assumptions, not in replacing the observed synthesis. Likewise, the preferred final language is not “the funnel plot showed publication bias” or “publication bias was absent”. A defensible conclusion states whether a small-study pattern was observed, how applicable and informative the chosen test was, which mechanisms were supported or remained plausible, what direct evidence about missing results was identified, and what formal risk-of-bias judgment followed from the appropriate framework.

PRISMA 2020 asks review authors to describe methods for assessing risk of bias due to missing results and to report the resulting assessments. This checklist supports that documentation, but completing it does not by itself establish PRISMA compliance or replace the ROB-ME or ROB-MEN tools.^14,15

Related methodological articles and research tools