How to Choose an Effect Size for Meta-Analysis

An effect size is not a detachable number that can be selected after extraction. It is an estimate on a defined scale, attached to a particular outcome, comparison, population, time point and statistical model. The Cochrane Handbook distinguishes binary, continuous, ordinal, count or rate, and time-to-event data because the available measures and their interpretations differ across those data structures.¹ Treating every result as a generic “effect size” hides those differences at the point where they matter most.

The word size can also mislead. The choice of measure is not a contest to find the largest numerical value. A risk ratio, odds ratio, risk difference, mean difference and standardized mean difference place effects on different scales. Their values cannot be compared as though they were alternative readings from the same ruler. The selection must follow the scientific question, not the apparent magnitude or statistical significance of the result.^1,2 Accessing free systematic review and meta-analysis templates helps establish standardized documentation routines across all synthesis stages.

Start with the target effect, not the statistic

Three questions establish the analysis target. What outcome construct matters? What comparison is being made? What effect of that comparison is the review trying to estimate? The first question prevents a shared numeric scale from disguising different concepts. The second fixes the order of groups, exposures or conditions. The third identifies the estimand: the target effect in a defined population, time and analysis context. Prespecified synthesis groups help keep those choices aligned with the review question.³ Only after those questions are clear should the team ask which effect measure can express the target with the available data.

For a randomized intervention review, the target may be the effect of assignment to the intervention at a prespecified follow-up. For a non-randomized association, the target may be a covariate-adjusted contrast. For prognosis, it may be a relative rate over time. Two studies can report the same statistical measure and still estimate different targets because their populations, adjustment sets, time windows or treatment strategies differ. Conversely, studies can report different instruments yet estimate a sufficiently similar construct to support a standardized analysis. The measure does not settle comparability by itself.^2,3

Available data matter, but convenience does not have final authority. If the prespecified target requires a risk ratio and reports omit the necessary events, totals or compatible adjusted estimates, the response is to retrieve data, justify a different measure, separate syntheses or acknowledge that quantitative synthesis is not feasible. Quietly changing the analysis because a different column is easier to extract turns a data limitation into an undocumented change of question. Cochrane guidance emphasizes collecting the group summaries, effect estimates and uncertainty needed by the planned synthesis.⁴

Choosing among measures for continuous outcomes

Mean difference

The mean difference preserves the outcome unit and estimates the average difference between groups. It is usually the most interpretable route when studies use the same instrument, version and unit, and when the scale has the same substantive meaning across populations. If blood pressure is reported in the same unit, for example, a pooled mean difference can be read in that unit. This does not guarantee that pooling is sensible. Follow-up time, measurement protocol, score direction, final values versus changes, and design corrections still need agreement.¹

A useful rule is not “same outcome means MD.” It is “same construct, compatible measurement unit and defensible common interpretation may support MD.” A scale called pain can have different ranges or anchors. A laboratory value can be expressed in different units. A change score and a final score can sometimes enter the same mean-difference synthesis, but the variance and assumptions need correct handling. Record the actual route.^1,2

Standardized mean difference

The standardized mean difference is commonly used when studies measure the same underlying construct using different scales.¹ Standardization divides a group difference by a within-study standard deviation, and small-sample corrections are commonly applied. This produces a common numerical unit, but it does not standardize the construct, study population or instrument quality. Combining a depression inventory with a general wellbeing scale is not justified merely because both can be converted into standard-deviation units.

The denominator also affects meaning. Populations with more variable scores can produce smaller standardized estimates for the same raw difference. Measurement reliability and study design can contribute to that variation. Methodological evaluations of SMD estimators and their sampling variances show that small samples and distributional conditions matter.⁸ The analysis record should therefore identify whether the software reports Cohen’s d, Hedges’ g or another correction, how the variance was calculated and how score direction was harmonized.

Interpretation should return to the outcome context. Universal small, medium and large labels are not substitutes for a meaningful difference on the actual construct. If possible, re-express the result using a familiar instrument, a minimally important difference or another justified reference. If re-expression requires extra assumptions, report them.^1,9

Ratio of means and other routes

A ratio of means expresses a multiplicative contrast and may be useful when the scale has a meaningful zero and proportional change is scientifically interpretable. Empirical work has compared it with MD and SMD across continuous-outcome meta-analyses.⁶ It is not suitable simply because studies use different scales, and it is not defined sensibly for every measurement scale. The question is whether a ratio such as 0.80 has a stable meaning for the outcome, not whether the software can calculate it.

Choosing among measures for binary outcomes

Binary data commonly support a risk ratio, odds ratio or risk difference. A risk ratio compares probabilities. An odds ratio compares odds. A risk difference subtracts probabilities. Their null values are 1, 1 and 0 respectively. Each can be mathematically valid while answering a different numerical question.¹

Relative and absolute effects should not be treated as rivals when both are needed for decision-making. A relative measure may be chosen for synthesis, while absolute effects are later estimated for one or more credible comparator risks. That re-expression must preserve the uncertainty and show the assumed baseline risk. The number needed to treat is derived from an absolute effect and baseline risk for interpretation; Cochrane guidance does not recommend pooling NNTs directly.¹

Event coding is part of the measure. For odds ratios, reversing event and non-event reciprocates the estimate. For risk differences, it changes the sign. For risk ratios, switching from an adverse event to its complement can change the numerical result and its precision in less intuitive ways. Decide whether the event means recovery, failure, occurrence or absence before extraction, and state which side of the null favours which group.¹

Sparse data require more than choosing a label. Zero cells, double-zero trials, small studies and group imbalance can affect estimation and standard errors. Some common corrections can introduce bias or instability. Research on risk-ratio meta-analysis illustrates why blanket rules are unsafe.⁷ Prespecify the sparse-data method with statistical support and report sensitivity to consequential choices. A free guide cannot replace that case-specific analysis.^1,7

Counts, rates, time-to-event outcomes, ordinal data and correlations

Counts allow more than one event per participant. Rates add exposure time, such as events per person-year. A rate ratio compares event rates, while a rate difference expresses an absolute difference in those rates. The analysis needs event counts and person-time or a compatible model-based estimate and uncertainty. Reducing recurrent events to a binary indicator can discard information and answer a different question.¹

Time-to-event outcomes incorporate when an event occurs and how censoring is handled. A hazard ratio is common, but it is not a risk ratio over the entire follow-up. Its interpretation depends on the study model and assumptions, including whether a proportional-hazards interpretation is defensible. If studies report only risks at a fixed time, medians or survival curves, recovering a compatible estimate may require specialist methods. Do not relabel one of these summaries as a hazard ratio.¹

Ordinal outcomes can be analysed as categories, dichotomized at a threshold, treated as approximately continuous or modelled through proportional odds. Every route has consequences. Thresholds can differ across studies, and dichotomization loses ordering information. A proportional-odds model uses more of the ordinal structure but adds an assumption and may be harder to interpret. The effect measure should follow the outcome definition and available reporting, not a preference for a particular software module.^1,2

Correlations are bounded and their sampling properties change near the bounds, so conventional workflows commonly transform them before inverse-variance synthesis. The record should name the transformation and variance method. Recent methodological criticism suggests that standard approaches may be biased in some settings, but emerging evidence should be presented as an advanced qualification rather than a universal replacement rule. When the choice could materially change conclusions, plan sensitivity analysis and specialist review.^1,2

Understand the analysis scale, display scale and null

Difference measures use a null value of 0. Ratio measures use a null value of 1 on the original display scale and 0 after logarithmic transformation. Ratio measures such as risk ratios, odds ratios, rate ratios and hazard ratios are generally analysed on the natural-log scale because that scale is symmetric around the null and supports appropriate statistical calculations.¹ Software commonly back-transforms the pooled result for display.

Direction must be fixed as carefully as the null. Higher scores can mean improvement on one scale and deterioration on another. An experimental-versus-comparator risk ratio below 1 can represent benefit when the event is harmful and harm when the event is desirable. Forest-plot labels should make the direction visible, but the data record must establish it first. A reversed sign can look statistically plausible and still invert the scientific conclusion.¹

Check compatibility, multiplicity and statistical dependence

Pooling requires more than a common measure. Studies must address a sufficiently similar construct and synthesis question. Time points, analysis populations, outcome definitions and adjustment sets can change the estimand. Standardization cannot erase those differences. If compatibility is uncertain, separate the estimates, investigate the source of difference or use narrative synthesis rather than forcing a single pooled number.^1,3

Study design changes the variance and sometimes the estimate. Cluster-randomized trials require an analysis that respects clusters. Crossover and paired studies involve correlated observations. Multi-arm studies can reuse a comparator. Multiple outcomes or time points from the same participants are dependent. Treating each estimate as independent can give one study too much influence. Cochrane reporting guidance specifically identifies cluster, crossover, paired and multi-arm structures as unit-of-analysis concerns.²

Multiplicity also requires an outcome-selection rule. A study may report several scales for the same construct, several eligible time points, adjusted and unadjusted estimates, or multiple analysis populations. Selecting the estimate with the smallest P-value is not a solution. Prespecify an outcome hierarchy or use a statistical model that handles dependence. Record deviations when the planned estimate is unavailable.^2,5

Interpret the estimate on its own scale

Statistical compatibility and practical importance are different judgments. A confidence interval describes uncertainty around the estimate under the model; it does not identify the smallest meaningful effect or establish certainty in the evidence. A relative effect also needs a credible baseline risk before it can describe an absolute change. A standardized effect needs context before standard-deviation units become meaningful. Cochrane guidance recommends planning interpretation, including possible re-expression, at the protocol stage.⁹

Do not interpret an odds ratio as a risk ratio, a hazard ratio as a risk over a fixed follow-up, or an SMD as a change in the original instrument. Do not call a result important because the confidence interval excludes the null. The effect measure tells you how the estimate is expressed. Clinical, educational, policy or practical importance requires outcome-specific thresholds, consequences, baseline conditions and stakeholder values.^1,9

Report the decision so another analyst can reproduce it

A complete methods statement names the outcome and time point, effect measure, comparison order, data source, transformations, variance method, dependence correction, software and version. It explains how different scales or definitions were harmonized, how multiple estimates were selected and how missing statistics were recovered. It also identifies prespecified sensitivity analyses and deviations. PRISMA 2020 provides the reporting framework, but a concise sentence in a manuscript cannot substitute for the underlying analysis record.⁵

For ratio measures, state that analysis occurred on the log scale and that results were back-transformed where relevant. For SMD, state the estimator and small-sample correction. For adjusted estimates, state which estimate was preferred and why. For cluster or paired designs, state the correction or model. For effect directions, state how scores or events were aligned. These details determine what the pooled number means.^1,5

Common errors and why they fail

Worked decisions

Continuous symptoms measured by different instruments

A fictional review evaluates a therapy for the same prespecified anxiety construct at eight to twelve weeks. Studies use three validated scales with different ranges. The team considers MD, SMD and ratio of means. MD would require a common unit that the reports do not share. A ratio is not substantively stable because the scales do not have a common meaningful zero. The team selects a bias-corrected SMD, aligns higher scores to worse symptoms, flags variation in reliability and population SDs, records the software variance method and plans interpretation through a familiar scale and outcome-specific thresholds. The decision is defensible only if the instruments measure the same construct closely enough.^1,8

Binary recovery with varying baseline risk

A fictional review studies recovery after a brief intervention. It selects a risk ratio for the primary synthesis because recovery is consistently defined and relative probability matches the protocol question. It does not call the result an odds ratio. Because baseline recovery varies across settings, the team also plans absolute re-expression at several justified comparator risks rather than treating one risk difference as universal. A subgroup with very few events is referred for sparse-data analysis rather than processed through an automatic correction.^1,7

Time to relapse with incomplete reporting

A fictional review targets time to first relapse. Several studies report adjusted hazard ratios, while others report only the proportion relapsed at one year. The team does not combine those quantities under a single label. It seeks recoverable log hazard ratios and standard errors, records model and adjustment details and keeps fixed-time risks in a separate synthesis if compatible hazard estimates cannot be obtained. The missing data change feasibility, not the underlying target.^1,4

Conclusion

Choose an effect measure by preserving the meaning of the outcome and the target effect. Data type narrows the candidate family. Scale, direction, study design, available uncertainty, compatibility and interpretation determine whether a candidate is defensible. Document the choice before calculation, execute it in validated statistical software and keep the assumptions visible when the result is reported. That sequence prevents a convenient statistic from quietly replacing the scientific question. Explore complete guidance in the Meta-analysis methods, effect sizes and forest plots topic library.^1,5

Frequently asked questions