Effect Size Selection Worksheet for Meta-analysis

A meta-analysis cannot repair a mismatch between what studies measured and what a reviewer intends to claim. Before any study-level estimate enters software, the review team needs a controlled record of the outcome, the contrast, the unit, the direction and the assumptions that allow estimates to be compared. Cochrane distinguishes binary, continuous, ordinal, count or rate, and time-to-event outcome data because each type supports different effect measures and different routes to an estimate.¹ Treating every result as a generic “effect size” hides those differences at the point where they matter most.

The task is not to find the most sophisticated statistic. It is to select a scale that answers the prespecified synthesis question and can be estimated consistently across eligible studies. A mean difference can preserve the original measurement unit. A standardized mean difference can permit synthesis across different instruments, but changes the unit and introduces assumptions about the meaning of a standard deviation. A risk ratio, odds ratio and risk difference can all be computed from binary outcomes, yet they express different comparisons and behave differently when baseline risk varies. Hazard ratios concern event timing and censoring, not simply whether an event occurred. The right decision therefore depends on the scientific target as well as the data table.^1,2 Accessing free systematic review and meta-analysis templates helps establish standardized documentation routines across all synthesis stages.

Why effect-measure selection belongs in the protocol and analysis record

Selection made after viewing study results is vulnerable to convenience and outcome-driven switching. The protocol should identify the intended measure for each outcome and explain important contingencies, such as how different scales, adjusted estimates, rare events, clustered allocation or missing variances will be handled. Prespecified grouping also helps keep the analysis aligned with the review question.³ PRISMA 2020 expects review authors to report the effect measures and synthesis methods used.⁵ Reporting becomes more credible when the manuscript can be traced to a dated decision record rather than reconstructed after the analysis.

A useful record separates three concepts. The outcome construct is what matters, such as depressive symptoms, mortality or time to relapse. The estimand is the target effect under a defined comparison and population, such as the effect of assignment to an intervention at twelve weeks. The effect measure is the numerical scale used to express an estimate of that target, such as a mean difference or risk ratio. These terms are related, but they are not interchangeable. A review can use a familiar effect measure and still target the wrong effect if the comparison, time point or analysis population does not match the protocol. The extraction plan must also retain the group summaries, estimates and uncertainty needed to implement the selected route.⁴

Match the measure to the outcome and the intended meaning

Continuous outcomes

Use a mean difference when studies assess the same construct on the same scale and a difference in that scale is meaningful. The result keeps the original unit, which can make interpretation direct. This apparent simplicity does not eliminate judgment. A five-point difference has to refer to comparable versions of the instrument, comparable score direction, a defensible time point and a population for which that difference has the intended meaning. Change scores and final values may sometimes be combined on a mean-difference scale, but the extraction and variance methods must be compatible and prespecified.¹

A standardized mean difference is considered when studies measure the same underlying construct using different instruments. Standardization places each difference in standard-deviation units, allowing numerical combination across scales. It does not make unlike constructs equivalent. It also means that the denominator depends on variation within studies, which may reflect population heterogeneity, measurement reliability and study design as well as the underlying effect. The team should record the correction used for small samples, the variance method implemented by the software and the direction assigned to higher scores. Advanced evaluations show that SMD estimation and variance estimation can behave differently under small samples and non-ideal distributions.^1,8

Do not convert a standardized estimate into a universal label such as small, medium or large and stop there. Interpretation should return to the outcome, population, instrument properties and a meaningful reference where one is available. A ratio of means is another possible continuous-outcome measure when a multiplicative comparison is scientifically sensible and the scale has a meaningful zero. Empirical work has examined it as an alternative to difference measures, not as a universal replacement.⁶

Binary outcomes

Risk ratio, odds ratio and risk difference are not alternative names for the same result. A risk ratio compares probabilities, an odds ratio compares odds and a risk difference compares absolute probabilities. The first two have a null value of 1; the risk difference has a null value of 0. Odds ratios can diverge materially from risk ratios when events are common, so interpreting an odds ratio as though it were a risk ratio can exaggerate the apparent change.¹

Relative and absolute measures also transport differently across baseline risks. A stable relative effect can imply different absolute effects in populations with different comparator risks. A risk difference may be immediately interpretable in one setting but vary across settings because baseline risk changes. Selection should therefore consider whether the review primarily needs a relative comparison for synthesis, an absolute comparison for a defined population, or both synthesis and later re-expression. The number needed to treat is derived from an absolute difference for interpretation; it is not itself an effect measure to pool directly.¹

Sparse events create further constraints. Zero cells, double-zero studies, imbalance and small samples can affect whether a study-level measure and standard error can be estimated reliably. The worksheet records the planned handling and the software method, but it deliberately does not prescribe a continuity correction or a sparse-data model. Those choices require the full analysis context. Methodological work on risk ratios, for example, identifies pitfalls without establishing that one binary measure is best in every review.^1,7

Counts, rates, time-to-event outcomes, ordinal data and correlations

A count records how many events a participant experiences; a rate relates events to exposure time. Treating recurrent events as a binary yes/no outcome discards frequency and follow-up information. A rate ratio may be suitable when event counts and person-time or a compatible adjusted estimate are available, but its assumptions and the definition of exposure time must be recorded. Time-to-event data add censoring and follow-up timing. Hazard ratios are common, yet their interpretation depends on how the underlying study model represents hazards and time. An event risk at a fixed time and a hazard ratio are not interchangeable merely because both concern the same clinical event.¹

Ordinal outcomes may be analysed through a defensible continuous, binary or proportional-odds route depending on the scale, reporting and model. Dichotomizing a rich scale can lose information and make thresholds differ across studies. Correlations are often transformed before conventional inverse-variance synthesis because their raw sampling distribution is problematic near its bounds. Recent methodological debate about correlation meta-analysis should be treated as an advanced qualification. The core decision record should name the transformation, variance method, software and interpretation rather than claim that an emerging method has already replaced established practice.^1,2

Audit compatibility before pooling

Two estimates with the same label are not automatically compatible. A mean difference at four weeks and a mean difference at twelve months may answer different questions. Two risk ratios can reverse direction if one study codes recovery as the event and another codes non-recovery. An SMD for anxiety and an SMD for general psychological distress may use the same unit while measuring different constructs. Compatibility is a scientific judgment supported by the protocol, not a property created by standardization.¹

Dependence deserves explicit attention. Cluster-randomized trials assign groups rather than independent individuals. Crossover and paired designs produce correlated observations. Multi-arm trials can contribute the same comparator more than once. Repeated measures and multiple time points can create several eligible estimates from the same participants. If those estimates are treated as independent, their precision can be overstated. The worksheet asks for the design and adjustment so the analysis team can verify that the study-level estimate and variance match the statistical unit.²

Adjusted and unadjusted estimates create another compatibility decision. In non-randomized studies, an adjusted estimate may address confounding better than a crude comparison, but different adjustment sets can target different conditional effects. The record should list the covariates and identify which estimate the protocol prioritizes. A common numeric scale does not guarantee a common estimand.²

How to use the worksheet

1. Complete one record for each outcome and synthesis group, not one record for the whole review.
2. Describe the construct and estimand before selecting a candidate measure.
3. Inventory the statistics and study designs actually available. Do not assume every report contains the needed variance.
4. Audit scale, direction, timing and statistical dependence across studies.
5. Record the candidate measure, alternatives considered and the reason for the decision.
6. Run the calculation in validated statistical software, then return to record the software, version and output location.
7. Obtain methodological review for unresolved transformations, dependence, sparse data or incompatible estimands.

Illustrative record: different depression scales

Completion check

A record is ready for analysis only when another qualified team member can reconstruct why the measure was selected and how each eligible study will contribute a compatible estimate and variance. Empty fields are not the only sign of incompleteness. A polished rationale can still fail if it never defines the event, ignores a reversed scale, combines unlike time points, treats correlated estimates as independent or relies on a software default that no one has documented.

• The construct, comparison, population and time point are explicit.
• The estimand is distinguished from the numerical measure.
• The available statistics support the proposed study-level estimate.
• The null value, direction and analysis scale are correct.
• Compatibility across instruments, definitions and designs is justified.
• Transformations and unit-of-analysis corrections have an owner.
• The software, version, output location and review decision are recorded.
• Any departure from the protocol is connected to an amendment or deviation record.

Document uncertainty in the decision itself

An effect-measure record should not create false certainty. Some studies may report insufficient variance information, use unclear event definitions, or provide estimates adjusted for incompatible covariate sets. Record these limitations as unresolved rather than filling gaps from convention. If a recovery method, correlation assumption or conversion is planned, name the required inputs, justify the method, identify who will verify it, and retain the calculation with the analysis output.

Alternative measures may remain defensible. The record should explain why the primary measure best matches the estimand and how a reasonable alternative would change interpretation. Sensitivity analysis is useful when it tests a material assumption, but it should not become a search for the most favourable result. Preserve the prespecified decision, label deviations and report whether alternatives alter the substantive conclusion.⁵

Interpretation plans belong in the record because they reveal whether the chosen scale serves the review question. State the null, direction, meaningful thresholds and any planned absolute re-expression. For standardized effects, identify outcome-specific anchors or external interpretive evidence where available. For relative effects, specify the baseline risk source needed for absolute translation. This planning prevents a mathematically valid estimate from being reported in language it cannot support.¹

Conclusion

Effect-measure selection is a chain of scientific and statistical decisions, not a menu choice made after data entry. A strong record begins with the outcome and target effect, names the data and design constraints, explains the selected scale, audits compatibility and preserves the software and approval trail. Once that chain is visible, calculation can proceed in validated statistical software without outsourcing judgment to the program.⁵