Primary-Study Overlap in Umbrella Reviews and Second-Order Meta-Analysis

By Dr. Esmaeel Saeedy Robat, Founder of MetaSyn Academy · Published meta-analyst (Nature Human Behaviour, 2026) · Evidence checked August 2026

Start with the right question: what exactly is overlapping?

“Primary-study overlap” is often used as though it described one simple condition. In practice, several structures can sit underneath the same label. Two reviews may cite the identical trial report. They may cite different papers from the same trial. They may use different waves of the same cohort, different intervention contrasts that share a control group, or different outcomes measured in the same participants. These situations all matter, but they do not necessarily create the same amount or type of statistical dependence.

The first distinction is between a study and a report. A single underlying study can generate a protocol, conference abstract, main results paper, secondary analysis and long-term follow-up. If those publications are entered as separate rows merely because the citations differ, the evidence matrix can exaggerate the number of distinct studies and change the apparent overlap. Recent matrix-construction research explicitly examined publication-thread assumptions and showed that reasonable choices about study identity, scope and structural missingness can materially alter both overall and pairwise CCA values.⁹

The second distinction is between bibliographic overlap and analytic overlap. A study may occur somewhere in both reviews but contribute to the target outcome in only one. Conversely, a modest whole-review overlap may become concentrated almost entirely in the particular outcome you want to synthesize. PRIOR therefore asks authors, when applicable, to describe overlap at the level of comparison and outcome, and Cochrane notes that overlap may need to be assessed separately across comparisons.⁶¹³

The evidence matrix is part of the method, not data-entry housekeeping

A citation or evidence matrix is the basic representation used to map overlap. Reviews sit in columns; underlying primary studies sit in rows; cells indicate whether each study appears in each review. Pieper and colleagues used this structure to formalize CCA, and GROOVE later extended the workflow by providing overall and pairwise CCA plus graphical inspection.¹³

The apparent simplicity is deceptive. Hennessy and Johnson likewise emphasize that CCA should be used as part of a considered overlap assessment rather than as a stand-alone decision rule.² Before you type 1s and 0s, freeze three rules.

The 2025 online/2026 volume meta-research study is important because it quantified this problem. Across seven sampled overviews reconstructed under 16 matrix scenarios, overall CCA ranged from 1.2% to 13.5% and pairwise CCA from 0% to 15.7%, depending on matrix-construction assumptions. In the broader sample of 193 overviews, 44.0% did not address overlap at all, while only 11.9% reported use of an overview-specific reporting guideline.⁹ The lesson is not that CCA is unreliable. It is that a CCA value is only interpretable when the matrix that generated it is reproducible.

CCA: the exact calculation and the boundary of its interpretation

The standard overall CCA is:

Pieper’s original illustrative matrix contained four reviews, 22 total study occurrences and 12 unique primary publications. The calculation was (22 − 12)/(4 × 12 − 12) = 10/36 = 0.28, or 28%.¹ The numerator counts repeated occurrences beyond the first appearance of each unique study. The denominator is the maximum possible repeated area for a matrix with the same number of unique studies and reviews.

The mathematical boundaries are useful for checking implementations. If every study appears in only one review, N = r and CCA = 0. If every unique study appears in every review, N = rc and CCA = 1 (100%). With only one review, the denominator is zero and CCA is undefined; there is no between-review overlap to summarize. An empty matrix is likewise invalid.

For exactly two reviews, CCA simplifies to the Jaccard similarity coefficient. Let A and B be the sets of studies in the two reviews. Total occurrences are |A| + |B|, unique studies are |A ∪ B|, and repeated occurrences are therefore |A ∩ B|. Pairwise CCA becomes:

This identity is useful because it clarifies what pairwise CCA describes: set similarity. It still does not describe statistical covariance between pooled effects.

The traditional CCA bands are descriptive conventions, not decision thresholds

Pieper and colleagues proposed the familiar labels 0–5% slight, 6–10% moderate, 11–15% high and above 15% very high.¹ These labels are convenient for description, but later work cautions against turning them into mechanistic rules. Bracchiglione and colleagues note that the categories have not been meta-epidemiologically validated as consequence thresholds, and recent matrix-construction work questions uncritical use of the bands when reasonable methodological assumptions can move an overview between categories.⁴⁹

Why overall CCA should not be the end of the analysis

One global number can hide a highly uneven structure. Bracchiglione and colleagues evaluated 30 overviews and found a median overall CCA of 6.7%. Across 3,138 review pairs, however, the median proportion of pairs categorized as slight was 52.8%, while the median proportion categorized as very high was 28.3%. Six of the 30 overviews had at least 80% of their review pairs in the very-high category.⁴ The global value and the local pattern can therefore tell different stories.

GROOVE was designed partly around this problem. It calculates the overall CCA and the CCA for each possible pair of reviews, allowing investigators to see whether the overlap is dispersed across the matrix or concentrated in a few review pairs.³ Pairwise inspection is particularly useful before any rule that prioritizes one review over another, because it identifies the actual pairs or families that need attention. When the matrix becomes too large for simple Venn-style displays, heatmaps, UpSet-style displays and network representations can also make overlap structure easier to inspect.¹⁶

Pairwise CCA has a containment problem when review sizes are very different

Pairwise CCA/Jaccard compares the intersection with the union. That is a coherent set-similarity measure, but it can be unintuitive when one review is much smaller than another. Imagine Review A contains five studies and Review B contains 50. If all five studies in A are also in B, A contributes no unique primary study relative to B, yet the pairwise Jaccard value is only 5/50 = 10%. Kirvalidze and colleagues highlighted the need for care when interpreting pairwise CCA in such imbalanced cases.¹¹

A transparent complement is to show directional containment:

In the example above, A contained in B = 100%, while B contained in A = 10%. That description exposes the nesting that Jaccard compresses. The MetaSyn workspace reports both directions for this reason. It does not replace pairwise CCA with another claimed standard; it simply shows the underlying set relationship more transparently.

Whole-review overlap and outcome-level overlap can lead to different decisions

Suppose two systematic reviews each contain studies of depression, anxiety and sleep. Your umbrella review is interested only in depression. A whole-review matrix that contains every study from both reviews may be useful for describing broad redundancy, but it can misrepresent the overlap relevant to the depression estimate. The inferentially relevant question is which primary studies actually contributed to the depression result in each review.

PRIOR item 9b explicitly asks authors, when applicable, to describe how primary-study overlap was identified and managed at the comparison and outcome level and to specify how overlap was illustrated or quantified for each outcome.⁶ Cochrane similarly advises separate assessment when reviews contain multiple comparisons.¹³ Kho and colleagues operationalized an outcome-centric approach in which the review set can change by outcome; in their applications, multiple reviews were required for 19 of 46 outcomes and the observed outcome-level CCA values ranged from 0% to 71.4%.¹²

This is one of the most important practical distinctions for second-order work. If Review A’s depression estimate and Review B’s depression estimate are candidates for pooling, the most relevant overlap map is the set of primary effects underlying those two estimates. A whole-review CCA is not a substitute for that analytic map.

Structural missingness is useful, but it should remain visibly optional

GROOVE introduced structural missingness, or structural zeros, for study-review cells that could not possibly contain an inclusion. Chronology is the clearest example: a systematic review whose eligible search ended in 2015 could not have included a study published in 2018. GROOVE allows such cells to be marked and calculates an adjusted pairwise CCA.³

The important caveat is in the original methods paper itself: structural missingness was introduced as a new, optional feature and described as a complementary approach requiring further study.³ Later matrix meta-research shows that structural-missingness assumptions can materially affect CCA, which strengthens the case for documenting them but does not validate one universal rule for every type of structural zero.⁹

For practical work, chronology is usually the least ambiguous case. Language restrictions, scope restrictions and outcome restrictions need more care because they may represent review design choices rather than structural impossibility. If an adjusted analysis is used, preserve the unadjusted matrix and record exactly which cells were treated as structurally impossible and why.

Weighted CCA adds information, but it does not become covariance

Standard CCA treats a small pilot study and a very large study as equal units of overlap. Ying and colleagues proposed weighted corrected covered area (wCCA) to address that limitation by weighting primary studies using the square root of sample size.¹⁰ The method is peer reviewed and useful as a complementary description of informational overlap.

It should not be overinterpreted. The square root of sample size is not a universal reconstruction of the actual inverse-variance weight across every effect metric and design. wCCA does not estimate the covariance between review-level pooled effects, and it has not replaced standard CCA in current overview guidance. For that reason, the first MetaSyn workspace below keeps the validated binary matrix and ordinary CCA as the default and treats wCCA as an advanced method to consult in its original paper rather than an automatic extra output.

How to use the workspace without letting the number make the decision

The tool is intentionally conservative. It calculates things that can be calculated from a binary evidence matrix, then stops. It does not tell you which review to delete. It does not infer a correlation between meta-analytic estimates. It does not convert CCA into a variance inflation factor. It does not automatically downgrade certainty of evidence. Those are not missing features; they are methodological boundaries.

Use the output in three passes. First, inspect the matrix itself. A surprising result often comes from an unresolved study thread, an outcome that was included at the wrong scope, or a review update that is almost entirely nested inside its predecessor. Second, inspect the overall and pairwise values together. The global CCA tells you about the matrix as a whole; the pairwise table tells you where the repetition sits. Third, read the containment columns. If one review is almost entirely contained in another, ask what unique information, population, method or outcome the smaller review contributes before deciding whether both need to enter the same synthesis.

The generated methods text deliberately records the matrix scope and row unit because those decisions are part of reproducibility. Add your review-selection rationale, structural-missingness assumptions and sensitivity plan manually if they affected the analysis.

What should you do after overlap is detected?

Cochrane does not prescribe one universal strategy. Its current overview guidance recognizes several possible responses, depending on the overview purpose and the planned analysis. Authors can include all relevant reviews while ensuring that each primary study’s outcome data are extracted only once; they can select among overlapping reviews using prespecified criteria such as recency, quality, relevance or comprehensiveness; or, in some descriptive contexts, they can retain all relevant reviews while explicitly documenting the overlap and its limitations.¹³ Pollock and colleagues’ decision-tool work likewise shows that inclusion choices involve trade-offs rather than one universally superior rule.¹⁴¹⁵

A sensitivity analysis is particularly valuable when the main analysis includes review-level estimates despite a nontrivial overlap concern. The pruning rule should be determined without looking for the result you prefer. For example, within a clearly identified review family, the protocol might retain the review that most closely matches the target PICO and use methodological quality and recency as prespecified tie-breakers. Re-running the model after applying that rule shows whether the conclusion is fragile to repeated evidence. It does not prove that the pruned model is the only correct model.

Second-order meta-analysis changes the problem from redundancy to dependence

A descriptive overview can acknowledge overlap without mathematically modelling it. A second-order meta-analysis cannot simply assume that repeated review-level estimates are independent. Schmidt and Oh’s formal second-order framework defines the input as statistically independent and methodologically comparable first-order meta-analyses; they explicitly state that the primary studies or samples in one first-order meta-analysis should not also appear in the others.⁷

Why does this matter? Under a deliberately simplified fixed-weight model, let (hat{ heta}_A = sum_i a_iY_i) and (hat{ heta}_B = sum_i b_iY_i), where the weights are normalized and fixed, primary-study estimates are independent across studies, and the exact same study-level estimate (Y_i) is reused in both reviews. Then the covariance contributed by the shared studies is:

The equation makes the limitation of CCA obvious. Knowing that a study appears in both reviews does not tell you its sampling variance, its weight in each first-order meta-analysis, whether both reviews used the same outcome or comparison, or whether the underlying effects themselves are correlated. Random-effects first-order meta-analyses add further complexity because the weights depend on heterogeneity estimates that may differ between reviews. Different effect metrics, adjusted versus unadjusted estimates, shared participants with different outcomes, multi-arm trials and different follow-up times can all change the covariance structure.

This is why a pooled effect, its standard error and a CCA value are generally insufficient to reconstruct the cross-review covariance. Two review pairs can have the same CCA and very different sampling relationships. When detailed contributing effects, variances and weights are available, covariance-aware modelling may be possible in principle. But at that point, analysts should also ask whether reconstructing or directly re-analysing the unique primary-study evidence is methodologically cleaner for the question at hand.

What happens if positive covariance is ignored?

The safest general statement is about the variance calculation. The variance of a weighted combination of correlated estimates contains off-diagonal covariance terms. Setting those terms to zero treats shared information as if it were new independent information. With positive dependence, standard errors and confidence intervals can therefore become too optimistic, and hypothesis tests can become anti-conservative. Heterogeneity statistics and moderator tests are also affected because their usual reference distributions and variance assumptions rely on the dependence structure being correctly represented. The exact direction and magnitude of distortion in quantities such as ( au^2) are not universal and should not be asserted from CCA alone.

Why ordinary RVE is not a plug-in fix for this structure

Robust variance estimation is valuable when multiple dependent effect sizes are organized within independent higher-level clusters. Pustejovsky and Tipton expanded the working-model framework for correlated and hierarchical effects and emphasized that robust inference protects against misspecification of a working covariance model; it does not make the data hierarchy irrelevant.⁸ In the review-overlap problem, the dependence can be cross-cutting: Primary Study X may appear in Reviews A and B, while Study Y appears in B and C. Treating each review as an independent cluster therefore does not automatically satisfy the usual RVE setup.

The evidence review for this resource did not identify a directly validated, turnkey procedure in which conventional meta-analytic RVE takes one pooled estimate per review plus a CCA matrix and produces calibrated second-order inference. That conclusion is deliberately narrow. It does not claim that no covariance-aware robust method could ever be developed; it means that “RVE handles dependence” is not sufficient justification for using standard RVE to handle this particular cross-review dependence.

Returning to primary studies can solve duplication, but it changes the project

When review-level covariance cannot be recovered and quantitative synthesis is still essential, one option is to return to the unique primary studies. This removes duplicate representation at the review level and lets the analyst apply one consistent effect-size definition and statistical model. It can be a clean solution, but it should not be described as universally superior. Cochrane notes that an overview is defined by using systematic reviews as the unit of searching, inclusion and analysis; if the project shifts to searching, extracting and analysing primary studies directly, the work may become a different form of evidence synthesis.¹³

Schmidt and Oh also discuss the alternative of conducting a full first-order meta-analysis of the primary studies rather than a second-order synthesis, with advantages and disadvantages in workload, moderator representation and scope.⁷ The correct question is not “Which method is always best?” but “Which level of evidence is appropriate for the inference we are trying to make, and do we have enough information to analyse that level defensibly?”

High overlap plus discordant conclusions is a diagnostic signal

Two highly overlapping reviews can still reach different conclusions. That is not paradoxical. The reviews may differ in search dates, inclusion criteria, outcome definitions, extracted time points, effect metrics, handling of multi-arm studies, heterogeneity estimators, subgroup rules, risk-of-bias decisions, small-study-effect adjustments or certainty judgments. Lunny and colleagues have documented the broader problem of overlapping, discordant and problematic data in overviews, and current Cochrane guidance emphasizes the additional complexity created when overlapping reviews differ in conduct, quality, reporting or conclusions.⁵¹³

When the evidence sets are very similar but the conclusions diverge, do not ask CCA to choose a winner. Audit the synthesis decisions. Start with the exact target PICO, then compare which primary studies contributed to the outcome, what numerical data were extracted, and how those data were transformed and pooled. Review-level methodological quality can inform selection, but it should be considered alongside relevance, comprehensiveness and recency rather than used as a single automatic rule.

Overlap is not a separate GRADE downgrade

Current GRADE domains do not contain a standalone “primary-study overlap” downgrade. The overlap problem sits upstream: if repeated evidence is quantitatively mishandled, the resulting precision, heterogeneity or evidence selection may themselves be misleading. The correct response is to address the duplication and dependence problem before interpreting certainty, not to invent an additional numerical GRADE penalty for CCA.

Reporting: make the decision trail reconstructable

PRIOR separates several aspects of overlap reporting. Item 8b asks how overlap in populations, interventions, comparators and/or outcomes of systematic reviews was identified and managed during selection. Item 9b asks, where applicable, how primary-study overlap was identified and managed at comparison/outcome level during data collection and how the degree of overlap was illustrated or quantified. Item 17 asks authors to describe the extent of primary-study overlap across the included systematic reviews.⁶ Cochrane’s Chapter V similarly recommends mapping which primary studies appear in which reviews, at minimum using a citation matrix, and describing the nature and amount of overlap; CCA may be calculated.¹³

A strong Methods description should therefore report enough detail to rebuild the analysis:

A worked decision example

Imagine an umbrella review identifies four meta-analyses of the same intervention and outcome. Review A contains 20 contributing trials, Review B 18, Review C 8 and Review D 22. The whole-review CCA is moderate. Pairwise inspection, however, shows that all eight trials in Review C also appear in Review A, while A contains 12 additional trials. The directional containment is therefore C→A = 100%, even though the pairwise Jaccard value is 8/20 = 40%. Review D shares only three trials with A and none with C.

The first decision is not to remove Review C because a number looks high. Ask what C adds. Is it an older review that A updates? Does it cover a different population or use a different risk-of-bias standard? Does its pooled estimate use the same eight trials for the same outcome, or does it include a different time point? If C is a superseded member of the same review family and contributes no unique evidence to the target outcome, a prespecified rule may reasonably select A. If C answers a distinct subgroup question, retaining it descriptively may be appropriate even though its study set is nested.

If A, B and D are then proposed as inputs to a second-order meta-analysis, repeat the overlap assessment using the trials that actually generated the three pooled outcome estimates. If the shared-study pattern is nontrivial and the covariance cannot be reconstructed, standard inverse-variance pooling does not become defensible merely because the overall CCA is labeled “moderate.” The analysis plan must then choose a more defensible route: further review selection, an overlap-pruned sensitivity model, no second-order pooling, or a move back to unique primary-study effects.

What this page can settle, and what it cannot

The evidence base is strong enough to support several clear practices. Build an explicit study-by-review matrix. Resolve publication threads. Match the matrix to the actual synthesis question. Use overall CCA as a global summary and pairwise CCA to inspect local structure. Show containment when review sizes make Jaccard hard to interpret. Treat traditional CCA categories as descriptive labels, not exclusion rules. Report the assumptions that created the matrix. Most importantly, distinguish repeated evidence from the statistical covariance that repeated evidence may induce.

Other questions remain genuinely unsettled. Structural-missingness adjustments are still assumption-dependent. wCCA is a useful newer measure but not a universal replacement for CCA. There is no general function that turns CCA into a review-level covariance matrix. Conventional RVE does not become a turnkey solution simply because the data are dependent. Those are not reasons to hide overlap; they are reasons to make the analysis more transparent and the claims more proportionate.

Related MetaSyn Academy resources

References for the numbered citations are supplied in the separate MetaSyn reference block and should be pasted at the bottom of this page.