Systematic Review Screening: Title, Abstract and Full-Text Selection

The eligibility statement and the screening decision are not the same thing

Study selection is frequently treated as a simple, clerical sorting sequence. Methodologically, it is a repeated classification problem executed under conditions of incomplete information. The protocol defines the intended evidence boundary; screening tests whether each retrieved report supplies enough evidence to place its underlying study inside or outside that boundary. While PRISMA 2020 visually maps the resulting pathway, a diagram alone does not make the underlying decisions reproducible.^1,2

Disagreement between reviewers is not the primary threat. Two reviewers can agree perfectly while applying a scientifically inappropriate or biased interpretation. Conversely, high disagreement rates during piloting provide valuable data—they expose hidden assumptions before they distort the entire evidence base. A defensible selection process controls the criterion applied, the evidence available at each stage, the reviewer action, and the recorded resolution. The purpose of duplicate assessment is to reduce the probability that one reviewer’s oversight or private interpretation silently changes the review population.^3,4

Operationalize each eligibility criterion

A broad protocol heading like “eligible population” is not a screening rule. You must define observable characteristics, edge cases, and rules for handling mixed-sample studies. Distinguish strict eligibility criteria from variables collected later during data extraction. If an eligibility rule requires an amendment during the review, preserve the old version, the rationale, the date, and the approving role.

Pilot the form before screening the full set

1. Select a sample of records representing obvious inclusions, obvious exclusions, and highly ambiguous edge cases.
2. Have reviewers apply the draft criteria independently, recording their reasons alongside their decisions.
3. Compare disagreements to identify unclear wording, missing rules, or inconsistent reviewer assumptions.
4. Revise the criteria, repeat the pilot if necessary, and freeze a dated version prior to launching the main screen.

While some guidance suggests example agreement targets, no universal kappa or percentage threshold applies perfectly across all review types. Pre-specify a suitable rule for your field and explain how the pilot shaped the final criteria. Interpret agreement statistics alongside their category distribution, recognizing that high agreement on a flawed criterion does not make the rule scientifically valid.^6,8

Keep records, reports, studies, and decisions strictly separated

A reliable screening system separates the imported record, the retrieved report, the underlying study, and the review decision itself. Deduplication acts on records; retrieval targets reports; eligibility ultimately concerns studies; and an inclusion or exclusion decision is an event tied to a reviewer, criterion version, and timestamp. Collapsing these entities leads directly to characteristic errors: companion publications get counted as separate trials, conference abstracts are excluded despite full articles containing eligible data, or the same study inflates the flow diagram multiple times.^2,3

This separation determines the unit of counting. A database record may disappear during deduplication, a report may remain unavailable despite retrieval attempts, and several reports may support one included study. Each event belongs in a different section of the audit trail. Stable identifiers allow you to make corrections later without breaking the connection between the citation history, the eligibility evidence, and the reported PRISMA totals.

Use the two screening stages for fundamentally different questions

Write stage-specific rules before reviewers encounter the main dataset. At the first stage, a negative decision must rely strictly on information explicit enough to satisfy an exclusion rule. At full text, reviewers should locate the evidence for every criterion, drawing on appendices, registries, or linked companion reports when protocol methodology permits. A record retained as “unclear” at the first stage is not provisionally included; it has merely avoided an irreversible judgment made with inadequate evidence. Keep this distinction visible in training materials and screening forms.

Screening order heavily influences human judgment. A reviewer who has just rejected fifty irrelevant records may apply an artificially strict threshold to the fifty-first ambiguous abstract. Conversely, recognizing a prominent author or journal can prompt inclusion without the scrutiny applied to an unknown paper. While completely blinding reviewers to authors or journals is rarely practical, highly structured criteria, independent initial decisions, and periodic team calibration checks can significantly mitigate the influence of fatigue and recognition bias.

Retain an unclear abstract unless a criterion explicitly excludes it

Abstracts are inherently incomplete summaries. If a required population characteristic or specific outcome is absent from the abstract, distinguish “not reported” from “definitively absent”. Retrieve the full text whenever the record remains plausibly eligible. A broad, over-inclusive first-stage rule reduces the risk of prematurely discarding eligible studies before sufficient evidence is available.

This asymmetric treatment of uncertainty is a deliberate feature of evidence synthesis. A false exclusion at the first stage is extremely difficult to discover later because the full report never enters the assessment pipeline. A false retention consumes extra retrieval and screening time, but remains entirely correctable. While this balance can be modified in massive reviews through validated machine prioritization, the review team must explicitly state how records were ordered and what safeguards bounded the risk of missing eligible evidence.

Match reviewer independence to the required synthesis standard

For Cochrane intervention reviews, the final decision regarding whether a retrieved report meets eligibility criteria must be made independently by at least two people. Duplicate title and abstract screening is highly desirable, but not mandated as a universal rule. JBI scoping-review guidance expects two or more reviewers to screen independently at both the title/abstract and full-text stages. Other organizations and protocols set varying requirements. Always follow the governing standard and transparently report the process you actually used.^4,5

Resolve conflicts without erasing the original independent decisions

Preserve each reviewer’s initial vote, the specific criterion that triggered the disagreement, the discussion outcome, and the involvement of any third-party adjudicator. Recurrent conflicts usually signal a defective operational rule rather than reviewer inattention. If a rule’s wording changes during screening to resolve these conflicts, document whether earlier records were reassessed under the new version.

Review conflict patterns continuously during conduct. Repeated disagreements over a population boundary likely indicate an undefined mixed-sample rule; repeated conflicts about publication types reveal reviewers confusing a report format with a study design. Corrective action might involve a dated clarification, new test examples, or targeted reassessment. The goal is to refine the decision system while maintaining an honest, auditable history of its development. Simply asking reviewers to “be more consistent” will never fix a criterion written in language that supports multiple defensible interpretations.

Link companion reports directly to the underlying study

A trial registry entry, conference abstract, primary journal article, and long-term follow-up paper may all describe a single study. Reviewers screen reports as individual documents, but must maintain a study-report linkage ledger so companion publications are not evaluated or synthesized as independent trials. Your final flow diagram must report included studies and included reports separately.

Initiate linkage as soon as a possible relationship surfaces, keeping it open to revision. Shared authorship, identical settings, exact sample sizes, overlapping recruitment dates, intervention dosages, and registration IDs can suggest a relationship, but none are infallible. Record the rationale for merging reports while preserving the original report identifiers. If two apparently separate reports later prove to describe the identical study, you can revise the study-level counts without losing the report-level screening history.

Multiple reports often contribute different fragments of information to a single eligibility decision. A clinical registry may establish the study design, the primary article describes the sample, and a secondary follow-up report provides the required time point. Reviewers must not exclude each incomplete report separately when the combined study documentation satisfies the protocol. Conversely, the existence of an eligible companion report does not mean every publication attached to the study is relevant. Report-level relevance and study-level inclusion remain connected but distinct judgments.

Limit automation to a human-governed workflow

Prioritization algorithms and text-mining tools can sort records or assist with bounded clerical tasks. If used, record the specific tool, version, training protocol, stopping rule, and human verification process. No universal active-learning stopping rule applies across all review topics, and automated scores must never be presented as autonomous final inclusion decisions.

Evidence regarding semi-automated screening suggests potential workload reductions, but methodologies, training datasets, and stopping policies vary wildly. Published evaluations do not support the universal claim that an algorithm can replace accountable human selection across all review designs. Treat model output as a recorded input: preserve the software version, the prioritization rule, the screening order, the human verification steps, and explicitly log any records excluded without human review.^7,9,10

Amend criteria without rewriting the history

Screening frequently exposes eligibility complications the protocol did not anticipate. Clarifying wording without moving the scientific boundary is acceptable, but substantively altering population, design, intervention, outcome, or report eligibility constitutes a formal protocol amendment. Record the rationale, decision maker, date, new version number, and affected records. Applying a changed interpretation only to subsequent citations creates severe, time-dependent selection bias.

Amendments must not be driven by a preference for observed results. If a newly discovered study forces a boundary change, the review must document the scientific justification and apply the revised criterion consistently to all comparable records. While prospective protocol registration helps readers distinguish planned selection from post-hoc modification, the internal screening log provides the operational proof that the change was implemented fairly.

Criterion versions must travel with exported data. A spreadsheet column containing only the current rule is useless if earlier decisions were made under a previous version. Retain a version identifier on every single decision event. When re-screening is required, log the new decision as a fresh event rather than overwriting the old one. This event-based architecture allows you to explain exactly why inclusion counts shifted between protocol conduct, conference presentation, manuscript submission, and future review updates.

Treat quality control as evidence about the process

Quality control measures should be proportionate to the consequences of an error and aligned with the applicable methodological standard. Random verification of a single-reviewer first stage, duplicate assessment of all retrieved reports, targeted audits of automated exclusions, and reconciliation of unusual reason patterns answer entirely different questions. Pre-specify who performs the check, how records are sampled, what defines an error, and the corrective action required. A small verification sample that detects a critical exclusion error should trigger an investigation of the specific criterion and potentially mandate broader reassessment, not just the correction of a single row.

Workload pressure directly impacts methodological rigor. Fatigue, prolonged screening sessions, uneven record allocations, and looming deadlines alter reviewer behavior. Operational plans should utilize manageable batches, track progress without rewarding rapid exclusion, and provide clear escalation routes for difficult cases. These controls do not eliminate human judgment; they stabilize the conditions under which judgment is exercised. The final publication should acknowledge any material constraints that could have compromised selection, such as incomplete retrieval, partial duplicate assessment, or reliance on an unvalidated automated stopping procedure.

Report sufficient detail to reconstruct the selection method

A publishable methods section must identify the screening platform, the reviewers involved at each stage, the degree of independence, piloting details, conflict resolution strategies, automation usage, retrieval protocols, report-to-study linkage, and the handling of protocol amendments. The flow diagram then reconciles the numerical movement of records through identification, retrieval, eligibility, and inclusion. It should never be used to mask an absent decision method or replace a detailed exclusion log.^1,2

Internal documentation will naturally be far more granular than the final manuscript. Preserve stable IDs, duplicate decisions, specific reason codes, free-text citations, timestamps, and criterion versions even if the journal only publishes aggregate counts. This separation enables concise public reporting without sacrificing auditability. It also future-proofs the review: a later update team can locate the exact prior decision and determine whether a new protocol boundary or a newly published report alters a study’s status.

Perform numerical reconciliation directly from decision records, not by forcing records to fit a desired diagram. The number of reports sought for retrieval cannot be accurately inferred from the number assessed if retrieval failures went unrecorded. Likewise, the number of included reports cannot simply default to the number of included studies. Before publication, calculate every flow box from a specified unit and test the arithmetic across all routes. Any unexplained residual is a data-quality failure, not a formatting issue.

Rigorous documentation also supports transparent departures from the protocol. If the team was forced to alter the reviewer structure, introduce machine prioritization, broaden a criterion, or abandon retrieval of unavailable reports, the methods and limitations section must state exactly what occurred and why. A systematic review remains highly informative after a justified protocol deviation; it becomes impossible to appraise when the deviation is concealed.

Limitations

This guide outlines a general systematic-review selection workflow. Specialist evidence-synthesis designs may demand different reviewer structures, sampling rules, or decision categories. Always follow the relevant methodological guidance, approved protocol, and institutional requirements.

No screening form or digital tool can guarantee complete retrieval, perfect interpretation, or absolute freedom from selection bias. Calibration samples may fail to capture the difficult edge cases encountered later, and inter-rater agreement can remain artificially high when both reviewers share the same underlying misconception. These realities justify active, continuous methodological oversight throughout screening rather than blind reliance on a single initial pilot or numerical threshold.

Conclusion

Defensible study selection is the product of explicit eligibility rules, stage-appropriate evaluation of evidence, preserved independent decisions, and traceable conflict resolution. Title-and-abstract screening must protect plausibly eligible records from premature exclusion; full-text assessment must test the complete operational criteria and yield one accountable outcome per study. When records, reports, studies, and individual decisions remain properly linked, the final PRISMA flow diagram functions as a verifiable summary of the review process rather than a retrospective approximation.

Frequently asked questions