Skip to Main Content
Monash Health Library
Click here to chat with a librarian

Systematic Review Guide

Quality assessment is a critical step in systematic reviews, used to evaluate the methodological rigor and risk of bias in the studies included. This process ensures that the evidence synthesised is reliable and valid, contributing to trustworthy conclusions.

Reviewers typically use standardized tools or checklists tailored to the study design, and assessments are often conducted independently by two reviewers to maintain objectivity. The results of the quality assessment help inform the interpretation of findings and may influence the weighting or inclusion of studies in the final analysis.

Quality assessment is a critical step in systematic reviews, used to evaluate the methodological rigor and risk of bias in the studies included. This process ensures that the evidence synthesised is reliable and valid, contributing to trustworthy conclusions.

Reviewers typically use standardised tools or checklists tailored to the study design, and assessments are often conducted independently by two reviewers to maintain objectivity. 

Why is it important to appraise the included studies in a review?

  • Identifies risk of bias that could affect results and conclusions.
  • Informs weighting of evidence in synthesis and interpretation.
  • Supports transparency by showing how judgments about study quality were made.
  • Helps explain heterogeneity – differences in study quality can clarify inconsistent findings.

How do I appraise studies?

  1. Select an appropriate appraisal tool
    1. Choose a tool that matches the study design (e.g., RCT, cohort, qualitative).
    2. Common tools: Cochrane Risk of Bias, Newcastle-Ottawa Scale, CASP checklists, JBI tools.
  2. Assign two independent reviewers
  3. Appraise each study
    1. Appraisal can be done in Covidence for studies of the same type (e.g. all RCTs)
  4. Record and summarise
  5. Use appraisal in synthesis
    1. Weight studies according to quality in narrative or quantitative synthesis.
  6. Report on the appraisal process in your review

Recommended resources:

Webinar - Evaluating a Research Paper

Critical appraisal tools and checklists facilitate a systematic, structured and transparent approach to this process. They prompt you to consider specific aspects of each study that you appraise.

Common critical appraisal tools

Tool Study design
Latitudes Network Multiple study designs
JBI (Joanna Briggs Institute) RCTs, observational, economic evaluations, quasi-experimental, expert opinion, policy
CASP RCTs, cohort, case-control, qualitative, systematic reviews, Clinical Prediction Rule
CEMB (Centre for Evidence Based Medicine) RCTs, cohort, case-control, IDP reviews

Tools and checklists for specific study designs

Tool Study design
Downs & Black Multipurpose
Mixed Method Appraisal Tool (MMAT) Multipurpose
Cochrane Risk of Bias RCTs
Jadad Scale RCTs
PEDro Scale RCTs
Cochrane ROBINS-I Observational studies
Cochrane ROBINS-E Observational studies
The Newcastle-Ottawa Scale (NOS) for Case Control Studies  Observational studies
QUADAS-2  Diagnostic studies
QUAPAS - Quality Assessment of Prognostic Accuracy Studies  Prognostic studies
AMSTAR 2  Systematic reviews
ROBIS Tool  Systematic reviews
AACODS Grey literature
CHEC List: Consensus Health Economic Criteria  Economic evaluations
PROTECT Checklist  Drug adverse events
EPHPP tool for quantitative studies  Public health studies

AGREE II instrument

 Clinical practice guidelines