Skip to Main Content
Monash Health Library


Click here to chat with a librarian

A critical step within a systematic review is to assess each individual study for risk of bias. This is known as critical appraisal, quality assessment or risk of bias assessment. The purpose of the assessment is to validate the quality and relevance of each included study.

Key appraisal considerations include: 

  • the methodological quality of a study
  • the possibility of biases in the study design, the results or the conclusions
  • design flaws in the study that raise questions about validity of findings
  • overestimation by the researcher of intervention effect -- the extent to which the results can be believed.
  • whether the study findings can be generalised to the research context (external validity)

Rigorous and standardised appraisal of each included study is conducted after the full text screening stage. Appraisal can occur simultaneously with data extraction. 

"Risk-of-bias assessments should be performed independently by at least two people. Doing so can minimize errors in assessments and ensure that the judgement is not influenced by a single person’s preconceptions" -- Cochrane handbook Chapter 7


Appraisal process:

  1. Select an appropriate appraisal tool/checklist - Use a tool specifically designed to assess bias for the study design(s) of your included studies. For example, an RCT checklist for an RCT study design. This may mean you need to use multiple tools, covering a wide range of study designs. See the Tools & checklists for critical appraisal section below for for links.

  2. Pilot your preferred tool(s) - Two reviewers should pilot a few studies to ensure that the tool is comprehensive enough for all aspects of the included studies and to identify any differences in understanding of the tool before proceeding.

  3. Document the appraisal process - In the protocol, include which tools you intend to use, how many reviewers, if tools will be piloted and any cut off scores or inclusion/exclusion parameters that will be taken into account during the appraisal process. If these decisions are impacted by the screening process, changes must be noted and justified in your manuscript.

  4. Report risk of bias assessment -  In the review methods section, include a summary of the appraisal process as per your protocol with appraisal tools used, how many reviewers, if piloted and other relevant information pertaining to the appraisal process. Include high level summary of risk of assessment judgements. Results of risk of bias assessment can also be presented in a table or graph, where the judgments for each appraisal domain for each study are presented alongside their descriptive justification.

Can l exclude studies at the appraisal stage?

  • Studies are not usually excluded from results due to poor quality, or high risk of bias. They are retained in the list of included studies with a quality judgement (e.g. low bias, moderate bias, high bias) for each appraisal domain used, and an explanation to support the appraisal judgement. 
  • If you choose to exclude studies completely, you should cite them as meeting the inclusion criteria and explain why they were excluded - PRISMA 2020 checklist item #16b

Critical appraisal tools and checklists facilitate a systematic, structured, and transparent approach to this process. They prompt you to consider specific aspects of each study that you appraise.

Which checklist should I use?

There are many different critical appraisal checklists available -- choose one that best fits the study design you are appraising, and has been developed and validated by a reputable body such as the Joanna Briggs Institute (JBI) or Cochrane.

If you will be appraising more than one type of study, choose checklists published by the same body where possible. For example, if you need to appraise both RCTs and cohort studies, consider using the respective CASP checklists for these study designs.

Where can I find critical appraisal tools & checklists?

The below repositories each contain multiple critical appraisal tools and checklists for a wide variety of study designs.

  • Latitudes Network
    • Collection of tools for use in evidence synthesis projects such as systematic reviews. Search for a tool by study design. 
  • JBI (Joanna Briggs Institute)
    • Contains 15 critical appraisal tools of typical study designs as well as economic evaluations, quasi-experimental, expert opinion, policy etc. 
  • CASP (Critical Appraisal Skills Program)
    • CASP offers checklists for a wide variety of study designs in PDF, Word, or online versions. Includes a Clinical Prediction Rule checklist.
  • CEMB (Centre for Evidence Based Medicine)
    • Contains typical study designs with the addition of a IDP (Individual Patient Data) review checklist. 
  • SIGN Checklists
    • Scottish Intercollegiate Guidelines Network (SIGN) contains tools for various study designs, plus notes on their use.

Additional tools & checklists for specific study designs

  • Multipurpose tools 
  • Randomised controlled trials (RCTs)
    • Cochrane Risk of Bias (ROB) 2.0 -- Templates tailored to various RCT types, e.g. cluster-randomized trials and cross-over trials. 
    • Jadad Scale -- Simple tool based on only three domains: random assignment, double blinding, and patient withdrawal/dropout.
    • PEDro Scale -- Developed by Physiotherapy Evidence Database for assessing clinical trials across various fields of study (not just physio).
  • Observational studies (Including cohort studies, case-control studies)
  • Diagnostic studies
    • QUADAS-2 -- Developed by Bristol Medical School to evaluate primary diagnostic accuracy studies.
  • Prognostic studies 
  • Systematic reviews & meta-analyses (Note: these checklists are for appraising the quality of other published systematic reviews)
    • AMSTAR 2 -- Developed for systematic reviews which include randomised and/or non-randomised studies of healthcare interventions.
    • ROBIS Tool -- Assesses the risk of bias in systematic reviews of interventions, diagnosis, prognosis, or aetiology.

Tools & checklists for systematic reviews on specific topics

Consider the below options if your review topic is focused on economic evaluations, drug adverse events, or if your review will inform public health practice.

Tools & checklists for other types of evidence

Non-academic sources of evidence must also be critically appraised -- the below checklists have been developed for this purpose.

  • Grey literature (government and industry reports, theses, conference papers etc.) -- Use the AACODS Checklist developed at Flinders University.
  • Clinical practice guidelines -- Refer to the AGREE II instrument.

Both PRISMA and the Cochrane Handbook have sections outlining requirements for the critical appraisal of studies as part of a systematic review. 

Note: it is important to remember that once you have completed your systematic review others may appraise your work. Keep in mind the critical appraisal criteria for systematic reviews while conducting your own review to ensure it is of high quality. 

Monash Health acknowledges the Traditional Custodians of the land, the Wurundjeri and Boonwurrung peoples, and we pay our respects to them, their culture and their Elders past, present and future.

We are committed to creating a safe and welcoming environment that embraces all backgrounds, cultures, sexualities, genders and abilities.