Skip to Main Content
Monash Health Library
Journals A-Z | Renew Your Loans | New eBooks | Mind & Matter
If you have an exact title add ti= (eg. ti=Emergency Medicine)


Click here to chat with a librarian

A critical step within a systematic review is to assess each individual study for risk of bias. This is known as critical appraisal, quality assessment or risk of bias assessment. The purpose of the assessment is to validate the quality and relevance of each included study.

Key appraisal considerations include: 

  • the methodological quality of a study
  • the possibility of biases in the study design, the results or the conclusions
  • design flaws in the study that raise questions about validity of findings
  • overestimation by the researcher of intervention effect -- the extent to which the results can be believed.
  • whether the study findings can be generalised to the research context (external validity)

Rigorous and standardised appraisal of each included study is conducted after the full text screening stage. Appraisal can occur simultaneously with data extraction. 

"Risk-of-bias assessments should be performed independently by at least two people. Doing so can minimize errors in assessments and ensure that the judgement is not influenced by a single person’s preconceptions" -- Cochrane handbook Chapter 7


Appraisal process:

  1. Select an appropriate appraisal tool/checklist - Use a tool specifically designed to assess bias for the study design(s) of your included studies. For example, an RCT checklist for an RCT study design. This may mean you need to use multiple tools, covering a wide range of study designs.
    Critical Appraisal Tools   
     

  2. Pilot your preferred tool(s) - Two reviewers should pilot a few studies to ensure that the tool is comprehensive enough for all aspects of the included studies and to identify any differences in understanding of the tool before proceeding.

  3. Document the appraisal process - In the protocol, include which tools you intend to use, how many reviewers, if tools will be piloted and any cut off scores or inclusion/exclusion parameters that will be taken into account during the appraisal process. If these decisions are impacted by the screening process, changes must be noted and justified in your manuscript.

  4. Report risk of bias assessment -  In the review methods section, include a summary of the appraisal process as per your protocol with appraisal tools used, how many reviewers, if piloted and other relevant information pertaining to the appraisal process. Include high level summary of risk of assessment judgements. Results of risk of bias assessment can also be presented in a table or graph, where the judgments for each appraisal domain for each study are presented alongside their descriptive justification.

Can l exclude studies at the appraisal stage?

  • Studies are not usually excluded from results due to poor quality, or high risk of bias. They are retained in the list of included studies with a quality judgement (e.g. low bias, moderate bias, high bias) for each appraisal domain used, and an explanation to support the appraisal judgement. 
  • If you choose to exclude studies completely, you should cite them as meeting the inclusion criteria and explain why they were excluded - PRISMA 2020 checklist item #16b

Both PRISMA and the Cochrane Handbook have sections outlining requirements for the critical appraisal of studies as part of a systematic review. 

Note: it is important to remember that once you have completed your systematic review others may appraise your work. Keep in mind the critical appraisal criteria for systematic reviews while conducting your own review to ensure it is of high quality. 

Monash Health acknowledges the Traditional Custodians of the land, the Wurundjeri and Boonwurrung peoples, and we pay our respects to them, their culture and their Elders past, present and future.

We are committed to creating a safe and welcoming environment that embraces all backgrounds, cultures, sexualities, genders and abilities.