Appraising
Learning Objectives
This chapter will help you:
- identify the risk of bias in the literature.
- critically appraise literature.
- identify retracted studies.
“The critical appraisal and quality assessment of primary research are key stages in systematic review and evidence synthesis. These processes are driven by the need to determine how far the primary research evidence, singly and collectively, should inform findings and, potentially, practice recommendations.” (Carroll & Booth, 2015, p. 149).
Critical appraisal considers whether a study can be trusted. It assesses if it has been conducted in a way that minimises errors or bias and how applicable the results are to the population you are interested in (Booth et al., 2022). For systematic reviews, critical appraisal is required. For scoping reviews this stage is generally not included, but depending on the purpose, researchers may choose to include an assessment of the study quality and risk of bias (Munn et al., 2018; Peters et al., 2020). If this is done, justification for including this step should be reported.
Critical appraisal is sometimes referred to as risk of bias to better describe evaluating how a study was conducted to reduce the potential for biased results and not actual bias (Stone et al., 2023).
Why critically appraise
Any review seeks to minimise bias in the results to ensure accuracy. This is especially important in systematic reviews as one aim is to influence practice. Critical appraisal means you evaluate studies in an objective and impartial way to identify problems that could have impacted the results and therefore possibly affect your review findings (Booth et al., 2022).
This short video by Cochrane explains the importance of critical appraisal.
Cochrane Crowd. (2023, September 28). Module 3: Assessing the evidence [Video]. YouTube. https://youtu.be/qdkGPAIULtM
To critically appraise research, you need to understand the methodologies, strengths and weaknesses of study designs fairly well. The Centre for Evidence-Based Medicine from Oxford University has some great resources on study designs.
Appraisal of studies
The way studies are designed and conducted can lead to the risk of bias in results (Boutron et al., 2023). The Catalogue of Bias lists types of bias that can occur in research and how results may be affected.
For the critical appraisal to be accurate and not influenced by one person’s opinion, it should be done by at least two reviewers, with disagreements resolved by discussion or another author. The assessment needs to be transparent to readers, so give the tool used for evaluation and justify your assessment. This is often presented in a table (Boutron et al., 2023).
Tools and checklists
Tools or checklists are often used in critical appraisal as they ensure the same elements are evaluated. The ones used will depend on the study types being assessed. If there are multiple study types, try and use tools from the same source for better consistency.
- ROBINS-I (risk of bias in non-randomized studies – interventions)
- RoB 2 (risk of bias in randomised trials)
- CASP checklists (variety of study designs)
- JBI critical appraisal tools (variety of study designs)
- SIGN checklists (variety of study designs)
This short video by Carrie Price, a medical librarian, describes how to use some of the different checklists to critically appraise journal articles.
Price, C. (2021, April 11). Critical appraisal of research evidence in the health professions [Video]. YouTube. https://youtu.be/Vn0HN5_YP9c
Activity 14
Conflicts of interest
It is important to consider if studies may have been influenced by conflicts of interest. These could be financial, such as a company paying for a study on a medication they sell, or non-financial, such as a researcher’s beliefs, relationships with others, or membership of a group (Boutron et al., 2023).
If conflicts of interest are found, you need to decide how likely it is that it impacted the study. If it is unlikely, you probably don’t need to worry about it. If it could potentially have influenced the study, you will need to keep this in mind when analysing the results of your review.
Risk of bias in review results
Review results can be impacted by missing information from your included articles. This could be due to authors not reporting all results or choosing not to publish at all if the results were not significant (Boutron et al., 2023; Page et al., 2023).
If your review has many studies with a high risk of bias, the certainty of your results may be lower. This means you need to consider how to approach studies with a high risk of bias and document this in your protocol. You could:
- Exclude them from your analysis, but you should compare how results would differ if they were included.
- Present multiple analyses according to the risk of bias, so one with all studies, and one with only studies with a low risk of bias.
- Include all studies and discuss the risk of bias in your review.
- Use statistical adjustment of the results from each study to try and remove the risk of bias (Boutron et al., 2023).
The Cochrane Handbook has more details about how to assess your review results for a risk of bias.
Retracted studies
When selecting and assessing studies to include in a review, be aware of the possibility of retracted articles in your results. Articles can be retracted for several reasons, such as:
- unreliable findings due to error, fraud, or falsification
- plagiarism
- redundant publication
- copyright or authorisation infringement
- unethical research
- manipulated peer review process
- undisclosed conflicts of interest (Committee on Publication Ethics Council, 2019, p. 2).
When retracted articles are included in systematic reviews or meta-analyses, the results of the review may be inaccurate. One meta-analysis found that ivermectin was beneficial in treating COVID-19, however after several of the included studies were retracted due to plagiarism, ethical concerns and inaccurate data collection, a reanalysis excluding those studies found ivermectin had no benefit (Hill et al., 2022).
It can be challenging to identify when an article has been retracted because of inconsistent practices across databases. Suelzer et al. (2021) found one database had no retraction labels for 23 retracted articles.
Some suggested methods for avoiding including retracted articles in reviews are:
- Update to the latest version of EndNote, which includes a feature that identifies retracted articles in your library.
- Search for retraction notices for your included articles in databases such as Retraction Watch, or using techniques described in Cochrane Technical Supplement 4: Searching for and Selecting Studies.
- Check the included articles just before publication in case any were retracted while you were undertaking the study.
- Consult a librarian.
Example
See below for the critical appraisal method for the following systematic review.
Sullivan, O., Curtin, M., Flynn, R., Cronin, C., Mahony, J. O., & Trujillo, J. (2024). Telehealth interventions for transition to self-management in adolescents with allergic conditions: A systematic review. Allergy, 79, 861-883. https://doi.org/10.1111/all.15963
“The Mixed Methods Appraisal Tool (MMAT) was used to assess the methodological quality of each study.38 This tool allows the appraisal of five categories of empirical studies: qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies and mixed methods studies. Quality appraisal was completed by MOS, with a random 33% cross-checked by MC. Findings were not used to exclude studies that otherwise met the inclusion criteria, as recommended in the tools’ user guide.38” (Sullivan et al., 2024, p. 863).
Further reading
Carroll, C., & Booth, A. (2015). Quality assessment of qualitative evidence for systematic review and synthesis: Is it meaningful, and if so, how should it be performed? Research Synthesis Methods, 6(2), 149–154. https://doi.org/10.1002/jrsm.1128
References
Booth, A., Sutton, A., Clowes, M., & Martyn-St James, M. (2022). Systematic approaches to a successful literature review (3rd ed.). Sage Publications.
Boutron, I., Page, M. J., Higgins, J. P. T., Altman, D. G., Lundh, A., & Hróbjartsson, A. (August, 2023). Considering bias and conflicts of interest among the included studies. In J. P. T. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page, & V. A. Welch (Eds.), Cochrane handbook for systematic reviews of interventions (Version 6.4.) Cochrane. https://training.cochrane.org/handbook/current/chapter-07
Carroll, C., & Booth, A. (2015). Quality assessment of qualitative evidence for systematic review and synthesis: Is it meaningful, and if so, how should it be performed? Research Synthesis Methods, 6(2), 149–154. https://doi.org/10.1002/jrsm.1128
Committee on Publication Ethics Council. (2019). Retraction guidelines. https://doi.org/10.24318/cope.2019.1.4
Hill, A., Mirchandani, M., & Pilkington, V. (2022). Ivermectin for COVID-19: Addressing potential bias and medical fraud. Open Forum Infectious Diseases, 9(2), Article ofab645. https://doi.org/10.1093/ofid/ofab645
Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Medical Research Methodology, 18, Article 5. https://doi.org/10.1186/s12874-017-0468-4
Page, M. J., & Higgins. J. P. T. (2016). Rethinking the assessment of risk of bias due to selective reporting: A cross-sectional study. Systematic Reviews, 5. Article 108. https://doi.org/10.1186/s13643-016-0289-2
Page, M. J., McKenzie, J. E., & Higgins, J. P. T. (2018). Tools for assessing risk of reporting biases in studies and syntheses of studies: A systematic review. BMJ Open, 8(3), Article e019703. http://dx.doi.org/10.1136/bmjopen-2017-019703
Page, M. J., Higgins, J. P. T., & Sterne, J. A. C. (August, 2023). Assessing risk of bias due to missing results in a synthesis. In J. P. T. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page, & V. A. Welch (Eds.), Cochrane handbook for systematic reviews of interventions (Version 6.4.) Cochrane. https://training.cochrane.org/handbook/current/chapter-13
Peters, M. D. J., Godfrey, C., McInerney, P., Munn, Z., Tricco, A. C., & Khalil, H. (2020). Scoping reviews. In E. Aromataris, & Z. Munn (Eds.), JBI manual for evidence synthesis. https://jbi-global-wiki.refined.site/space/MANUAL/355862497/10.+Scoping+reviews
Stone, J. C., Barker, T. H., Aromataris, E., Ritskes-Hoitinga, M., Sears, K., Klugar, M., Leonardi-Bee, J., & Munn, Z. (2023). From critical appraisal to risk of bias assessment: Clarifying the terminology for study evaluation in JBI systematic reviews. JBI Evidence Synthesis, 21(3), 472–477. https://doi.org/10.11124/JBIES-22-00434
Suelzer, E. M., Deal, J., Hanus, K., Ruggeri, B. E., & Witkowski, E. (2021). Challenges in identifying the retracted status of an article. JAMA Network Open, 4(6), Article e2115648. https://doi.org/10.1001/jamanetworkopen.2021.15648
A systematic error or deviation from the truth in results.
How research is done, including how information is collected and analysed and why a particular method was chosen.
Where an individual may preference, or be perceived to preference, their own interests or obligations over their duties and responsibilities as a researcher.
An assessment of the likelihood that the intervention effect will not be substantially different from what the research found.
The removal of an already published article from a journal.
Taking someone else's work or ideas and passing them off as one's own.
Where a research report is read and commented on by people with similar expertise and interests. It helps to check the quality of the research.
