Importance of replicable reviews
Learning Objectives
This sub-chapter will help you:
- understand the issue of poor quality reviews.
- understand the importance of a reproducible study.
“….poorly conducted and/or reported systematic reviews can have wide-ranging ramifications on societal outcomes.” (Uttley et al., 2023, p. 30)
The problem with reviews
Any time you conduct research, you want it to hold up to scrutiny. This is no different for scoping or systematic reviews. Your name will be included as an author, so you want to make sure you are associated with quality work.
There has been a significant increase in the number of reviews published. From 2000 to 2019, the number of systematic reviews in PubMed increased over 20 times, but the amount of all articles only increased by 2.6 times (Hoffmann et al., 2021). Another analysis of PubMed found an increase of 2,728% in systematic reviews from 1986 to 2015 (Ioannidis, 2016).
While the number of systematic reviews published has increased, many are poor quality, which impacts applications for practice and guidelines. A study by Siemens et al. (2021) on systematic reviews of cancer treatments found 88.1% were of critically low quality, while in bariatrics, 99% were of critically low quality and 78% had a high risk of bias (Storman et al., 2019).
A living systematic review found the most common issues were the lack of a registered protocol, only a single reviewer, flawed risk of bias assessments, and not including grey literature. The website created as part of this project, Systematic Reviewlution, is a great place to explore potential problems to avoid them in your review (Uttley et al., 2023).
The authors grouped problems into four categories and proposed solutions for each.
- comprehensive: Include an information specialist or librarian in the search development, and consider exclusion criteria and limits.
- rigorous: Follow methodological guidelines and develop a protocol.
- transparent: Follow reporting guidelines, make protocols and data accessible, and report on any changes from the protocol.
- objective: Work in a skilled team, make sure questions are meaningful and important, and disclose potential conflicts of interest (Uttley et al., 2023).
This highly entertaining video by health librarian Carrie Price describes 10 ways your systematic review sucks. You can keep these in mind when conducting yours! Be aware, this video contains some bleeped out swear words, so if you are uncomfortable with that, feel free to skip watching.
Price, C. (2023, November 28). Ten reasons why your systematic review sucks [Video]. YouTube. https://youtu.be/KVJvo8pVjlA
If you don’t watch the video, the ten reasons your systematic review needs more work are:
- You did it by yourself!
- You didn’t read the guidance!
- You didn’t do a protocol!
- You didn’t search systematically!
- You decided to Google the articles!
- You didn’t like the number of results so you limited to only a couple of years!
- You used a filter to only limit to humans!
- You didn’t bother with critical appraisal!
- You figured it out as you went!
- You named it ambiguously for ultimate confusion!
Further reading
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., & Henry, D. A. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, Article j4008. https://doi.org/10.1136/bmj.j4008
References
Hoffmann, F., Allers, K., Rombey, T., Helbach, J., Hoffmann, A., Mathes, T., & Pieper, D. (2021). Nearly 80 systematic reviews were published each day: Observational study on trends in epidemiology and reporting over the years 2000-2019. Journal of Clinical Epidemiology, 138, 1-11. https://doi.org/10.1016/j.jclinepi.2021.05.022
Ioannidis, J. P. A. (2016). The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Millbank Quarterly, 94(3), 485-514. https://doi.org/10.1111/1468-0009.12210
Siemens, W., Schwarzer, G., Rohe, M. S., Buroh, S,. Meerpohl, J. J., & Becker, G. (2021). Methodological quality was critically low in 9/10 systematic reviews in advanced cancer patients: A methodological study. Journal of Clinical Epidemiology, 136, 84-95. https://doi.org/10.1016/j.jclinepi.2021.03.010
Storman, M., Storman, D., Jasinska, K. W., Swierz, M. J., & Bala, M. M. (2019). The quality of systematic reviews/meta-analyses published in the field of bariatrics: A cross-sectional systematic survey using AMSTAR 2 and ROBIS. Obesity Reviews, 21, Article e12994. https://doi.org/10.1111/obr.12994
Uttley, L., Quintana, D. S., Montgomery, P., Carroll, C., Page, M. J., Falzon, L., Sutton, A., & Moher, D. (2023). The problems with systematic reviews: A living systematic review. Journal of Clinical Epidemiology, 156, 30-41. https://doi.org/10.1016/j.jclinepi.2023.01.011
The risk of a systematic error in the research that could detract from the truth.
A systematic review which is continually updated, incorporating relevant new evidence as it becomes available.
The plan for a piece of research, describing in detail what the researchers plan to do.
Material that is not published in a peer reviewed journal or a chapter in a book. It includes internal reports, committee minutes, conference papers, fact sheets, newsletters.
