8: Methods

Photo by Google DeepMind from Pexels used under Pexels Licence.

The methods section is possibly the most important section of a review. It needs to be described in sufficient detail so that others can replicate the review, and in doing so, also provides you with a guide for conducting the review.

Tip: It can help to write the methods in the future tense – what you plan to do – and then change to the past tense when complete.

The methods section of a review should detail how the particular studies are found, selected, appraised and how data extracted from the studies are managed. These steps are important to establishing the rigour and credibility of the review.

Ideally several steps in a review are conducted concurrently by two review authors independently, and then their decisions compared once the task is complete to see if there is any disagreement. If disagreements occur, these need to be discussed; if agreement still cannot be reached, a third person should be involved. This process reduces bias or systematic error and should be documented in the methods section where used. However, such steps are commonly omitted in learning exercises, and are not discussed further here, although you should see references to such approaches as you look at the reviews linked throughout this resource.

Finding studies

Finding studies for a review includes referring to the type of studies that are to be included and the process(es) for choosing them. Reviews commonly include research articles published in peer-reviewed literature, but sometimes studies from grey literature are included. This book confines itself to peer-reviewed publications, as this is most common.

Regardless of the scope of the literature to be searched, the methods section of a review needs to clearly relate to the research question or objective of the review.

Steps that must be covered in the methods:

  • Search strategy: describes how potentially relevant research papers that could answer the question (or meet the objectives) will be identified through their titles and keywords, and the health databases that will be searched. More than one strategy may be used.
  • Study selection: describes the process for choosing the papers to include in the review from the broader group of papers that the search has identified.
  • Critical appraisal: describes how the quality of each of the selected papers to be included in the review will be judged.
  • Data extraction: details the information that will be extracted from each included paper.
  • Data synthesis: outlines the process to summarise the extracted data to answer the research question.

This section will explore search strategies, study selection and critical appraisal, while sections 9 and 10 look at data extraction and synthesis.

Search strategies

The titles of potential papers to be included in a review are identified through a search of electronic databases. This is where you can call on experts to help you, but you do need to do some work first. Think about:

  • keywords or terms relevant to your topic
  • possible databases that could be used
  • the time period for the search (i.e. the years to cover; gold standard reviews do not set time limits, but these take a lot of time)
  • languages (there are no language restrictions on gold standard systematic reviews).

It is important to be realistic about what you can achieve from a search, especially when you are still developing your review skills. It is more important to understand the search process and how it can affect the quality of your work.

Different questions lead to different search strategies, and best practice requires transparency to minimise bias where possible.

Further reading

The following paper,[1] in the environmental literature, lists some common sources of bias in conducting reviews and the problems bias can cause, such as selection bias when the studies chosen for inclusion are not representative of the evidence base available.

Eight problems with literature reviews and how to fix them

Activity: Developing your search skills

There are a number of resources that can help you develop your searching skills. This University of Sydney Library video explains how to develop good search strategies.

Source: UniSydneyLibrary, YouTube

Designing search strategies with STARLITE

STARLITE[2] is a tool that can help you to think about your search strategy and your study selection. STARLITE is another mnemonic that provides a framework for reporting search strategies. This tool will help you describe the search and study selection parts of the methods section. You can see how this was done in this published overview. The elements in the STARLITE tool are explained in Box 1.

Box 1: STARLITE – a framework for reporting search strategies

STARLITE element Description
S

Sampling strategy

 This refers to the overall strategy to the search for studies. The three main types of sampling strategies are:

  • Comprehensive strategies, which attempt to identify all relevant studies on the topic. This is the approach used in a Cochrane review.
  • Selective strategies, which attempt to identify all relevant studies but only within specified limits. This approach is often used in a rapid review.
  • Purposive strategies, which focus on samples of titles from specific disciplines, years or journals. This approach is commonly used in purpose-specific reviews.
T

Type of studies

 There are two options for describing the types of studies to be included in a review. These are to either:

  • Fully report, which means to specify the actual study types (e.g. randomised controlled trials, studies using grounded theory) or study designs to be included in the review.
  • Partially report, where an ‘umbrella’ category such as ‘qualitative studies’ is used without specifying the study designs to be included.
A

Approaches

 This refers to additional search strategies outside of electronic database searches used to find potential studies for the review. These commonly include:

  • Hand searching specific journals, for when journals are known to publish on particular topics, particularly if these are uncommon.
  • Citation snowballing, to check where studies that will be included in the review have been cited. This can be done using the citation records for a published article listed in electronic databases or Google Scholar.[3]
  • Checking the reference lists of studies that will be included in the review.
R

Range of years

 Electronic databases include titles across different time periods, so it is useful to report the range of years in which the search will be conducted. You can either:

  • Fully report, by including start and end dates with justification for time period chosen. In most cases this is preferred, as it is fully transparent.
  • Partially report, by including start and end dates but only determined by available coverage of databases. This requires readers to check the time period that databases cover.
L

Limits

 Limits in a search refer to functional limits that are applied for logistic reasons that do not alter the topic conceptually; e.g. the search may be limited to human participants or publication in specific languages.
I

Inclusion and exclusions

 Inclusions and exclusions in a search are conceptual limitations that mediate the scope of the topic area; e.g. these may limit the search to a specific geographical location or socio-economic environment, setting (such as a hospital or community) or to a specific focus of study (such as the study population or the intervention of interest).
T

Terms used

 This refers to the terms used in the actual searches. In a paper these may be:

  • Fully present, where the authors provide the example of the search strategy used in one or more of the electronic databases searched.
  • Partially present, where the authors report terminology used but without the specific evidence of the search syntax and operators.
E

Electronic sources

 There is considerable difference between electronic databases used for literature searches, so it is important to report the actual databases used (e.g. Medline). This helps in any attempt to replicate the search.

Study selection

Searching the specialist electronic databases in libraries can commonly lead to a long list of titles, many of which will not be included in the actual review.

The process used to select the particular studies to include in the review needs to be described in the methods section. This is for transparency and so that others can be confident that the studies are chosen fairly – that is, that no ‘cherry picking’ has occurred. Ideally the decision to include or exclude a particular paper is determined by criteria that are stipulated before the search is conducted. These are known as inclusion and exclusion criteria.

For beginner reviewers, particularly those undertaking practice exercises, it can be easier to understand useful criteria by using an iterative approach and reflecting on the process.

Tip: Work backwards by thinking about the papers you expect to find. Check how refining your question in different ways and changing your selection criteria affects your search strategy.

As you become more skilled, you should have sufficient knowledge of your area of interest to do this prospectively – that is, without anticipating what you will include. A non-iterative approach is important to the rigour or quality of a review as it reduces potential skewing bias or the evidence to be included.

Inclusion and exclusion criteria are used to ensure that the evidence you are using in the review is relevant to your research question. Common inclusion and exclusion criteria are the study population, the type of intervention and even the study design.

Study selection is an extension of the search process. If these criteria were used as search terms, it could confine the search too tightly to yield too few titles. Looser search terms will yield more titles but can include titles that are not suitable for the review. Clear criteria about what titles will be included or excluded from the review are needed.

Critical appraisal

The methods section should describe how the quality of each of the selected papers for inclusion in a review will be judged. Such judgements relate to how well the study was conducted and reported.

Study quality is important to the credibility or trustworthiness of the evidence a study provides. There is no single way to make judgements about study quality, as this depends on a range of factors. There are different ‘tools’, usually as types of checklists. These include the Critical Appraisal Skills Programme (CASP) series, risk of bias assessments and checklists for qualitative and other types of studies compiled by the Joanna Briggs Institute (JBI).[4]   [5]a research and development centre in South Australia promoting and supporting evidence-based practice (https://jbi.global/ebp) 

Study Quality Appraisal Tool

The Study Quality Appraisal Tool (SQAT) is designed to make the critical appraisal process simpler. It includes general statements that can be used with different study designs. It was developed for an undergraduate applied health research subject.[6]

SQAT has nine statements related to different aspects of research studies. Each statement has a scale where you indicate your level of agreement in terms of how that statement applies to the study in the paper you are appraising. For example, if you strongly agree with a statement, you circle the number corresponding to that response (in this case, 7).

However, you will have to work out exactly how you will apply the tool in your review. Some options are suggested below under Summarising appraisals. There is no one way to assess study quality and different people will make different judgements using the same tool. This is why it is important to describe the process that you use so that others can understand how you approached this step.

Ideally at least two members of the review team should appraise each paper independently. Then they compare the ratings they gave each item (or statement) and discuss any differences. After this discussion, the rating scores can be adjusted. If consensus cannot be reached, a third member of the review team can be consulted.

When using any appraisal tool, if you find that some issues are unclear, report whether this is because the study authors have described something poorly or not at all. Consider too whether any perceived lack of clarity possibly identifies a gap in your own knowledge and think what you might do to resolve that.

H5P activity: Study Quality Appraisal Tool (SQAT)

Choose a study relevant to an assessment task or an area of interest, and using SQAT, indicate on each scale your judgement as to how well the study addresses the corresponding statement.

Summarising appraisals

When using appraisal tools like SQAT, you need to think about how you will summarise your appraisals across the different statements or items and describe the process you use in your review. This summarising process should be decided before you appraise all the studies. Ideally, judgements about study quality are reported with other study details. Some options include:

  • Summing the scores for each item and allocating ratings for scores that fall within certain ranges – for example, if the score falls between 9 (lowest possible score) and 36 you could say this was a LOW QUALITY study, with scores between 37 and 50 indicating a MEDIUM QUALITY study etc.
  • Choosing one or two criteria that you think are particularly important and ranking studies on the basis of how they score for those criteria.
  • Any other way that makes sense to you and that you can justify.
  1. Haddaway, N. R., Bethel, A., Dicks, L. V., Koricheva, J., Macura, B., Petrokofsky, G., ... & Stewart, G. B. (2020). Eight problems with literature reviews and how to fix them. Nature Ecology & Evolution, 4(12), 1582–1589. https://doi.org/10.1038/s41559-020-01295-x ↵
  2. Booth, A. (2006). ‘Brimful of STARLITE’: Toward standards for reporting literature searches. Journal of the Medical Library Association, 94(4), 421. https://doi.org/10.1071/ah18108 ↵
  3. University of Sydney [UniSydneyLibrary]. (2018, July 24). Using keywords and subject headings [Video]. YouTube. https://www.youtube.com/watch?v=Temn-L4CbEs ↵
  4. https://libguides.library.usyd.edu.au/c.php?g=594445&p=6326999 ↵
  5. La Trobe University Library. (2024). Tools and guides. In Critical appraisal for health [online]. https://latrobe.libguides.com/criticalappraisal/tools-guides ↵
  6. Horey, D., & Nava Buenfil, F. (2016). 2AHR Study Quality Appraisal Tool [Tool]. La Trobe University. ↵
definition

Licence

Icon for the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Practical introduction to writing reviews in health research Copyright © 2024 by La Trobe eBureau is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted.

Share This Book