Evaluate your search
Learning Objectives
This sub-chapter will help you:
- test your search.
- correct errors in your search.
“Performing a high-quality electronic search of information resources is an essential contribution toward ensuring accuracy and completeness of the evidence base used in these reports. Furthermore, search quality has resource implications for the conduct of a review, because the number of records retrieved and screened is often very large.” (Sampson et al., 2009, p. 944).
The final stage of developing your search strategy is to test it out, evaluate it, and make any changes needed. It is very likely your search will need tweaking, so don’t get disheartened if it doesn’t work on the first try! Run your search as you have recorded it in the database you developed it in. Ideally, it will work without errors and retrieve the type and amount of results you expect including your seed papers. However, this rarely happens!
It is a good idea to log in to the database as you can then save your search. This saves you time as you won’t need to start from scratch if you need to run the search again later. The way you log in and save searches is different in each database, so refer to the help section in the database to find out how to do this.
Test it out
It is quickest to copy and paste your full search into the database, but this makes it difficult to identify errors. To copy and paste your search, you will need to:
- Double check all the field codes, operators and subject headings are correct for the database.
- Then simply copy and paste the search into the database.
- You may prefer to copy and paste each concept into a different line, and combine with AND.
It is easier to pick up errors if you search line by line and then combine them, but this takes longer. The following activity shows how to search this way.
Activity 6
The following video by Gerstein Science Information Center from the University of Toronto shows how to combine searches in Web of Science.
Gerstein Science Information Centre. (2023, September 30). Web of Science 4: Combining search lines [Video]. YouTube. https://youtu.be/9I96uYuK_9U
If your search does not run as planned, there are some common issues to look for.
- Syntax error: This message indicates a problem with the way your search is put together. It could indicate a missing quotation mark if you are using phrases, incorrect use of brackets or that the operators or field codes are for the wrong database. This type of error is easiest to pick up when searching line by line as you can identify the problematic part.
- Too many or too few results: Through your preliminary search development, you should have a rough idea of how much research there is about your topic. If the number of results is much smaller or larger than what you expected, check the Boolean operators. If you got too many results, you may have used OR to connect different concepts instead of AND. If you got too few results, you may have used AND to connect keywords or subject headings about a concept instead of OR. If you have left out an OR, many databases automatically use AND. If you used proximity operators, you may need to adjust the number of words in between keywords.
- Results are not relevant: Rank your results by relevance to see if the articles you expect to see are included. If you used a set of seed papers to develop your search, these should be in the results.
Activity 7
Finding seed papers in your results
A way to evaluate your search strategy is to see if your seed papers appear in the results. If they don’t, investigate why and modify your search.
Sort results by relevance, and if your papers are in the results they should appear towards the top. If you don’t see them, you can check if they are in the results by combining searches using AND and NOT. Use the database you expect has the greatest coverage for your discipline.
Activity 8
Now you need to investigate why those papers were not found by your search. Start by checking the article is indexed in the database you used. Do this by searching for the article title. If it is not in that database, find out which one it is in, and test your search in that one. Add the new database to your notes as one that you will use in your comprehensive search.
If the missing articles are in the original database, look at the subject headings and words in the title and abstract to find keywords and headings missing from your search. The author may have used a word you are unfamiliar with. There may have been an error in applying subject headings or it is very new and has not yet had subject headings applied. Add the missing keywords or subject headings into your search strategy and test it again to see if that retrieves the article.
Activity 9
Choose a database you have access to. (PubMed is open access if you don’t have access to a subscription database). Do a rough search for the question:
- For parents of young children, is family support or professional counseling more beneficial to parental stress?
Following the instructions above, check to see if the following seed papers are in your results. These are PubMed IDs, so don’t forget to search for them in the PMID field. If you are using PubMed, you can just copy and paste the list directly into the search bar.
- 18922679 OR 31785472 OR 36766996 OR 33599522 OR 34410778 OR 36824302
If any are missing, try to modify your search so they are included in the results.
PRESS checklist
Another great tool to help you evaluate your search strategy is the PRESS Checklist (Peer Review of Electronic Search Strategies) by McGowan et al., (2016). It is a structured way to examine a search strategy and includes:
- translation of the research question into concepts
- boolean and proximity operators
- subject headings
- keywords
- spelling, syntax, and how the concepts are combined
- limits and filters
Further reading
Salvador-Oliván, J. A., Marco-Cuenca, G., & Arquero-Avilés, R. (2019). Errors in search strategies used in systematic reviews and their effects on information retrieval. Journal of the Medical Library Association, 107(2), 210–221. https://doi.org/10.5195/jmla.2019.567
References
McGowan, J., Sampson, M., Salzwedel, D. M., Cogo, E., Foerster, V., & Lefebvre, C. (2016). PRESS peer review of electronic search strategies: 2015 guideline statement. Journal of Clinical Epidemiology, 75, 40-46. https://doi.org/10.1016/j.jclinepi.2016.01.021
Sampson, M., McGowan, J., Cogo, E., Grimshaw, J., Moher, D., & Lefebvre, C. (2009). An evidence-based practice guideline for the peer review of electronic search strategies. Journal of Clinical Epidemiology, 62(9), 944-952. https://doi.org/10.1016/j.jclinepi.2008.10.012
A systematically organised collection of information, such as journal articles.
A set of characters in a database that describes a particular kind of data, such as a title or abstract.
A character or string of characters used in a database to narrow the focus of the search.
A word or phrase that describes all material in a database on that subject.
Terms such as AND, OR, NOT which specify the relationship between concepts required for an item to be retrieved.
A group of known articles about a topic that are used to start developing a search, or test a search.
Also called adjacency operators. Commands entered into a database that allow you to search for two or more terms occurring in close proximity.
Also called hedges. A combination of search terms designed to retrieve studies meeting particular criteria, eg study design, population, geographic setting.
