Module 3.6 The Study Types
Remember that while general principles apply, critical appraisal will be a little different for each study type.
Consider the question types or study types below for more detail on applying these critical appraisal principles according to the context of your question or resource, including EBP calculations for each question type.
Study type: Diagnosis
Learn more about:
Validity
Validity
Are the objectives clearly stated?
Where do I look?
- Abstract
- Introduction
Did the authors have clear research objectives when the study began?
Are the analyses and presented results congruent with the original research objectives?
Did the clinicians face diagnostic uncertainty?
Where do I look?
- Patients and methods, specifically inclusion and exclusion criteria
Were subjects drawn from a group in which it was not known whether the condition of interest was present or absent?
Was the test evaluated in a representative spectrum of patients, similar to those that would be encountered in practice?
Where do I look?
- Patients and methods (usually displayed in Table 1)
Were the full spectrum of cases included – mild to severe, early to late?
Random or consecutive selection will minimise bias.
Was the reference standard test done regardless of the index test results? i.e. Did the results of the test being evaluated influence the decision to perform the ‘gold standard’?
Where do I look?
- Methods
Ideally, all study participants should have both the test being studied and the reference test.
In some situations where the reference test is expensive or invasive, a reasonable substitute is adequate follow up of those patients with negative index test results to see whether the diagnosis of interest remains truly absent over time.
Was there an independent, blind comparison between the index test and the ‘gold standard’?
Where do I look?
- Methods
The reference standard should be the current ‘best test’ that is closest to ‘truth’. This may include a combination of tests if that is the current way of determining the diagnosis.
If you are unsure whether the reference test was the current ‘gold standard’ you should look this up.
The investigator interpreting the results of each of the tests should be unaware of the results of the other test, i.e. blinded.
Were the methods for performing the test described in enough detail for the tests to be replicated?
Where do I look?
- Methods
- References
A reference may be given to another methodology paper which you will need to read.
What is the accuracy of the test?
Where do I look?
- Results
How does the test perform in the population being tested (post-test probabilities)?
Are results discussed in relation to the existing knowledge and is the discussion biased?
Where do I look?
- Discussion
The discussion should place results into a clinical context and the authors conclusions should be justified by the study results and where possible supported by evidence.
What level of evidence does this paper give?
Where do I look?
- Methods
Can you assign a Level of Evidence using the Oxford Centre for Evidence-Based Medicine (CEBM) Levels of Evidence Hierarchy?
Clinical importance
Clinical importance
Does the test improve patient outcomes?
Where do I look?
- Abstract
- Introduction
- Results
- Discussion
Consider whether patient care will differ for different test results.
Will these changes do more good than harm? Is the target disorder dangerous if left untreated?
Acknowledge and understand the risks of the new test.
Are confirmatory tests in patients with false positive index tests risky?
Ask yourself if effective treatment exists for patients with a positive test.
Applicability
Applicability
Are the results applicable to your patient or a patient in your care?
Where do I look?
- Introduction
- Discussion
Does the test perform differently for different severities of disease?
Does the test perform differently for populations with varying comorbidities?
Will the results change your management strategy?
Where do I look?
- Introduction
- Background reading
Think about the test and treatment thresholds for the health condition to be detected. Will you initiate treatment if this test is positive?
Can you reproduce the test and correctly interpret it in your clinical setting?
Where do I look?
- Results
- Discussion
Does the test yield the same result when applied to the same stable patient twice (is it reproducible?) Do different observers agree about the test results? Consider whether inter-observer variability has been analysed or addressed.
Ask yourself whether you have the same technology and expertise available.
How would I clearly express the results to a colleague or my patient?
Where do I look?
- Results
Extract data that helps you describe the study findings to a patient or colleague in plain English.
Can you perform EBP calculations to help you do this? Try putting the Positive Predictive Value (PPV) or Negative Predictive Value (NPV) into a sentence for your patient. For example, “based on this test, there’s about an 85% chance that you have the condition if your result is positive.”
EBP calculations
EBP calculations
| Gold standard positive (condition present) |
Gold standard negative (condition not present) |
|
| Test positive | True positive A |
False positive B |
| Test negative | False negative C |
True negative D |
| Sensitivity = A / (A + C) | Positive predictive value = A / (A + B) |
| Specificity = D / (B + D) | Negative predictive value = D / (C + D) |
- Likelihood ratio for a positive test result = sens/(1-spec)
- Likelihood ratio for a negative test result = (1-sens)/spec
- Pre-test probability (prevalence) = (A+C)/(A+B+C+D)
- Pre-test odds = prevalence/(1-prevalence)
Study type: Clinical practice guidelines
Critical appraisal of clinical practice guidelines requires a slightly different approach from other types of critical appraisal because the focus is not just on the evidence itself, but also on how the guideline was developed. While the same EBP apply, greater emphasis is placed on the rigour of the development process, the transparency of recommendations, and the applicability and flexibility of the guidelines across different clinical settings and patient populations.
Learn more about:
Validity
Validity
Are the objectives of the guideline clearly stated?
Where do I look?
- Introduction
The objectives of the guideline should be stated in an introduction setting out the purpose, scope, and target readership.
Who were the guideline authors?
Where do I look?
- Introduction
- Acknowledgements
- Appendix
The guideline should document authorship or group membership and may classify this by clinical interest.
Many guidelines are developed by a multidisciplinary group, thus involving important different perspectives in patient care.
Check that conflicts of interest are declared and dealt with adequately.
Assess the credibility of the authors.
Is the funding support for guideline development clearly identified?
Where do I look?
- Introduction or title page
- Acknowledgements
- Appendix
The agency or funding group should be identified. If external funding was received, look to see that conflicts of interest are declared and whether potential biases from the funding source were considered.
How did the authors identify and classify the major issues to be addressed, and have they described this process?
Where do I look?
- Introduction
- Appendix
An introduction or appendix should describe how the authors decided which questions were important, and how these questions were constructed.
Alternatively, the guideline may reference a publication describing the process and state that this process was followed.
Was a systematic review of evidence used to answer each question?
Where do I look?
- Introduction
- Appendix
Ideally a published systematic review or a comprehensive search for all relevant studies should be described or referenced.
Information should be adequate to ensure that the review methodology minimised bias.
Was follow-up sufficiently complete and was it long enough?
Where do I look?
- Introduction
- Recommendations
- Tables
- Appendix
Each recommendation should be supported by a level or grade of evidence.
The levels of evidence should be defined at some point in an introduction or appendix.
Clinical importance
Clinical importance
Did the authors assess the body of evidence and give an ‘evidence statement’ including benefits and risks before formulating each recommendation?
Where do I look?
- Body text
- Point form within text
- Tables
A description of methods used to assess the strength of the evidence should be included.
The authors should have taken the evidence identified in the systematic review into account in formulating each recommendation.
The key points of the evidence should be summarised for the reader.
Is each recommendation referenced to the published research?
Where do I look?
- Body text
- References
Readers should be able to identify the published research from the guideline, either with each recommendation or in the body of the text.
Applicability
Applicability
Have patient’s individual situations, values and preferences been discussed in recommending implementation of the guidelines?
Where do I look?
- Discussion
- Conclusion
The influence of individual patient variation and tailoring to the individual should be discussed.
Additional information on subgroups (e.g. elderly, comorbidities) should be presented if possible.
Have resource and economic considerations been discussed in recommending implementation of the guidelines?
Where do I look?
- Discussion
- Conclusion
The guidelines should consider cost-effectiveness and reduce inappropriate resource use.
Can the guidelines take into account clinically sensible variations in practice?
Where do I look?
- Body text
- Discussion
- Conclusion
Look at whether the guidelines can be implemented flexibly, and whether different management options are given where the evidence supports more than one alternative.
Is the guideline written in clear, unambiguous language?
Where do I look?
- Guideline
Language must be appropriate for the audience, or the guidelines will not be effective.
Are the guidelines recent or regularly updated?
Where do I look?
- Title page
- Appendix
- Addendum or supplementary guideline
Revisions should take place every three to five years, or more often (or with supplements) if the field is rapidly changing.
Have the guidelines been peer-reviewed, sent for public consultation, and reviewed by relevant professional groups?
Where do I look?
- Introduction
- Appendix
- Title page
Peer review ensures the validity of the guideline development process, the comprehensiveness of the guideline, and the applicability in a broad setting.
Professional group endorsement aids in dissemination and application of guidelines.
Study type: Therapy or Prevention
Learn more about:
Validity
Validity
Are the objectives clearly stated?
Where do I look?
- Title, abstract or final paragraph of the introduction
The therapy and the outcome(s) of interest will often be expressed in terms of a simple relationship but not necessarily a Population Intervention Comparison Outcomes (PICO) question.
Try to phrase the research question as a PICO question.
Was the study design suitable for the objectives?
Where do I look?
- Methods
A question of therapy is best answered by a randomised controlled clinical trial. Ensure the right population was studied to answer the study objectives.
Check whether the most appropriate control arm was used.
Were outcome measures objective?
Where do I look?
- Methods
Are the outcome measures objective and well validated?
If outcome measures were not objective, check whether observers and patients were blinded to treatment allocation and whether placebo or sham treatment was used.
Were the right outcome measures used?
Where do I look?
- Methods
Ask yourself if the right outcome measures were used for the study endpoints.
Was the assignment of patients to treatments randomised?
Where do I look?
- Methods
Any method of patient selection is less acceptable than randomisation. Ideally, randomisation will be central and computerised.
Were the groups comparable at baseline?
Where do I look?
- ‘Table 1’ usually shows baseline characteristics of participants.
Randomisation should ensure an even distribution of important variables between study arms. There should be some indication (i.e. p values) of whether differences between the two groups may affect the results, and the effect of that difference should be commented on in the discussion.
Look at whether the patients all began with a similar prognosis.
Was stratification performed for important prognostic variables?
Ensure the groups were treated the same other than the intervention.
Were all the patients who started the study accounted for at the end?
Where do I look?
- CONSORT diagram
- Results
Study drop-outs, cross-over and loss to follow up should be accounted for.
A CONSORT diagram or text in results should give details of what happened to all patients. Numbers should add up. Reasons for losses to follow up should be given, as this may not be random and may bias the study results.
Statistical methods may be needed to address this.
Were patients analysed in the groups to which they were randomised?
Where do I look?
- Methods or statistical methods section
Patients should be analysed for the efficacy of the intervention in the groups to which they were originally randomised, regardless of whether they received the treatment or not (intention to treat analysis or modified intention to treat).
Did they make statistical hypotheses before the study started?
Where do I look?
- Statistics
The authors should report:
- how the sample size (number of patients) was calculated
- whether all planned patients were recruited and if not, why not
- if the study design changed while the study was underway
Were the results combined appropriately if a meta-analysis was done? For example, was like combined with like?
Are the study results accurately described?
Where do I look?
- Results
- Discussion
Ask yourself if the results are presented clearly, objectively and in enough detail to allow readers to interpret them.
Check that:
- the numbers add up properly
- the conclusions follow from their results
- the discussion is unbiased
- the findings are placed in the context of other literature
Clinical importance
Clinical importance
Are the outcomes clinically relevant?
Where do I look?
- Methods
- References
Ensure that the study outcome measures have been shown to relate to clinically important outcomes.
How large was the treatment effect?
Where do I look?
- Results
- Conclusions
What is the magnitude of the difference between the treatment and control groups?
How clinically meaningful is the reported effect?
These questions can help you to determine whether the treatment had a meaningful and substantial impact on the clinical outcome.
How precise was the estimate of treatment effect?
Where do I look?
- Results
The ‘true’ effect is not known and the study results give us an estimate of the effect, together with a measure of precision.
We can gauge how close this estimate is to the ‘true’ effect using the 95% confidence interval (CI).
A narrow CI means that our estimate is likely to be quite close to the true effect. If the value corresponding to no effect falls within the 95% CI, the results are not statistically significant. These findings could be by play of chance alone.
Have side effects been adequately described and do the benefits outweigh the risks of the therapy?
Where do I look?
- Results
What proportion of patients suffered serious side effects? Did any patients suffer side effects that were worse than the effect of the disease?
Is a Relative Risk, Absolute Risk Reduction or Number Needed to Treat (harm) given?
Where do I look?
- Results
- Tables
If these are not given, you can use EBP calculations.
More information about Relative Risk (RR), Absolute Risk Reduction (ARR), and Number Need to Treat(NTT) (harm).
Applicability
Applicability
How would I clearly express the results to a colleague or my patient?
Where do I look?
- Abstract
- Results
- Discussion
If you are stuck, try doing some simple EBP calculations and putting NNT, ARR, or NNH into a simple sentence for your patient. Speaking to your colleague and a patient will require different approaches, as the way information is conveyed must be tailored to the audience.
More information about Relative Risk, Absolute Risk Reduction, and (RR), (ARR), and (NTT) (harm).
Does this paper answer your clinical question or have you changed your question to suit the literature?
Where do I look?
- Consider your patient
Check back with yourself to see if you have changed your question. Ask yourself if your altered question is more or less relevant to your patient.
Try another search or another paper if the paper answers questions that are not meaningful for your patient.
How similar were the included patients to your patient or population? Would your patient have been eligible for the clinical trials?
Where do I look?
- Patients and Methods
- Tables
See if the inclusion and exclusion criteria fit your patient.
Is treatment feasible and available in your clinical setting?
Where do I look?
- Consider your practice setting
Consider whether you can reproduce all important aspects of the intervention.
EBP calculations
EBP calculations
What is the Control Event Rate in the trial? (CER)
What is the Experimental Event Rate in the trial? (EER)
Relative risk reduction (RRR) = CER – EER
CER
Absolute risk reduction (ARR) = CER – EER
Number needed to treat (NNT) = 1
ARR
Online Clinical Significance Calculators:
- Mercer University MUSM Libraries: Calculators
- Chinese University of Hong Kong Centre for Clinical Research and Biostatistics C.I. Calculator: Odds Ratio, ARR, RRR, NNT, PEER
- UWA staff and students can use the Calculators available in the ClinicalKey database. Click on the ‘Other’ category then scroll to ‘Statistics’ to see calculators for NNT, NNH, risk, sample size and other relevant calculators for EBP.
Study type: Systematic reviews
Critical appraisal of systematic reviews also requires a slightly different approach to other critical appraisal tasks, again using the same EBP. More emphasis is placed on the methodology of the review, rather than the individual studies. If a meta-analysis is included, it is important to ensure that the studies have been appropriately combined.
Learn more about:
Validity
Validity
Did the review explicitly address a focused clinical question?
Where do I look?
- Introduction (title, abstract or final paragraph)
The main question being addressed should be clearly stated. The exposure, such as a therapy or diagnostic test, and the outcome(s) of interest will often be expressed in terms of a simple relationship but not necessarily a PICO question.
Was the search for relevant studies detailed and exhaustive?
Where do I look?
- Methods
- Results
Ideally includes a comprehensive search for all relevant studies in the major bibliographic databases (e.g. Medline, Cochrane, EMBASE, etc) and a search of reference lists from relevant studies, contact with experts, and conference abstracts. The search strategy should be included so that the search can be repeated.
The search should not be limited to English language only. The search strategy should include both MeSH terms and text words and should be reproducible.
The results section will outline the number of titles and abstracts retrieved and reviewed and the number of full-text studies retrieved.
Was the selection of primary studies reproducible and free from bias?
Where do I look?
- Results
- Figures
- Inclusion and exclusion criteria
Ideally the authors should define transparent inclusion and exclusion criteria for the review.
The selection of studies should be reproducible. The methods section should describe the inclusion and exclusion criteria for the review.
The results section will outline the number of studies included/excluded together with the reasons for exclusion. This information may be presented in a figure or flow chart.
Was the quality of included studies assessed, and were they of a high standard?
Where do I look?
- Methods
- Results
- Tables
The article should describe how the quality of each study was assessed using predetermined quality criteria appropriate to the type of clinical question (e.g. randomisation, blinding and completeness of follow-up). Results should be reproducible.
The methods section should describe the assessment of quality, and the criteria used (assessment of quality blinded to authors/title/journal is ideal).
The results section should provide information on the quality of the individual studies which may be tabulated.
Were all the important outcomes considered?
Where do I look?
- Methods
Study outcomes should have been defined appropriately and should consider all clinically relevant outcomes.
Are the individual studies adequately described?
Where do I look?
- Results
- Tables
- Appendix
Important characteristics of individual studies should be described succinctly.
The Results section should include a table or summary of important characteristics of included studies.
This may be an Appendix in Cochrane Reviews, or available as supplementary data online for other papers.
Were the results of primary studies combined appropriately?
Where do I look?
- Methods
- Statistics
- Results
Any meta-analysis should combine the same outcome measures from individual studies.
The results section should show which outcomes were combined.
How are the results presented and is this appropriate to the data?
Where do I look?
- Methods
- Statistics
- Figures
- Results
A systematic review can include a meta-analysis if the data are appropriate to combine in this way.
A meta-analysis combines the results of individual studies and produces a summary estimate of the intervention effect. This weights individual studies according to their size.
Results are expressed in a standard way, such as relative risk, odds ratio, or mean difference between groups.
Results are often displayed as a Forest plot, where individual studies are represented with a black square and horizontal line corresponding to the point effect of the study (where the square sits), the size of the study (size of the square), and the 95% CI (black line). A diamond at the bottom represents the pooled effect of all trials and the combined 95% CI. If the diamond does not overlap ‘1’, we know that the pooled effect is statistically significant.
Corresponding figures may include Odds Ratio or Hazard Ratio with 95% CI, weight (% of total) of the studies, and the number of events/patients for individual studies.
If the results are not suitable for meta-analysis, it is also valid to present them in a tabular form without statistical synthesis.
Example Forest Plot:
Were the results similar from study to study?
Where do I look?
- Methods
- Results
- Figures
Ideally, the results of the different studies should be similar or homogeneous. If heterogeneity exists the authors may estimate whether the differences are significant (chi-square test).
The results section should state whether the results are heterogeneous and discuss possible reasons for heterogeneity. The forest plot should show the results of the chi-square test for heterogeneity.
Has a sensitivity analysis been performed?
Where do I look?
- Methods
- Results
A sensitivity analysis asks whether the results would change if the study inclusion criteria were changed.
For example, what happens if we narrow the meta-analysis to include only adults? Or only rigorous studies? It may or may not be appropriate to perform sensitivity analyses.
Clinical importance
Clinical importance
Are the outcomes clinically relevant?
Where do I look?
- Methods
Check that the study outcome measures relate to the. clinically important outcomes This also relates to the research question and may involve both primary and secondary outcomes.
How large was the treatment effect in meta-analysis?
Where do I look?
- Results
- Figures
Have the results been presented in a way that you can understand them?
How precise was the estimate of treatment effect?
Where do I look?
- Results
- Figures
A 95% CI and p value give an estimate of the precision of the results.
Are the benefits worth the costs and potential toxicities?
Where do I look?
- Results
Have the authors also addressed toxicities and economic considerations in the review? Look for meta-analysis of toxicities, which may include only a subset of studies where this information was available from the original publication.
Is a Relative Risk, Absolute Risk Reduction or Number Needed to Treat (harm) given?
Where do I look?
- Results
- Tables
The meta-analysis can provide these summary measures of effect but depends on the data available and the specific analysis conducted. If not, you can calculate them.
More information about RR, ARR and NNT (harm)
Applicability
Applicability
Are the results discussed in relation to existing knowledge, and is the discussion biased?
Where do I look?
- Discussion
The discussion should place results into a clinical context and the authors conclusions should be justified by the study results.
How would I clearly express the results to a colleague or my patient?
Where do I look?
- Abstract
- Results
Try to extract data and describe the study findings to a patient or colleague in plain English. Use EBP calculations to help you do this.
Put a NNT, ARR, NNH into a sentence for your patient.
More information about RR, ARR and NNT (harm).
Does this paper answer your clinical question, or have you changed your question to suit the literature?
Ask yourself whether the paper helps you answer your clinical question.
Ask yourself if your altered question is more or less relevant to your patient. Try another search or another paper if the meta-analysis answers questions that are not meaningful for your patient.
How similar were the patients in the included trials to your patient or population?
Where do I look?
- Methods
- Tables
- Figures
Check whether your patient would have been eligible for the clinical trials.
Identify any important characteristics your patient has which have not been considered in the systematic review.
Look for a sub-set analysis that has been done for a group more like your patient.
Is treatment feasible and available in your clinical setting?
Where do I look?
- Consider your practice setting
Do you and your colleagues have the right skills to deliver this intervention?
Consider whether the intervention or test is funded by insurers, Medicare, or the PBS.
Study type: Prognosis
Learn more about:
Validity
Validity
Are the objectives clearly stated?
Where do I look?
- Title, abstract or final paragraph of the introduction
The main question being addressed should be clearly stated. The question will often be expressed in terms of a simple relationship but may not fit a PICO question.
Were objective and unbiased outcome criteria used?
Where do I look?
- Methods
- Statistical methods
This is simpler with clear outcomes like death.
Where the outcome requires clinical judgement (e.g. assessment of disability) then criteria for that outcome should be well defined and validated, and ideally assessors should be blinded to whether the patient had the potential prognostic factor.
Was there adjustment for important prognostic variables?
Where do I look?
- Methods
- Statistical methods
If the prognosis of two groups of patients is compared, their clinical characteristics should be similar at baseline, or the analysis should include statistical adjustments for prognostic variables.
Adjustment should also be made for treatments which may confound the study outcomes.
Did the study have a sufficiently large sample size?
Where do I look?
- Methods
- Statistical methods
Larger samples usually mean more precise results.
Online sample size calculators:
- Vanderbilt University Department of Biostatistics ‘PS’ Calculator
- ABS Sample Size Calculator
- UWA staff and students can use the Calculators available in the ClinicalKey database. Click on the ‘Other’ category then scroll to ‘Statistics’ to see calculators for NNT, NNH, risk, sample size and other relevant calculators for EBP.
Was the sample of patients representative, well-defined and at a similar point in the disease?
Where do I look?
- Patients
- Methods
Look at how the target disorder was defined in the inclusion criteria. Did they select the sample from a tertiary hospital setting or the community? Are all patients at the same point in the disease? Duration of disease may relate to prognosis.
Was follow-up sufficiently complete and was it long enough?
Where do I look?
- Methods
- Results
A flow diagram may show loss to follow up. Was the follow up time long enough for the outcome they were looking for to occur? Loss to follow up may not be random and may relate to either a good or a bad outcome.
A ‘sensitivity analysis’ can investigate the effects of loss to follow up. For example, the investigators may re-analyse the data assuming that all patients lost to follow up died. This can show how the ‘worst case scenario’ would affect the results.
Clinical importance
Clinical importance
How have the outcomes over time been quantified?
Where do I look?
- Results
May be expressed as a likelihood of an event in x years, months or weeks, an odds ratio, relative risk or displayed as a ‘time to event’ curve (Kaplan Meier Curve).
A Kaplan Meier curve shows how the chance of the event occurring changes over time. A survival curve is one type of time to event curve but time to any event may be graphed.
See BMJ’s Statistics at Square One – Survival Analysis for more information on Kaplan Meier curves.
How precise are the estimates of likelihood?
Where do I look?
- Results
A 95% CI should be given so that the reader can understand the precision of the result.
Are the results discussed in relation to existing knowledge and is the discussion biased?
Where do I look?
- Discussion
The discussion should place the results into a broader clinical context and the authors conclusions should be justified by the study results. This involves evaluating how the findings relate to existing research, their relevance to practice and the strength of the evidence supporting these conclusions.
What level of evidence does this paper give?
Where do I look?
- Methods
Try to assign a level of evidence using the Oxford CEBP Levels of Evidence Hierarchy.
Applicability
Applicability
Were the study patients and their management similar to those in my practice?
Where do I look?
- Methods (inclusion and exclusion criteria)
Decide if you can generalise the study population to your patient.
Would your patient have been included in the study?
Are they in the same setting?
Do they have the same severity of disease?
Are the results useful in patient management in your practice?
Where do I look?
- Consider your patient and practice
Does this paper answer your clinical question or have you changed your question to suit the available literature?
Look for important geographical and racial differences in the disease.
Consider whether your control treatment would be the same as the study control arm.
Ask yourself whether your patients present at the same disease stage.
How would I clearly express the results to a colleague or my patient?
Where do I look?
- Results
Try to extract data that helps you to describe the study findings to a patient or colleague in plain English.
Can you perform EBP calculations to help you do this?
Put a NNT, ARR, RRR, or NNH or survival figure into a sentence for your patient.
More information about Relative Risk, Absolute Risk Reduction, and Number Need to Treat (harm).
Study type: Aetiology/harm
Learn more about:
Validity
Validity
Are the objectives of the study clearly stated?
Where do I look?
- Introduction (title, abstract or final paragraph).
The main question being addressed should be clearly stated.
The question will often be expressed in terms of a simple relationship but may not fit a PICO question.
Were exposures and outcomes measured in the same way in the patient groups?
Where do I look?
- Methods
- Statistical methods
The Methods section should define the outcomes and describe how exposures were assessed. In cohort studies and RCTs there should be clear descriptions and definitions of outcomes, and patients in each group should be assessed equally carefully for these outcomes.
In case control studies, beware of:
- recall bias – increased chance that cases will carefully examine and recall their exposure.
- interview bias – more probing by interviewer in cases than in controls.
Look at whether the opportunity for exposure was the same in cases and controls, or whether the groups were dissimilar in this way.
Were the patient groups clearly defined and similar in prognosis other than exposure to the treatment or aetiological factor?
Where do I look?
- Patients
- Methods
How was the patient sample selected? A CT is the strongest design for studies of harm or aetiology, however it may not always be ethical to do a randomised trial to answer these types of questions.
If it is not feasible or ethical to do a RCT, a cohort study may be the best design. Have the characteristics of the different cohorts been well described?
Was follow-up sufficiently complete and was it long enough?
Where do I look?
- Methods
- Results
A flow diagram may show loss to follow up. Check whether the follow up time was long enough for the outcome/harm of interest to occur. Loss to follow up may not be random and may relate to either a good or bad outcome which may bias the findings.
A ‘sensitivity analysis’ can investigate the effects of loss to follow up. For example, the investigators may re-analyse the data assuming that all patients lost to follow up died. This can show how the ‘worst case scenario’ would affect the results.
Did the study have a sufficiently large sample size?
Where do I look?
- Statistical methods
Larger samples usually mean more precise results.
Online sample size calculators:
- Vanderbilt University Department of Biostatistics ‘PS’ Calculator
- ABS Sample Size Calculator
- UWA staff and students can use the Calculators available in the ClinicalKey database. Click on the ‘Other’ category then scroll to ‘Statistics’ to see calculators for NNT, NNH, risk, sample size and other relevant calculators for EBP.
Was there statistical adjustment for important differences between patient groups?
Where do I look?
- Statistical methods
In a study of aetiology or harm, the clinical characteristics of the groups should be similar at baseline, or the analysis should make statistical adjustments for prognostic variables. This is simpler in a RCT but more challenging for a cohort study or case-control study.
Is the temporal relationship plausible?
Where do I look?
- Results
- Discussion
An exposure must happen first to cause an outcome!
Check whether enough time passed for the outcome to have plausibly been caused by the exposure.
Is there evidence of a dose-response?
Where do I look?
- Results
- Discussion
If the risk of a bad outcome increases with the duration or amount of exposure, this is stronger evidence of aetiology.
Is the biological relationship plausible?
Where do I look?
- Introduction
- Discussion
Does an association between cause and effect make biological sense?
Is there evidence of causation from a withdrawal-rechallenge study?
Where do I look?
- Results
- Discussion
If an outcome resolved on withdrawal of the exposure, and reoccurred on re-exposure, this is stronger evidence that the exposure caused the outcome.
Clinical importance
Clinical importance
How strong is the association between exposure and outcome (harm, disease)?
Where do I look?
- Results
Often expressed as a (RR) or (OR). Relative risk cannot be used for case-control studies, as the number of exposed people per case (denominator) is unknown.
In case-control studies, an OR is used. Low OR or RR are hard to interpret from weaker study designs such as case-control or cohort studies, while very high OR or RR can be convincing even when the study design is not randomised.
More information about RR, ARR, NNT (harm).
How precise are the estimates of risk?
Where do I look?
- Results
A 95% CI should indicate the precision of the result.
More information about Relative Risk, Absolute Risk Reduction, and Number Need to Treat (harm).
Are the results discussed in relation to existing knowledge and is the discussion biased?
Where do I look?
- Discussion
The discussion should put the results into a clinical context and the authors conclusions should be justified by the study results.
What level of evidence does this paper give?
Where do I look?
- Methods
Try to assign a level of evidence using the Oxford CEBM Levels of Evidence Hierarchy.
Applicability
Applicability
Were the study patients and their management similar to those in my practice?
Where do I look?
- Methods (Inclusion and Exclusion criteria)
Can you generalise the study population to your patient? Check whether your patient has the same exposure, and whether their other risk factors are similar to the study group.
Are the results useful in your patient?
Where do I look?
- Results
How great is the risk in your patient? The OR and HR tell us about risk relative to an unexposed group.
What is the baseline risk in your patient if he/she is not exposed? How does that change on exposure? Try to calculate a NNT.
More information about Relative Risk, Absolute Risk Reduction, and Number Need to Treat (harm).
EBP calculations
EBP calculations
| Adverse outcome | Present (case) | Absent (control) | Total | |
|---|---|---|---|---|
| Exposed | Yes (Cohort) | a | b | a + b |
| a + c | b + d | a + b + c + d | ||
RCT, cohort study: Relative risk (RR) = [a/(a+b)]/[c/(c+d)]
Case-control study: Odds ratio (OR) = ad/bc
Calculations (2) for studies of aetiology/harm
Number needed to Harm (NNH)
Individualise for your patient by estimating their Event Rate for the adverse event if they were not exposed to the causative factor (PEER = Patient’s Expected Event Rate)
NNH = PEER (OR – 1) + 1
PEER (OR – 1) x (1 – PEER)
