top of page
Search

Constructing a Systematic Review

  • Writer: Daniel Amartya
    Daniel Amartya
  • May 8, 2020
  • 6 min read

How is a systematic review constructed?

  • A systematic review is simply a summary of all available evidence BUT made with a robust, systematic plan (there are steps that we need to follow and in order).

  • The search strategy is specific towards answering the research question and every component in the inclusion/exclusion of evidence should be backed with a reason

  • Each piece of research evidence is then rated for quality using a validated 'scoring' system. Here we identify how reliable the results of the included studies are. Although a systematic review provides the strongest piece of evidence, if the included studies are trash then .. (trash in, trash out)

  • Evidence tables are created to see the differences between the results across different studies

  • Individual studies are integrated into a summary, however it may not be possible to statistically combine the results of all the studies reviewed.


How do I do a systematic review?



STEP 1: Plan and assemble the team


Things to consider before you start a systematic review

  • Set aside some time since systematic reviews are known to take about 20 months

  • Work together with people who are experts in the field of interest. You will also need to work with a librarian (someone who can help construct the search strategy) and a statistician (someone who can assist with data analysis)

  • Reviewers to screen abstracts and read the full text



STEP 2: Define the question


Define the Question

  • A well-defined review question is essential to your systematic review

  • A good research question is the back bone if the review. Since the structure and sequence of your literature search depends on the research question.

  • A clinical question needs to be directly relevant to the patient or problem at hand and phrased in such a way to facilitate the search for an answer


FINER

FINER is an acronym for feasible, interesting, novel, ethical and relevant. The FINER framework is used for formulating research questions.

  • Feasible: is there an adequate number of subjects? technical expertise? Affordable in time and money? Manageable in scope?

  • Interesting: will it catch the attention of investigators, peers and the community?

  • Novel: confirms, refutes or extends to previous findings?

  • Ethical: amenable to a study that an institutional review board will approve?

  • Relevant: to scientific knowledge? to clinical and health policy? to future research?


PICO

A well-established format for structuring research questions is known by the acronym PICO, which stands for:

  • Patient or population

  • Intervention or indicator

  • Comparator or control

  • Outcome


STEP 3: Create a review protocol


Determine what studies get included in your systematic review:

  • Define your selection criteria (inclusion/exclusion criteria) - should relate to the areas defined by the PICO

  • Do you want to impose any limits? (e.g. language, publication type, study design)

  • How will I manage the collected data?

  • How will study selection be performed (who all will be involved in the process?)

  • What will I use to see if the study is of good or poor methodological quality?

  • What kind of data will extract?

  • How will I synthesise my results?


These are the questions you have to ask yourself when creating a review protocol.



Inclusion/exclusion criteria

  • Inclusion and exclusion criteria set the boundaries for the systematic review

  • Usually determined after setting the research question but before the search is conducted criteria

  • Usually recorded as a paragraph or table within the methods section of the systematic review

  • It may also be necessary to give the definitions, and source of the definition, used for particular concepts in the research question (e.g. definition of hypertension according the the American Heart Association 20XX guidelines)


Some of the categories included in an inclusion./exclusion criteria are:

  • Date (e.g. studies published within the last 5 years)

  • Exposure of interest: (e.g. only include patients that were implanted with device A, B, C and not D)

  • Geographic location of the study (e.g. only studies that were done in the Australian population)

  • Language (e.g. Only studies published in English language)

  • Participants (e.g. healthy men and women aged 21-50)

  • Peer review (all studies are peer-reviewed unless stated otherwise, 99% of the papers you find in PubMed and friends are peer-reviewed)

  • Reported outcomes (e.g. outcome of interest includes heart rate, blood pressure, rate of minor complication, rate of major complication, etc)

  • Setting

  • Study design (e.g. only include studies that are randomised controlled trials)

  • Type of publication (e.g. published studies and grey literature



Study eligibility: what are the consequences of being too inclusive or exclusive?

Too inclusive (search strategy and eligibility criteria too vague)

  • Scope is too large

  • Lose focus of the question

  • Main point may be lost

  • May be difficult to interpret


Too exclusive (search strategy and eligibility too specific)

  • Scope is too narrow

  • Potential to exclude important trials/studies

  • May end up not having enough evidence

  • If unaware, could lead to biased conclusions



Search strategy

  • The search strategy should be designed to identify the maximum number of studies relevant to the research question - should be systematic, transparent and reproducible

  • The goal of the literature search is to discover all studies that meet the inclusion criteria

  • Search comprehensively: terminology and databases

  • Search for unpublished studies, this helps reduce risk of publication bias (only including studies that are published)


What are some databases that you can use?

  • PubMed

  • Cochrane

  • Embase

  • CINAHL

  • PsycINFO

  • Web of Science



Publish protocol

  • High quality systematic reviews register their protocols prior to commencing

  • Some journals require you to have your protocol registered

  • You can register your protocol into PROSPERO - International prospective register of systematic reviews. However, it is not recommended that a protocol be published if you do not plan to commit to publishing the systematic review


STEP 4: Conduct search of literature


Conduct and document your search

Conduct the search using databases since you can save you searches in the database

Compile search results using reference management software (such as EndNote or Mendeley)


STEP 5: Screen studies (per protocol)


Identifying the right studies - the screening process

There are two stages in screening:

  • Preliminary: examine title and abstract to remove obviously irrelevant material/duplicates - no need to provide justification for you exclusions

  • Secondary: screen full text based on pre-set eligibility criteria (strictly follow) - must provide reasons why you excluded the documents


To reduce bias, you must have a minimum of TWO REVIEWERS to screen results (yourself and someone else from your team).


Things to document/keep record of when screening for papers:

  • How you searched for the papers in each database

  • What keywords or index terms were used

  • The date or when the search was conducted

  • How many results were retrieved

  • How many were removed as duplicates

  • The final number of discrete studies you subjected to your first sift through of study selection


PRISMA flow diagram

  • Depicts the flow of information through the different phases of a systematic review

  • Documents the number of studies that remain after each stage of the selection process

  • Maps the number of studies identified, included and excluded, and the reasons for being excluded


STEP 6: Appraise literature


Quality assessment and critical appraisal:

  • Should be done for all included studies

  • There are standardised critical appraisal tools available

  • Read the instructions for each tool carefully so that it is used properly

  • Quality may be used as an explanation. for differences in study results or to guide interpretation of findings, strength of inferences



STEP 7: Extract data


Data extraction:

  • Two reviewers complete this step: so they extract the study quality data (critical appraisal) using the same process data extraction (you can use a form to help)

  • Regular discussions of the progress between the two reviewers to ensure consistency

  • Only extract the relevant information: studies contain lots of information, only select those that you are interested in

  • Extraction should be comprehensive to prevent revisiting papers

  • Data extraction forms are available from Cochrane, Joanna Briggs Institute to assist with the process


STEP 8: Analysis/synthesise results


Data Synthesis


Data synthesis is when we compile and analyse the results of the included studies to answer the research questions. How we synthesise the data depends on whether the included studies are homogenous or heterogenous. The two types of data synthesis are:

  • Meta-analysis: statistical combination of results of each of the study to create a larger study

  • Narrative synthesis: description of results from each study to provide an overall "picture" - can be subjective


Heterogeneity

  • The difference in results should not be pooled in together

  • The variation between studies is often considered a weakness of a systematic review

  • Since if the studies are inconsistent across the studies, we must be wary of drawing a conclusion.


Drawing conclusion

  • General interpretation of the results in the context of other evidence and implications for practice

  • Factors to consider: quality of evidence of each outcome. benefits and harms, values and preferences of the patient

  • Also report factors that might influence a decision such as cost, effects of the intervention, availability of the resources. As these might indirectly affect the experiment

  • WHEN there is an inconclusive evidence, it is wrong to claim that it shows that an intervention has ' no effect' or is 'no different' from the control intervention


DO NOT REACH A CONCLUSION THAT GO BEYOND THE EVIDENCE


The importance of identifying outcomes

  • Preliminary classification of outcomes as critical, but not important

  • Basically this just shows how much you need to talk about one of the outcomes.


Grading the evidence

There are several tools you can use to help draw a conclusion:

  • GRADE - Grading of Recommendation Assessment Development and Evaluation

  • CEBM levels of Evidence, Oxford Centre for Evidence Based Medicine

  • AGREE-II Appraisal of Guideline for Research and Evaluation


STEP 9: Write up the results


Basically... Follow the PRISMA checklist!


STEP 10: PUBLISH

Recent Posts

See All
Study Design

Bradford Hill criteria The Bradford Hill criteria, otherwise known as Hill's criteria for causation, are a group of 9 principles that can...

 
 
 

Comments

bottom of page