how to critique a systematic research review article

• Search Menu
• Advance Articles
• Specialty Certificate Examination Cases
• Cover Archive
• Virtual Issues
• Trending Articles
• Author Guidelines
• Submission Site
• Open Access Options
• Reasons to Publish
• Reviewer Guidelines
• Self-Archiving Policy
• About Clinical and Experimental Dermatology
• About British Association of Dermatologists
• Editorial Board
• Advertising & Corporate Services
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Discussion and conclusions, learning points, funding sources, data availability, ethics statement, cpd questions, instructions for answering questions.

• < Previous

How to critically appraise a systematic review: an aide for the reader and reviewer

Conflicts of interest H.W. founded the Cochrane Skin Group in 1987 and was coordinator editor until 2018. The other authors declare they have no conflicts of interest.

• Article contents
• Figures & tables
• Supplementary Data

John Frewen, Marianne de Brito, Anjali Pathak, Richard Barlow, Hywel C Williams, How to critically appraise a systematic review: an aide for the reader and reviewer, Clinical and Experimental Dermatology , Volume 48, Issue 8, August 2023, Pages 854–859, https://doi.org/10.1093/ced/llad141

• Permissions Icon Permissions

The number of published systematic reviews has soared rapidly in recent years. Sadly, the quality of most systematic reviews in dermatology is substandard. With the continued increase in exposure to systematic reviews, and their potential to influence clinical practice, we sought to describe a sequence of useful tips for the busy clinician reader to determine study quality and clinical utility. Important factors to consider when assessing systematic reviews include: determining the motivation to performing the study, establishing if the study protocol was prepublished, assessing quality of reporting using the PRISMA checklist, assessing study quality using the AMSTAR 2 critical appraisal checklist, assessing for evidence of spin, and summarizing the main strengths and limitations of the study to determine if it could change clinical practice. Having a set of heuristics to consider when reading systematic reviews serves to save time, enabling assessment of quality in a structured way, and come to a prompt conclusion of the merits of a review article in order to inform the care of dermatology patients.

A systematic review aims to systematically and transparently summarize the available data on a defined clinical question, via a rigorous search for studies, a critique of the quality of included studies and a qualitative and/or quantitative synthesis. 1 Systematic reviews are at the top of the pyramid in most evidence hierarchies for informing evidence-based healthcare as they are considered of greater validity and clinical applicability than those study types lower down, such as case series or individual trials. 2

A good systematic review should provide an unbiased overview of studies to inform clinical practice. Systematic reviews can reconcile apparently conflicting results, add precision to estimating smaller treatment effects, highlight the evidence’s limitations and biases and identify research gaps. Guidelines are available to assist systematic reviewers to transparently report why the review was done, the authors’ methods and findings via the PRISMA checklist. 3

The sharp rise in systematic review publications over time raises concern that the majority are unnecessary, misleading and/or conflicted. 4 A review of dermatology systematic reviews noted that 93% failed to report at least one PRISMA checklist item. 5 Another review of a random sample of 140/732 dermatology systematic reviews in 2017 found 90% were low quality. 6 Some improvements have occurred: reporting standards compliance has improved slightly (between 2013 and 2017), 5 and several leading dermatology journals including the British Journal of Dermatology have changed editorial policies, mandating authors to preregister review protocols.

Given the surge in poor-quality systematic review publications, we sought to describe a checklist of seven practical tips from the authors’ collective experience of writing and critically appraising systematic reviews, hoping that they will assist busy clinicians to critically appraise systematic reviews both as manuscript reviewers and as readers and research users.

Read the abstract to develop a sense of the subject.

What was the motivation for completing the review?

Has the review protocol been published and have changes been made to it.

Review the reporting quality .

Review the quality of the article and the depth of the review question.

Consider the authors’ interpretation and assess for spin .

Summarize and come to a position .

Read the abstract to develop a sense of the subject

From the abstract, use the PICO (population, intervention, comparator and outcome) framework to establish if the subject, intervention and outcomes are relevant to clinical practice. Is the review question clear and appropriate?

Inspect the authors’ conflicts of interest and funding sources. Self-disclosed financial conflicts are often insufficiently described or not declared at all. 7 If you suspect conflicts for authors with no stated conflicts, briefly searching the senior authors’ names on PubMed, or the Open Payments website (for US authors) may reveal hidden conflicts. 8 Is the motivation for the systematic review justified in the introduction? Can new insights be formed by combining studies? If the systematic review is an update, what new available data justifies this? Search for similar recent systematic reviews (which may have been omitted intentionally). Is it a redundant duplicate review that adds little new useful information? 9 Has the author recently published reviews on similar subjects? Salami publications refer to authors chopping up a topic into smaller pieces to obtain maximum publications. 10

Search PROSPERO for publication of the review protocol. 11 A prepublished review protocol in a publicly accessible site offers reassurance that the systematic review followed a clear plan with prespecified PICO elements. Put bluntly, it reduces authors’ opportunity for deception by selective analysis and highlighting of results that are more likely to get published. If a protocol is found, assess deviation from this protocol and justification, if present. Protocol registration allows improved PRISMA reporting. 12 A registered protocol with reporting of deviations allows the reader to judge whether any modifications are justified, for example adjusting for unexpected challenges during analysis. 10

Review the reporting quality

Look for supplementary material detailing the PRISMA checklist. Commonly under-reported PRISMA items include protocol and registration, risk of bias across studies, risk of bias in individual studies, the data collection process and review objectives. 5 Adequate reporting quality using PRISMA does not necessarily indicate the review is clinically useful; however, it allows the reader to assess the study’s utility (see Table 1 ). Additional assessments of review quality are described below.

The relationship between systematic review reporting quality and study quality a

Adapted with permission from Williams. 21

Review the quality of the article and the depth of the review question

Distinct from quality of reporting completeness, assessing the review's quality allows for assessment of the overall clinical meaningfulness of the results. Does the PICO make sense in respect to this? The AMSTAR 2 critical appraisal instrument is useful in determining quantitative systematic review quality. 13 This checklist marks the key aspects of a systematic review and computes an outcome of the review quality. 14 If meta-analysis was performed, did the authors justify and use appropriate methods for statistical combination of results? Were weighted techniques used to combine results and adjusted for heterogeneity, if present? If heterogeneity was present, were sources of this investigated? Did authors assess the potential impact of the individual study’s risk of bias (RoB) and perform analysis to investigate the impact of RoB on the summary estimates of affect? See Table 2 for an example of a completed AMSTAR 2 checklist on a recently published poor-quality systematic review. 15

An example of assessment of the quality of a systematic review (Drake et al. ) 15 using the AMSTAR 2 checklist an explanation of which can be found at https://amstar.ca/Amstar-2.php

N/A, not applicable; PICO, population, intervention, comparator and outcome. a Denotes AMSTAR 2 critical domain. The overall confidence in the results of the review is dependent on such critical domains. When one critical domain is not satisfied, the confidence is rated as ‘low’ and the review may not provide an accurate and comprehensive summary of the available studies that address the question of interest. When more than one critical domain are not satisfied, the confidence in the results of the review is rated as ‘critically low’ and the review should not be relied on to provide an accurate and comprehensive summary of the available studies.

Quality checklists for assessment of qualitative research include Consolidated Criteria for Reporting Qualitative research (COREQ), Standards for Reporting Qualitative Research (SRQR) and Critical Appraisal Skills Programme (CASP). 16 Such checklists aim to improve identification of high-quality qualitative research in journal articles, as well as acting as a guide for conducting research. 16

Consider the authors’ interpretation and assess for spin

Spin is a distorted interpretation of results. This manifests itself in studies as (i) misleading reporting, (ii) misleading interpretation, and (iii) inappropriate extrapolation. 14 Are the conclusion’s clinical practice recommendations not supported by the studies’ findings? Is the title misleading? Is there selective reporting? These are the three most severe forms of spin occurring in systematic reviews. 17

Summarize and come to a position

Summarize the reviews main positives and negatives and establish if there is sufficient quality to merit changing clinical practice, or are fatal flaws present that nullify the review’s clinical utility? Consider internal validity (are the results true?) and external validity (are the results applicable to my patient group?). When applying the systematic review results to a particular patient, it may help to consider these points: (i) how similar are the study participants to my patient?; (ii) do the outcomes make sense to me?; (iii) what was the magnitude of treatment benefit? – work out the number needed to treat; 18 (iv) what are the adverse events?; and (v) what are my patient's values and preferences? 19

Although systematic reviews have potential for summarizing evidence for dermatological interventions in a systematic and unbiased way, the rapid expansion of poorly reported and poor-quality reviews (Table 3 ) is regrettable. We do not claim our checklist items (Table 4 ) are superior to other checklists such as those suggested by CASP, 20 but they are based on the practical experience of critical appraisal of dermatology systematic reviews conducted by the authors.

The top seven ‘sins’ of dermatology systematic reviews a

Adapted with permission from Williams. 10

Checklist of questions, considerations and tips for critical appraisal of systematic reviews

NNT, number needed to treat.

Considering each question suggested in our checklist when faced with yet another systematic review draws a timely conclusion on its quality and application to clinical practice, when acting as a reviewer or reader. Although the checklist may sound exhaustive and time-consuming, we recommend cutting it short if there are major red flags early on, such as absence of a protocol or assessment of RoB. Given the growing number of systematic reviews, having an efficient and succinct aide for appraising articles saves the reader time and energy, while simplifying the decision regarding what merits a change in clinical practice. Our intention is not to criticize others’ well-intentioned efforts, but to improve standards of reliable evidence to inform patient care.

Systematic reviews of randomized controlled trials offer one of the best methods to summarize the evidence surrounding therapeutic interventions for skin conditions.

The number of systematic reviews in the dermatology literature is increasing rapidly.

The quality of dermatology systematic reviews is generally poor.

We describe a checklist for the busy clinician or reviewer to consider when faced with a systematic review.

Key factors to consider include: determining the review motivation, establishing if the study protocol was prepublished, assessing quality of reporting and study quality using PRISMA, and AMSTAR 2 critical appraisal checklists, and assessing for evidence of spin.

Summarizing the main qualities and limitations of a systematic review will help to determine if the review is robust enough to potentially change clinical practice for patient benefit.

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

No new data generated.

Ethical approval: not applicable. Informed consent: not applicable.

Moher D , Liberati A , Tetzlaff J et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . Ann Intern Med 2009 ; 151 : 264 – 9 .

Google Scholar

Murad MH , Asi N , Alsawas M , Alahdab F . New evidence pyramid . Evid Based Med 2016 ; 21 : 125 – 7 .

Page MJ , McKenzie JE , Bossuyt PM et al.  The PRISMA 2020 statement: an updated guideline for reporting systematic reviews . BMJ 2021 ; 372 : n71 .

Ioannidis JP . The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses . Milbank Q 2016 ; 94 : 485 – 514 .

Croitoru DO , Huang Y , Kurdina A et al.  Quality of reporting in systematic reviews published in dermatology journals . Br J Dermatol 2020 ; 182 : 1469 – 76 .

Smires S , Afach S , Mazaud C et al.  Quality and reporting completeness of systematic reviews and meta-analyses in dermatology . J Invest Dermatol 2021 ; 141 : 64 – 71 .

Baraldi JH , Picozzo SA , Arnold JC et al.  A cross-sectional examination of conflict-of-interest disclosures of physician-authors publishing in high-impact US medical journals . BMJ Open 2022 ; 12 : e057598 .

Centers for Medicare & Medicaid Services . Open Payments Search Tool. About. Available at : https://openpaymentsdata.cms.gov/about (last accessed 22 April 2023).

Guelimi R , Afach S , Régnaux JP et al.  Overlapping network meta-analyses on psoriasis systemic treatments, an overview: quantity does not make quality . Br J Dermatol 2022 ; 187 : 29 – 41 .

Williams HC . Are dermatology systematic reviews spinning out of control? Dermatology 2021 ; 237 : 493 – 5 .

National Institute for Health and Care Research . About Prospero. Available at : https://www.crd.york.ac.uk/prospero/#aboutpage (last accessed 22 April 2023).

Barbieri JS , Wehner MR . Systematic reviews in dermatology: opportunities for improvement . Br J Dermatol 2020 ; 182 : 1329 – 30 .

Shea BJ , Reeves BC , Wells G et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both . BMJ 2017 ; 358 : j4008 .

AMSTAR . AMSTAR checklist. Available at : https://amstar.ca/Amstar_Checklist.php (last accessed 22 April 2023).

Drake L , Reyes-Hadsall S , Martinez J et al.  Evaluation of the safety and effectiveness of nutritional supplements for treating hair loss: a systematic review . JAMA Dermatol 2023 ; 159 : 79 – 86 .

Stenfors T , Kajamaa A , Bennett D . How to … assess the quality of qualitative research . Clin Teach 2020 ; 17 : 596 – 9 .

Yavchitz A , Ravaud P , Altman DG et al.  A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity . J Clin Epidemiol 2016 ; 75 : 56 – 65 .

Manriquez JJ , Villouta MF , Williams HC . Evidence-based dermatology: number needed to treat and its relation to other risk measures . J Am Acad Dermatol 2007 ; 56 : 664 – 71 .

Williams HC . Applying trial evidence back to the patient . Arch Dermatol 2003 ; 139 : 1195 – 200 .

CASP . CASP checklists. Available at : https://casp-uk.net/casp-tools-checklists/  (last accessed 22 April 2023).

Williams HC . Cars, CONSORT 2010, and clinical practice . Trials 2010 ; 11 : 33 .

Learning objective

To demonstrate up-to-date knowledge on assessing systematic reviews.

Which of the following critical appraisal checklists is useful for assessment of items that should be reported in a systematic review?

Which one of the following statements is correct?

The number of published systematic reviews in the dermatology literature is falling.

The quality of published dermatology systematic reviews is generally very good.

Publishing details of the PRISMA checklist in a systematic review indicates that the study quality is high.

External validity refers to the applicability of results to your patient group.

Internal validity refers to the applicability of results to your patient group.

Spin in systematic reviews can be described by which one of the following measures?

Authors declaring all conflicts of interest.

Title suggesting beneficial effect not supported by findings.

Adequate reporting of study limitations.

Conclusion formulating recommendations for clinical practice supported by findings.

Reporting a departure from study protocol that may modify interpretation of results.

PICO stands for which of the following.

PubMed, inclusion, comparator, outcome.

Population, items, comparator, outcome.

Population, intervention, context, observations.

Protocol, intervention, certainty, outcome.

Population, intervention, comparator, outcome.

Publication of a systematic review study protocol can be found at which source?

Cochrane Library.

ClinicalTrials.gov.

This learning activity is freely available online at https://oupce.rievent.com/a/TWWDCK

Users are encouraged to

Read the article in print or online, paying particular attention to the learning points and any author conflict of interest disclosures.

Reflect on the article.

Register or login online at https://oupce.rievent.com/a/TWWDCK and answer the CPD questions.

Complete the required evaluation component of the activity.

Once the test is passed, you will receive a certificate and the learning activity can be added to your RCP CPD diary as a self-certified entry.

This activity will be available for CPD credit for 5 years following its publication date. At that time, it will be reviewed and potentially updated and extended for an additional period.

Author notes

Email alerts, citing articles via.

• Recommend to Your Librarian
• Advertising and Corporate Services
• Journals Career Network

Affiliations

• Online ISSN 1365-2230
• Copyright © 2024 British Association of Dermatologists
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Appraising systematic reviews

Systematic reviews may or may not include a meta-analysis of the primary RCTs identified. Although systematic reviews of RCTs with meta-analysis are often said to provide the most compelling evidence of effectiveness and causality, not all systematic reviews are of the highest methodological quality. 

There are numerous checklists available. The SR toolbox  is an online catalogue providing summaries and links to the available guidance and software for each stage of the systematic review process including critical appraisal. Examples for systematic reviews include:

• AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews)
• CASP systematic review checklist.

The following framework forms an initial memory aid for assessing the quality of a systematic review.

• Framework for assessing systematic reviews

Content created by BMJ Knowledge Centre

• What is EBM?
• How to clarify a clinical question
• Searching for evidence
• Search strategies
• Case study of a search
• Appraise the evidence
• Evidence synthesis
• What is GRADE?
• Understanding statistics: BMJ Learning modules
• Understanding statistics: risk

EBM Toolkit home

Learn, Practise, Discuss, Tools

Critical Appraisal of Research Articles: Systematic Reviews

• Systematic Reviews
• Clinical Practice Guidelines
• Qualitative Studies

What is a Systematic Review?

A systematic review is a review of a clearly formulated queston that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from studies that are included in the review. Statistical methods may or may not be used to analyze and summarize the results of the included studies.

How to Find Systematic Reviews

1. Search the Cochrane Database of Systematic Reviews

2.  Using  PubMed , either use the 'Systematic Reviews' filter or add this to the end of your search 'AND (systematic review [ti])

3. If searching CINAHL , limit by publication type (select "Systematic Review").

Questions to Ask

• Is it a systematic review of the right type of studies which are relevant to your question?
• Does the methods section describe how all the relevant trials were found and assessed?    The paper should give a comprehensive account of the sources consulted in the search for relevant papers, the search strategy used to find them, and the quality and relevance criteria used to decide whether to include them in the review.
• The authors should include hand searching of journals and searching for unpublished literature.
• Were any obvious databases missed?
• Did the authors check the reference lists of articles and textbooks?
• Did they contact experts (to get their list of references checked for completeness and to try to find out about ongoing or unpublished research)?
• Did they use an appropriate search strategy; were important subject terms missed?
• Who were the study participants and how is their disease status defined?
• What intervention(s) were given, how, and in what setting?
• How were outcomes assessed?

      3.   Are the studies consistent, both clinically and statistically?

      4.   Compare with PRISMA

• Look at the most recent PRISMA checklist to see how well the authors documented the various preferred reporting items.

Appraisal Checklists for Systematic Reviews

• Critical Appraisals Skills Programme (CASP)
• Joanna Briggs Institute
• << Previous: Prognosis
• Next: Clinical Practice Guidelines >>

• Last Updated: Mar 1, 2024 11:56 AM
• URL: https://guides.himmelfarb.gwu.edu/CriticalAppraisal

• Himmelfarb Intranet
• Privacy Notice
• Terms of Use
• GW is committed to digital accessibility. If you experience a barrier that affects your ability to access content on this page, let us know via the Accessibility Feedback Form .
• Himmelfarb Health Sciences Library
• 2300 Eye St., NW, Washington, DC 20037
• Phone: (202) 994-2850
• [email protected]
• https://himmelfarb.gwu.edu

Handbook of Research Methods in Health Social Sciences pp 805–826 Cite as

Conducting a Systematic Review: A Practical Guide

• Freya MacMillan 2 ,
• Kate A. McBride 3 ,
• Emma S. George 4 &
• Genevieve Z. Steiner 5  
• Reference work entry
• First Online: 13 January 2019

2230 Accesses

1 Citations

It can be challenging to conduct a systematic review with limited experience and skills in undertaking such a task. This chapter provides a practical guide to undertaking a systematic review, providing step-by-step instructions to guide the individual through the process from start to finish. The chapter begins with defining what a systematic review is, reviewing its various components, turning a research question into a search strategy, developing a systematic review protocol, followed by searching for relevant literature and managing citations. Next, the chapter focuses on documenting the characteristics of included studies and summarizing findings, extracting data, methods for assessing risk of bias and considering heterogeneity, and undertaking meta-analyses. Last, the chapter explores creating a narrative and interpreting findings. Practical tips and examples from existing literature are utilized throughout the chapter to assist readers in their learning. By the end of this chapter, the reader will have the knowledge to conduct their own systematic review.

• Systematic review
• Search strategy
• Risk of bias
• Heterogeneity
• Meta-analysis
• Forest plot
• Funnel plot
• Meta-synthesis

This is a preview of subscription content, log in via an institution .

Buying options

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Barbour RS. Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ. 2001;322(7294):1115–7.

Article   Google Scholar  

Butler A, Hall H, Copnell B. A guide to writing a qualitative systematic review protocol to enhance evidence-based practice in nursing and health care. Worldviews Evid-Based Nurs. 2016;13(3):241–9.

Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80.

Dixon-Woods M, Bonas S, Booth A, Jones DR, Miller T, Sutton AJ, … Young B. How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res. 2006;6(1):27–44. https://doi.org/10.1177/1468794106058867 .

Greenhalgh T. How to read a paper: the basics of evidence-based medicine. 4th ed. Chichester/Hoboken: Wiley-Blackwell; 2010.

Google Scholar  

Hannes K, Lockwood C, Pearson A. A comparative analysis of three online appraisal instruments’ ability to assess validity in qualitative research. Qual Health Res. 2010;20(12):1736–43. https://doi.org/10.1177/1049732310378656 .

Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions (Version 5.1.0 [updated March 2011]). The Cochrane Collaboration; 2011.  http://handbook-5-1.cochrane.org/

Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, … Sterne JAC. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343. https://doi.org/10.1136/bmj.d5928 .

Hillier S, Grimmer-Somers K, Merlin T, Middleton P, Salisbury J, Tooher R, Weston A. FORM: an Australian method for formulating and grading recommendations in evidence-based clinical guidelines. BMC Med Res Methodol. 2011;11:23. https://doi.org/10.1186/1471-2288-11-23 .

Humphreys DK, Panter J, Ogilvie D. Questioning the application of risk of bias tools in appraising evidence from natural experimental studies: critical reflections on Benton et al., IJBNPA 2016. Int J Behav Nutr Phys Act. 2017; 14 (1):49. https://doi.org/10.1186/s12966-017-0500-4 .

King R, Hooper B, Wood W. Using bibliographic software to appraise and code data in educational systematic review research. Med Teach. 2011;33(9):719–23. https://doi.org/10.3109/0142159x.2011.558138 .

Koelemay MJ, Vermeulen H. Quick guide to systematic reviews and meta-analysis. Eur J Vasc Endovasc Surg. 2016;51(2):309. https://doi.org/10.1016/j.ejvs.2015.11.010 .

Lucas PJ, Baird J, Arai L, Law C, Roberts HM. Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews. BMC Med Res Methodol. 2007;7:4–4. https://doi.org/10.1186/1471-2288-7-4 .

MacMillan F, Kirk A, Mutrie N, Matthews L, Robertson K, Saunders DH. A systematic review of physical activity and sedentary behavior intervention studies in youth with type 1 diabetes: study characteristics, intervention design, and efficacy. Pediatr Diabetes. 2014;15(3):175–89. https://doi.org/10.1111/pedi.12060 .

MacMillan F, Karamacoska D, El Masri A, McBride KA, Steiner GZ, Cook A, … George ES. A systematic review of health promotion intervention studies in the police force: study characteristics, intervention design and impacts on health. Occup Environ Med. 2017. https://doi.org/10.1136/oemed-2017-104430 .

Matthews L, Kirk A, MacMillan F, Mutrie N. Can physical activity interventions for adults with type 2 diabetes be translated into practice settings? A systematic review using the RE-AIM framework. Transl Behav Med. 2014;4(1):60–78. https://doi.org/10.1007/s13142-013-0235-y .

Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Med Res Methodol. 2001;1:2. https://doi.org/10.1186/1471-2288-1-2 .

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. https://doi.org/10.1371/journal.pmed.1000097 .

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1. https://doi.org/10.1186/2046-4053-4-1 .

Mulrow CD, Cook DJ, Davidoff F. Systematic reviews: critical links in the great chain of evidence. Ann Intern Med. 1997;126(5):389–91.

Peters MDJ. Managing and coding references for systematic reviews and scoping reviews in EndNote. Med Ref Serv Q. 2017;36(1):19–31. https://doi.org/10.1080/02763869.2017.1259891 .

Steiner GZ, Mathersul DC, MacMillan F, Camfield DA, Klupp NL, Seto SW, … Chang DH. A systematic review of intervention studies examining nutritional and herbal therapies for mild cognitive impairment and dementia using neuroimaging methods: study characteristics and intervention efficacy. Evid Based Complement Alternat Med. 2017;2017:21. https://doi.org/10.1155/2017/6083629 .

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, … Higgins JP. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355. https://doi.org/10.1136/bmj.i4919 .

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57. https://doi.org/10.1093/intqhc/mzm042 .

Tong A, Palmer S, Craig JC, Strippoli GFM. A guide to reading and using systematic reviews of qualitative research. Nephrol Dial Transplant. 2016;31(6):897–903. https://doi.org/10.1093/ndt/gfu354 .

Uman LS. Systematic reviews and meta-analyses. J Can Acad Child Adolesc Psychiatry. 2011;20(1):57–9.

Download references

Author information

Authors and affiliations.

School of Science and Health and Translational Health Research Institute (THRI), Western Sydney University, Penrith, NSW, Australia

Freya MacMillan

School of Medicine and Translational Health Research Institute, Western Sydney University, Sydney, NSW, Australia

Kate A. McBride

School of Science and Health, Western Sydney University, Sydney, NSW, Australia

Emma S. George

NICM and Translational Health Research Institute (THRI), Western Sydney University, Penrith, NSW, Australia

Genevieve Z. Steiner

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Freya MacMillan .

Editor information

Editors and affiliations.

School of Science and Health, Western Sydney University, Penrith, NSW, Australia

Pranee Liamputtong

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this entry

Cite this entry.

MacMillan, F., McBride, K.A., George, E.S., Steiner, G.Z. (2019). Conducting a Systematic Review: A Practical Guide. In: Liamputtong, P. (eds) Handbook of Research Methods in Health Social Sciences. Springer, Singapore. https://doi.org/10.1007/978-981-10-5251-4_113

Download citation

DOI : https://doi.org/10.1007/978-981-10-5251-4_113

Published : 13 January 2019

Publisher Name : Springer, Singapore

Print ISBN : 978-981-10-5250-7

Online ISBN : 978-981-10-5251-4

eBook Packages : Social Sciences Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

This paper is in the following e-collection/theme issue:

Published on 28.3.2024 in Vol 11 (2024)

Translating Suicide Safety Planning Components Into the Design of mHealth App Features: Systematic Review

Authors of this article:

• Kim Gryglewicz 1 * , MSW, PhD   ; 
• Victoria L Orr 2 * , BS   ; 
• Marissa J McNeil 2 * , MA   ; 
• Lindsay A Taliaferro 3 , MPH, PhD, CHES   ; 
• Serenea Hines 2 , MSW   ; 
• Taylor L Duffy 2 , BS   ; 
• Pamela J Wisniewski 4 * , PhD  

1 School of Social Work, University of Central Florida, Orlando, FL, United States

2 Center for Behavioral Health Research & Training, University of Central Florida, Orlando, FL, United States

3 Department of Population Health Sciences, University of Central Florida, Orlando, FL, United States

4 Department of Computer Science, Vanderbilt University, Nashville, TN, United States

*these authors contributed equally

Corresponding Author:

Kim Gryglewicz, MSW, PhD

School of Social Work

University of Central Florida

12805 Pegasus Drive HS I

Orlando, FL, 32816

United States

Phone: 1 14078232954

Email: [email protected]

Background: Suicide safety planning is an evidence-based approach used to help individuals identify strategies to keep themselves safe during a mental health crisis. This study systematically reviewed the literature focused on mobile health (mHealth) suicide safety planning apps.

Objective: This study aims to evaluate the extent to which apps integrated components of the safety planning intervention (SPI), and if so, how these safety planning components were integrated into the design-based features of the apps.

Methods: Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, we systematically analyzed 14 peer-reviewed studies specific to mHealth apps for suicide safety planning. We conducted an analysis of the literature to evaluate how the apps incorporated SPI components and examined similarities and differences among the apps by conducting a comparative analysis of app features. An independent review of SPI components and app features was conducted by downloading the available apps.

Results: Most of the mHealth apps (5/7, 71%) integrated SPI components and provided customizable features that expanded upon traditional paper-based safety planning processes. App design features were categorized into 5 themes, including interactive features, individualized user experiences, interface design, guidance and training, and privacy and sharing. All apps included access to community supports and revisable safety plans. Fewer mHealth apps (3/7, 43%) included interactive features, such as associating coping strategies with specific stressors. Most studies (10/14, 71%) examined the usability, feasibility, and acceptability of the safety planning mHealth apps. Usability findings were generally positive, as users often found these apps easy to use and visually appealing. In terms of feasibility, users preferred using mHealth apps during times of crisis, but the continuous use of the apps outside of crisis situations received less support. Few studies (4/14, 29%) examined the effectiveness of mHealth apps for suicide-related outcomes. Positive shifts in attitudes and desire to live, improved coping strategies, enhanced emotional stability, and a decrease in suicidal thoughts or self-harm behaviors were examined in these studies.

Conclusions: Our study highlights the need for researchers, clinicians, and app designers to continue to work together to align evidence-based research on mHealth suicide safety planning apps with lessons learned for how to best deliver these technologies to end users. Our review brings to light mHealth suicide safety planning strategies needing further development and testing, such as lethal means guidance, collaborative safety planning, and the opportunity to embed more interactive features that leverage the advanced capabilities of technology to improve client outcomes as well as foster sustained user engagement beyond a crisis. Although preliminary evidence shows that these apps may help to mitigate suicide risk, clinical trials with larger sample sizes and more robust research designs are needed to validate their efficacy before the widespread adoption and use.

Introduction

Suicide is one of the leading causes of death in the United States, accounting for >45,000 deaths annually [ 1 ]. Over the last decade, suicide rates have doubled for youth aged 10 to 24 years [ 2 ] and have steadily increased for racial and ethnic minority youth [ 1 , 3 , 4 ]. Suicide ideation and attempt rates have also risen [ 5 , 6 ], especially among youth and minoritized populations [ 5 , 7 - 11 ]. Numerous studies have shown that untreated mental illness, limited or lack of available care, and low perceived need for mental health treatment are common, yet preventable, suicide risk antecedents [ 12 - 19 ]. Moreover, stigma, difficulties recognizing suicide warning signs, preferences for self-reliance and autonomy, fear of burdening others, and negative treatment experiences can negatively affect help-seeking intentions and engagement in mental health services [ 20 - 24 ].

Researchers have identified various suicide prevention strategies to reduce the public health problem of suicide [ 25 , 26 ]. Safety planning is an integral component of suicide care [ 27 ] and has been empirically validated for reducing suicidality [ 28 , 29 ]. The process of safety planning involves collaboration between a clinical and client, as well as with the at-risk individual and their support network. This means that the support network could also be part of the safety planning process [ 30 ]. Safety planning involves jointly identifying, problem-solving, and communicating strategies to keep an individual safe if a crisis arises. Core strategies focus on uncovering warning signs or triggers that precede an emotional event, identifying and reinforcing the use of healthful self-management strategies to cope with distress, encouraging the use of positive socialization strategies for distraction and support, creating a network of external support and professional contacts to solicit assistance and support, and reducing access to lethal means [ 31 ]. The individualized nature of creating a safety plan (ie, a written document detailing the plan to keep an individual safe during a crisis) allows the person at risk of suicide the ability to incorporate culturally relevant and meaningful strategies, thereby making these plans useful and relevant for diverse populations [ 30 , 32 ].

Suicide safety planning is a brief intervention that has been used in both acute and clinical settings [ 31 , 33 , 34 ] and as a self-help tool [ 35 ]. Overall, researchers have found this intervention to be feasible, acceptable, and useful to facilitate support and reduce suicide risk [ 32 , 33 , 35 - 37 ]. Researchers have found safety plans and related interventions, such as crisis response planning [ 38 ], to be effective in reducing the risk of hospitalization, increasing engagement in mental health treatment, and promoting the use of healthful coping strategies when used alongside other therapeutic approaches [ 33 , 34 , 36 , 39 , 40 ]. Although safety planning has shown initial success in reducing suicidal urges and offering a sense of hope to individuals in crisis [ 41 ], some clinicians and researchers have criticized this process [ 42 , 43 ]. For example, safety planning encourages clinicians to revisit and update safety plans with their clients over time [ 44 ], which can prove challenging if service use barriers prevent clients from reaccessing care or if clients misplace or throw away their paper-based safety plan.

Considering these challenges, mobile health (mHealth) technologies could offer a timely and effective solution to address some of the criticisms directed at traditional safety planning methods. mHealth, particularly the use of apps, represents a common tool used by consumers with access to mobile phones [ 45 , 46 ]. In addition, mHealth has garnered attention as a practical and convenient method for implementing mental health interventions [ 47 ], with increase in the quantity and functionality of applications and tools resulting in increased use [ 48 ]. In general, mHealth apps have been used to effectively help individuals identify and manage symptoms of various mental health problems and conditions such as depression, anxiety, substance abuse, posttraumatic stress, and eating disorders [ 49 , 50 ]. Thus, incorporating mHealth apps into mental health treatment and adjunctive interventions may prove beneficial.

Furthermore, incorporating mHealth apps into established evidence-based interventions may also serve as a culturally inclusive way of disseminating treatment to younger, more technologically savvy generations who also happen to demonstrate higher rates of suicidal thoughts and behaviors than adults [ 6 ]. mHealth apps may also help address service use barriers and risk factors (eg, stigma) that hinder individuals from seeking help and participating in treatment for suicidality. Combining suicide safety planning practices with mHealth apps may combat accessibility concerns as well, including a commonly reported flaw of the traditional intervention—the reliance on a paper format [ 35 ]. Given the widespread proliferation of mHealth apps for suicide prevention, there is a need to examine the components and features that have been incorporated into the design of suicide safety planning apps.

The purpose of this systematic literature review was to first assess the extent to which suicide safety planning mHealth apps integrated the 6 steps or components of a widely used safety planning intervention (SPI) developed by Stanley and Brown [ 31 ] (research question [RQ] 1). Next, we independently reviewed available mHealth suicide safety planning apps via download from iOS and Android app stores to assess the integration of SPI components and to categorize different app design features used to personalize the end users’ experience (RQ2). We also examined the evidence on the effectiveness of these apps in terms of usability, acceptability, app engagement, and suicide-related outcomes (RQ3). This review aims to synthesize the extant research to inform suicide prevention efforts, clinical practice, and future development of suicide safety planning mHealth apps.

In accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 statement guidelines ( Multimedia Appendix 1 [ 51 ]), a comprehensive systematic review of existing literature on suicide safety planning via mHealth apps was conducted. The process is described in the following sections.

Systematic Literature Review

Eligibility criteria.

The inclusion criteria for the reviewed research studies were as follows: (1) a primary focus on suicide safety planning involving the use of a mHealth app, (2) publication in a peer-reviewed article written in English, and (3) availability of the full text of the article. Studies were excluded if (1) the word suicide, safety plan, or app was not included in the title; (2) they included other forms of mHealth technologies as the primary focus (eg, web-based applications); (3) the apps were designed with safety planning as a secondary focus (ie, not exclusively for suicide safety planning, not intended as a crisis intervention, or use of safety planning as a secondary tool to other treatment modalities); and (4) they were part of other systematic reviews or meta-analyses. We included studies across the entire system development life cycle (eg, formative evaluations and 1 group pre-posttest designs) owing to limited research on the topic and the relatively recent emergence of such research.

Information Sources

The following 5 bibliographic databases were used to systematically review the literature: PsycINFO, PubMed, ACM Digital Libraries, Academic Search Premier, and ERIC. We limited our results to articles published between January 2000 and May 2023. All databases were last searched on July 2, 2023.

Search Strategy

We used the following keywords to search for the topic of interest in each scientific database: “Safety Plan*” AND (“Applications” OR “Apps”); (“Suicide” OR “Safety Plan*”) AND (“Applications” OR “Apps”); “Suicide Interven*” AND (“Applications” OR “Apps”); “Suicide Prevent*” AND (“Applications” OR “Apps”); “Suicide Contract” AND (“Applications” OR “Apps”); “mHealth” AND “Suicide”; “Crisis Response” AND “Plan*.” Asterisks were added to search for words that began with the preceding letters (eg, prevent*: prevent, prevention, and preventing). An example of the search strategy outlined above is provided in Multimedia Appendix 2 .

Selection Process

Citations obtained from electronic databases were imported into Zotero (version 6.0.16). Two reviewers (KG and VLO) independently screened the articles to remove duplicates and assessed inclusion and exclusion criteria by title and abstract. For articles about which the reviewers were uncertain after the title and abstract review, 4 reviewers independently analyzed the full-text articles to determine whether they met the inclusion criteria. The reviewers discussed discrepancies until they reached a consensus. The references of all articles that met the inclusion criteria were reviewed and cross-referenced for additional relevant articles. We included all eligible studies (N=14) in this systematic review ( Figure 1 [ 51 ]).

Data Collection Process

Data from eligible studies were analyzed using the Cochrane Collaboration’s data extraction template for included studies (version 1.8) [ 52 ]. We added study-specific items to the template to answer RQ1 and RQ2. Specifically, to answer RQ1, we reviewed articles describing each mHealth app and coded, using a dichotomous (yes or no) coding scheme, for the following SPI components: (1) personal warning signs, (2) coping strategies, (3) ways to distract oneself through social activities, (4) identification of and ways to access trusted individuals (eg, family and friends) for support, (5) identification of and ways to access community supports (eg, mental health professionals, nonmental health adult supports, crisis, or emergency services), and (6) information about keeping the environment safe (eg, restricting access to lethal means). To answer RQ2, we downloaded available mHealth apps via the Apple App Store or Google Play Store or contacted app developers to conduct an independent review of SPI components and app features described in the articles. Next, we created codes to describe app features, organized and categorized codes based on similarities, and generated 5 themes to capture the core aspects of features. To answer RQ3, we extracted both qualitative and quantitative findings reported on primary and secondary outcomes. We categorized the study outcomes into 3 main research themes.

Two reviewers coded 2 research articles to assess interrater reliability based on the coding template and made refinements as necessary (eg, added operational definitions to describe SPI components and provided examples of app features). Once finalized, the reviewers used the template to extract the data from the remaining studies. Data items included (1) general article information (eg, author, publication year, and country); (2) study methods (eg, aims and research design); (3) study characteristics (eg, sample size, sample demographics, and setting); (4) SPI intervention characteristics (RQ1); (5) mHealth app design features (RQ2) and primary and secondary outcomes (RQ3); and (6) study implications and future directions ( Multimedia Appendix 3 [ 42 , 43 , 53 - 64 ]). A similar process was used to independently code the SPI components and app features of the mHealth apps available for download.

Risk of Bias Assessment

The risk of bias for each study was assessed by 2 independent reviewers (KG and VLO) using Joanna Briggs Institute (JBI) appraisal tools for quasi-experimental [ 65 ] and qualitative research study designs [ 66 ]. For studies that included mixed methods designs, we used both tools as recommended by the JBI. Each appraisal tool used a rating scale with yes, no, unclear, and nonapplicable responses. The overall appraisal rating was based on the following categories: include, exclude, and seek further information. Disagreements between the reviewers were discussed until they reached a consensus ( Multimedia Appendix 4 [ 65 , 66 ]).

Synthesis of Results

Owing to the heterogeneity of the study designs, participants, and outcomes collected, we could not perform a meta-analysis of the identified studies in this review. Therefore, we present a narrative synthesis of the study findings.

Study Selection

The initial search of electronic databases and hand-searched references resulted in a total of 46,397 peer-reviewed articles. After duplicate records were removed, 21,151 studies remained. Titles were screened for relevancy (eg, relating to suicide, suicide safety planning, and mHealth apps), and 20,970 articles were excluded. A total of 181 abstracts were reviewed. Following full-text reviews of 54 articles, 40 articles were removed (15 studies did not include an mHealth app and 25 were not intended as a suicide safety planning app). A total of 14 articles met the inclusion criteria (refer to Figure 1 for breakdown).

Study Characteristics

The detailed study characteristics of the selected articles (N=14) are presented in Multimedia Appendix 3 . Most studies (12/14, 86%) were conducted outside the United States [ 42 , 53 - 63 ]. The year range of the selected articles was between 2015 and 2023.

Study Design

As shown in Tables 1 and 2 , a total of 7 mHealth suicide safety planning apps were studied across the 14 articles in our data set ( Multimedia Appendix 3 ). We classified the articles based on the research design (ie, formative feedback, usability assessment, single cohort pre-posttest, and random control trial protocol). Formative designs assessed SPI components and features to guide app development [ 43 , 56 , 61 , 64 ], whereas usability designs assessed interface design issues and functionality (eg, task difficulty and time to complete tasks) [ 55 , 60 , 61 , 64 ]. Other studies evaluated the acceptability or feasibility of a fully developed mHealth app [ 54 , 58 - 60 , 62 , 63 ]. Across these studies, participants rated the frequency and duration of app use; ease of navigation; and level of satisfaction, comfort, confidence, or engagement in using the app.

a N/A: not applicable.

Sample Characteristics

Across studies, the study sample varied in age, type of participant (eg, youth or adults at risk of suicide and clinicians collaborating with suicidal clients), and setting (eg, suicide prevention clinic and pediatric inpatient facility). Among studies that recruited participants to inform or evaluate mHealth suicide safety planning apps [ 43 , 54 - 56 , 58 - 64 ], the sample size ranged from 11 to 36 participants. However, after reporting dropout rates, sample sizes dropped to as low as 2 participants and as high as 22 participants.

Integration of SPI Components Within mHealth Apps

Most articles (5/7, 71%) describing the mHealth apps incorporated SPI components into the design of their apps [ 54 , 58 , 61 , 63 , 64 ] ( Table 3 ). Creating a safe environment from lethal means was the missing component in 29% (2/7) of the apps [ 55 , 56 ].

a BoMM: Brake of My Mind.

b SPC: safety planning component.

c SPC or app feature included in the app.

d MHP: mental health professional.

e SPC or app feature missing in the app.

f Denotes innovative app features aligned with SPI components.

g Feature included in the app that was not mentioned in the article.

We used the JBI quasi-experimental appraisal tool [ 65 ] to assess the risk of bias across 5 studies [ 55 , 58 - 60 , 63 ]. These studies did not include a control or comparison group, increasing the threat to internal validity. Pre- and posttest measures were used to assess the immediate effects of the mHealth apps. However, the lack of repeated outcome measures over time, selection bias (nonrandom samples), and small sample sizes pose a risk of bias within and across these studies.

The qualitative appraisal checklist tool [ 66 ] was used to assess the risk of bias in 4 studies [ 43 , 54 , 56 , 62 ]. Across 2 studies [ 43 , 54 ], the cultural or theoretical orientation of the researchers and their influence on the research process was unclear. These issues were noted in the other 2 studies [ 56 , 62 ] as well. In these studies [ 56 , 62 ], it was also difficult to identify the philosophical perspective and congruity between the research methods, data analysis, and interpretation. The studies included more of a description of the design of the apps and included general perceptions from stakeholders.

The remaining studies [ 61 , 64 ] were assessed using both the quasi-experimental and qualitative appraisal tools owing to their mixed methods designs. In both studies, it was unclear whether the researchers’ cultural or theoretical orientation, their influence on the research, and the adequate representation of the participants and their voices were addressed. Other key issues included the lack of a control or comparison group, nonrandom and small sample sizes, and the use of posttest measures to assess usability at only 1 time point. JBI appraisal results are included in Multimedia Appendix 4 .

On the basis of our independent review of available mHealth suicide safety planning apps, SPI components described in each article were verified in 71% (5/7) of the apps [ 54 , 56 , 58 , 61 , 63 ]. The app features described in the articles were also confirmed in these apps. App features not highlighted in the articles but found within the apps are listed in Table 3 . We were unable to verify SPI components and app features in 2 of the reviewed apps in the literature [ 55 , 64 ].

Comparative Analysis of SPI Components and App Features

In our analysis of the literature and available mHealth apps for download, we synthesized the commonalities of app features and categorized them into 5 broad themes: interactive features, individualized user experience, interface design, guidance and training, and privacy and sharing. These features are described in the following sections.

Interactive Features

Three of the suicide safety planning mHealth apps [ 54 , 56 , 61 ] allowed users to associate suicide warning signs or precipitating stressors with their personalized coping strategies (aligns with SPI 1 and 2 in Table 3 ). O’Grady et al [ 61 ] stressed the importance of including this feature in apps, as this functionality can serve to preemptively address an impending crisis before it fully manifests. Most of the suicide safety planning mHealth apps (6/7, 86%) also included social distractor features in which users had access to their phone’s camera with the ability to upload or view media content (eg, pictures, quotes, music, activities, videos, and inspirational stories; SPI 3) [ 54 , 55 , 58 , 61 , 63 , 64 ]. In the BackUp app [ 63 ], loved ones, trusted supports, and suicidal users were able to upload media and share content to inspire hope and distract users from negative thinking.

Each mHealth app also included a built-in feature for users to save and contact trusted individuals within their social support networks (SPI 4). Typically, users entered contact information into the mHealth app directly or linked to their contact directories. A unique feature of the MYPLAN app [ 54 ] allowed users to create prewritten messages that they could send to their social supports during times of distress. Although this feature was created to inform loved ones of the app user’s emotional state during a crisis, participants (ie, app users) noted concerns about messages being misunderstood, whereas relatives felt that messages could minimize emotional states or provide inaccurate information about the app user’s safety. All apps included the ability to access community supports such as mental health professionals (SPI 5). Three apps [ 54 , 55 , 61 ] included GPS capabilities, which enabled users to search for nearby counseling agencies or emergency services, and, after selecting a search result, users received directions for quick access (SPI 5 and 6). The ED-SAFE app [ 64 ] included a referral search engine that allowed users to find behavioral health care by specialty and zip code. Emergency service numbers, mostly displayed via a phone icon or brief words (eg, “Crisis”), were clearly visible (listed on all pages) in 57% (4/7) of the mHealth apps [ 56 , 58 , 61 , 63 ], which is the suggested ethical guideline from prior work [ 67 ]. Three apps did not include access to emergency service numbers on all pages but provided them somewhere else within the app [ 54 , 55 , 64 ].

Individualized User Experience

All apps (7/7, 100%) allowed users to continually add to or revise their safety plans. Examples included the addition of new warning signs, reasons for living, and identifying coping strategies. None of the apps maintained a historical record of the previous safety plans or provided a visual mechanism to track daily, weekly, or monthly patterns based on stressors encountered or coping strategies used. Other personalization aspects included the ability to enable or disable therapeutic modalities [ 61 ], the inclusion of web-based resources to take an aptitude and personality test [ 55 ], exercises to express moods [ 55 ], and mood tracking [ 55 , 56 , 61 ]. In addition, all apps had built-in features to make esthetic customizations, such as personalizing the home screen, changing the color palate, and adding background pictures [ 54 - 56 , 58 , 61 , 63 ]. In 57% (4/7) of the apps, notifications were enabled to remind users about using their safety plan or skills to practice [ 54 , 56 , 61 , 63 ].

Interface Design

Several studies used iterative feedback from content and app design experts to create easy-to-navigate interfaces [ 58 , 61 , 63 ]. To enhance the navigation experience, a simple layout, clear or user-friendly language, and accessibility features were important design considerations included in some mHealth apps [ 54 , 58 , 61 , 64 ]. For example, SafePlan ’s layout mimicked the paper version of the safety plan to better transition users from using the paper version to the app [ 61 ].

Guidance and Training

In-app tutorials or instructional videos were included in 86% (6/7) of the suicide safety planning mHealth apps [ 54 - 56 , 58 , 63 , 64 ]. Some of these tutorials focused on how to use the app, whereas others explained the safety planning process. For example, the BeyondNow app [ 58 ] included a video outlining the process of safety planning and links to other helpful information. The most extensive tutorials were seen in the companion app to ED-SAFE [ 64 ], where tutorials could be received from a female provider, a male community member, or an avatar. The mHealth suite of apps also included self-care education materials about suicidality, safety plans, and life plans. In addition, the BackUp app [ 63 ] provided supportive contacts with web-based information on ways to identify warning signs and strategies to talk with suicidal individuals. The Brake of My Mind app [ 55 ] included an introduction from the developer with additional web-based resources to increase app usability.

Privacy and Sharing

Researchers also highlighted app privacy and sharing capabilities as important features to consider when designing mHealth suicide safety planning apps. Given the personal nature of the information saved, most mHealth apps required a username and password to log in [ 54 - 56 , 61 , 63 , 64 ]. For example, ED-SAFE [ 64 ] used the username and password feature to verify user identity and connect information collected in the emergency department setting to the mHealth app. Other apps disabled GPS for location tracking or did not use external servers to store users’ information for privacy and security concerns [ 61 , 63 ]. Several apps (5/7, 71%) included features allowing users to share self-monitoring data or share safety plans with clinicians or trusted individuals [ 54 , 56 , 58 , 61 , 64 ]. For instance, ED-SAFE [ 64 ] allowed users to share safety plans as well as appointment information, self-care education, helplines, referrals, and distractions through password-protected privileges given to authorized family members.

mHealth App Evidence of Effectiveness

The qualitative and quantitative findings were categorized into 3 main research themes: app usability and acceptability, app use and engagement, and suicide-related outcomes.

App Usability and Acceptability Findings

Across 71% (10/14) of the studies [ 54 - 56 , 58 - 64 ] that assessed the initial usability or acceptability of mHealth suicide safety planning apps, stakeholders’ experiences testing the mHealth apps were generally positive. Four studies [ 55 , 60 , 61 , 64 ] included standard rating scales (ie, System Usability Scale [ 68 ]) to assess the perceived usability of their apps, and scores exceeded the minimum usability standards (ie, >70). The remaining studies used qualitative feedback from focus groups, case reports, and open-ended questionnaires. For example, in the study by Buus et al [ 54 ], participants found the MYPLAN safety planning app useful in recognizing patterns of impending crises and for reinforcing personalized strategies to cope with distress. In describing the benefits of the BeyondNow safety planning app, participants in the study by Melvin et al [ 58 ] reported developing a sense of hope and connection from using the app. Researchers have attributed these findings to the accessibility of the app and its customizable features. According to the authors, stakeholders regarded apps as highly intuitive, easy to use, and visually appealing interface in terms of the design [ 59 , 61 , 62 , 64 ].

App Use and App Engagement

Five studies examined app use over time [ 58 - 60 , 63 , 64 ]. Overall app engagement and use were minimal. Across 3 studies, >70% of the participants used the apps at least once during the testing period, which ranged from 1 to 10 weeks [ 58 , 59 , 63 ]. In the study by Melvin et al [ 58 ], 77% (17/22) of the participants reported using the mHealth app “occasionally” or “a lot,” including to make changes to safety plans. Most participants also reported using the mHealth app during a suicidal crisis (15/22, 68%) or when experiencing suicidal ideation (18/22, 82%). Increased frequency of app use during a crisis or among participants with high levels of suicide ideation was reported in studies by Pauwels et al [ 63 ] and Muscara et al [ 59 ]. Larkin et al [ 64 ] reported that 2 (40%) out of 5 participants reported downloading the ED-SAFE patient mHealth app after discharge. Low uptake rates were mostly attributed to the participants’ forgetfulness to download the app. Although most participants acknowledged the benefits of using mHealth suicide safety planning apps during times of crisis [ 58 , 63 ], participant feedback from the study by Muscara et al [ 59 ] suggested that participants did not believe or were unsure whether the use of the BeyondNow safety planning app could help them manage their symptoms or keep individuals safe during a crisis. Only 35% (6/17) of the participants favored using the app in the future. Conversely, participants in the study by Nuij et al [ 60 ] noted that easy access to the Backup mHealth app provided a sense of reassurance and helped to deter suicidal thoughts.

Suicide-Related Outcomes

Suicide-related outcomes were examined across 29% (4/14) of the small-scale pilot studies (with sample sizes ranging between 3 and 22) [ 55 , 58 , 59 , 63 ]. The study by Jeong et al [ 55 ] assessed the Theory of Planned Behavior constructs, including attitudes, subjective norms, perceived behavioral control, and intentions toward engaging in suicide attempts, using a pre-posttest design with a small (N=3) sample of adolescent survivors of suicide attempts. The results showed statistically significant changes in attitudes, perceived behavioral control, and intentions, suggesting that the suicide safety planning app helped to positively shift attitudes toward life and reduce beliefs and intentions to engage in self-harm behavior.

Suicide coping or resilience was evaluated in 2 studies using pre-posttest designs [ 58 , 59 ]. Both studies used the same safety planning app (ie, BeyondNow ) to examine the changes in protective factors. Melvin et al [ 58 ] found a statistically significant increase in suicide-related coping among youth and adult participants (n=22). This finding suggests an increase in knowledge and confidence to use internal coping strategies and external resources to manage suicide ideation. However, the researchers did not observe statistically significant changes in suicide resilience (ie, the perceived ability to manage suicidal thoughts and feelings). In contrast, Muscara et al [ 59 ] found a significant increase in 1 subscale of suicide resilience, emotional stability (ie, the ability to regulate emotions), among youth participants (N=17) in their study.

Suicidal ideation or self-harm behavior were measured in 3 studies [ 58 , 59 , 63 ]. In an open-label, single-group design, Melvin et al [ 58 ] found statistically significant reductions in both the severity and intensity of suicide ideation following exposure to an 8-week trial that evaluated the clinical effectiveness of using the BeyondNow suicide safety planning app as an adjunct to treatment as usual (ie, existing mental health services). In an evaluation of the same mHealth app, but with the addition of a personalized toolbox app (ie, BlueIce ), instead of treatment as usual, Muscara et al [ 59 ] also found a reduction in suicide ideation and self-harm behaviors (ie, attempts to harm oneself with and without suicidal intent). However, these findings were not conclusive or statistically significant owing to the small sample size and lack of a control group. Pauwels et al [ 63 ] found a similar, nonsignificant decrease in suicide ideation scores in a study examining pre-posttest changes following exposure to the BackUp suicide safety planning app. Although these studies provide some evidence of clinical utility, these researchers noted study limitations and the need for further evaluation using randomized controlled trials (RCTs).

Principal Findings

The primary aim of this study was to conduct a comprehensive analysis of the integration and inclusion of the SPI components developed by Stanley and Brown [ 31 ] in the design of mHealth suicide safety planning apps. The secondary aim was to synthesize and assess the research methods of studies that reported on the effectiveness of these apps. Implications of these findings and practical recommendations for future directions in mHealth suicide safety planning research are described in the following sections.

Integrating Components of Suicide Safety Planning Into mHealth Apps

Overall, most apps included the core components of the SPI developed by Stanley and Brown [ 31 ], such as the identification of suicide warning signs, coping strategies, and supportive persons. Therefore, the results from this review provide evidence of some level of successful integration of SPI components into mHealth suicide safety planning apps (RQ1). Lethal means safety was 1 component that was not incorporated in 2 of the apps reviewed. Reducing access to lethal means is a critical part of suicide safety planning [ 31 ] and warrants inclusion in mHealth apps as it brings attention to methods that could be used to attempt or die by suicide if not removed from a user’s environment.

An important aspect of suicide safety planning is access to one’s safety plan. In this review, having access to safety plans at any time [ 54 , 55 , 58 , 60 , 61 ] and being able to continually revise the plan were considered benefits over traditional paper-based safety planning. In some apps, users could create associations between different suicide safety planning components (SPCs; eg, triggers and coping strategies) to better contextualize their experiences and create actional plans for mitigating crises [ 54 , 56 , 61 ]. We recommend that additional linkages between the SPCs be included to further personalize users’ experiences.

Despite the integration of SPI components within mHealth suicide safety planning app designs, we also identified important gaps in the literature that warrant the attention of app designers, researchers, and mental health professionals who may use this type of technology within their clinical practice. For instance, researchers have consistently emphasized the importance of completing the initial safety plan alongside a knowledgeable clinician [ 42 , 54 , 58 , 61 ] to ensure that at-risk users and loved ones understand the components and purpose of a safety plan. However, many of the analyzed apps allowed users to complete the safety plan without the recommended clinical support, and in some cases, they lacked disclaimers. Therefore, additional guidance from a professional when using mHealth suicide safety planning apps would further serve to assist users and ensure that the safety planning process is carried out as intended.

This review also found that most of the apps did not go beyond the traditional SPCs of paper-based protocols to integrate more interactive features that could potentially improve adherence or engagement. For instance, daily or weekly check-ins have been shown to improve adherence in other mHealth contexts, such as for smoking cessation [ 69 ] and the management of schizophrenia [ 70 ]. Visualization graphs of patterns or trends in suicide warning signs, triggers, and coping behaviors logged over time may serve to increase engagement and improve outcomes, as visualizing behavior change over time has been recommended in other mHealth contexts [ 71 ], such as alcohol reduction [ 72 ]. Furthermore, other meaningful ways to actively and continuously engage one’s support contacts (eg, clinicians, parents, and family members) and to reinforce the use of healthful coping strategies would be an advantageous direction for future exploration in mHealth app design. Beyond general support contacts, prior research has found that parental support is a significant protective factor against youth suicide [ 73 , 74 ]. For youth, in particular, it may be advantageous to include parents, family members, or other trusted adults in the mHealth suicide safety planning process to increase uptake, enhance help-seeking and coping behaviors, and reinforce ways to keep one’s environment safe. However, future research would need to carefully design and evaluate such interventions to ensure they are effective before making these interventions widely available through the dissemination of mHealth apps for suicide safety planning.

Another variation across the apps was that some apps provided default values for suicide SPCs (eg, suggested coping strategies), whereas others did not. Therefore, an area of future research could be to study whether providing default values is beneficial or detrimental to the safety planning process. Finally, rather than training focused on the technical aspects of using the mHealth app, there is a need to include psychoeducation for suicide safety planning [ 75 ], especially related to coping strategies and lethal means restriction, which should be modeled as a collaborative process between at-risk users and their support systems [ 76 ].

Usability and Design Considerations for mHealth Suicide Safety Planning Apps

Overall, our review highlights three important recommendations to consider when designing safety planning mHealth apps (RQ2): the need to (1) encourage end user collaboration in the design and implementation of the intervention, (2) incorporate personalization or customization capabilities, and (3) develop appropriate privacy safeguards to prevent liability and address other safety concerns that may arise when integrating mental health care and technology. A key strength of most studies in our review was the interdisciplinary collaboration between app developers, computer scientists, and clinical researchers that facilitated the design, development, and evaluation of the various mHealth suicide safety planning apps. In addition, multiple stakeholders were included in the design process, including individuals at risk of suicide, clinicians, usability experts, parents, and extended family members. Only in 1 instance, end users engaged who were not considered part of the target population of at-risk users (eg, students). We strongly recommend that future research continue to include researchers from across multiple disciplines (eg, psychology, public health, social work, medicine, computer science, and human-computer interaction), intended end users, and mental health professionals across each stage of the research process. For instance, researchers from different disciplines may be able to raise important threats to validity during the research design process that could lead to more robust study designs.

A key weakness highlighted within several studies was limited uptake or sustained use of the mHealth suicide safety planning apps over time. Such findings shed suspicion on the feasibility of this type of intervention being effective outside of research, regardless of the high usability and acceptability ratings. Some studies attributed lack of use to the reduction of suicidal behaviors over time, but others suggested that the suicide safety planning process, as designed to be carried out within the apps, was only suited for in-crisis situations and not appropriate for sustained use over time. Although this may be the case, it is also possible that the lack of interactive or engaging features within the apps made them less appealing to users. Being able to customize and personalize app features may help to enhance the user’s experience and increase app engagement. Many of the apps included social distractions (ie, music and pictures) or other features, such as diary cards, which might help increase overall app engagement during noncrisis periods. However, as suicidality is episodic, future research should be conducted to understand how different modalities or features (eg, mood tracking, journaling, mindfulness, and art) could be combined with suicide safety planning in a complementary way for long-term use and engagement. Future work should also consider leveraging advanced technologies and assessments, such as artificial intelligence and ecological momentary assessments [ 77 , 78 ], that could be used to anticipate heightened suicide risk and prompt users to engage in the mHealth app suicide safety planning process when they need it most.

Threats to Validity and Inconclusive Clinical Outcomes Associated With the Use of mHealth Suicide Safety Planning Apps

This review provides some preliminary evidence suggesting that suicide safety planning via mHealth apps could be an easy-to-use mechanism to provide individualized care to those who may otherwise go unserved due to common treatment barriers (RQ3), such as poor accessibility to service providers, lack of knowledge about suicide, and stigmatizing beliefs about help seeking [ 20 - 24 ]. At the same time, several threats to validity were uncovered by our assessment of risk bias, which can inform directions for future research. First, the robustness of the qualitative studies could be improved by stating the positionality of the researchers as well as a clear justification for the design of the mHealth apps. In some cases, articles were published by interdisciplinary teams, whereas in other cases, authors appeared to be from a single discipline (eg, computer science). Details about the composition and expertise of the research team are important, as well-implemented mHealth apps require interdisciplinary skill sets that span clinical, design-based, and technical expertise. Furthermore, the quantitative studies analyzed in our review were constrained by small sample sizes and no published RCTs. Among the pre-posttest studies conducted thus far, the clinical outcomes were inconclusive.

As such, RCTs with control groups, random assignment, and repeated measure outcomes assessed over time are needed in the future to evaluate the efficacy of using suicide safety planning mHealth apps compared with traditional paper-based safety plans [ 54 , 57 ], specifically related to reducing suicidal urges and behaviors and increasing use of coping strategies, as well as increased engagement in crisis and mental health services after the crisis. When doing so, researchers should recruit larger samples to ensure that the results are conclusive and can be generalized to the populations of interest. Furthermore, additional use metrics collected by the apps to track behavioral data associated with using different app features, such as user engagement with the 6 components of the SPI developed by Stanley and Brown [ 31 ], should be considered to better understand the potential mediating factors and behaviors that may influence clinical outcomes. Although the usability of the apps would be an important consideration to control for in future studies, it is necessary to move beyond such measures to determine the efficacy of mHealth apps in reducing suicide-related outcomes. In summary, the inclusion of more advanced study design methodologies and recommendations from lessons learned in future mHealth apps could serve to mitigate suicide risk and promote overall safety.

Limitations and Future Research

This systematic review included 14 peer-reviewed articles that designed, developed, and evaluated mHealth apps for suicide safety planning. There are several limitations of this study that should be addressed in future research. First, although our search process was comprehensive, it is possible that our keywords missed relevant articles and mHealth apps that should have been included in the review. Second, as many of the apps described in the articles were not publicly available for download, we requested access from the corresponding authors to conduct our review. In 2 cases, we were unable to gain access to the apps; therefore, our analysis was based on the description of those apps based on the published paper. As such, it may be possible that some features were not described in the original papers; thus, they were not included in our review. Future research should also consider conducting a systematic feature analysis of mHealth suicide safety planning apps that are publicly available for download but not studied within the peer-reviewed literature. Finally, a limited number of published RCTs at the time of the review restricted our ability to report on app use and suicide-related outcomes. As such, the main call-to-action from this review is the need to move beyond usability studies of newly developed mHealth suicide safety planning apps to robust clinical research designs to examine their efficacy in reducing suicidality among at-risk user populations.

Conclusions

Overall, most articles included in this review did little to evaluate the efficacy of mHealth suicide safety planning apps beyond usability assessments, signaling that these apps and corresponding research are still in their infancy in terms of validating clinical outcomes. Although most of the mHealth safety planning apps included in our review are not yet downloadable and broadly available for public use, the prevalence and popularity of mHealth suicide prevention and mental health support apps on the open market that have been deployed without rigorous peer-reviewed research is a concern. As such, there is a critical need for future research to ensure that mHealth apps for suicide safety planning integrate the lessons learned from empirical user-based and clinical research, are upheld to high ethical mental health care standards, and show clinical efficacy for reducing suicidality before the apps are released to end users. This is especially true given the delicate and important goal of preventing suicide among at-risk populations. It is promising to see that future randomized clinical trials have been registered to build upon this important preliminary work on mHealth suicide safety planning apps.

Acknowledgments

The authors would like to thank the following authors who provided access to their apps to conduct this feature analysis: Niels Buus and JL Stovgaard Larsen (MYPLAN); Glenn Melvin (BeyondNow); James Duggan (SafePlan); and Lea Meier, Caroline Gurtner, and François von Kaenel (SERO).

Conflicts of Interest

None declared.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist.

Example search strategy.

Detailed summary of the selected articles and key findings (N=14).

Critical appraisal results.

• WISQARS™ — web-based injury statistics query and reporting system. Centers for Disease Control and Prevention. URL: https://www.cdc.gov/injury/wisqars/index.html [accessed 2023-01-11]
• Curtin SC, Heron M. Death rates due to suicide and homicide among persons aged 10-24: United States, 2000-2017. NCHS Data Brief. Oct 2019.(352):1-8. [ Medline ]
• Congressional Black Caucus. Ring the alarm: the crisis of black youth suicide in America. The National Action Alliance for Suicide Prevention. Dec 2019. URL: https://theactionalliance.org/resource/ring-alarm-crisis-black-youth-suicide-america [accessed 2023-06-05]
• Ramchand R, Gordon JA, Pearson JL. Trends in suicide rates by race and ethnicity in the United States. JAMA Netw Open. May 03, 2021;4(5):e2111563. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gaylor EM, Krause KH, Welder LE, Cooper AC, Ashley C, Mack KA, et al. Suicidal thoughts and behaviors among high school students - youth risk behavior survey, United States, 2021. MMWR Suppl. Apr 28, 2023;72(1):45-54. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Plemmons G, Hall M, Doupnik S, Gay J, Brown C, Browning W, et al. Hospitalization for suicide ideation or attempt: 2008-2015. Pediatrics. Jun 2018;141(6):e20172426. [ CrossRef ] [ Medline ]
• Johns MM, Lowry R, Haderxhanaj LT, Rasberry CN, Robin L, Scales L, et al. Trends in violence victimization and suicide risk by sexual identity among high school students - youth risk behavior survey, United States, 2015-2019. MMWR Suppl. Aug 21, 2020;69(1):19-27. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Peña JB, Wyman PA, Brown CH, Matthieu MM, Olivares TE, Hartel D, et al. Immigration generation status and its association with suicide attempts, substance use, and depressive symptoms among latino adolescents in the USA. Prev Sci. Dec 2008;9(4):299-310. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Johns MM, Lowry R, Andrzejewski J, Barrios LC, Demissie Z, McManus T, et al. Transgender identity and experiences of violence victimization, substance use, suicide risk, and sexual risk behaviors among high school students - 19 states and large urban school districts, 2017. MMWR Morb Mortal Wkly Rep. Jan 25, 2019;68(3):67-71. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stone DM, Jones CM, Mack KA. Changes in suicide rates - United States, 2018-2019. MMWR Morb Mortal Wkly Rep. Feb 26, 2021;70(8):261-268. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jin SS, Dolan TM, Cloutier AA, Bojdani E, DeLisi L. Systematic review of depression and suicidality in child and adolescent (CAP) refugees. Psychiatry Res. Aug 2021;302:114025. [ CrossRef ] [ Medline ]
• Fortune S, Stewart A, Yadav V, Hawton K. Suicide in adolescents: using life charts to understand the suicidal process. J Affect Disord. Jun 2007;100(1-3):199-210. [ CrossRef ] [ Medline ]
• Moskos M, Olson L, Halbern S, Keller T, Gray D. Utah youth suicide study: psychological autopsy. Suicide Life Threat Behav. Oct 2005;35(5):536-546. [ CrossRef ] [ Medline ]
• Pagura J, Fotti S, Katz LY, Sareen J, Swampy Cree Suicide Prevention Team. Help seeking and perceived need for mental health care among individuals in Canada with suicidal behaviors. Psychiatr Serv. Jul 2009;60(7):943-949. [ CrossRef ] [ Medline ]
• Sankey M, Lawrence R. Brief report: classification of adolescent suicide and risk-taking deaths. J Adolesc. Dec 2005;28(6):781-785. [ CrossRef ] [ Medline ]
• Pisani AR, Wyman PA, Petrova M, Schmeelk-Cone K, Goldston DB, Xia Y, et al. Emotion regulation difficulties, youth-adult relationships, and suicide attempts among high school students in underserved communities. J Youth Adolesc. Jun 2013;42(6):807-820. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rajappa K, Gallagher M, Miranda R. Emotion dysregulation and vulnerability to suicidal ideation and attempts. Cogn Ther Res. Nov 9, 2011;36:833-839. [ CrossRef ]
• Hung P, Busch SH, Shih YW, McGregor AJ, Wang S. Changes in community mental health services availability and suicide mortality in the US: a retrospective study. BMC Psychiatry. Apr 25, 2020;20(1):188. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hester RD. Lack of access to mental health services contributing to the high suicide rates among veterans. Int J Ment Health Syst. Aug 18, 2017;11:47. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Curtis C. Youth perceptions of suicide and help-seeking: ‘they'd think I was weak or “mental”’. J Youth Stud. Jul 28, 2010;13(6):699-715. [ CrossRef ]
• Aguirre Velasco A, Cruz IS, Billings J, Jimenez M, Rowe S. What are the barriers, facilitators and interventions targeting help-seeking behaviours for common mental health problems in adolescents? A systematic review. BMC Psychiatry. Jun 11, 2020;20(1):293. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Czyz EK, Horwitz AG, Eisenberg D, Kramer A, King CA. Self-reported barriers to professional help seeking among college students at elevated risk for suicide. J Am Coll Health. 2013;61(7):398-406. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hom MA, Stanley IH, Joiner TE. Evaluating factors and interventions that influence help-seeking and mental health service utilization among suicidal individuals: a review of the literature. Clin Psychol Rev. Aug 2015;40:28-39. [ CrossRef ] [ Medline ]
• Wilson CJ, Deane FP. Adolescent opinions about reducing help-seeking barriers and increasing appropriate help engagement. J Educ Psychol Consult. 2001;12(4):345-364. [ CrossRef ]
• Ougrin D, Tranah T, Stahl D, Moran P, Asarnow JR. Therapeutic interventions for suicide attempts and self-harm in adolescents: systematic review and meta-analysis. J Am Acad Child Adolesc Psychiatry. Feb 2015;54(2):97-107.e2. [ CrossRef ] [ Medline ]
• Pfeiffer PN, King C, Ilgen M, Ganoczy D, Clive R, Garlick J, et al. Development and pilot study of a suicide prevention intervention delivered by peer support specialists. Psychol Serv. Aug 2019;16(3):360-371. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Evidence. Zero Suicide. URL: https://zerosuicide.edc.org/evidence [accessed 2023-10-25]
• Ferguson M, Rhodes K, Loughhead M, McIntyre H, Procter N. The effectiveness of the safety planning intervention for adults experiencing suicide-related distress: a systematic review. Arch Suicide Res. 2022;26(3):1022-1045. [ CrossRef ] [ Medline ]
• Nuij C, van Ballegooijen W, de Beurs D, Juniar D, Erlangsen A, Portzky G, et al. Safety planning-type interventions for suicide prevention: meta-analysis. Br J Psychiatry. Aug 2021;219(2):419-426. [ CrossRef ] [ Medline ]
• Conti EC, Jahn DR, Simons KV, Edinboro LP, Jacobs ML, Vinson L, et al. Safety planning to manage suicide risk with older adults: case examples and recommendations. Clin Gerontol. 2020;43(1):104-109. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stanley B, Brown GK. Safety planning intervention: a brief intervention to mitigate suicide risk. Cogn Behav Pract. May 2012;19(2):256-264. [ CrossRef ]
• Chesin MS, Stanley B, Haigh EA, Chaudhury SR, Pontoski K, Knox KL, et al. Staff views of an emergency department intervention using safety planning and structured follow-up with suicidal veterans. Arch Suicide Res. Jan 02, 2017;21(1):127-137. [ CrossRef ] [ Medline ]
• Stanley IH, Hom MA, Joiner TE. A systematic review of suicidal thoughts and behaviors among police officers, firefighters, EMTs, and paramedics. Clin Psychol Rev. Mar 2016;44:25-44. [ CrossRef ] [ Medline ]
• Stanley B, Brown GK, Brenner LA, Galfalvy HC, Currier GW, Knox KL, et al. Comparison of the safety planning intervention with follow-up vs usual care of suicidal patients treated in the emergency department. JAMA Psychiatry. Sep 01, 2018;75(9):894-900. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Labouliere CD, Stanley B, Lake AM, Gould MS. Safety planning on crisis lines: feasibility, acceptability, and perceived helpfulness of a brief intervention to mitigate future suicide risk. Suicide Life Threat Behav. Feb 2020;50(1):29-41. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Czyz EK, King CA, Biermann BJ. Motivational interviewing-enhanced safety planning for adolescents at high suicide risk: a pilot randomized controlled trial. J Clin Child Adolesc Psychol. 2019;48(2):250-262. [ CrossRef ] [ Medline ]
• Stewart KL, Darling EV, Yen S, Stanley B, Brown GK, Weinstock LM. Dissemination of the safety planning intervention (SPI) to university counseling center clinicians to reduce suicide risk among college students. Arch Suicide Res. 2020;24(sup1):75-85. [ CrossRef ] [ Medline ]
• Rudd MD, Joiner TE, Rajab MH. Treating Suicidal Behavior: An Effective, Time-Limited Approach. New York City, NY. Guilford Publications; 2001.
• Asarnow JR, Berk M, Hughes JL, Anderson NL. The SAFETY program: a treatment-development trial of a cognitive-behavioral family treatment for adolescent suicide attempters. J Clin Child Adolesc Psychol. 2015;44(1):194-203. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bryan CJ, Mintz J, Clemans TA, Leeson B, Burch TS, Williams SR, et al. Effect of crisis response planning vs. contracts for safety on suicide risk in U.S. Army Soldiers: a randomized clinical trial. J Affect Disord. Apr 01, 2017;212:64-72. [ CrossRef ] [ Medline ]
• Little V, Neufeld J, Cole AR. Integrating safety plans for suicidal patients into patient portals: challenges and opportunities. Psychiatr Serv. Jun 01, 2018;69(6):618-619. [ CrossRef ] [ Medline ]
• Nuij C, van Ballegooijen W, Ruwaard J, de Beurs D, Mokkenstorm J, van Duijn E, et al. Smartphone-based safety planning and self-monitoring for suicidal patients: rationale and study protocol of the CASPAR (Continuous Assessment for Suicide Prevention And Research) study. Internet Interv. May 05, 2018;13:16-23. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kennard BD, Biernesser C, Wolfe KL, Foxwell AA, Craddock Lee SJ, Rial KV, et al. Developing a brief suicide prevention intervention and mobile phone application: a qualitative report. J Technol Hum Serv. Oct 01, 2015;33(4):345-357. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Moscardini EH, Hill RM, Dodd CG, Do C, Kaplow JB, Tucker RP. Suicide safety planning: clinician training, comfort, and safety plan utilization. Int J Environ Res Public Health. Sep 04, 2020;17(18):6444. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Czyz EK, Arango A, Healy N, King CA, Walton M. Augmenting safety planning with text messaging support for adolescents at elevated suicide risk: development and acceptability study. JMIR Ment Health. May 25, 2020;7(5):e17345. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Anderson M, Jiang J. Teens, social media and technology 2018. Pew Research Center. May 31, 2018. URL: https://www.pewresearch.org/internet/2018/05/31/teens-social-media-technology-2018/ [accessed 2024-02-28]
• Comer JS. Introduction to the special series: applying new technologies to extend the scope and accessibility of mental health care. Cogn Behav Pract. Aug 2015;22(3):253-257. [ CrossRef ]
• Lal S. E-mental health: promising advancements in policy, research, and practice. Healthc Manage Forum. Mar 2019;32(2):56-62. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Turvey CL, Roberts LJ. Recent developments in the use of online resources and mobile technologies to support mental health care. Int Rev Psychiatry. 2015;27(6):547-557. [ CrossRef ] [ Medline ]
• Shingleton RM, Richards LK, Thompson-Brenner H. Using technology within the treatment of eating disorders: a clinical practice review. Psychotherapy (Chic). Dec 2013;50(4):576-582. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. Mar 29, 2021;372:n71. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ryan R, Horey D, Oliver S, McKenzie J, Prictor M, Santesso N, et al. Cochrane Consumers and Communication Group Standard protocol text and additional guidance for review authors. La Trobe University. Mar 2019. URL: https:/​/opal.​latrobe.edu.au/​articles/​journal_contribution/​Standard_Protocol_Text_and_Additional_Guidance_for_Review_Authors/​6692969 [accessed 2023-05-19]
• Andreasson K, Krogh J, Bech P, Frandsen H, Buus N, Stanley B, et al. MYPLAN -mobile phone application to manage crisis of persons at risk of suicide: study protocol for a randomized controlled trial. Trials. Apr 11, 2017;18(1):171. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Buus N, Erlangsen A, River J, Andreasson K, Frandsen H, Larsen JL, et al. Stakeholder perspectives on using and developing the MYPLAN suicide prevention mobile phone application: a focus group study. Arch Suicide Res. 2020;24(1):48-63. [ CrossRef ] [ Medline ]
• Jeong YW, Chang HJ, Kim JA. Development and feasibility of a safety plan mobile application for adolescent suicide attempt survivors. Comput Inform Nurs. Aug 2020;38(8):382-392. [ CrossRef ] [ Medline ]
• Meier L, Gurtner C, Nuessli S, Miletic M, Bürkle T, Durrer M. SERO - a new mobile app for suicide prevention. Stud Health Technol Inform. May 16, 2022;292:3-8. [ CrossRef ] [ Medline ]
• Melia R, Francis K, Duggan J, Bogue J, O'Sullivan M, Young K, et al. Using a safety planning mobile app to address suicidality in young people attending community mental health services in Ireland: protocol for a pilot randomized controlled trial. JMIR Res Protoc. Feb 21, 2023;12:e44205. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Melvin GA, Gresham D, Beaton S, Coles J, Tonge BJ, Gordon MS, et al. Evaluating the feasibility and effectiveness of an Australian safety planning smartphone application: a pilot study within a tertiary mental health service. Suicide Life Threat Behav. Jun 2019;49(3):846-858. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Muscara F, Ng O, Crossley L, Lu S, Kalisch L, Melvin G, et al. The feasibility of using smartphone apps to manage self-harm and suicidal acts in adolescents admitted to an inpatient mental health ward. Digit Health. Nov 26, 2020;6:2055207620975315. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nuij C, van Ballegooijen W, de Beurs D, de Winter RF, Gilissen R, O'Connor RC, et al. The feasibility of using smartphone apps as treatment components for depressed suicidal outpatients. Front Psychiatry. Sep 27, 2022;13:971046. [ FREE Full text ] [ CrossRef ] [ Medline ]
• O'Grady C, Melia R, Bogue J, O'Sullivan M, Young K, Duggan J. A mobile health approach for improving outcomes in suicide prevention (SafePlan). J Med Internet Res. Jul 30, 2020;22(7):e17481. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Skovgaard Larsen JL, Frandsen H, Erlangsen A. MYPLAN - a mobile phone application for supporting people at risk of suicide. Crisis. May 2016;37(3):236-240. [ CrossRef ] [ Medline ]
• Pauwels K, Aerts S, Muijzers E, De Jaegere E, van Heeringen K, Portzky G. BackUp: development and evaluation of a smart-phone application for coping with suicidal crises. PLoS One. Jun 21, 2017;12(6):e0178144. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Larkin C, Djamasbi S, Boudreaux ED, Varzgani F, Garner R, Siddique M, et al. ReachCare mobile apps for patients experiencing suicidality in the emergency department: development and usability testing using mixed methods. JMIR Form Res. Jan 27, 2023;7:e41422. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tufanaru C, Munn Z, Aromataris E, Campbell J, Hopp L. Chapter 3: systematic reviews of effectiveness. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. Adelaide, South Australia. Joanna Briggs Institute; 2020.
• Lockwood C, Munn Z, Porritt K. Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. Int J Evid Based Healthc. Sep 2015;13(3):179-187. [ CrossRef ] [ Medline ]
• De Jaegere E, Portzky G, van den Berg M. Ethical guidelines for technology-based suicide prevention programmes. European Regions Enforcing Actions against Suicide. 2014. URL: https:/​/www.​researchgate.net/​publication/​260297369_Ethical_Guidelines_for_Technology-Based_Suicide_Prevention_Programmes [accessed 2024-02-28]
• Brooke J. SUS: a 'quick and dirty' usability scale. In: Usability Evaluation In Industry. Boca Raton, FL. CRC Press; 1996.
• McClure JB, Anderson ML, Bradley K, An LC, Catz SL. Evaluating an adaptive and interactive mHealth smoking cessation and medication adherence program: a randomized pilot feasibility study. JMIR Mhealth Uhealth. Aug 03, 2016;4(3):e94. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stubbe DE. Enhancing adherence: using mobile health technology to improve self-management for individuals with schizophrenia. Focus (Am Psychiatr Publ). Oct 2020;18(4):424-427. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Short CE, DeSmet A, Woods C, Williams SL, Maher C, Middelweerd A, et al. Measuring engagement in eHealth and mHealth behavior change interventions: viewpoint of methodologies. J Med Internet Res. Nov 16, 2018;20(11):e292. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bell L, Garnett C, Qian T, Perski O, Williamson E, Potts HW. Engagement with a behavior change app for alcohol reduction: data visualization for longitudinal observational study. J Med Internet Res. Dec 11, 2020;22(12):e23369. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hatchel T, Merrin GJ, Espelage AD. Peer victimization and suicidality among LGBTQ youth: the roles of school belonging, self-compassion, and parental support. J LGBT Youth. 2019;16(2):134-156. [ CrossRef ]
• Flouri E, Buchanan A. The protective role of parental involvement in adolescent suicide. Crisis. 2002;23(1):17-22. [ CrossRef ] [ Medline ]
• Abbott-Smith S, Ring N, Dougall N, Davey J. Suicide prevention: what does the evidence show for the effectiveness of safety planning for children and young people? - a systematic scoping review. J Psychiatr Ment Health Nurs. Oct 2023;30(5):899-910. [ CrossRef ] [ Medline ]
• Asarnow JR, Zullo L, Ernestus SM, Venables CW, Goldston DB, Tunno AM, et al. "Lock and protect": development of a digital decision aid to support lethal means counseling in parents of suicidal youth. Front Psychiatry. Oct 6, 2021;12:736236. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Auerbach RP, Lan R, Galfalvy H, Alqueza KL, Cohn JF, Crowley RN, et al. Intensive longitudinal assessment of adolescents to predict suicidal thoughts and behaviors. J Am Acad Child Adolesc Psychiatry. Sep 2023;62(9):1010-1020. [ CrossRef ] [ Medline ]
• Bhatt P, Liu J, Gong Y, Wang J, Guo Y. Emerging artificial intelligence-empowered mHealth: scoping review. JMIR Mhealth Uhealth. Jun 09, 2022;10(6):e35053. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by J Torous; submitted 14.09.23; peer-reviewed by HL Tam, B Leckning; comments to author 06.10.23; revised version received 19.12.23; accepted 31.12.23; published 28.03.24.

©Kim Gryglewicz, Victoria L Orr, Marissa J McNeil, Lindsay A Taliaferro, Serenea Hines, Taylor L Duffy, Pamela J Wisniewski. Originally published in JMIR Mental Health (https://mental.jmir.org), 28.03.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.

share this!

March 29, 2024

This article has been reviewed according to Science X's editorial process and policies . Editors have highlighted the following attributes while ensuring the content's credibility:

fact-checked

Climate adaptation research applied 'in real-time'

by SciDev.Net

With global South countries already bearing the devastating consequences of climate change, adaptation research needs to have immediate on-the-ground impact, while still being scientifically rigorous, say climate action specialists in a review published in Climate Services .

"With the climate crisis before us, we don't have time to sit back and do a conventional research program of two, or five, or 10 years, and then use the research itself," says Jesse DeMaria-Kinney, head of secretariat of the Adaptation Research Alliance (ARA).

The effects of climate change, fueled by the greenhouse gases humans are pumping into the atmosphere, are being felt acutely in the global South as temperatures rise, seasons shift, and extreme weather events, such as floods and storms, become more frequent and intense.

Adaptation, which involves changing ecological, social or economic systems to make them better able to weather the risks of climate change, "is a critical component of the long-term global response to climate change to protect people, livelihoods and ecosystems," according to the United Nations Framework Convention on Climate Change.

But the difficulty is that research, as it is traditionally undertaken, has long lead times and that model is not fit for a rapidly changing climate.

"Decisions and actions have to be made now and they need to be made on the best available evidence," says DeMaria-Kinney. "But we need to build flexibility into research and that flexibility has to be informed by a continual research and iterative process that runs parallel to implementation."

Action-oriented research

Last year, at the UN climate summit COP28 in Dubai, ARA announced that it had mobilized more than £3 million (US$3.8 million) in investments for action-oriented research that addresses pressing adaptation needs of those most vulnerable to climate impacts.

Formally launched in 2021, the ARA is a global coalition of organizations committed to action-orientated research for adaptation. Its 250 members range from intergovernmental organizations such as the United Nations Environmental Program to small community-based organizations.

"Action-orientated research is a paradigm shift in the way that the ARA sees research being done on climate change adaptation," DeMaria-Kinney says.

"This kind of research really focuses on ensuring impact for those on the frontlines of climate change, building capacity throughout the research processes, and the research actually being done with the end users."

Action-orientated research is different from traditional research because it happens alongside the implementation of findings on the ground, DeMaria-Kinney explains, adding that it focuses on "learning while doing."

He stresses that it should be driven by the needs of affected communities, working with those communities to co-design projects and find solutions that will have genuine societal impact.

One of the major investments announced by ARA was for the new Research 4 Impact (R4I) Hub, set up as part of the Climate Adaptation and REsilience (CLARE) research program, jointly designed and run by the UK's Foreign, Commonwealth and Development Office and Canada's International Development Research Center (IDRC).

"We're in a decisive decade," says Bruce Currie-Alder, who leads the climate team at the IDRC. "We often know enough to act" and with action-oriented research, "you use research as a learning tool in real-time," implementing and testing findings immediately to determine what worked and what didn't, he explains.

As an example he points to flood preparedness in communities in West Africa and the research that can be done ahead of an actual flood event to determine the most effective action. In October 2022, more than 3.4 million people were displaced following floods in Nigeria, Chad, Niger, Burkina Faso, Mali and Cameroon.

"What's the tailoring that needs to be done at a community level?" Currie-Alder asks. If a community were flooded, would its residents be able to receive cash transfers to tide them over during the flood and in its aftermath? "What types of measures are needed 72 hours before the water starts rising? These are researchable questions," he says.

Research findings could be implemented immediately to prepare the community for the next flood, he explains, and scientists could then research whether the interventions made a difference and how they could be improved.

Research for impact

The R4I Hub's new Opportunities Fund aims to translate research and existing knowledge into practical applications for communities in the global South. Project funding ranges from C$15,000 (US$11,00) to C$60,000 (US$44,000), and interventions need to be completed within a year, Currie-Alder says. It is open to governments and quasi-government agencies, and non-governmental and civil society organizations that want to put evidence into action.

"Over the years, I've heard people say things like, 'I don't have time to wait for a new research project to get up and running and develop answers—I only have three months to get something in front of the minister and influence this particular investment,"' he adds. "This is the responsive need we're hoping that the hub will be able to address."

There are many funding opportunities available, from large international funds such as the Green Climate Fund to more modest national efforts, but small interventions which need evidence can fall through the cracks, says Currie-Alder.

For example, perhaps "there's a community investing its local funds and trying to think about the best bet in terms of local infrastructure, whether it's a drainage channel or a new road," he explains.

"These are things that sometimes go under the radar of a big research agenda. You don't go to a university and say, 'I want a Ph.D. student to do this.'" But the R4I Opportunity Fund could be able to mobilize existing expertise and research to assist.

The fund is looking for organizations that already have a clear sense of the project they need guidance on and the sort of support they need. This support could, for example, be the help of a soil scientist, an energy and water systems optimization specialist, or understanding the research around adaptation decisions.

"We're keen to learn from the hub's activities over 2024 and 2025 and then see whether its funding needs to be bigger, and if it needs to offer a greater spectrum of funding options," Currie-Alder says.

Collaboration on the ground

Jenny Frankel-Reed, a senior program officer with the agricultural development team at the Bill & Melinda Gates Foundation, tells SciDev.Net, "We need to increase the relevance of scientific inquiry around climate action." The research should also be run by the affected regions, she says, adding, "In Sub-Saharan Africa, there's inequity both in terms of the impacts of climate and also who is generating the solutions."

The foundation has pledged £300,000 (US$380,000) to facilitate "co-creation" workshops for small-scale farmers in two African countries to identify research opportunities collaboratively. It is still deciding where the workshops will be based.

"It's always worth the time and the expense to do that [collaborative co-design] work well because the results are more durable, the buy-in is stronger, the questions are clearer—there are many advantages," says Frankel-Reed. This is one of the fundamental principles of action-oriented adaptation research.

About 70 percent of smallholder farmers in Africa rely on rainfed farming systems and this type of agriculture is particularly vulnerable to climate change, with its shifting seasons, variable temperatures, and extreme weather events .

"There's an urgency to climate adaptation that requires our research to be shaped by the people who are affected and really collaborate with the people who will use it," says Frankel-Reed. "It also needs to be done in a way that is going to build capacity around the world so that people are able to solve their own challenges around climate adaptation as well."

"There's a demand for this kind of research," adds DeMaria-Kinney. "That demand is seen by the ARA going from 33 when we launched at COP26 [in 2021] to having 250 members."

Action-oriented adaptation research is "flipping" the traditional research model around, says Currie-Alder. "As opposed to saying, 'What's your interesting idea and how does that influence the real work?', you're saying, 'What is the opportunity for impact, and what is the knowledge that is needed for that?'"

Provided by SciDev.Net

Explore further

Feedback to editors

A cosmic 'speed camera' just revealed the staggering speed of neutron star jets in a world first

11 hours ago

Saturday Citations: 100-year-old milk, hot qubits and another banger from the Event Horizon Telescope project

14 hours ago

Curiosity rover searches for new clues about Mars' ancient water

17 hours ago

Study says since 1979 climate change has made heat waves last longer, spike hotter, hurt more people

Scientist taps into lobsters' unusual habits to conquer the more than 120-year quest to farm them

Mar 29, 2024

Blind people can hear and feel April's total solar eclipse with new technology

Mapping the best route for a spacecraft traveling beyond the sun's sphere of influence

Researchers outline new approach in search for dark matter through future DUNE research project

Researchers reveal evolutionary path of important proteins

Study identifies protein responsible for gas vesicle clustering in bacteria

Relevant physicsforums posts, iceland warming up again - quakes swarming.

46 minutes ago

Unlocking the Secrets of Prof. Verschure's Rosetta Stones

‘our clouds take their orders from the stars,’ henrik svensmark on cosmic rays controlling cloud cover and thus climate.

Mar 27, 2024

Higher Chance to get Lightning Strike by Large Power Consumption?

Mar 20, 2024

A very puzzling rock or a pallasite / mesmosiderite or a nothing burger

Mar 16, 2024

Earth's earliest forest discovered in SW England

Mar 8, 2024

More from Earth Sciences

Related Stories

COP28: Countries promise action on food and farming

Dec 11, 2023

A guide on how to use climate data to inform human adaptation

Jan 11, 2024

COP28: Climate cash exclusion for conflict zones to end

Climate adaptation funds are not reaching front-line communities: What needs to be done about it

Nov 27, 2023

Opinion: COP28 agreement on adapting to climate change kicks the real challenge down the road

Dec 15, 2023

Report: As climate impacts accelerate, finance gap for adaptation efforts at least 50% bigger than thought

Nov 2, 2023

Recommended for you

Largest ice shelf in Antarctica lurches forward once or twice each day

Researchers find WWI and WWII bombs in the ground are becoming more volatile

Fukushima fallout transport longevity revealed by North Pacific ocean circulation patterns

Mar 28, 2024

Atmospheric observations in China show rise in emissions of a potent greenhouse gas

Study finds landfill point source emissions have an outsized impact and present opportunity to tackle US waste methane

Let us know if there is a problem with our content.

Use this form if you have come across a typo, inaccuracy or would like to send an edit request for the content on this page. For general inquiries, please use our contact form . For general feedback, use the public comments section below (please adhere to guidelines ).

Please select the most appropriate category to facilitate processing of your request

Thank you for taking time to provide your feedback to the editors.

Your feedback is important to us. However, we do not guarantee individual replies due to the high volume of messages.

E-mail the story

Your email address is used only to let the recipient know who sent the email. Neither your address nor the recipient's address will be used for any other purpose. The information you enter will appear in your e-mail message and is not retained by Phys.org in any form.

Newsletter sign up

Get weekly and/or daily updates delivered to your inbox. You can unsubscribe at any time and we'll never share your details to third parties.

More information Privacy policy

Donate and enjoy an ad-free experience

We keep our content available to everyone. Consider supporting Science X's mission by getting a premium account.

E-mail newsletter

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Otolaryngol Head Neck Surg

Systematic and other reviews: criteria and complexities

Robert t. sataloff.

1 Editor-in-Chief, Journal of Voice, Philadephia, USA

2 Editor Emeritus, Ear, Nose and Throat Journal, Philadephia, USA

Matthew L. Bush

3 Assistant Editor, Otology & Neurotology, Lexington, USA

Rakesh Chandra

4 Editor-in-Chief, Ear, Ear, Nose and Throat Journal, Nashville, USA

Douglas Chepeha

5 Editor-in-Chief, Journal of Otolaryngology – Head & Neck Surgery, Toronto, Canada

Brian Rotenberg

6 Editor-in-Chief, Journal of Otolaryngology – Head & Neck Surgery, London, Canada

Edward W. Fisher

7 Senior Editor, Journal of Laryngology and Otology, Birmingham, UK

David Goldenberg

8 Editor-in-Chief, Operative Techniques in Otolaryngology – Head and Neck Surgery, Hershey, USA

Ehab Y. Hanna

9 Editor-in-Chief, Head & Neck, Houston, USA

Joseph E. Kerschner

10 Editor-in-Chief, International Journal of Pediatric Otorhinolaryngology, Milwaukee, USA

Dennis H. Kraus

11 Co-Editor-in-Chief, Journal of Neurological Surgery Part B: Skull Base, New York, USA

John H. Krouse

12 Editor-in-Chief, Otolaryngology – Head and Neck Surgery, Philadelphia, USA

13 Editor-in-Chief, OTO-Open, Philadelphia, USA

14 Editor-in-Chief, Journal for Oto-Rhino-Laryngology, Head and Neck Surgery, Philadelphia, USA

15 Editor-in-Chief, World Journal of Otorhinolaryngology – Head and Neck Surgery, Philadelphia, USA

Michael Link

16 Co-Editor-in-Chief, Journal of Neurological Surgery Part B: Skull Base, Rochester, USA

Lawrence R. Lustig

17 Editor-in-Chief, Otology & Neurotology, New York, USA

Samuel H. Selesnick

18 Editor-in-Chief, The Laryngoscope, New York, USA

Raj Sindwani

19 Editor-in-Chief, American Journal of Rhinology & Allergy, Cleveland, USA

Richard J. Smith

20 Editor-in-Chief, Annals of Otology, Rhinology & Laryngology, Iowa City, USA

James Tysome

21 Editor-in-Chief, Clinical Otolaryngology, Cambridge, UK

Peter C. Weber

22 Editor-in-Chief, American Journal of Otolaryngology, Boston, USA

D. Bradley Welling

23 Editor-in-Chief, Laryngoscope Investigative Otolaryngology, Boston, USA

Review articles can be extremely valuable. They synthesize information for readers, often provide clarity and valuable insights into a topic; and good review articles tend to be cited frequently. Review articles do not require Institutional Review Board (IRB) approval if the data reviewed are public (including private and government databases) and if the articles reviewed have received IRB approval previously. However, some institutions require IRB review and exemption for review articles. So, authors should be familiar with their institution’s policy. In assessing and interpreting review articles, it is important to understand the article’s methodology, scholarly purpose and credibility. Many readers, and some journal reviewers, are not aware that there are different kinds of review articles with different definitions, criteria and academic impact [ 1 ]. In order to understand the importance and potential application of a review article, it is valuable for readers and reviewers to be able to classify review articles correctly.

Systematic reviews

Authors often submit articles that include the term “systematic” in the title without realizing that that term requires strict adherence to specific criteria. A systematic review follows explicit methodology to answer a well-defined research question by searching the literature comprehensively, evaluating the quantity and quality of research evidence rigorously, and analyzing the evidence to synthesize an answer to the research question. The evidence gathered in systematic reviews can be qualitative or quantitative. However, if adequate and comparable quantitative data are available then a meta-analysis can be performed to assess the weighted and summarized effect size of the studies included. Depending on the research question and the data collected, systematic reviews may or may not include quantitative meta-analyses; however, meta-analyses should be performed in the setting of a systematic review to ensure that all of the appropriate data were accessed. The components of a systematic review can be found in an important article by Moher et al. published in 2009 that defined requirements for systematic reviews and meta-analyses [ 2 ].

In order to optimize reporting of meta-analyses, an international group developed the Quality of Reporting of Meta-Analyses (QUOROM) statement at a meeting in 1996 that led to publication of the QUOROM statement in 1999 [ 3 ]. Moher et al. revised that document and re-named the guidelines the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). The PRISMA statement included both meta-analyses and systematic reviews, and the authors incorporated definitions established by the Cochrane Collaboration [ 4 ]. The PRISMA statement established the current standard for systematic reviews. To qualify as a systematic review, the methods section should acknowledge use of the PRISMA guidelines, and all PRISMA components should be incorporated strictly in all facets of the paper from the research question to the discussion. The PRISMA statement includes a checklist of 27 items that must be included when reporting a systematic review or meta-analysis [ 2 ]. A downloadable version of this checklist can be used by authors, reviewers, and journal editorial staff to ensure compliance with recommended components [ 5 ]. All 27 will not be listed in this brief editorial (although authors and reviewers are encouraged to consult the article by Moher et al. and familiarize themselves with all items), but a few will be highlighted.

The research question, as reflected in the title, should be a hypothesis-based specific research inquiry. The introduction must describe the rationale for the review and provide a specific goal or set of goals to be addressed. The type of systematic review, according to the Cochrane Collaboration, is based on the research question being asked and may assess diagnostic test accuracy, review prognostic studies evidence, evaluate intervention effect, scrutinize research methodology, or summarize qualitative evidence [ 6 ].

In the methods section, the participants, interventions, comparisons, outcomes and study design (PICOS) must be put forward. In addition to mentioning compliance with PRISMA, the methods section should state whether a review protocol exists and, if so, where it can be accessed (including a registration number). Systematic reviews are eligible for registration in the International Prospective Register of Systematic Reviews (PROSPERO) as established at the University of York (York, UK). When PROSPERO is used (it is available but not required for systematic reviews), registration should occur at the initial protocol stage of the review, and the final paper should direct to the information in the register. The methods section also must include specific study characteristics including databases used, years considered, languages of articles included, specific inclusion and exclusion criteria for studies; and rationale for each criterion must be included. Which individuals specifically performed searches should be noted. Electronic search strategy (with a full description of at least one electronic search strategy sufficient to allow replication of the search), process for article selection, data variables sought, assumptions and simplifications, methods for assessing bias risk of each individual study (such as selective reporting in individual studies) and utilization of this information in data synthesis, principal summary measures (risk ratio, hazard ratio, difference in means, etc.), methods of data management and combining study results, outcome level assessment, and other information should be reported.

The results section should include the number of studies identified, screened, evaluated for eligibility (including rationale for exclusion), and those included in the final synthesis. A PRISMA flow diagram should be included to provide this information succinctly [ 7 ]. The results also should include the study characteristics, study results, risk of bias within and across studies, and a qualitative or quantitative synthesis of the results of the included studies. This level of rigor in acquiring and evaluating the evidence of each individual study is one of the criteria that distinguishes systematic reviews from other categories. If the systematic review involves studies with paired samples and quantitative data, a summary of data should be provided for each intervention group along with effect estimates and confidence intervals for all outcomes of each study. If a meta-analysis is performed, then synthesized effect size should be reported with confidence intervals and measures of consistency (i.e. – data heterogeneity such as I 2 ) for each meta-analysis, and assessment of bias risk across studies. A forest plot, which provides a graphical presentation of the meta-analysis results, should be included.

The discussion section should summarize the main findings commenting on the strength of evidence for each outcome, as well as relevance to healthcare providers, policymakers and other key stake-holders; limitations of the study and outcomes; and conclusions highlighting the interpretation of results in the context of other research, and implications for future research.

Without adhering to of all of these criteria and the others listed in the PRISMA statement and checklist, the review does not qualify to be classified as “systematic”.

Meta-analyses

Meta-analyses, when feasible based on available and comparable quantitative data, supplement a systematic review evaluation, by adding a secondary statistical analysis of the pooled weighted outcomes of similar studies. This adds a level of objectivity in the synthesis of the review’s findings. Meta-analyses are appropriate when at least 2 individual studies contain paired samples (experimental group and control group) and provide quantitative outcome data and sample size. Studies that lack a control group may over-estimate the effect size of the experimental intervention or condition being studied and are not ideal for meta-analyses [ 8 ]. It also should be remembered that the conclusions of a meta-analysis are only as valid as the data on which the analysis is based. If the articles included are flawed, then the conclusions of the meta-analysis also may be flawed. Systematic reviews and meta-analyses are the most rigorous categories of review.

Other types of reviews

Mixed methods reviews.

Systematic reviews typically contain a single type of data, either qualitative or quantitative; however, mixed methods reviews bring together a combination of data types or study types. This approach may be utilized when quantitative data, in the setting of an intervention study, only provide a narrow perspective of the efficacy or effectiveness of the intervention. The addition of qualitative data or qualitative studies may provide a more complete picture of the knowledge, attitudes, and behaviors of clinicians, patients or researchers regarding that intervention. This type of review could involve collecting either the quantitative or the qualitative data using systematic review methodology, but often the qualitative data are gathered using a convenience sampling. Many qualitative studies provide useful insights into clinical management and/or implementation of research interventions; and incorporating them into a mixed methods review may provide valuable perspective on a wide range of literature. Mixed methods reviews are not necessarily systematic in nature; however, authors conducting mixed methods reviews should follow systematic review methodology, when possible.

Literature and narrative reviews

Literature reviews include peer-reviewed original research, systematic reviews, and meta-analyses, but also may include conference abstracts, books, graduate degree theses, and other non-peer reviewed publications. The methods used to identify and evaluate studies should be specified, but they are less rigorous and comprehensive than those required for systematic reviews. Literature reviews can evaluate a broad topic but do not specifically articulate a specific question, nor do they synthesize the results of included studies rigorously. Like mixed method reviews, they provide an overview of published information on the topic, although they may be less comprehensive than integrative reviews; and, unlike systematic reviews, they do not need to support evidence-based clinical or research practices, or highlight high-quality evidence for the reader. Narrative reviews are similar to literature reviews and evaluate the same scope of literature. The terms sometimes are used interchangeably, and author bias in article selection and data interpretation is a potential concern in literature and narrative reviews.

Umbrella reviews

An umbrella review integrates previously published, high-quality reviews such as systematic reviews and meta-analyses. Its purpose is to synthesize information in previously published systematic reviews and meta-analyses into one convenient paper.

Rapid review

A rapid review uses systematic review methodology to evaluate existing research. It provides a quick synthesis of evidence and is used most commonly to assist in emergent decision-making such as that required to determine whether COVID-19 vaccines should receive emergent approval.

Scoping, mapping, and systematized reviews

If literature has not been reviewed comprehensively in a specific subject that is varied and complex, a mapping review (also called scoping review) may be useful to organize initial understanding of the topic and its available literature. While mapping reviews may be helpful in crystallizing research findings and may be published, they are particularly useful in helping to determine whether a topic is amenable to systematic review, and to help organize and direct the approach of the systematic review or other reviews of the subject. Systematized reviews are used most commonly by students. The systematized review provides initial assessment of a topic that is potentially appropriate for a systematic review, but a systematized review does not meet the rigorous criteria of a systematic review and has substantially more limited value. Additional types of reviews exist including critical review, state-of-the-art review, and others.

Reviews can be invaluable; but they also can be misleading. Systematic reviews and meta-analyses provide readers with the greatest confidence that rigorous efforts have attempted to eliminate bias and ensure validity, but even they have limitations based upon the strengths and weaknesses of the literature that they have assessed (and the skill and objectivity with which the authors have executed the review). Risks of bias, incomplete information and misinformation increase as the rigor of review methodology decreases. While review articles may summarize research related to a topic for readers, non-systematic reviews lack the rigor to answer adequately hypothesis-driven research questions that can influence evidence-based practice. Journal authors, reviewers, editorial staff, and should be cognizant of the strengths and weaknesses of review methodology and should consider them carefully as they assess the value of published review articles, particularly as they determine whether the information presented should alter their patient care.

Authors’ contributions

The author(s) read and approved the final manuscript.

Declarations

The authors declare no competing interests.

This article is co-published in the following journals: Journal of Voice, Otology & Neurotology, Ear, Nose and Throat Journal, Journal of Laryngology and Otology, Operative Techniques in Otolaryngology – Head and Neck Surgery, Head & Neck, International Journal of Pediatric Otorhinolaryngology, Journal of Neurological Surgery Part B: Skull Base, Otolaryngology – Head and Neck Surgery, World Journal of Otorhinolaryngology – Head and Neck Surgery, The Laryngoscope, American Journal of Rhinology & Allergy, Annals of Otology, Rhinology & Laryngology, Clinical Otolaryngology, American Journal of Otolaryngology, Laryngoscope Investigative Otolaryngology.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

• Alzheimer's disease & dementia
• Arthritis & Rheumatism
• Attention deficit disorders
• Autism spectrum disorders
• Biomedical technology
• Diseases, Conditions, Syndromes
• Endocrinology & Metabolism
• Gastroenterology
• Gerontology & Geriatrics
• Health informatics
• Inflammatory disorders
• Medical economics
• Medical research
• Medications
• Neuroscience
• Obstetrics & gynaecology
• Oncology & Cancer
• Ophthalmology
• Overweight & Obesity
• Parkinson's & Movement disorders
• Psychology & Psychiatry
• Radiology & Imaging
• Sleep disorders
• Sports medicine & Kinesiology
• Vaccination
• Breast cancer
• Cardiovascular disease
• Chronic obstructive pulmonary disease
• Colon cancer
• Coronary artery disease
• Heart attack
• Heart disease
• High blood pressure
• Kidney disease
• Lung cancer
• Multiple sclerosis
• Myocardial infarction
• Ovarian cancer
• Post traumatic stress disorder
• Rheumatoid arthritis
• Schizophrenia
• Skin cancer
• Type 2 diabetes
• Full List »

share this!

March 28, 2024

This article has been reviewed according to Science X's editorial process and policies . Editors have highlighted the following attributes while ensuring the content's credibility:

fact-checked

trusted source

Research finds no evidence probiotics offer benefit during pregnancy

by Burnet Institute

A systematic review of the use of probiotics during pregnancy on pregnancy-related conditions found no evidence that probiotics had any effect on the conditions.

Published today in the American Journal of Obstetrics & Gynecology Maternal-Fetal Medicine , the research analyzed trials of thousands of women who took probiotics during pregnancy and the impact on pregnancy-related conditions, including preeclampsia.

Probiotics are usually taken as a standalone supplement or fortified food and are sometimes used by women during pregnancy.

Burnet Senior Research Officer and one of the study's lead authors Dr. Annie McDougall said the research team analyzed data from 29 clinical trials to assess the efficacy and safety of probiotic use during pregnancy.

"Our research, which analyzed trials involving more than 7000 pregnant women, found there was no benefit of probiotics on any pregnancy-related conditions," she said.

"This paper shows there is no evidence that probiotics should be prescribed during pregnancy to improve pregnancy-related outcomes."

Dr. McDougall said more research was needed into the effects of probiotic use during pregnancy on preeclampsia and other conditions, but the research had revealed other interesting findings.

"One of the surprising findings from our research was the role of gut microbiome in pregnant women who are under-nourished or live in places with poor sanitation ," she said.

"There is still a lot that remains unknown about the impact of gut microbiome on pregnancy-related conditions and more research is needed before specific guidelines for pregnancy can be developed."

"We are hoping to build on this research to learn more about the impact of gut microbiome during pregnancy in under-nourished populations and in low- to middle-income countries."

The study is part of broader research being conducted through the Accelerating Innovation for Mothers (AIM) project, which is focused on improving maternal and infant health globally and reducing maternal and infant deaths. The project aims to find innovative medicines, devices, and diagnostics for pregnancy-specific conditions such preeclampsia, preterm labor, and impaired fetal growth in low- and middle-income countries .

As a result of this study, the AIM team is now working towards developing target product profiles (TPPs) on interventions for pregnant women who have gut microbiome dysfunction.

Explore further

Feedback to editors

Researchers produce grafts that replicate the human ear

17 hours ago

An infamous 'inflammasome'—a rogue protein complex—appears to underlie a rare and disabling autoimmune disorder

Mar 29, 2024

Researchers discover skin biomarkers in infants that predict early development of food allergies

Veterans help provide greater insight into Klinefelter and Jacobs syndromes

High-resolution images reveal similarities in protein structures between Alzheimer's disease and Down syndrome

How blocking a neural receptor responsible for addiction could reduce alcohol use

Study finds few hospitals promoting potentially predatory medical payment products

COVID-19 research: Study reveals new details about potentially deadly inflammation

Enhanced melanoma vaccine offers improved survival for men

How music choices can affect productivity

Related stories.

New insights into medicines for preeclampsia

Nov 22, 2022

Medication use during pregnancy is common in women with preeclampsia

Nov 6, 2019

Preeclampsia and preterm birth risk may be reduced by calcium dose lower than current WHO standard

Jan 10, 2024

New research finds that people who use cannabis early in pregnancy are at increased risk of poor pregnancy outcomes

Feb 9, 2023

Expecting? Stay heart-healthy for you and your baby

Dec 27, 2023

How to care for your heart during a pregnancy

Jul 5, 2023

Recommended for you

Steady rise in US suicides among adolescents, teens

Women with obesity do not need to gain weight during pregnancy, new study suggests

Mar 28, 2024

Positive associations found between premenstrual disorders and perinatal depression

Amount of protein consumed by pregnant women modulates the facial appearance of offspring, study finds

Miscarriages linked to health risks in later pregnancies

Prolonged use of certain hormone drugs linked to increased brain tumor risk

Mar 27, 2024

Let us know if there is a problem with our content

Use this form if you have come across a typo, inaccuracy or would like to send an edit request for the content on this page. For general inquiries, please use our contact form . For general feedback, use the public comments section below (please adhere to guidelines ).

Please select the most appropriate category to facilitate processing of your request

Thank you for taking time to provide your feedback to the editors.

Your feedback is important to us. However, we do not guarantee individual replies due to the high volume of messages.

E-mail the story

Your email address is used only to let the recipient know who sent the email. Neither your address nor the recipient's address will be used for any other purpose. The information you enter will appear in your e-mail message and is not retained by Medical Xpress in any form.

Newsletter sign up

Get weekly and/or daily updates delivered to your inbox. You can unsubscribe at any time and we'll never share your details to third parties.

More information Privacy policy

Donate and enjoy an ad-free experience

We keep our content available to everyone. Consider supporting Science X's mission by getting a premium account.

E-mail newsletter