conclusions of a systematic review with narrative synthesis

Log in using your username and password

Search More Search for this keyword Advanced search
Latest content
For authors
Browse by collection
BMJ Journals More You are viewing from: Google Indexer

Mhairi Campbell 1 ,
Srinivasa Vittal Katikireddi 1 ,
Amanda Sowden 2 ,
Joanne E McKenzie 3 ,
Hilary Thomson 1
1 MRC/CSO Social and Public Health Sciences Unit , University of Glasgow , Glasgow , UK
2 Centre for Reviews and Dissemination , University of York , York , UK
3 School of Public Health and Preventive Medicine , Monash University , Melbourne , Victoria , Australia
Correspondence to Ms Mhairi Campbell; Mhairi.Campbell{at}glasgow.ac.uk

Introduction Reliable evidence syntheses, based on rigorous systematic reviews, provide essential support for evidence-informed clinical practice and health policy. Systematic reviews should use reproducible and transparent methods to draw conclusions from the available body of evidence. Narrative synthesis of quantitative data (NS) is a method commonly used in systematic reviews where it may not be appropriate, or possible, to meta-analyse estimates of intervention effects. A common criticism of NS is that it is opaque and subject to author interpretation, casting doubt on the trustworthiness of a review’s conclusions. Despite published guidance funded by the UK’s Economic and Social Research Council on the conduct of NS, recent work suggests that this guidance is rarely used and many review authors appear to be unclear about best practice. To improve the way that NS is conducted and reported, we are developing a reporting guideline for NS of quantitative data.

Methods We will assess how NS is implemented and reported in Cochrane systematic reviews and the findings will inform the creation of a Delphi consensus exercise by an expert panel. We will use this Delphi survey to develop a checklist for reporting standards for NS. This will be accompanied by supplementary guidance on the conduct and reporting of NS, as well as an online training resource.

Ethics and dissemination Ethical approval for the Delphi survey was obtained from the University of Glasgow in December 2017 (reference 400170060). Dissemination of the results of this study will be through peer-reviewed publications, and national and international conferences.

evidence synthesis
health policy

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/

https://doi.org/10.1136/bmjopen-2017-020064

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

This study will be the first to develop a consensus-based reporting guideline for narrative synthesis of quantitative data (NS) in systematic reviews.

The study follows the recommended methodology for developing reporting standards.

The online Delphi survey of international experts in NS will be an effective method of gaining reliable consensus from a group of experts.

The reporting guideline and the supplementary materials developed to support use of existing guidance will aid the implementation of best practice conduct and reporting of NS.

Introduction

Well-conducted systematic reviews are important for informing clinical practice and health policy. 1 In some reviews, meta-analysis of effect estimates may not be possible or sensible. For example, data may be insufficient to allow calculation of standardised effect estimates, the effect metrics arising from different study designs may not be amenable to synthesis (eg, those arising from interrupted time series and randomised trials), or high levels of statistical heterogeneity may mean that presenting an average effect is misleading. For reviews of quantitative data where statistical synthesis is not possible, narrative synthesis of quantitative data (NS) is often the alternative method of choice. A major concern about NS is that it lacks transparency and therefore introduces bias into the synthesis. 2 3 This is an important criticism, which raises questions about the validity and utility of reviews using NS, and ultimately increases the risk of adding to research waste. 4 NS involves collating study findings into a coherent textual narrative, with descriptions of differences in characteristics of the studies including context and validity, often using tables and graphs to display results. 5 6 Published guidance for NS funded by the UK’s Economic and Social Research Council (ESRC) describes techniques for promoting transparency between review level data and conclusions; these include graphical and structured tabulation of the data. 5 However, a recent analysis of systematic reviews of public health interventions suggests that this guidance is rarely used. 7

Relative to developments in meta-analysis or statistical synthesis, and synthesis of qualitative data in the past decade, work to support improved conduct and transparent reporting in NS has been scarce. While a reporting guideline has been developed for systematic reviews and meta-analysis, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), 8 the focus of the synthesis items is on meta-analysis of effect estimates, with no items for alternative approaches to synthesis. The Cochrane Methodological Expectations of Cochrane Intervention Reviews (MECIR) standards for conducting and reporting Cochrane reviews specify one general item referring to non-quantitative synthesis or non-statistical synthesis, and do not have any items specifically for NS. 2 Reporting guidelines have had some impact on improving the reporting for randomised trials and may have similar benefits for improving the reporting of methods and results from NS. 9

There is a growing demand for reviews addressing complex questions, and which incorporate diverse sources of data. Cochrane, a global leader in evidence synthesis of health and public health interventions, has recognised this. 10 Following the prioritisation of relevance and breadth of coverage in the Cochrane strategy, 10 it is likely that the proportion of Cochrane reviews addressing complex questions will increase; this may result in increased use of NS methods. Realising the need for improved implementation and reporting of NS methods, the Cochrane Strategic Methods Fund has funded the ICONS-Quant project: Improving Conduct and Reporting of Narrative Synthesis of Quantitative Data. This paper presents the protocol for the work that will be undertaken.

ICONS-Quant

The ICONS-Quant project aims to improve the implementation of NS methods through enhancing existing guidance on the conduct of NS and developing a reporting guideline. Provision of reporting guidelines alone will not necessarily lead to improved research conduct; provision of explanatory guidance, dissemination, endorsement and support for adherence is also necessary. 11 We will produce materials to support the implementation of best practice in the application of NS methods, and improved reporting. While our focus is on Cochrane reviews, the key outputs of the project will be of use for reviews published elsewhere and will be made freely available. We will:

describe current practice in conduct and reporting of NS in Cochrane reviews;

achieve expert consensus on reporting standards for NS;

provide support for those involved in NS through the provision of enhanced guidance on NS conduct and online training resources.

We intend ICONS-Quant guideline to be used in combination with the PRISMA guidelines. 8 The PRISMA guidelines provide items relating to the various stages of review conduct, for example, providing a clear abstract, explaining the literature search strategy, reporting methods to assess risk of bias. The ICONS-Quant reporting guideline will focus on the methods of synthesis, relating most closely to expanding on PRISMA Item 14 ‘synthesis of results’, outlining details that require to be reported to promote transparency in NS.

Methods and analysis

The ICONS-Quant project will be conducted over a period of 24 months from May 2017. Here, we outline the development of a reporting guideline for NS and supporting materials for existing guidance. In line with recommendations for best practice in developing reporting guidelines, 11 we will:

identify the need for the ICONS-Quant guideline (Work Programme One);

conduct a Delphi survey and consensus meeting (Work Programme Two);

enhance existing guidance on NS (Work Programme Three);

develop learning materials for implementation of NS (Work Programme Four).

Below we outline the Project Advisory Group (PAG) and the research that will be conducted within each Work Programme. Details of the ICONS-Quant project have been registered with the Enhancing the Quality and Transparency of Health Research Network, which provides a database of reporting guidelines in development ( http://www.equator-network.org/library/reporting-guidelines-under-development/ #74).

Project Advisory Group

We have established an ICONS-Quant PAG which will provide governance for the project as well as expert advice. The ICONS PAG includes named project collaborators from Cochrane Review Groups (Effective Practice and Organisation of Care, Consumers and Communication, and Tobacco Addiction), a representative with experience of NS from the Campbell Collaboration Methods Group and a user representative from the National Institute for Health and Care Excellence.

Work Programme One: assessment of current reporting and conduct of NS in Cochrane reviews

Previously we investigated current practice in the conduct and reporting of NS in systematic reviews of public health interventions. 7 Work Programme One will extend this exercise to assess use of NS methods and their reporting across all Cochrane Review Groups. We will identify all Cochrane reviews published between April 2016 and April 2017 and screen them to determine the method of synthesis for the primary outcome. Reviews will be included for further examination if the method for reporting the synthesis of the primary outcomes relies on text. We will identify those that use NS or that synthesise studies using text only, whether or not the authors refer to the use of NS or textual methods for synthesis. Reviews will be excluded if they are empty, include only one study, report on diagnostic test accuracy, or are a review of methodology. We will record how the synthesis has been conducted and reported. We will use the existing data extraction template designed for our previous assessment of NS in public health reviews. This template is based on key sources of best practice for NS, 12–15 including the ESRC guidance on the conduct of NS. 5 Questions relate to use of theory; investigation of differences across included studies and reported findings; transparency of links between data and text (including data visualisation tools used); assessment of robustness of the synthesis; and adequacy of description of NS methods. 5 Using a similar format to our review of NS in public health reviews, 16 we will tabulate the extracted data. This will allow description of:

the extent of reporting of NS methods: the amount and type of detail included;

the range of approaches and tools used to narratively synthesise data;

how conceptual and methodological heterogeneity is managed;

review authors’ reflection on robustness of synthesis.

The results of this exercise will be used to inform development of the initial checklist for inclusion in the Delphi survey.

Work Programme Two: Delphi survey

A Delphi consensus survey will be conducted. This is the standard approach to elicit expert opinion for the purposes of developing consensus-based reporting guidelines. 17 18 The results of the assessment exercise in Work Programme One, in conjunction with key texts on NS, 12–15 19 20 findings from the previous assessment of reporting NS in public health reviews 16 and input from the ICONS PAG, will be used to develop the initial items for Round One of the Delphi survey. An expert panel will then be consulted to inform the development of the Delphi survey. The panel will be identified by the project team and members of the ICONS PAG, and will comprise 15–20 authors and methodologists experienced in or familiar with the purpose and conduct of NS. A videoconference with the expert panel will be used to present findings from Work Programme One and a draft of the proposed Delphi survey. Participants’ input will be recorded and used to refine the Delphi survey.

The Delphi online survey will use a questionnaire to achieve consensus on the content and wording of reporting items considered to capture the pertinent details of NS. The online platform will be created by the MRC/CSO Social and Public Health Sciences Unit, University of Glasgow, using a web-based platform recently developed for this purpose. The platform facilitates personalised invitations to participate, password-protected logins and personalised reminders, and enables data collation for quantitative and qualitative analyses. There will be two rounds of the survey, with a third version conducted if necessary to gain consensus among participants.

Participants will include members of the ICONS PAG and others experienced in NS. Suitable participants will be identified by the project team, through recommendations from the ICONS PAG and through the data extraction exercise described in Work Programme One. The data extraction exercise will help articulate identified gaps in reporting of methods and findings of NS where transparency is particularly lacking. The identified gaps will be used when drafting reporting item questions for the Delphi consensus exercise, to improve transparency in NS. We will invite a maximum of 100 individuals to participate and they will be recruited via their workplace email address. We will ask for their professional opinion on the content of a draft reporting guideline. The invitation will outline the aim of the Delphi survey, the process involved and the time commitment, and include a participant information sheet. Individuals who accept the invitation will be asked to take part in each round of the survey. It will be clearly stated that at any stage, a respondent can opt out of the Delphi survey. The survey will ask participants to provide details of their job category; no personal information will be collected. Respondents will be asked to use their email address to log in to the survey. This information will be used only to verify the appropriate use of the survey and will not be used in the analysis. The Delphi survey will involve implied consent: it will be made clear to participants that by responding to the survey, they are consenting to participate in the study. It will be explained to respondents that: their responses will not be linked to their identity (deidentified); only researchers will have access to the data; and the data will be stored on a password-encrypted computer and stored and destroyed in accordance with Medical Research Council guidelines.

The Delphi survey will consist of closed and open-ended questions. Round One of the survey will provide an introduction to the project and instructions for the survey. The participants will be invited to rank each of the proposed guideline items on a 4-point Likert scale (essential, desirable, possible, omit, used in previous Delphi surveys for developing reporting guidelines 21 22 ). For each item, the participants will be invited to provide comments. A reminder email will be sent approximately 2 weeks after the initial invitation. Round One will close approximately 4 weeks after the first invitations are issued. Responses to Round One of the Delphi will be exported verbatim into a Microsoft Excel spreadsheet and collated. Responses to the scale rating will be summarised as counts and percentage frequencies. The free-text content will be collated and summarised. The results from both the quantitative and qualitative data collation will be used to inform the development of Round Two of the Delphi and the content of the final guideline checklist. Redrafting of the Delphi survey items will be conducted in discussion with all study group members within 1 month of closure of the round.

All participants from Round One will be invited to take part in Round Two. In Round Two, the proposed checklist items will be presented in three sections:

Items that reached high consensus in Round One and that are expected to be included in the final checklist. These items will have an a priori agreement of >70% approval, as recommended by Diamond et al . 23 Participants will not be asked to rate these items again but will be asked to comment on whether they agree with the inclusion of each item in the checklist and to provide comments if they disagree or with suggestions to clarify the wording of the items.

Items that have been significantly altered or are additional as a result of Round One. The participants will be invited to rate these items on the 4-point scale and provide comments on each item.

Items that were rated as ‘omit’ in Round One and that are not expected to be included in the final checklist. Participants will not be asked to rate these items; they will be invited to provide their opinion on the removal of these items from the final checklist.

If there is a substantial lack of consensus remaining following Round Two of the Delphi, a third round will be prepared and conducted. Round Three will follow the same format as Round Two, providing the reporting guideline items in three sections: items that are expected to be included in the final guideline; those significantly altered; and items that will be removed from the final checklist.

Consensus meeting

An expert panel of individuals experienced in NS methods will be invited to participate in the consensus meeting to finalise the content of the guideline. It is anticipated that this will be held as a face-to-face meeting at the Cochrane colloquium in 2018 in Edinburgh, UK. If this is not possible, an online consensus meeting will be conducted using webinar software. If necessary, an additional virtual meeting will be held to accommodate different time zones of invitees. At the consensus meeting the reporting guideline items developed from the Delphi survey will be discussed, with priority given to establishing consensus on the content and wording of items for which the level of consensus is less clear.

Work Programme Three: enhancement of existing guidance on NS methods

We will produce materials to support the current guidance that includes information on the rationale for, as well as implementation of each stage of NS. This will be developed as a supplement to the reporting guideline items. The enhanced guidance will be accessible to novice reviewers and will provide examples of good practice to illustrate how methods of NS may be used. The findings of Work Programmes One and Two, the assessment of current reporting of NS and the Delphi consensus will be used to inform development of enhanced guidance on NS. 5 Cochrane Review Groups who publish reviews incorporating NS will be identified through the process of Work Programme One. We anticipate that these will include a range of Cochrane Review Groups and examples will be developed which are relevant to all groups. An overview of methodological tools which can be used to support NS and which have been developed since publication of the ESRC guidance in 2006 will also be incorporated. The PAG will be asked for comments on the draft guidance before it is piloted.

Work Programme Four: development of learning materials on implementation of NS

Training materials based on the guidance developed in Work Programmes Two and Three will be produced to promote improved use of NS methods. We have secured support from Cochrane Training to collaborate in Work Programme Four. We will deliver two to three live participatory webinars (to allow for different time zones) to present the agreed guidance developed in Work Programme Two. One webinar will be recorded and provided on a web page, along with a record of the questions raised in the webinar, and any other frequently asked questions that emerge.

In addition, an online training module on NS will be developed in collaboration with Cochrane Training and a specialist e-learning company. The module will include a mix of didactic and participatory teaching methods involving assessment and interpretation of data and syntheses. We will work with Cochrane colleagues to incorporate the reporting items into the MECIR standards, and offer to update the relevant chapters of the Cochrane Handbook.

Ethics and dissemination

Dissemination of the results of this study will be through peer-reviewed publications, and national and international conferences. In addition, the objectives of Work Programmes Three and Four are to distribute and encourage use of the ICONS-Quant guideline through webinars and an online training module.

Posada FB ,
Haines A , et al
Higgins J ,
Lasserson T ,
Chandler J , et al
Valentine JC ,
Wilson SJ ,
Rindskopf D , et al
Glasziou P ,
Altman DG ,
Bossuyt P , et al
Roberts H ,
Sowden A , et al
Petticrew M ,
Rehfuess E ,
Noyes J , et al
Campbell M ,
Thomson H ,
Katikireddi SV , et al
Liberati A ,
Tetzlaff J , et al
Shamseer L ,
Altman DG , et al
10. ↵ Cochrane Collaboration . Cochrane strategy to 2020 . 2015 http://community.cochrane.org/organizational-info/resources/strategy-2020
Schulz KF ,
Simera I , et al
Armstrong R ,
Jackson N , et al
Williamson PR
Higgins JP ,
20. ↵ World Health Organization . WHO handbook for guideline development . Geneva : World Health Organization , 2014 .
Hoffmann TC ,
Glasziou PP ,
Boutron I , et al
Craig P , et al
Diamond IR ,
Feldman BM , et al

Contributors HT conceived the idea of the study. HT, SVK, AS, JEM and MC designed the study methodology. MC prepared the first draft of the protocol manuscript and all authors critically reviewed and approved the final manuscript.

Funding This project was supported by funds provided by the Cochrane Strategic Methods Fund. MC, HT and SVK receive funding from the UK Medical Research Council (MC_UU_12017-13 and MC_UU_12017-15) and the Scottish Government Chief Scientist Office (SPHSU13 and SPHSU15). SVK is supported by an NHS Research Scotland Senior Clinical Fellowship (SCAF/15/02). JEM is supported by a National Health and Medical Research Council (NHMRC) Australian Public Health Fellowship (1072366).

Disclaimer The views expressed in the protocol are those of the authors and not necessarily those of Cochrane or its registered entities, committees or working groups.

Competing interests HT and SVK are Cochrane editors. JEM is a co-convenor of the Cochrane Statistical Methods Group.

Patient consent Not required.

Ethics approval University of Glasgow College of Social Sciences Ethics Committee (reference number400170060)

Provenance and peer review Not commissioned; externally peer reviewed.

Read the full text or download the PDF:

Which review is that? A guide to review types.

Which review is that?
Review Comparison Chart
Decision Tool
Critical Review
Integrative Review
Narrative Review
State of the Art Review
Narrative Summary
Systematic Review
Meta-analysis
Comparative Effectiveness Review
Diagnostic Systematic Review
Network Meta-analysis
Prognostic Review
Psychometric Review
Review of Economic Evaluations
Systematic Review of Epidemiology Studies
Living Systematic Reviews
Umbrella Review
Review of Reviews
Rapid Review
Rapid Evidence Assessment
Rapid Realist Review
Qualitative Evidence Synthesis
Qualitative Interpretive Meta-synthesis
Qualitative Meta-synthesis
Qualitative Research Synthesis
Framework Synthesis - Best-fit Framework Synthesis
Meta-aggregation
Meta-ethnography
Meta-interpretation
Meta-narrative Review
Meta-summary
Thematic Synthesis
Mixed Methods Synthesis

Narrative Synthesis

Bayesian Meta-analysis
EPPI-Centre Review
Critical Interpretive Synthesis
Realist Synthesis - Realist Review
Scoping Review
Mapping Review
Systematised Review
Concept Synthesis
Expert Opinion - Policy Review
Technology Assessment Review
Methodological Review
Systematic Search and Review

Narrative’ synthesis’ refers to an approach to the systematic review and synthesis of findings from multiple studies that relies primarily on the use of words and text to summarise and explain the findings of the synthesis. Whilst narrative synthesis can involve the manipulation of statistical data, the defining characteristic is that it adopts a textual approach to the process of synthesis to ‘tell the story’ of the findings from the included studies. As used here ‘narrative synthesis’ refers to a process of synthesis that can be used in systematic reviews focusing on a wide range of questions, not only those relating to the effectiveness of a particular intervention. (Popay et al. 2006)

A systematic literature review and narrative synthesis on the risks of medical discharge letters for patients’ safety

Christine Maria Schwarz 1 ,
Magdalena Hoffmann ORCID: orcid.org/0000-0003-1668-4294 1 , 2 ,
Petra Schwarz 3 ,
Lars-Peter Kamolz 1 ,
Gernot Brunner 1 &
Gerald Sendlhofer 1 , 2

BMC Health Services Research volume 19 , Article number: 158 ( 2019 ) Cite this article

27k Accesses

44 Citations

5 Altmetric

Metrics details

The medical discharge letter is an important communication tool between hospitals and other healthcare providers. Despite its high status, it often does not meet the desired requirements in everyday clinical practice. Occurring risks create barriers for patients and doctors. This present review summarizes risks of the medical discharge letter.

The research question was answered with a systematic literature research and results were summarized narratively. A literature search in the databases PubMed and Cochrane Library for Studies between January 2008 and May 2018 was performed. Two authors reviewed the full texts of potentially relevant studies to determine eligibility for inclusion. Literature on possible risks associated with the medical discharge letter was discussed.

In total, 29 studies were included in this review. The major identified risk factors are the delayed sending of the discharge letter to doctors for further treatments, unintelligible (not patient-centered) medical discharge letters, low quality of the discharge letter, and lack of information as well as absence of training in writing medical discharge letters during medical education.

Conclusions

Multiple risks factors are associated with the medical discharge letter. There is a need for further research to improve the quality of the medical discharge letter to minimize risks and increase patients’ safety.

Peer Review reports

The medical discharge letter is an important communication medium between hospitals and general practitioners (GPs) and an important legal document for any queries from insurance carriers, health insurance companies, and lawyers [ 1 ]. Furthermore, the medical discharge letter is an important document for the patient itself.

A timely transmission of the letter, a clear documentation of findings, an adequate assessment of the disease as well as understandable recommendations for follow-up care are essential aspects of the medical discharge letter [ 2 ]. Despite this importance, medical discharge letters are often insufficient in content and form [ 3 ]. It is also remarkable that writing of medical discharge letters is often not a particular subject in the medical education [ 4 ]. Nevertheless, the medical discharge letter is an important medical document as it contains a summary of the patient’s hospital admission, diagnosis and therapy, information on the patient’s medical history, medication, as well as recommendations for continuity of treatment. A rapid transmission of essential findings and recommendations for further treatment is of great interest to the patient (as well as relatives and other persons that are involved in the patients’ caring) and their current and future physicians. In most acute care hospitals, patients receive a preliminary medical discharge letter (short discharge letter) with diagnoses and treatment recommendations on the day of discharge [ 5 ]. Unfortunately, though, the full hospital medical discharge letter, which is often received with great delay, is an area of constant conflict between GPs and hospital doctors [ 1 ]. Thus the medical discharge letter does not only represent a feature of process and outcome quality of a clinic, but also influences confidence building and binding of resident physicians to the hospital [ 6 ].

Beside the transmission of patients’ findings from physician to physician, the delivery of essential information to the patient is an underestimated purpose of the medical discharge letter [ 7 ]. The medical discharge letter is often characterized by a complex medical language that is often not understood by the patients. In recent years, patient-centered/patient-directed medical discharge letters are more in discussion [ 8 ]. Thus, the medical discharge letter points out risks for patients and physicians while simultaneously creating barriers between them.

A systematic review of the literature was undertaken to identify patient safety risks associated with the medical discharge letter.

Search strategy

A systematic literature search was conducted using the electronic databases PubMed and Cochrane Database. Additionally, we scanned the reference lists of selected articles (snowballing). The following search terms were used: “discharge summary AND risks”, “discharge summary AND risks AND patient safety” and “discharge letter AND risks” and “discharge letter AND risks AND patient safety”. We reviewed relevant titles and abstracts on English and German literature published between January 2008 and May 2018 and started the search at the beginning of February 2018 and finished it at the end of May 2018.

Eligibility criteria

In this systematic review, articles were included if the title and/or abstract indicated the report of results of original research studies using quantitative, qualitative, or mixed method approaches. Studies in paediatric settings or studies that do not handle possible risks of the medical discharge letter were excluded, as well as reports, commentaries and letters. Electronic citations, including available abstracts of all articles retrieved from the search, were screened by two authors to select reports for full-text review. Duplicates were removed from the initial search. Nevertheless, during the search of articles the selection, publication as well as language bias must be considered. Thereafter, full-texts of potentially relevant studies were reviewed to determine eligibility for inclusion. In the following Table 1 inclusion and exclusion criteria for the studies are listed. Afterwards, key outcomes and main results were summarized. Differences were resolved by consensus. Finally, a narrative synthesis of studies meeting the inclusion criteria was conducted. Reference management software MENDELEY (Version 1.19.3) was used to organise and store the literature.

Data extraction

The data extraction in form of a table was used to summarize study results. The two authors extracted the data relating to author, country, year, study design, and outcome measure as well as potential risk factors to patient safety directly into a pre-formatted data collection form. After data extraction, the literature was discussed and synthesized into themes. The evaluation of the single studies was done using checklists [STROBE (combined) and the Cochrane Data collection form for intervention reviews (RCTs and non-RCTs)]. Meta-analysis was not considered appropriate for this body of literature because of the wide variability of studies in relation to research design, study population, types of interventions and outcomes.

Then a narrative synthesis was performed to synthesize the findings of the different studies. Because of the range of very different studies that were included in this systematic review, we have decided that a narrative synthesis constitutes the best instrument to synthesise the findings of the studies. First, a preliminary synthesis was undertaken in form of a thematic analysis involving searching of studies, listing and presenting results in tabular form. Then the results were discussed again and structured into themes. Afterwards, summarizing of included studies in a narrative synthesis within a framework was performed by one author.

This framework consisted of the following factors: the individuals and the environment involved in the studies (doctors, hospitals), the tools and technology (such as discharge letter delivery systems), the content of the medical discharge letter (such as missing content, quality of content), the accuracy and timeliness of transfer. These themes were discussed in relation to potential risks for patient’s safety. All articles that were included in this review were published before. The framework of this study was chosen following a previously published systematic review dealing with patient risks associated with telecare [ 9 ].

The initial literature search in the two online databases identified 940 records. From these records, 65 full text articles were screened for eligibility. Then 36 full-text articles were excluded because they pertained to patient transfer within the hospital or to another hospital, or to patient hand-over situations. Finally, 29 studies were included in this review. Included studies are listed in Table 2 . All document types were searched with a focus on primary research studies. The results of the search strategy are shown in Fig. 1 .

Flow chart literature search strategy

From these 29 studies, 13 studies dealt with the quality analysis of discharge letters, 12 studies with delayed transmission of medical discharge letters and just as many with the lack of information in medical discharge letters. Only few studies dealt with training on writing medical discharge letters and with understanding of patients of their medical discharge letters. The descriptive information of the included articles is presented in Table 2 . Overall quality of the articles was found to be acceptable, with clearly stated research questions and appropriate used methods.

Risk factors

In the following the identified major risk factors concerning the medical discharge letter are presented in a narrative summary.

Delayed delivery

The medical discharge letters should arrive at the GP soon after hospital discharge to ensure the quickest possible further treatment [ 4 ]. If letters are delivered weeks after the hospital stay, a continuous treatment of the patient cannot be ensured. Furthermore, the author of the medical discharge letter will no longer have current data after the discharge of the patient, which may result in a loss of important information [ 10 ]. Interfaces between different treatment areas and organizational units are known to cause a loss of information and a lack of quality in patient handling [ 11 ]. The improvement of information transfer between different healthcare providers during the transition of patients has been recommended to improve patient care [ 12 , 13 ]. Delayed communication of findings may lead to a lack of continuity of care and suboptimal outcomes, as well as decreased satisfaction levels for both patients and GPs [ 14 , 15 , 16 ]. In a review of Kripalani et al., it was shown that 25% of discharge summaries were never received by GPs [ 17 ]. This has several negative consequences for patients. Li et al. [ 18 ] found that a delayed transmission or absence of the medical discharge summary is related to patient readmission, and a study by Gilmore-Bykovskyi [ 19 ] found a strong relationship between patients whose discharge summaries omitted designation of a responsible clinician/clinic for follow-up care and re-hospitalisation and/or death. A Swedish study by Carlsson et al. [ 20 ] points out that a lack of accuracy and continuity in discharge information on eating difficulties may increase risk of undernutrition and related complications. A study of Were et al. [ 18 ] investigated pending lab results in medical discharge summaries and found that only 16% of tests with pending results were mentioned in the discharge summaries, and Walz et al. [ 21 ] found that approximately one third of the sub-acute care patients had pending lab results at discharge, but only 11% of these were documented in the medical discharge summaries.

Quality, lack of information

Medical discharge letters are a key communication tool for patient safety issues [ 17 ]. Incomplete and insufficient medical discharge letters increase the risks of readmission and myriad other complications [ 22 ]. Langelaan et al. (2017) evaluated more than 2000 medical discharge letters and found that in about 60% of the letters essential information was missing, such as a change of the existing medication, laboratory data, and even data on the patients themselves [ 23 ]. Accurate and complete medical discharge summaries are essential for patient safety [ 17 , 24 , 25 ]. Addresses; patient data, including duration of stay; diagnoses; procedures; operations; epicrisis and therapy recommendations; as well as findings in the appendix; are minimum requirements that are supposed to be included in the medical discharge letter [ 4 ]. However, it was found that key components are often lacking in medical discharge letters, including information about follow-up and management plans [ 23 , 26 ], test results [ 27 , 28 , 29 ], and medication adjustments [ 30 , 31 , 32 , 33 , 34 , 35 ]. In a review of Wimsett et al. [ 36 ] key components of a high-quality medical discharge summary were identified in 32 studies. These important components were discharge diagnosis, the received treatment, results of investigations as well as follow-up plans.

Accuracy of patients’ medication information is important to ensure patient safety. Hospital doctors expect GPs to continue with the prescribed (or modified) drug therapy. However, the selection of certain drugs is not always transparent for the GPs. A study by Grimes et al. [ 30 ] found that a discrepancy in medication documentation at discharge occurred in 10.8% of patients. From these patients nearly 65.5% were affected by discrepancies in medication documentation. The most prevalent inconsistency was drug omission (20.9%). Only 2% of patients were contacted, although general patient harm was assessed. A Swedish study of 2009 [ 37 ] investigated the quality improvement of medical discharge summaries. A higher quality of discharge letter led to an average of 45% fewer medication errors per patient.

A recent study by Tong et al. [ 38 ] revealed a reduced rate of medication errors in medical discharge summaries that were completed by a hospital pharmacist. Hospital pharmacists play a key role in preparing the discharge medication information transferred to GPs upon patient discharge and should work closely with hospital doctors to ensure accurate medication information that is quickly communicated to GPs at transitions of care [ 39 ]. Most hospitals have introduced electronic systems to improve the discharge communication, and many studies found a significant overall improvement in electronic transfer systems due to better documentation of information about follow-up care, pending test results, and information provided to patients and relatives [ 40 , 41 , 42 ]. Mehta et al. [ 43 ] found that the changeover to a new electronic system resulted in an increased completeness of discharge summaries from 60.7 to 75.0% and significant improvements in levels of completeness in certain categories.

Writing of medical discharge letter is missing in medical education

Both junior doctors as well as medical students reported that they received inadequate guidance and training on how to write medical discharge summaries [ 44 , 45 ] and recognized that higher priority is often given to pressing clinical tasks [ 46 ]. Research into the causes of prescribing errors by junior doctors at hospitals in the UK has revealed that latent conditions like organizational processes, busy environments, and medical care for complex patients can lead to medication errors in the medical discharge summary [ 47 ].

Fortunately, some study results demonstrate that information and education on writing medical discharge letters would enhance communication to the GPs and prevent errors during the patient discharge process [ 37 ]. Minimal formal teaching about writing medical discharge summaries is common in most medical schools [ 39 , 46 ]; however, a study by Shivji et al. has shown that simple, intensive educational sessions can lead to an improvement in the writing process of medical discharge summaries and communication with primary care [ 48 ].

Since the medical discharge letter should meet specific quality criteria, senior physicians and/or the head physician correct(s) and validate(s) the letter. The medical discharge letter therefore represents an essential learning target [ 8 ]. Training activities and workshops are necessary for junior doctors to improve writing medical discharge letters [ 44 , 49 ]. It might be also useful for young doctors to use checklists or other structured procedures to improve writing [ 4 ]. Maher et al. showed that the use of a checklist enhanced the quality (content, structure, and clarity) of medical discharge letters written by medical students [ 50 ].

In the following Table 3 main risk factors of the medical discharge letter are summarized.

The results of this systematic literature research indicate notable risk factors relating to the medical discharge letter. In a study by Sendlhofer et al., 360 risks were identified in hospital settings [ 51 ]. From these, 176 risks were scored as strategic and clustered into “top risks”. Top risks included medication errors, information errors, and lack of communication, among others. During this review, these potential risk factors were also identified in terms of the medical discharge letter.

Delayed sending and low quality of medical discharge letters to the referring physicians, may adversely affect the further course of treatment. However, a study of Spencer et al. has determined rates of failures in processing actions requested in hospital discharge summaries in general practice. It was found that requested medication changes were not made in 17% and patient harm occurred in 8% in relation to failures [ 52 ].

Despite the existence of reliable standards [ 53 ] many physicians are not adequately trained for writing medical discharge letters during their studies. Regular trainings and workshops and standardized checklists may optimize the quality of the medical discharge letter. Furthermore, electronic discharge letters have the potential to easily and quickly extract important information such as diagnoses, medication, and test results into a structured discharge document, and offer important advantages such as reliability, speed of information transfer, and standardization of content. Comprehensive discharge letters reduce the readmission rate and increase safety and quality by discharging of the patient. A missing structure, as well as a complex language, illegible handwriting, and unknown abbreviations, make reading medical discharge letters more complicated [ 4 ]. At least, poor patient understanding of their diagnosis and treatment plans and incomprehensible recommendations can adversely impact clinical outcome following hospital discharge. Many studies confirm that inadequate communication of findings [ 3 , 39 , 54 ] is an important risk factor in patients’ safety [ 51 ].

Most medical information in the discharge letter is not understood by patients (as well as relatives and other persons that are involved in the patients’ caring) and patients themselves do not receive a comprehensible medical discharge letter. The content of the medical discharge letter is often useless for the patient due to its medical terminology and content that is not matching with the patient’s level of knowledge or health literacy [ 55 , 56 , 57 ]. Poor understanding of diagnoses and related discharge plans are common among patients and family members and often accompanied by unplanned hospital readmissions [ 58 , 59 , 60 , 61 ]. In a study by Lin et al., it was shown that a patient-directed discharge letter enhanced understanding for hospitalization and for recommendations. Furthermore, verbal communication of the letter contents, explanation of every section of the medical discharge letter, and the opportunity for discussion and asking questions improved patient comprehension [ 7 ]. A study by O’Leary et al. showed that roughly 80–95% of patients with breast tumours want to be informed and educated about their illness, treatment, and prognosis [ 62 ].

High quality of care is characterized by a patient-centered communication, where the patient’s personal needs are also in focus [ 63 ]. Translation of medical terms in reports and letters leads to a better understanding of the disease and, interestingly, the avoidance of medical terms did not lead to deterioration in the transmission of information between the treating physicians. Moreover, it was found that the minimisation of medical terminology in medical discharge letters improved understanding and perception of patients’ ability to manage chronic health conditions [ 64 ]. In effect, it is clear that patient-centered communication improves outcome, mental health, patient satisfaction and reduces the use of health services [ 65 ].

Strengths and limitations

We have identified key problems with the medical discharge summaries that negatively impact patients’ safety and wellbeing. However, there is a heterogeneous nature of the included studies in terms of study design, sample size, outcomes, and language. Only two reviewers screened the studies for eligibility and only full-text articles were included in the literature review; furthermore, only the databases Pubmed and Cochrane library were screened for appropriate studies. Due to these constraints, there is a chance that other relevant studies may have been missed.

High-quality medical discharge letters are essential to ensure patient safety. To address this, the current review identified the major risk factors as delayed sending and low quality of medical discharge letters, lack of information and patient understanding, and inadequate training in writing medical discharge letters. In future, research studies should focus on improving the communication of pending test results and findings at discharge, and on evaluating the impact that this improved communication has on patient outcomes. Moreover, a simple patient-centered medical discharge letter may improve the patient’s (as well as family members’ and other caregivers’) understanding of disease, treatment and post-discharge recommendations.

Abbreviations

General practitioner

Randomized Controlled Trial

STrengthening the Reporting of OBservational studies in Epidemiology

United Kingdom

Kreße B, Dinser R. Anforderungen an Arztberichte- ein haftungsrechtlicher Ansatz. Medizinrecht. 2010;28(6):396–400.

Google Scholar

Möller K-H, Makoski K. Der Arztbrief - Rechtliche Rahmen- Bedingungen. 2015;5:186–94.

Van Walraven C, Weinberg AL. Quality assessment of a discharge summary system. CMAJ. 1995;152(9):1437–42.

PubMed Google Scholar

Unnewehr M, Schaaf B, Friederichs H. Die Kommunikation optimieren. Dtsch Arztebl Int. 2013;110(37):831–4.

Roth-Isigkeit A, Harder S. Die Entlassungsmedikation im Arztbrief. Eine explorative Befragung von Hausärzten/−innen. Vol. 100, Medizinische Klinik. 2005. p. 87–93.

Bohnenkamp B. Arbeitsorganisation: Der Arztbrief - Viel mehr als nur lästige Pflicht. Vol. 113, Deutsches Ärzteblatt International. 2016. p. 2–4.

Lin R, Tofler G, Spinaze M, Dennis C, Clifton-Bligh R, Nojoumian H, Gallagher R, et al. Patient-directed discharge letter (PADDLE)-a simple and brief intervention to improve patient knowledge and understanding at time of hospital discharge. Hear Lung Circ. 2012;21:S312.

Hammerer P. Patientenverständliche Arztbriefe und Befunde. Springer Medizin. 2018;33(2):119–23.

Guise V, Anderson J, Wiig S. Patient safety risks associated with telecare: a systematic review and narrative synthesis of the literature. BMC Health Serv Res. 2014;14(1):588.

Raab, A., & Drissner A. Einweiserbeziehungsmanagement: Wie Krankenhäuser erfolgreich Win-Win-Beziehungen zu niedergelassenen Ärzten aufbauen. Kohlhammer Verlag; 2011. 240 p.143.

Hart D. Vertrauen, Kooperation, Organisation. Berlin Heidelberg: Springer Verlag; 2006. p. 845–7.

Cook RI. Gaps in the continuity of care and progress on patient safety. BMJ. 2000;320(7237):791–4.

CAS PubMed Google Scholar

Duggan C, Feldman R, Hough J, Bates I. Reducing adverse prescribing discrepancies following hospital discharge. Int J Pharm Pract. 1998;6(2):77–82.

Polyzotis PA, Suskin N, Unsworth K, Reid RD, Jamnik V, Parsons C, et al. Primary care provider receipt of cardiac rehabilitation discharge summaries are they getting what they want to promote long-term risk reduction. Circ Cardiovasc Qual Outcomes. 2013;6(1):83–9.

Poon EG, Gandhi TK, Sequist TD, Murff HJ, Karson AS, Bates DW. “I wish i had seen this test result earlier!”: Dissatisfaction with test result management systems in primary care. Vol. 164, Archives of Internal Medicine. 2004. p. 2223–8.

Coleman EA, Berenson RA. Lost in transition: Challenges and opportunities for improving the quality of transitional care. Vol. 141, Annals of Internal Medicine. 2004. p. 533–6.

Kripalani S, LeFevre F, Phillips CO, Williams MV, Basaviah P, Baker DW. Deficits in communication and information transfer between hospital-based and primary care physicians: implications for patient safety and continuity of care. JAMA. 2007;297(8):831–41.

Were MC, Li X, Kesterson J, Cadwallader J, Asirwa C, Khan B, et al. Adequacy of hospital discharge summaries in documenting tests with pending results and outpatient follow-up providers. J Gen Intern Med. 2009;24(9):1002–6.

Gilmore-Bykovskyi AL, Kennelty KA, Dugoff E, Kind AJH. Hospital discharge documentation of a designated clinician for follow-up care and 30-day outcomes in hip fracture and stroke patients discharged to sub-acute care. BMC Health Serv Res. 2018;18(1).

Carlsson E, Ehnfors M, Eldh AC, Ehrenberg A. Accuracy and continuity in discharge information for patients with eating difficulties after stroke. J Clin Nurs. 2012;21(1–2):21–31.

Walz SE, Smith M, Cox E, Sattin J, Kind AJH. Pending laboratory tests and the hospital discharge summary in patients discharged to sub-acute care. J Gen Intern Med. 2011;26(4):393–8.

Horwitz LI, Jenq GY, Brewster UC, Chen C, Kanade S, Van Ness PH, et al. Comprehensive quality of discharge summaries at an academic medical center. J Hosp Med. 2013;8(8):436–43.

Alderton M, Callen JL. Are general practitioners satisfied with electronic discharge summaries? Vol. 36, Health Information Management Journal. 2007. p. 7–12.

Harel Z, Wald R, Perl J, Schwartz D, Bell CM. Evaluation of deficiencies in current discharge summaries for dialysis patients in Canada. J Multidiscip Healthc. 2012;5:77–84.

Philibert I, Barach P. The European HANDOVER Project: A multi-nation program to improve transitions at the primary care - Inpatient interface. BMJ Quality and Safety. 2012;21(SUPPL. 1).

Greer RC, Liu Y, Crews DC, Jaar BG, Rabb H, Boulware LE. Hospital discharge communications during care transitions for patients with acute kidney injury: a cross-sectional study. BMC Health Serv Res. 2016;16:449.

Belleli E, Naccarella L, Pirotta M. Communication at the interface between hospitals and primary care: a general practice audit of hospital discharge summaries. Aust Fam Physician. 2013;42(12):886–90.

Roy CL, Poon EC, Karson AS, Ladak-Merchant Z, Johnson RE, Maviglia SM, et al. Patient safety concerns arising from test results that return after hospital discharge. Ann Intern Med. 2005;143(2):121–8.

Gandara E, Moniz T, Ungar J, Lee J, Chan-Macrae M, O’Malley T, et al. Communication and information deficits in patients discharged to rehabilitation facilities: an evaluation of five acute care hospitals. J Hosp Med. 2009;4(8):E28–33.

Grimes T, Delaney T, Duggan C, Kelly JG, Graham IM. Survey of medication documentation at hospital discharge: implications for patient safety and continuity of care. Ir J Med Sci. 2008;177(2):93–7.

Uitvlugt EB, Siegert CEH, Janssen MJA, Nijpels G, Karapinar-Çarkit F. Completeness of medication-related information in discharge letters and post-discharge general practitioner overviews. Int J Clin Pharm. 2015;37(6):1206–12.

Ooi CE, Rofe O, Vienet M, Elliott RA. Improving communication of medication changes using a pharmacist-prepared discharge medication management summary. Int J Clin Pharm. 2017;39(2):394–402.

Perren A, Previsdomini M, Cerutti B, Soldini D, Donghi D, Marone C. Omitted and unjustified medications in the discharge summary. Qual Saf Heal Care. 2009;18(3):205–8.

CAS Google Scholar

Garcia BH, Djønne BS, Skjold F, Mellingen EM, Aag TI. Quality of medication information in discharge summaries from hospitals: an audit of electronic patient records. Int J Clin Pharm. 2017;39(6):1331–7.

Monfort AS, Curatolo N, Begue T, Rieutord A, Roy S. Medication at discharge in an orthopaedic surgical ward: quality of information transmission and implementation of a medication reconciliation form. Int J Clin Pharm. 2016;38(4):838–47.

Wimsett J, Harper A, Jones P. Review article: Components of a good quality discharge summary: A systematic review. Vol. 26, EMA - Emergency Medicine Australasia. 2014. p. 430–8.

Bergkvist A, Midlöv P, Höglund P, Larsson L, Bondesson Å, Eriksson T. Improved quality in the hospital discharge summary reduces medication errors-LIMM: Landskrona integrated medicines management. Eur J Clin Pharmacol. 2009;65(10):1037–46.

Tong EY, Roman CP, Mitra B, Yip GS, Gibbs H, Newnham HH, et al. Reducing medication errors in hospital discharge summaries: a randomised controlled trial. Med J Aust. 2017;206(1):36–9.

Yemm R, Bhattacharya D, Wright D, Poland F. What constitutes a high quality discharge summary? A comparison between the views of secondary and primary care doctors. Int J Med Educ. 2014;5:125–31.

O’Leary KJ, Liebovitz DM, Feinglass J, Liss DT, Evans DB, Kulkarni N, et al. Creating a better discharge summary: improvement in quality and timeliness using an electronic discharge summary. J Hosp Med. 2009;4(4):219–25.

Chan S. P Maurice a, W pollard C, Ayre SJ, Walters DL, Ward HE. Improving the efficiency of discharge summary completion by linking to preexisiting patient information databases. BMJ Qual Improv Reports. 2014;3(1):1–5.

Lehnbom EC, Raban MZ, Walter SR, Richardson K, Westbrook JI. Do electronic discharge summaries contain more complete medication information? A retrospective analysis of paper versus electronic discharge summaries. Heal Inf Manag J. 2014;43(3):4–12.

Mehta RL, Baxendale B, Roth K, Caswell V, Le Jeune I, Hawkins J, et al. Assessing the impact of the introduction of an electronic hospital discharge system on the completeness and timeliness of discharge communication: A before and after study. BMC Health Serv Res. 2017;17(1).

Heaton A, Webb DJ, Maxwell SRJ. Undergraduate preparation for prescribing: the views of 2413 UK medical students and recent graduates. Br J Clin Pharmacol. 2008;66(1):128–34.

Maxwell S, Walley T. Teaching safe and effective prescribing in UK medical schools: A core curriculum for tomorrow’s doctors. Vol. 55, British Journal of Clinical Pharmacology. 2003. p. 496–503.

Frain JP, Frain AE, Carr PH. Experience of medical senior house officers in preparing discharge summaries. Br Med J. 1996;312(7027):350.

Dornan T, Investigator P, Ashcroft D, Lewis P, Miles J, Taylor D, et al. An in depth investigation into causes of prescribing errors by foundation trainees in relation to their medical education. EQUIP study. Vol. 44, Methods. 2010.

Shivji FS, Ramoutar DN, Bailey C, Hunter JB. Improving communication with primary care to ensure patient safety post-hospital discharge. Br J Hosp Med. 2015;76(1):46–9.

Cresswell A, Hart M, Suchanek O, Young T, Leaver L, Hibbs S. Mind the gap: Improving discharge communication between secondary and primary care. BMJ Qual Improv Reports. 2015;4(1):u207936.w3197.

Maher B, Drachsler H, Kalz M, Hoare C, Sorensen H, Lezcano L, et al. Use of Mobile applications for hospital discharge letters - improving handover at point of practice. Int J Mob Blended Learn. 2013;5(4):29.

Sendlhofer G, Brunner G, Tax C, Falzberger G, Smolle J, Leitgeb K, et al. Systematic implementation of clinical risk management in a large university hospital: the impact of risk managers. Wien Klin Wochenschr [Internet]. 2015; 127:1–11. Available from: https://doi.org/10.1007/s00508-014-0620-7

Spencer RA, Spencer SEF, Rodgers S, Campbell SM, Avery AJ. Processing of discharge summaries in general practice: a retrospective record review. Br J Gen Pract. 2018;68(673):e576–85.

Carpenter I. A Clinician’s guide to record standards - part 2: standards for the structure and content of medical records and communications when patients are admitted to hospital. Acad Med R Coll R Coll Physicians. 2008;10(5):24.

GMC. Good practice in prescribing and managing medicines and devices. Good Medical Practice. 2013. p. 1–11.

Jolly BT, Scott JL, Sanford SM. Simplification of emergency department discharge instructions improves patient comprehension. Ann Emerg Med. 1995;26(4):443–6.

Williams DM, Counselman FL, Caggiano CD. Emergency department discharge instructions and patient literacy: a problem of disparity. Am J Emerg Med. 1996;14(1):19–22.

Jolly BT, Scott JL, Feied CF, Sanford SM. Functional illiteracy among emergency department patients: a preliminary study. Ann Emerg Med. 1993;22(3):573–8.

Soler RS, Juvinyà Canal D, Noguer CB, Poch CG, Brugada Motge N, Garcia D m, Gil M. Continuity of care and monitoring pain after discharge: patient perspective. J Adv Nurs. 2010;66(1):40–8.

Grover G, Berkowitz CD, Lewis RJ. Parental recall after a visit to the emergency department. Clin Pediatr (Phila). 1994;33(4):194–201.

Witherington EMA, Pirzada OM, Avery AJ. Communication gaps and readmissions to hospital for patients aged 75 years and older: observational study. Qual Saf Heal Care. 2008;17(1):71–5.

Marcantonio ER, McKean S, Goldfinger M, Kleefield S, Yurkofsky M, Brennan T. a. Factors associated with unplanned hospital readmission among patients 65 years of age and older in a Medicare managed care plan. Am J Med. 1999;107(1):13–7.

O’Leary KA, Estabrooks CA, Olson K, Cumming C. Information acquisition for women facing surgical treatment for breast cancer: Influencing factors and selected outcomes. Vol. 69, Patient Education and Counseling. 2007. p. 5–19.

Vogel BA, Helmes AW, Bengel J. Arzt-Patienten-Kommunikation in der Tumorbehandlung: Erwartungen und Erfahrungen aus Patientensicht. Zeitschrift für Medizinische Psychol. 2006;15(4):149–61.

Wernick M, Hale P, Anticich N, Busch S, Merriman L, King B, et al. A randomised crossover trial of minimising medical terminology in secondary care correspondence in patients with chronic health conditions: impact on understanding and patient reported outcomes. Intern Med J. 2016;46(5):596–601.

Woods SS, Schwartz E, Tuepker A, Press NA, Nazi KM, Turvey CL, et al. Patient experiences with full electronic access to health records and clinical notes through the my healthevet personal health record pilot: Qualitative study. J Med Internet Res. 2013;(15, 3).

Weiskopf NG, Rusanov A, Weng C. Sick patients have more data: the non-random completeness of electronic health records. MIA Symp. 2013;2013:1472–7.

Choudhry AJ, Baghdadi YMK, Wagie AE, Habermann EB, Heller SF, Jenkins DH, et al. Readability of discharge summaries: with what level of information are we dismissing our patients? In: Am J Surg. 2016. p. 631–6.

Li JYZ, Yong TY, Hakendorf P, Ben-Tovim D, Thompson CH. Timeliness in discharge summary dissemination is associated with patients’ clinical outcomes. J Eval Clin Pract. 2013;19(1):76–9.

Download references

Acknowledgements

Not applicable.

This research project was part of a project funded by the Gesundheitsfonds Steiermark. The funders had no role in study design, data collection and analyses, decision to publish, or preparation of the manuscript.

Availability of data and materials

All data generated or analysed during this study are included in this published article and its supplementary information files.

Author information

Authors and affiliations.

Research Unit for Safety in Health, c/o Division of Plastic, Aesthetic and Reconstructive Surgery, Department of Surgery, Medical University of Graz, Graz, Austria

Christine Maria Schwarz, Magdalena Hoffmann, Lars-Peter Kamolz, Gernot Brunner & Gerald Sendlhofer

Executive Department for Quality and Risk Management, University Hospital Graz, Auenbruggerplatz 1/3, 8036, Graz, Austria

Magdalena Hoffmann & Gerald Sendlhofer

Carinthia University of Applied Science, Feldkirchen, Austria

Petra Schwarz

You can also search for this author in PubMed Google Scholar

Contributions

CS wrote the manuscript; CS, MH and PS performed the literature search; LK contributed to the conception of this work; GB contributed to the interpretation of data and GS supervised the project. All authors were critically revising the manuscript and all authors have read and approved the final manuscript.

Corresponding author

Correspondence to Magdalena Hoffmann .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Schwarz, C.M., Hoffmann, M., Schwarz, P. et al. A systematic literature review and narrative synthesis on the risks of medical discharge letters for patients’ safety. BMC Health Serv Res 19 , 158 (2019). https://doi.org/10.1186/s12913-019-3989-1

Download citation

Received : 09 August 2018

Accepted : 06 March 2019

Published : 12 March 2019

DOI : https://doi.org/10.1186/s12913-019-3989-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Discharge letter
Discharge summary
Patient safety
Hospital discharge
Systematic review

BMC Health Services Research

ISSN: 1472-6963

General enquiries: [email protected]

conclusions of a systematic review with narrative synthesis

Open access
Published: 21 March 2024

Systems approaches to scaling up: a systematic review and narrative synthesis of evidence for physical activity and other behavioural non-communicable disease risk factors

Harriet Koorts ORCID: orcid.org/0000-0003-1303-6064 1 ,
Jiani Ma 1 ,
Christopher T. V. Swain 2 , 3 ,
Harry Rutter 4 ,
Jo Salmon 1 &
Kristy A. Bolton 1

International Journal of Behavioral Nutrition and Physical Activity volume 21 , Article number: 32 ( 2024 ) Cite this article

27 Accesses

10 Altmetric

Metrics details

Non-communicable diseases (NCDs) are the leading causes of death worldwide. Systems approaches have potential for creating sustainable outcomes at scale but have rarely been used to support scale up in physical activity/nutrition promotion or NCD prevention more generally. This review aimed to: (i) synthesise evidence on the use of systems approaches in scaling up interventions targeting four behavioural risk factors for NCDs; and (ii) to explore how systems approaches have been conceptualised and used in intervention implementation and scale up.

Seven electronic databases were searched for studies published 2016–2021. Eligible studies targeted at least one of four NCD behavioural risk factors (physical inactivity, tobacco use, alcohol consumption, diet), or described evaluation of an intervention planned for or scaled up. Studies were categorised as having a (i) high , (ii) moderate , or (iii) no use of a systems approach. A narrative synthesis of how systems approaches had been operationalised in scale up, following PRISMA guidelines.

Twenty-one intervention studies were included. Only 19% ( n = 4) of interventions explicitly used systems thinking to inform intervention design, implementation and scale up (targeting all four risk factors n = 2, diet n = 1, tobacco use n = 1). Five studies (‘high use’) planned and implemented scale up with an explicit focus on relations between system elements and used system changes to drive impact at scale. Seven studies (‘moderate use’) considered systems elements impacting scale-up processes or outcomes but did not require achieving system-level changes from the outset. Nine studies (‘no use’) were designed to work at multiple levels among multiple agencies in an intervention setting, but the complexity of the system and relations between system elements was not articulated. We synthesised reported barriers and facilitators to scaling up, and how studies within each group conceptualised and used systems approaches, and methods, frameworks and principles for scaling up.

In physical activity research, and NCD prevention more broadly, the use of systems approaches in scale up remains in its infancy. For researchers, practitioners and policymakers wishing to adopt systems approaches to intervention implementation at scale, guidance is needed on how to communicate and operationalise systems approaches in research and in practice.

Trial registration

PROSPERO (CRD42021287265).

Introduction

Non-communicable diseases (NCDs), such as cardiovascular disease, cancer and type 2 diabetes, are the leading causes of death worldwide, contributing to more than 41 million deaths globally [ 1 ]. Major behavioural risk factors for NCDs include physical inactivity, tobacco use, alcohol consumption and an unhealthy diet [ 2 ]. The high burden of NCDs causes substantial economic losses worldwide, and deaths from NCDs disproportionately affect low- and middle-income countries [ 1 ]. Whilst there are multiple targeted action plans for the prevention of NCDs, such as the Global Action Plan for the Prevention and Control of Non-communicable Diseases 2013–2030 [ 2 ], and for specific risk factors such as the Global Action Plan for Physical Activity (GAPPA) [ 3 ] and Global Alcohol Action Plan 2022–2030 [ 4 ]; many of the NCD risk factors have reached pandemic proportions. For example, annually, tobacco use causes over 8 million deaths https://www.who.int/news-room/fact-sheets/detail/tobacco , around 3.2 million deaths are attributed to physical inactivity [ 5 ], over 3 million deaths result from the harmful use of alcohol [ 6 ], and 2.8 million deaths are as a result of being overweight or obese [ 7 ]. By 2030, global health-care costs of physical inactivity alone are estimated to exceed INT$520 billion [ 8 ]. Increasing implementation of evidence-based solutions to reduce the risk of NCDs at scale is a thus global priority of the World Health Organization (WHO) [ 9 ]. Despite burgeoning evidence for the effectiveness of different interventions to prevent NCDs; researchers, practitioners and policymakers have limited access to effective ways of scaling up NCD risk factor interventions, which are essential for global shifts in health [ 10 , 11 ].

Scaling up presents a complex set of challenges. They are complex, not only due to the factors underpinning NCD risk factors, but also in the nature of the processes required to achieve impact at large scale. Interventions intended for scale up should thus be planned with consideration of complexity [ 12 , 13 , 14 ]. Addressing one element within a complex public health problem (e.g., through a discrete intervention targeting a specific factor) is unlikely to achieve desirable population-wide effects at scale [ 10 ]. For physical inactivity, intervention effects can be attenuated at scale [ 15 ], and yet the mechanisms underpinning outcomes of scaling up, which may contribute to these attenuated effects, are often complex and poorly understood [ 16 ]. Large-scale interventions span many different community contexts and adapted in response to these contexts, posing problems for the attribution of impact during evaluation [ 12 ]. The scaling process itself often involves multiple delivery strategies or systems, and the vast number of settings, contexts and systems affected during scale-up can extend beyond the capacity for data collection [ 13 ]. This lack of knowledge of the complex interactions between factors when scaling up poses difficulties when generalising ‘effective’ approaches to successful scaling.

Systems approaches provide a framework for exploring the multitude of interdependent elements that influence a problem, and the scaling up of solutions for that problem, as they can help establish the relations between factors, how they change over time, and acknowledge that effective actions are required across political, social, cultural, economic and scientific domains within the system [ 17 ]. All the NCD risk factors themselves have a multitude of interdependent elements that are complex, interconnected and have interacting influences [ 18 , 19 , 20 ]. Traditional, linear ‘blueprint approaches’ to scaling up that are observed in global health initiatives may not adequately fit the dynamic and unpredictable ways in which health services, organisations, and communities expand and are sustained [ 21 ].

Complexity and systems theory can make a valuable contribution to understanding and addressing population health scale up [ 13 ]. Systems approaches are also theorised to be influential in creating sustainable outcomes at scale [ 13 , 14 ]. However, in public health generally, there is limited evidence for the impacts of adopting a systems approach, with a paucity of detailed descriptions of their operationalisation. In health systems research, a meta-analysis of 35 studies identified a significant improvement to patient and service outcomes when a systems approach informed healthcare design and delivery [ 22 ]. However, in NCD behavioural risk factor research, systems approaches have had mixed adoption. For example, in tobacco cessation research, for almost 30 years systems approaches have been recognised and systems level strategies applied in health services [ 23 ], whereas for physical activity there are few well described examples of interventions that have been planned, delivered and evaluated using systems approaches and systems analysis methods [ 24 ].

For researchers, practitioners and policymakers wishing to achieve population level impact of interventions, and adopt a systems approach to intervention implementation at scale, greater guidance is needed on ways to achieve this and how to operationalise systems approaches in the context of public health scale up. Given that in public health generally, evidence to demonstrate the value of applying a systems approach is still emerging [ 25 ], and there is little research that has examined systems-based practice [ 26 ] or how systems approaches are applied in scaling up in public health [ 13 , 14 ]; we sought to contribute to addressing these gaps in the current review.

The objective of this paper is to synthesise the evidence of how systems approaches have been used to inform scaling up in physical activity. However, to explore scaling up of interventions targeting behavioural risk factors for NCDs more comprehensively, the scope was broadened to include the three other key NCD behavioural risk factors: tobacco use, alcohol consumption, and diet. As NCD prevention often operationalises ‘diet’ in terms of obesity, we consider diet and obesity jointly; herein ‘diet/obesity’. Given the current lack of published evidence describing implementation of systems approaches in public health [ 27 ], and the fact that interventions targeting physical activity are often designed as multicomponent interventions targeting multiple NCD risk factors combined (i.e., physical activity and diet), this strategy also provides an opportunity to learn from other areas of public health.

The specific aims of this review are as follows. Firstly, to identify how systems approaches have been used to inform and understand: (i) approaches and strategies to scaling up interventions targeting four behavioural risk factors for NCD; and (ii) barriers and facilitators to scaling up of these interventions. Secondly, how the term ‘systems’ has been conceptualised and used in the broader context of intervention implementation and scale up in these studies will be identified.

This review was prospectively registered with PROSPERO (registration number CRD42021287265) and follows the Preferred Reporting Items for Systemic Reviews and Meta-Analyses (PRISMA) guidelines [ 28 ]. The PRISMA checklist is presented in Additional file 1 .

Definition of a systems approach to scaling up

Systems-based public health is an evolving field, with no widely agreed definition of what it entails [ 22 , 29 ]. A ‘systems approach’, however, is generally understood as an approach that takes into account a multiplicity of interacting factors across a system, and the ways in which that system responds and adapts to interventions within it [ 17 ]. ‘Scale up’ refers to “ deliberate efforts to increase the impact of successfully tested interventions, to benefit a greater number of people and to foster policy and programme development” [ 30 ]. Whilst scale up processes commence once an intervention has been developed, in practice settings, scale up can occur before an intervention has been tested within a research trial [ 31 ]. ‘Dissemination’ is another term often used when referring to at-scale implementation, as it refers to an active approach to spreading an evidence-based intervention using planned strategies [ 32 ]. However, unlike ‘scale up’, dissemination need not include efforts to maximise the scale of interventions. More recently, a ‘systems approach to scale up’ has been defined as an approach that “ prioritises the behaviour and function of the system, with a focus on relations between a number of system elements, using system-level levers and dynamic system changes to drive impact at scale ” [ 14 ]. A systems approach to scale up emphasises that scale up activities (i.e., activities such as obtaining and maintaining the resources and implementation capacity needed for large-scale intervention delivery), should focus on generating changes within a system to achieve the desired population level outcome [ 14 ]. For example, the characteristics of the target system(s) that scaling occurs within are considered from the outset of scale up planning, in order to identify how best to reorientate that system to achieve the desired impacts on health [ 14 ]. Given that there is huge variance in definitions for terminology used in systems science more broadly [ 29 ], for the purposes of this review, we use the definition of a systems approach to scaling up from Koorts and Rutter (2021) [ 14 ], which informs the analytical framework for data synthesis and interpretation of our findings.

Eligibility criteria

Studies were eligible for inclusion based on the following criteria: 1) they involved interventions with a primary outcome targeting at least one of the four main behavioural risk factors for NCDs (physical inactivity, tobacco use, alcohol consumption and diet); and 2) interventions were conducted/planned for implementation in a real-world setting with a focus on scale up/scale up outcomes (effectiveness, scale-up, dissemination, translation or implementation studies [including randomised controlled trials with a focus on scale up/scale up outcomes], or protocols) or an evaluation of a previously scaled intervention (i.e., scaled up in a real-world setting). Eligible studies must have also included the term ‘system(s)’ and described either the approach/strategy taken during scale up or barriers and/or facilitators to the scale up process or outcomes. Exclusion criteria included: 1) studies testing intervention efficacy only or without a focus on scale up/scale up outcomes (e.g., randomised controlled trials, feasibility, and pilot studies); 2) reviews; and 3) studies applying or testing a policy (i.e., no intervention was implemented). For the purposes of this review, we defined an ‘intervention’ as “a set of actions with a coherent objective to bring about change or produce identifiable outcomes” [ 33 ], and excluded those described as a policy, strategy or government regulation.

Information sources and search strategy

The following online databases were searched online for peer reviewed English language articles published on or after January 1st, 2016, until 31st October 2021: EBSCOHost, Medline, CINHAIL, Sportdiscus, Global Health, PsychINFO and EMBASE. This search time frame (2016–21) was chosen in response to recent calls in public health for the use of systems approaches to address complex population level problems [ 20 ], and it would enable us to capture more recent interventions that were scaled post publication of key global action plans (e.g., [ 2 , 3 ] and [ 4 ]). Grey literature was searched via Google Advance and the first ten pages were screened for inclusion. The search strategy (Additional file 2 ) was developed and tested in consultation with a Deakin University research librarian, drawing on previous systematic reviews of related topics (e.g., [ 22 , 34 , 35 , 36 ]), and informed by the PICO(T) (participants/population; intervention; comparator; outcomes, time) methodological approach, as recommended by Cochrane reviews [ 37 , 38 ]. In this paper, Participants/populations were any age; the Intervention needed to target at least one of the four main behavioural risk factors for chronic disease (physical inactivity, tobacco use, alcohol consumption, diet); the Comparator was not required as this review focused on studies conducted in real-world settings; the Outcomes included the approaches and strategies to scaling up, barriers and facilitators experienced, and how ‘systems’ has been conceptualised and used; and the Time was 2016–2021.

Study selection

Search results were imported into data management software Covidence ( https://www.Covidence.org ), and duplicates from the search were automatically removed. One researcher (NR) screened article titles against inclusion and exclusion criteria. All abstracts and full texts were screened by two authors independently (CS, JM). Where discrepancies in study inclusions occurred, a consensus agreement was made by four authors (CS, JM, HK and KB). Where there was incomplete information to determine scale up approaches or scale up frameworks used, reference lists and forward searching was undertaken by JM and HK.

Quality appraisal

The Mixed Methods Appraisal Tool (MMAT) version 2018 [ 39 ] was used to appraise the quality of included studies as it encompasses multiple study designs (e.g., qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies) which was reflective of the included study designs. JM and KB conducted the quality appraisal independently using the MMAT screening questions and relevant checklist questions (by each of the five study design categories) in the Covidence software. Options for each question were yes, no or can’t tell. Any disagreements in quality appraisal were resolved by discussion between JM and KB and coming to a consensus decision. As per MMAT instructions, an overall score from each question was not calculated [ 39 ].

Data extraction

Data were extracted independently by two authors (CS, JM), with other authors (HK, KB) consulted for clarification where necessary. Data extraction included: author, year of publication, title, country or region of intervention scale-up, study design, aim, adaptions made (e.g. intervention, approach, setting), level of scale up (e.g., state/national level), intervention duration, target population, target behaviour (physical inactivity, tobacco use, alcohol consumption, diet), implementation setting, target intervention, name/number of organisations/stakeholders involved in scale up, role of organisations/stakeholders involved in scale up (e.g., funder, evaluator), how the term ‘system(s)’ has been conceptualised and used in the broader context of intervention implementation and scale up, framework/definition of scale up, method/approach to scale up, evaluation design for intervention effectiveness and evaluation of scale-up, data collected, and reported barriers and facilitators to scale up. Extracted data were tabulated (by JM and HK) to present study characteristics and results.

Data synthesis

Following guidance on narrative synthesis methodology [ 40 ], JM created a qualitative textual description for each included study, containing information on the scale up approach described, and how the term ‘systems’ and systems approaches were conceptualised, used or informed each study. Studies were required to have included the term ‘system(s) in order to be eligible for inclusion in the review and were not required to have described or implied any use of a systems approach to scaling up. Based on the textual descriptions and in accordance with the main aims of this review, two authors (HK and KB) independently created an initial set of categories ( n = 8 and n = 10, respectively) to capture how systems approaches had been used to inform and understand: (i) approaches and strategies to scaling up public health interventions; (ii) barriers and facilitators to scaling up; (iii) the evaluation of scale up processes and outcomes; and (iii) how the term ‘systems’ was conceptualised and used in each study. Initial categories were discussed and refined by HK, KB, CS and JM until consensus was reached, to produce a final set of six categories that comprised the final analytical framework for data synthesis (see Table 1 below).

Based on the definition of a systems approach to scaling up [ 14 ], the six categories within the analytical framework were assigned to one of three groups, according to their alignment with the definition: (Group 1; Analytical framework category 1 and 2) High use of systems approaches in scaling up (e.g., studies in which systems thinking informed the intervention design, implementation and scale up approach, the intervention had a focus on system changes); (Group 2; Analytical framework category 3 and 4) Moderate use of systems approaches in scaling up (e.g., studies in which the intervention had a focus on system changes or the role/influence of system factors, but did not explicitly adhere to the definition of a systems approach to scale up) and; (Group 3; Analytical framework category 5 and 6) No use of systems approaches in scaling up (e.g., studies in which the intervention may involve multiple strategies in multiple settings/sectors targeting different levels, but did not explicitly adhere to the definition of a systems approach to scale up). Table 1 presents the six categories within the analytical framework, against the three levels of a systems approach to scaling up.

Using the analytical framework (Table 1 ), a narrative synthesis was undertaken by JM with HK. Key findings and descriptions in each study were coded in line with the analytical framework. Themes were used to describe the approach to scale up and conceptualisation/use of the term ‘systems’ and a systems approach. Comparisons between themes were identified to present a synthesis of findings across studies, individual findings are only reported where themes were unique to an individual intervention or study. Qualitative textual descriptions relating to the barriers and facilitators to scaling up were summarised using an inductive thematic analysis approach. Descriptions were aggregated according to major themes, which informed the key barriers and facilitators reported in the Results.

The search generated 22 eligible papers, corresponding to 21 intervention studies (two papers [protocol and outcomes] addressed the same intervention). Figure 1 presents the PRISMA flowchart, which displays the number of studies screened, assessed, and included/excluded for the final review. Results reported correspond to data contained in the 22 eligible papers.

PRISMA flow diagram

Study sample characteristics

A description of the included papers ( n = 22) is presented in Table 2 . Of the 21 intervention studies we included in this review, 19 were discrete interventions [ 12 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ], one study involved evaluation of 16 discrete interventions [ 61 ], and one study involved evaluation of six recommended evidence-based strategies [ 62 ]. For the purposes of this review, studies that included multiple discrete programmes or strategies (e.g., [ 61 , 62 ]), are reported as one intervention. Twenty of the 21 interventions were already being implemented at scale, only one intervention was planned for scale up in a real-world setting, with their evaluation focussing on potential scalability for future roll out [ 59 ].

According to World Bank categorisations [ 64 ], the 21 interventions were implemented in five high income countries (United States of America [ 41 , 42 , 49 , 56 , 57 , 61 , 62 ], Australia [ 43 , 44 , 48 , 50 , 51 , 59 , 60 ], Canada [ 45 , 47 , 52 , 53 ], New Zealand [ 12 ] and the UK [ 58 ]), and one low income country (Ghana [ 46 , 63 ]).

Seven interventions targeted physical activity and diet [ 27 , 42 , 44 , 47 , 57 , 58 , 60 ], four targeted diet [ 41 , 45 , 51 , 59 ], four targeted physical activity [ 50 , 52 , 53 , 62 ], two targeted tobacco use [ 49 , 56 ], two targeted all four behaviour risk factors (physical inactivity, tobacco use, alcohol consumption and diet) [ 12 , 48 ], and one targeted diet and malnutrition [ 46 ]. One study targeted physical activity, tobacco use, diet, UV exposure and preventive care (e.g., cancer screening and human papillomavirus vaccination) [ 61 ], but the tobacco control and HPV vaccine elements of the intervention were reported as under way and were not included by the authors as part of their evaluation [ 61 ].

Interventions targeted health improvements or behaviour change among children/adolescents and families [ 41 , 42 , 43 , 44 , 46 , 47 , 50 , 51 , 53 , 55 , 60 , 61 , 63 ], older adults [ 52 ], smokers [ 49 , 56 ], men [ 58 ], and general community members, including community based partner organisations/practitioners (e.g., food banks and the YMCA) [ 12 , 45 , 47 , 48 , 49 , 57 , 59 , 61 , 62 ]. Interventions targeted multiple community-based settings, such as schools [ 41 , 43 , 44 , 46 , 50 , 51 , 53 , 61 , 63 ], childcare settings [ 44 , 47 , 61 ], faith-based organisations [ 41 , 57 ], local health districts and local governments [ 44 , 48 ], workplaces [ 43 ], professional football clubs [ 58 ], supermarkets [ 59 ], coordinated care organisations [ 49 ], cancer centres [ 56 ], and in the community at a broader environmental/policy level [ 12 , 42 , 45 , 48 , 49 , 52 , 60 , 61 , 62 ].

Of the 22 papers reviewed, there was one qualitative study [ 45 ], one type 2 hybrid effectiveness implementation trial [ 52 ], one group randomised trial [ 57 ], two quasi-experimental designs [ 43 , 51 ], three case studies [ 12 , 44 , 48 ], four cluster randomised controlled trials [ 46 , 50 , 53 , 63 ], five mixed method evaluations [ 41 , 47 , 59 , 61 , 62 ], and five summary/descriptive articles describing intervention and scale up processes/outcomes generally [ 42 , 49 , 56 , 58 , 60 ].

Approaches and strategies to scaling up

There were varying approaches and strategies used to reach at-scale implementation (Table 2 ). Fifteen (71%) of the 21 interventions were described as ‘designed for scale’. These interventions were developed based on prior evidence from other programmes, and scaling required strong research-practice partnerships and existing resources customised for stakeholders. Of the 15 interventions designed for scale, 10 (67%) were implemented without the reported need for small scale pilot trials, and were scaled across a single community [ 62 ], multiple communities within one state [ 48 , 60 , 61 ], across a whole state/province [ 41 , 44 , 47 ], across multiple states [ 42 ], or nationally [ 12 , 45 ]. Five (33%) of the 15 interventions followed a traditional translation pathway of a phased approach from small-scale controlled efficacy testing to plan for scale up [ 59 ], and from small-scale controlled efficacy/feasibility testing to then effectiveness and implementation testing at a state [ 50 , 57 ] and province level [ 52 , 53 ].

Four interventions were expanded due to earlier effectiveness outcomes, and ongoing government funding and support. Two of these three interventions started as smaller community projects that were incrementally expanded to reach multiple communities or settings state-wide [ 43 , 51 ], and two began as pilot trials that were incrementally expanded to reach nationally [ 46 , 58 ]. Two interventions targeting tobacco reduction were scaled as part of existing health system infrastructure. One was the extension of more than two decades of established state level policies and evidence-based initiatives [ 49 ], and the other was integrated as part of an existing large, comprehensive health system [ 56 ].

Barriers and facilitators to scaling up

Forty-one textual descriptions relating to barriers and facilitators to scaling up were summarised from nineteen included studies (Table 2 ). Six themes emerged, which included four facilitators (committed stakeholder engagement, capacity building in the local workforce, flexibility in delivery and implementation, and ongoing programme feedback and improvement) and two barriers (competing interests and priorities, and insufficient time and resources).

Facilitator 1: Committed stakeholder engagement

Full engagement of stakeholders was key for scale-up success, as reported in several studies [ 12 , 41 , 42 , 43 , 45 , 47 , 48 , 49 , 53 , 56 , 57 , 58 , 59 , 60 , 62 ]. Early involvement and participation of stakeholders in the planning and implementation of initiatives were also identified as important [ 12 , 57 , 58 ]. End users and leadership were the most mentioned stakeholders/aspects that played a key role in the scale up process. A weight management programme targeting men with overweight or obesity leveraged the popularity of professional football among target users, and existing organisational structure of local football clubs, as delivery systems to provide a sustainable model for wider implementation of the programme [ 58 ]. Several studies reported that the initiatives, and implementation process, were driven by the local contexts and needs and resulted in scale-up successes [ 12 , 41 , 43 , 48 , 62 ]. Strategically engaging leadership and facilitating alignment in goals and missions across different organisations and communities was associated with a greater degree of collective actions [ 12 , 45 , 56 , 59 , 60 ].

Facilitator 2: Capacity building in local workforce

Efforts in capacity building [ 12 , 41 , 53 , 57 , 58 ] were common in the included scale-up initiatives. Capacity building for the implementation workforce was necessary, especially in systems thinking guided initiatives. This could include providing adequate resources and funding to support existing infrastructures. Understanding and responding to the adaptive nature of systems were recognised as a key skillset for local workforce [ 12 , 48 ].

Facilitator 3: Flexibility in delivery and implementation

Initiatives that had a multi-component or flexible delivery model were more acceptable to local implementation as they allow for autonomy [ 12 , 53 ]. The flexibility also enabled initiatives to be appropriately adapted to fit local context and place characteristics [ 58 , 62 ]. By contrast, rigid and controlled approaches in the initiatives appeared to hinder implementation [ 45 , 48 ].

Facilitator 4: Ongoing programme feedback and improvement

Establishing and utilising an ongoing evaluation and feedback system was perceived instrumental in scale up [ 57 , 59 , 60 ]. In a whole-of-community child obesity prevention programme, ongoing monitoring and feedback of programme achievements to community organisations informed quality improvement and innovation, which resulted in a significant increase in the promotion of community settings implementing the evidence-based practice [ 60 ].

Barrier 1: Competing interests and priorities

Implementation of initiatives was challenged by conservative/risk averse attitudes of local organisations [ 45 ], competing priorities of local organisations [ 43 ], and competing interests of industries [ 12 , 59 ]. For example, organisations were reluctant to adopt the initiative due to a low perception of risk versus benefits, i.e., potential low relative advantage [ 45 ]. Another healthy supermarket intervention described the challenges in promoting sales of healthy foods due to perceived customer demand and supplier contract agreements [ 59 ].

Barrier 2: Insufficient time and resources

Lack of appropriate resources such as funding was identified as a barrier to successful scale-up [ 12 , 45 , 53 , 55 , 59 ]. For one intervention that targeted different points of the system [ 12 ], siloed and competitive funding approaches in the government impacted the financial security of the initiative.

Use of systems approaches to scaling up

Tables 3 , 4 and 5 summarise how studies categorised in the high, moderate and no use of a systems approach groups, respectively, articulated how ‘systems’ was conceptualised or used, in what ways a systems approach was adopted, and the methods/theoretical frameworks or principles applied to study scale up processes or outcomes.

High use of a systems approach to scaling up

Five studies were categorised as demonstrating high use of a systems approach to scaling up [ 12 , 44 , 48 , 51 , 56 ] (Table 3 ). Interventions in this group targeted physical activity and diet ( n = 2), diet ( n = 1), tobacco use ( n = 1) and all four behavioural risk factors ( n = 1). Table 6 provides a case example of a high use systems approach. In these studies, systems thinking informed the intervention design or implementation and scale up approach, or the intervention had a focus on system changes. In these studies, there was an explicit focus on relations between system elements and using system changes to drive impact at scale. Four of the five studies explicitly used systems thinking to inform the intervention design, implementation, and scale up processes, with the goal of systems change (Table 3 ).

Conceptualisation and use of ‘systems’ and a systems approach

All five studies in the high use group involved engagement and collaboration with communities and stakeholders in the process and evaluation of scaling up. Four studies explicitly used systems thinking to inform the intervention design, implementation, and scale up processes, with the goal of systems change [ 12 , 48 , 51 , 56 ]. Systems thinking was reflected in this process by exploring connections, priorities, and common interests between community groups and organisations [ 48 , 56 ], identifying systems elements and existing infrastructure that supports scale-up [ 51 ], conceptualising systems change in the prevention paradigm and infrastructure [ 12 ], and quantifying the connection between systems players and how it contributes to the success of scale-up [ 44 ].

Methods, theoretical frameworks and/or principles adopted to study scale up processes or outcomes

Three studies in the high use category employed theoretical frameworks to guide the systems thinking practice. One obesity prevention study applied the Analysis Grid for Environments Linked to Obesity (ANGELO) framework [ 65 ] to guide the development and implementation of scale-up, defining the key priorities while considering existing capacity within a system [ 51 ]. Another study adapted the Cancer Care Continuum as a systems framework to identify the necessary resources and players needed for tobacco treatment integration in each of the cancer care stages (e.g., prevention, diagnosis) [ 56 ]. This same study used the Consolidated Framework for Implementation Research (CFIR) [ 66 ] to describe and evaluate implementation. The WHO’s Prevention System Building Blocks [ 67 ] was applied to one study that targeted all four behavioural risk factors, to identify facilitators and areas for improvement in scale-up of obesity prevention in the domain of workforce, leadership, relationships and networks, resources, and knowledge and data [ 12 ]. Three studies in the high use category defined the measures or system level changes resulting from the scale up process [ 12 , 44 , 48 ]. For example, one study used a quantitative systems analysis approach (i.e., social network analysis) to evaluate the diffusion of knowledge of the intervention among stakeholder groups [ 44 ], whereas another developed ad-hoc qualitative indicators to capture the presence and/or absence of system-level changes from stakeholders’ perspectives [ 12 ].

Moderate use of a systems approach to scaling up

Seven studies were categorised as moderate use of a systems approach to scaling up [ 41 , 42 , 43 , 49 , 52 , 53 , 57 ] (Table 4 ). Interventions in this group targeted physical activity and diet ( n = 3), physical activity ( n = 2), diet ( n = 1), and tobacco use ( n = 1). Table 7 provides a case example of a moderate use systems approach. All studies in the group, as described in the papers, recognised the importance of systems thinking in resolving scale-up challenges, however, systems thinking was not embedded from the outset to identify what would be required to achieve system-level changes. System analysis methods were not necessarily used to study or address barriers and facilitators to the scale up process or outcomes. In these studies, there was an implicit focus on systems change, as reflected in the intervention design and evaluation that targeted multiple sectors and components.

For the moderate use group, where systems were involved during scale up, this included via the intervention: (i) targeting different system levels (i.e., targeting multilevel organisations) (e.g., [ 41 , 43 ]), or points of the system (i.e., improving system barriers to access and to affordability) (e.g., [ 49 ]); (ii) involving organisations that had an influence at different system levels (e.g., schools) (e.g., [ 53 ]); or (iii) including intervention strategies that had a specific focus on system changes at an organisational (i.e., changing faith based organisations’ engagement in health promotion) (e.g., [ 57 ]) or policy/environmental level (i.e., empowering communities) (e.g., [ 42 ]). For example, two studies recognised the wide range of determinants on diet and/or physical activity and designed strategies to promote an environment that supports health behaviours (e.g., improve built environment for more opportunities in physical activity) [ 42 , 43 ]. Further, one study recognised the interconnectedness of multiple sectors in smoking cessation and planned for accountability and incentives to support the continued partnership to improve access to cessation services [ 49 ].

Of the seven included studies, two studies utilised conceptual models of scaling up (i.e., [ 30 , 68 ]) to guide implementation of interventions targeting physical activity in older adults [ 52 ], and in schools [ 53 ]. Four of the seven studies described or defined the system level changes resulting from the scale up process [ 41 , 42 , 43 , 52 ], with a focus on community capacity as an indicator of system level changes. All studies utilised a series of strategies (e.g., educational activities) across multiple sectors (e.g., community, education) to strengthen the community capacity that supports healthy eating and/or physical activity. None of the studies, as reported in the papers, mentioned the measure on the change in community capacity.

No use of a systems approach to scaling up

Nine studies were categorised as not using a systems approach to scaling up [ 45 , 46 , 47 , 50 , 58 , 59 , 60 , 61 , 62 ] (Table 5 ). Interventions in this group targeted physical activity and diet ( n = 3), diet ( n = 2), physical activity ( n = 2), diet and malnutrition ( n = 1), and physical activity, tobacco use, diet, UV exposure, and inadequate preventive care ( n = 1). Table 8 provides a case example. These studies adopted an approach to scaling up that included multiple sectors, settings and intervention components, although, as reported in the papers, the complexity of the system and the relations between the system elements were not explicitly targeted or articulated.

Where systems were involved during scale up in the ‘no use’ group, this included via the intervention: (i) targeting different system levels (i.e., targeting multilevel organisations, e.g., [ 62 ]) or settings within the system (e.g., [ 60 ]); (ii) involving organisations that had an influence at different system levels (e.g., representing food retailers, Non-farmers, advertisers) (e.g., [ 45 , 46 , 59 ]); or (iii) including intervention strategies that had a specific focus on system changes at an organisational level (i.e., changing childcare provider capacity to engage in health promotion or increasing football club coaches capacity to deliver health promotion within their existing organisations) (e.g., [ 47 , 58 ]). All studies in this group involved multiple strategies in multiple sectors and may have involved targeting different systems; however, the strategies and sectors were not considered in a relational way or tended to focus on one of few points of within the system. For example, in one study, only school meals service and distribution were focussed on, even though the intervention was designed as a multi-sectoral strategy to increase food production, household income, and food security [ 46 ]. Several multicomponent interventions did not describe or explicitly target systems change (e.g., [ 50 ]), but acknowledged the role and influence of system factors (e.g., [ 47 ]). Whilst some studies acknowledged that there must be sufficient organisational or system support for effective scale up, where the term ‘system(s)’ was mentioned, this was referred in context of: ‘evaluation systems’ (e.g., [ 62 ]), ‘monitoring systems’ (e.g., [ 46 , 60 ]), implementation ‘delivery system’ (e.g., [ 50 , 58 ]), or in the broader general context of the social/childcare/health system that the intervention took place (e.g., [ 45 , 59 ]). Where ‘systems’ were not explicitly mentioned, factors relevant to systems change could still be utilised (i.e., establishing community infrastructure and developing community action plans to support capacity at scale) (e.g., [ 61 ]) (Table 5 ).

None of the nine studies included in the ‘no use’ group described using a scale up framework. One study retrospectively applied the PRACTIS guide [ 69 ], which is a framework that incorporates a systems thinking perspective on effective implementation and scale up, to describe the scale up of an intervention targeting professional football clubs [ 58 ]. However, the authors acknowledge that the PRACTIS guide was not available to guide intervention scale up from the initial stages. Concepts within the social system, relevant to Rogers’ Diffusion of Innovations [ 70 ], informed the evaluation of two studies, to establish scalability of an intervention [ 59 ] and actual scale up outcomes [ 45 ]. Five of the nine studies described system level changes resulting from scale up, mainly on outcomes as a result of multiple strategies implemented in interventions [ 47 , 50 , 61 , 62 ] or as a result of strong policy alignment [ 60 ].

Quality appraisal findings

Quality assessment findings are presented in Additional file 3 . In general, there were a lack of details in the reporting of the sampling strategy and whether the study sample is representative of the target population, and the risk of nonresponse bias for quantitative survey was rarely considered. For mixed methods studies, it was often unclear how divergences and inconsistencies between quantitative and qualitative results were addressed.

Systems approaches and complexity science have potential for enhancing both the development and the scaling up of health interventions [ 20 , 21 , 71 ], but there is a paucity of knowledge of whether and how systems approaches have been adopted when scaling up in public health [ 14 ]. In this paper, we systematically explore the use of systems approaches to scaling up prevention of four behavioural risk factors for NCDs; physical inactivity, tobacco use, alcohol consumption and unhealthy diet. Of the studies included in this review, interventions targeted a mix of behavioural risk factors across different ages and settings, however, almost all studies were conducted in high income countries. Despite the fact that the majority of interventions in this review were described as having been designed for scale, only four (19%) of the 21 interventions explicitly used systems thinking to inform the intervention design, implementation, and scale up processes, with the goal of systems change. Whilst studies often included multiple sectors and intervention components; recognition of the complexity of the system and the relations between the system elements were not explicitly targeted or articulated.

Systems approaches are not imperative for scale up, and there is a lack of robust evidence that a systems approach leads to better outcomes for sustainability and impact at scale. However, this review illustrates that current conceptualisations of what constitutes designing an intervention for scale does not necessarily include a consideration of the impact of systems or principles of systems thinking. This is despite the fact that successful scale up includes when an evidence-based intervention becomes embedded in a system(s) to achieve long-term, sustainable health impact [ 10 ]. We also identified that interventions described as designed for scale were reliant on strong research-practice partnerships. Facilitators to scale up included committed stakeholder engagement, capacity building in the local workforce, and flexibility in delivery and implementation. Conversely, barriers included competing interests and priorities, and insufficient time and resources. These barriers and facilitators are consistent with previous scale up research [ 72 , 73 ]. Studies in this review also reported challenges relating to measuring system changes, including the time required to detect long-term changes [ 51 ], and challenges with appropriate measures for population reach and engagement [ 48 ], which is consistent with research that explored some of the tensions of scaling up in physical activity [ 13 ].

More than two thirds of interventions included were described as designed for scale, whereas less than one third of interventions followed a traditional translation pathway of small-scale controlled efficacy testing, to effectiveness and implementation at scale. This finding suggests that there may have been a move away from the traditional translation pipeline that begins with controlled research trials, to one that considers effective real-world translation, as has been recommended for more than two decades [ 74 ]. It also suggests that there is greater consideration of translation and population impact early in the research process, which has also been recommended for over a decade [ 75 ]. Nonetheless, despite recommendations for the use of complex systems models in public health [ 20 ], recognition of the impact of system factors varied greatly across studies. Of those studies that did adopt a systems approach to scale up (the ‘high use’ group), most of the interventions targeted all four behavioural risk factors for NCDs we included in this review. In this high use group, there was an explicit focus on relations between system elements and using system changes to drive impact at scale. Of the studies that did not adopt a systems approach (the ‘no use’ group), most of the interventions targeted physical activity and diet.

The varied appreciation, adoption and implementation of systems approaches for the four NCD behavioural risk factors we included in this review may reflect the overall lack of scale up approaches that adopted systems perspectives. For example, in physical activity promotion research, systems approaches have largely been underutilised and systems concepts need to be engaged more robustly in physical activity interventions [ 24 ]. In physical activity scale up research in particular, systems approaches are perceived as important, but may not be seen as feasible to achieve in practice [ 10 ]. In obesity prevention research, which can include interventions targeting physical activity and diet, there is a lack of evidence for whole of systems approaches [ 34 ]. Complex systems perspectives have been applied to study alcohol reduction and associated harms, however their application has remained predominantly at the individual or local level [ 19 ]. Consistent with public health promotion more broadly [ 20 ], alcohol reduction interventions have often been reductionist with a focus on easily modifiable risk factors and high risk groups [ 76 ], despite advocacy for a focus on the real-world systems that alcohol consumption and harms are created and shaped by [ 19 ]. For smoking cessation, the benefits of approaches that systemically integrate into or incorporate clinical settings have long been recognised, however, their adoption has been inconsistent and implementation slow [ 77 ]. For several decades, tobacco control has become increasingly complex and thus the use of systems thinking has long been encouraged [ 78 ]. However, the effectiveness of system change interventions on tobacco cessation rates and system level outcomes at scale, remains less clear [ 79 ].

Whilst our review showed that socio-ecological models have been an integral part of designing, implementing, and evaluating physical activity interventions, the interconnectedness among these elements in scale up was rarely considered. Implementation of a suite of activities across multiple settings or levels need not mean a systems approach has been adopted [ 80 ]. Our findings support this, as studies in both the high and no use systems approach groups included interventions targeting multiple settings and levels. Yet, only studies in the high use group (e.g., [ 48 ]) explicitly applied systems thinking in practice and embedded systems approaches from the outset.

Consistent with the definition of a systems approach to scale up that we used in this paper [ 14 ], scale up exists on a continuum and scale up need not adopt a systems approach. As is shown in this review, scale up approaches can include linear, intervention-orientated expansive approaches that prioritise the spread of interventions into existing systems, through to approaches that sit within a complex systems paradigm that begin by considering the characteristics of the target system(s) that scaling occurs within [ 14 ]. Scaling health interventions in a traditional, linear way through efficacy to effectiveness and scale up trials is well documented (e.g., [ 73 ]). Our review draws attention to the fact that there is a lack of published evidence for ways to operationalise systems approaches when scaling interventions, including how to select and apply relevant theories and frameworks. Of the 21 interventions we included, only three employed a theoretical framework to guide systems thinking practice. Of the studies that did, to some extent, adopt a systems approach to scale up (i.e., the high and moderate use groups), system-level change and the scale up processes and outcomes that drive the change at scale were not well defined, and often the reporting was brief and lacked theoretical explanation. When a systems approach was applied and concisely reported, it provided a more comprehensive view of a problem and can help identify potential barriers and opportunities for scaling up interventions (e.g., [ 56 ]), however, our quality appraisal highlighted that most studies lacked information on key parameters.

To progress the field and better equip researchers, practitioners and policymakers to invest in efforts for scaling up, it has been recommended that NCD prevention adopts new paradigms and perspectives that incorporate systems thinking [ 18 , 20 , 81 ]. This includes using frameworks that incorporate a systems thinking perspective on how to achieve effective outcomes at scale [ 13 ] and adopt new ways of accounting for the complex systems in which interventions are implemented [ 82 ]. Complexity and systems theory can be used to understand and approach population health scale-up by considering the system as a whole prior to intervention design, development and implementation. For example, Scaling Readiness assessments involve both quantitative and qualitative data collection, and can be used to ascertain the characteristics of the system that influence prospective interventions prior to investment [ 83 ]. The Intervention Scalability Assessment Tool (ISAT) [ 84 ] has also been used to assess prospective scalability of interventions into existing systems, through a participatory process with stakeholders. For frameworks that guide scale up planning, the PRACTIS guide [ 69 ] adopts a systems thinking perspective to scaling up that can inform both scale up planning [ 85 ] and evaluation [ 58 ]. Specifically, PRACTIS workshops (i.e., [ 86 ]) involve a participatory process and co-design process with stakeholders, which allows for the systematic identification and documentation of data that are influential for intervention uptake, political support and community sustainability across multiple levels of the system. For data collection that can account for the complex systems in which interventions are implemented, resources such as the Consolidated Framework for Implementation Research (CFIR) [ 66 ] outline key factors influencing implementation of interventions, including ways of collecting data to account for these multilevel factors at scale. A key component of the Designing for Dissemination and Sustainability (D4DS) logic model [ 85 ], is designing a research product with the end in mind. This includes, understanding the characteristics of systems by using methods (e.g., system dynamics modelling [ 87 ]) to anticipate and plan for adaptation of interventions in response to changes in context over time [ 85 ]. Use of these frameworks and tools is recommended for improving operationalisation of systems approaches in the context of public health scale up.

Building the evidence base in systems approaches to scaling up, for physical activity and other NCD behavioural risk factors, has the potential to improve how we communicate and operationalise systems approaches when scaling for widespread and sustainable impact, in both research and in practice.

Strengths and limitations

A key strength of this review is that we utilised a narrative synthesis approach which enabled us to account for different scale up approaches and the varied terminology used to describe systems and systems approaches. Due the complicated and understudied nature of the topic we address in this paper, and thus the extended time required to extract, analyse, and reach consensus on the use of a systems approach to scaling up; we acknowledge the gap between data extraction and publishing. However, by combining a narrative synthesis approach with systematic screening, independent data categorisation, and quality assessment of methodological rigour, the robustness of our review is enhanced. The data were extracted by two independent researchers, and the analytical framework was developed, and data synthesised by three independent researchers. Given the lack of a universally agreed-upon definition of a 'systems approach' and the broad use of systems language, our methodology enabled us to capture implicit and explicit systems methods and approaches. Our analytical framework also allowed us to identify whether scaling up adhered to a systems approach, even if the term 'systems approach' was not explicitly used in the study.

Our systematic search process meant that we were able to identify relevant articles, however, as our data synthesis was reliant on information contained within the published papers, it is unknown to what extent systems or systems approaches were considered in the scale up process by decision-makers but were simply not reported in the published articles. We undertook reference list searches for studies where information was incomplete or unclear, however, this was only conducted for the scale up strategy and framework used, and not for all aspects of scale up. There is thus the potential that additional information related to scale up may have been published elsewhere. In addition, we also only report on the country of scale up directly relevant to the papers included, whereas interventions may have been expanded to other countries elsewhere. For example, for one intervention we acknowledge that adaptations had occurred for implementation in other countries (e.g., [ 58 ]), which we do not report on in our results. In this review, we focused on the four major behavioural risk factors for NCDs [ 2 ]. We acknowledge that the WHO physical activity guidelines also include recommendations related to sedentary behaviour [ 88 ]. Sedentary behaviour is considered a distinct health risk factor to physical inactivity [ 89 , 90 ], and thus future reviews may wish to broaden the scope of included studies to align with global guidelines. To our knowledge, this review is the first to categorise the use of systems approaches in scale up, however, future research is needed to quantify different levels. Finally, non-English publications were excluded from this review.

Systems approaches allow for consideration of complexity at scale, facilitating different ways of planning for and interpreting challenges that are associated with scale up. Systems approaches also align with some of the key facilitators to successful scale up. By acknowledging the interconnectedness among various components of a system and ensuring their efficient and effective collaboration; systems approaches can potentially lead to more successful and impactful scale up outcomes. This review showed that the use of systems approaches when scaling up interventions targeting key behavioural risk factors for NCDs (physical inactivity, tobacco use, alcohol consumption and diet) is still in its infancy and there is a need for high quality studies. In particular for population level physical activity promotion, this presents a huge gap in knowledge. For decision-makers wishing to adopt or support a systems approach to intervention implementation at scale, greater guidance is needed on what is required to achieve this, and how to communicate and operationalise systems approaches in both research and practice.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Non-communicable disease

World Health Organization

Global Burden of Disease Collaborative Network, Global Burden of Disease Study 2019 (GBD 2019) Results (2020, Institute for Health Metrics and Evaluation – IHME) https://vizhub.healthdata.org/gbd-results/ . 2019.

World Health Organization. Global action plan for the prevention and control of noncommunicable diseases 2013–2030. Geneva: World Health Organization; 2013.

Google Scholar

World Health Organization. Global Action Plan on Physical Activity 2018–2030: more active people for a healthier world. Geneva: World Health Organization; 2018.

World Health Organization. Global Alcohol Action Plan 2022–2030. Geneva: Switzerland; 2022.

World Health Organization. Global status report on physical activity 2022. Geneva: World Health Organization; 2022.

https://www.who.int/news-room/fact-sheets/detail/alcohol . Cited 2023 31st March.

https://www.who.int/news-room/facts-in-pictures/detail/6-facts-on-obesity . Cited 2023 31st March.

Santos AC, et al. The cost of inaction on physical inactivity to public health-care systems: a population-attributable fraction analysis. Lancet Glob Health. 2023;11(1):e32–9.

Article CAS PubMed Google Scholar

on, W.H.O.I.H.-l.C. and NCDs. Think piece: why is 2018 a strategically important year for NCDs? Geneva: World Health Organization; 2018.

Reis RS, et al. Scaling up physical activity interventions worldwide: stepping up to larger and smarter approaches to get people moving. Lancet. 2016;388:1337.

Article PubMed PubMed Central Google Scholar

Murphy J, et al. Advocating for implementation of the global action plan on physical activity: challenges and support requirements. J Phys Act Health. 2023;20(1):10–9.

Article PubMed Google Scholar

Matheson A, et al. Strengthening prevention in communities through systems change: lessons from the evaluation of Healthy Families NZ. Health Promot Int. 2020;35(5):947–57.

Koorts H, et al. Tensions and paradoxes of scaling up: a critical reflection on physical activity promotion. Int J Environ Res Public Health. 2022;19(21):14284.

Koorts H, Rutter H. A systems approach to scale-up for population health improvement. Health Res Pol Syst. 2021;19(1):27.

Article Google Scholar

Lane C, et al. How effective are physical activity interventions when they are scaled-up: a systematic review. Int J Behav Nutr Phys Act. 2021;18(1):16.

Koorts H, et al. Mechanisms of scaling up: combining a realist perspective and systems analysis to understand successfully scaled interventions. Int J Behav Nutr Phys Act. 2021;18(1):42.

Rutter H, et al. Systems approaches to global and national physical activity plans. Bull World Health Organ. 2019;97(2):162–5.

Allender S, et al. Translating systems thinking into practice for community action on childhood obesity. Obes Rev. 2019;20(S2):179–84.

McGill E, et al. Applying a complex systems perspective to alcohol consumption and the prevention of alcohol-related harms in the 21st century: a scoping review. Addiction. 2021;116(9):2260–88.

Rutter H, et al. The need for a complex systems model of evidence for public health. Lancet. 2017;390(10112):2602–4.

Paina L, Peters DH. Understanding pathways for scaling up health services through the lens of complex adaptive systems. Health Policy Plan. 2012;27(5):365–73.

Komashie A, et al. Systems approach to health service design, delivery and improvement: a systematic review and meta-analysis. BMJ Open. 2021;11(1): e037667.

Fiore MC, Keller PA, Curry SJ. Health system changes to facilitate the delivery of tobacco-dependence treatment. Am J Prev Med. 2007;33(Supplement 6):S349–56.

Nau T, et al. A scoping review of systems approaches for increasing physical activity in populations. Health Res Pol Syst. 2022;20(1):104.

Jebb SA, et al. Systems-based approaches in public health: Where next?. Canadian Academy of Health Sciences & The Academy of Medical Sciences; 2021.

Carey G, et al. Systems science and systems thinking for public health: a systematic review of the field. BMJ Open. 2015;5:e009002.

Bolton KA, et al. Generating change through collective impact and systems science for childhood obesity prevention: The GenR8 change case study. Plos One. 2022;17(5):e0266654.

Article CAS PubMed PubMed Central Google Scholar

Page MJ, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372: n71.

Peters DH. The application of systems thinking in health: why use systems thinking? Health Res Pol Syst. 2014;12(1):51.

Simmons R, Shiffman J. Scaling up health service innovations: a framework for action, in scaling up health service delivery. Geneva: World Health Organization; 2007. p. 1–30 F.P. Simmons R, Ghiron L,, Editor.

Indig D, et al. Pathways for scaling up public health interventions. BMC Public Health. 2017;18(1):68.

Rabin BA, et al. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14(2):117–23.

Rychetnik L, et al. Criteria for evaluating evidence on public health interventions. J Epidemiol Community Health. 2002;56:119–27.

Bagnall AM, et al. Whole systems approaches to obesity and other complex public health challenges: a systematic review. BMC Public Health. 2019;19(1):8.

Palmer M, et al. The effectiveness of smoking cessation, physical activity/diet and alcohol reduction interventions delivered by mobile phones for the prevention of non-communicable diseases: a systematic review of randomised controlled trials. Plos One. 2018;13(1): e0189801.

Rehm J, et al. Alcohol use and dementia: a systematic scoping review. Alzheimer’s Res Ther. 2019;11(1):1.

McKenzie JE, et al. Defining the criteria for including studies and how they will be grouped for the synthesis. In: Higgins JPT, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions. England: Cochrane; 2022.

Riva JJ, et al. What is your research question? An introduction to the PICOT format for clinicians. J Can Chiropr Assoc. 2012;56(3):167–71.

PubMed PubMed Central Google Scholar

Hong QN, et al. The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Educ Inf. 2018;34:285–91.

Popay J, et al. Guidance on the conduct of narrative synthesis in systematic reviews: A product from the ESRC Methods Programme. 2006. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=ed8b23836338f6fdea0cc55e161b0fc5805f9e27 .

Berman M, et al. Evaluation of the healthy lifestyles initiative for improving community capacity for childhood obesity prevention. Prev Chronic Dis. 2018;15:E24.

Betancourt K, et al. Empowering one community at a time for policy, system and environmental changes to impact obesity. Ethn Dis. 2017;27(Suppl 1):347–54.

Bolton KA, et al. The outcomes of health-promoting communities: being active eating well initiative-a community-based obesity prevention intervention in Victoria Australia. Int J Obes (Lond). 2017;41(7):1080–90.

Conte KP, et al. Dynamics behind the scale up of evidence-based obesity prevention: protocol for a multi-site case study of an electronic implementation monitoring system in health promotion practice. Implement Sci. 2017;12(1):146.

Fernandez MA, et al. Factors influencing the adoption of a healthy eating campaign by federal cross-sector partners: a qualitative study. BMC Public Health. 2016;16(1):904.

Gelli A, et al. A school meals program implemented at scale in Ghana increases height-for-age during midchildhood in girls and in children from poor households: a cluster randomized trial. J Nutr. 2019;149(8):1434–42.

Hassani K, et al. Implementing appetite to play at scale in British Columbia: evaluation of a capacity-building intervention to promote physical activity in the early years. Int J Environ Res Public Health. 2020;17(4):1132.

Joyce A, et al. The “Practice Entrepreneur” - An Australian case study of a systems thinking inspired health promotion initiative. Health Promot Int. 2018;33(4):589–99.

CAS PubMed Google Scholar

Livingston CJ, et al. Reducing tobacco use in Oregon through multisector collaboration: aligning medicaid and public health programs. Prev Chronic Dis. 2020;17:E155.

Lonsdale C, et al. Scaling-up an efficacious school-based physical activity intervention: study protocol for the “Internet-based Professional Learning to help teachers support Activity in Youth” (iPLAY) cluster randomized controlled trial and scale-up implementation evaluation. BMC Public Health. 2016;16(1):873.

Malakellis M, et al. School-based systems change for obesity prevention in adolescents: outcomes of the Australian Capital territory “It’s Your Move!” Aust N Z J Public Health. 2017;41(5):490–6.

McKay HA, et al. Status quo or drop-off: do older adults maintain benefits from choose to move-a scaled-up physical activity program-12 months after withdrawing the intervention? J Phys Act Health. 2021;18(10):1236–44.

Nettlefold L, et al. Scaling up action schools! BC: how does voltage drop at scale affect student level outcomes? A cluster randomized controlled trial. Int J Environ Res Public Health. 2021;18(10):5182.

Sacher PM, et al. Addressing childhood obesity in low-income, ethnically diverse families: outcomes and peer effects of MEND 7–13 when delivered at scale in US communities. Int J Obes. 2019;43(1):91–102.

Article CAS Google Scholar

Sutherland R, et al. Scale-up of the Physical Activity 4 Everyone (PA4E1) intervention in secondary schools: 24-month implementation and cost outcomes from a cluster randomised controlled trial. Int J Behav Nutr Phys Act. 2020;18(1):137.

Tong EK, et al. The emergence of a sustainable tobacco treatment program across the cancer care continuum: a systems approach for implementation at the university of California Davis comprehensive cancer center. Int J Environ Res Public Health. 2020;17(9):3241.

Wilcox S, et al. Faith, activity, and nutrition randomized dissemination and implementation study: countywide adoption, reach, and effectiveness. Am J Prev Med. 2018;54(6):776–85.

Hunt K, et al. Scale-up and scale-out of a gender-sensitized weight management and healthy living program delivered to overweight men via professional sports clubs: the Wider Implementation of Football Fans in Training (FFIT). Int J Environ Res Public Health. 2020;17(2):584.

Blake MR, et al. The ‘Eat Well @ IGA’ healthy supermarket randomised controlled trial: process evaluation. Int J Behav Nutr Phys Act. 2021;18(1):36.

Wolfenden L, et al. From demonstration project to changes in health systems for child obesity prevention: the legacy of “Good for Kids, Good for Life.” Aust N Z J Public Health. 2020;44(1):3–4.

Rechis R, et al. Be Well Communities™: mobilizing communities to promote wellness and stop cancer before it starts. Cancer Causes Control. 2021;32(8):859–70.

Davis SM, Cruz TH, Kozoll RL. Research to practice: implementing physical activity recommendations. Am J Prev Med. 2017;52(3 Suppl 3):S300–s303.

Gelli A, et al. Evaluation of alternative school feeding models on nutrition, education, agriculture and other social outcomes in Ghana: rationale, randomised design and baseline data. Trials. 2016;17:37.

The World Bank Group, High income data. The World Bank Group; 2019.

Swinburn B, Egger G, Raza F. Dissecting obesogenic environments: the development and application of a framework for identifying and prioritizing environmental interventions for obesity. Prev Med. 1999;29:563–70.

Damschroder LJ, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):1–15.

World Health Organization. Everybody’s business - strengthening health systems to improve health outcomes: WHO’s framework for action. Geneva: World Health Organization; 2007.

Yamey G. Scaling up global health interventions: a proposed framework for success. Plos Med. 2011;8(6):e1001049.

Koorts H, et al. Implementation and scale up of population physical activity interventions for clinical and community settings: the PRACTIS guide. Int J Behav Nutr Phys Act. 2018;15(1):51.

Rogers E. Diffusion of Innovations. 5th ed. New York: Free Press; 2003.

Greenhalgh T, Papoutsi C. Spreading and scaling up innovation and improvement. BMJ. 2019;365:2068.

Mangham LJ, Hanson K. Scaling up in international health: what are the key issues? Health Policy Plan. 2010;25(2):85–96.

World Health Organization. Practical guidance for scaling up health service innovations. Geneva, Switzerland: World Health Organization; 2009.

Glasgow R, Lichtenstein E, Marcus A. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003;93(8):1261–7.

World Health Organization. Beginning with the end in mind: planning pilot projects and other programmatic research for successful scaling up. Geneva: Switzerland: World Health Organization; 2011.

Atkinson J-A, et al. Harnessing advances in computer simulation to inform policy and planning to reduce alcohol-related harms. Int J Public Health. 2018;63(4):537–46.

Pipe AL, Evans W, Papadakis S. Smoking cessation: health system challenges and opportunities. Tob Control. 2022;31(2):340–7.

National Cancer Institute. Greater than the sum: systems thinking in Tobacco control. U.S MD: Bethesda; 2007.

Thomas D, et al. System change interventions for smoking cessation. Cochrane Database Syst Rev. 2017;2(2):Cd010742.

PubMed Google Scholar

Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol. 2009;43(3–4):267–76.

Braithwaite J, et al. When complexity science meets implementation science: a theoretical and empirical analysis of systems change. BMC Med. 2018;16(1):63.

Jago R, et al. Rethinking children’s physical activity interventions at school: a new context-specific approach. Front Public Health. 2023;11:1149883.

Sartas M, et al. Scaling readiness: science and practice of an approach to enhance impact of research for development. Agric Syst. 2020;183:102874.

Lee K, et al. The intervention scalability assessment tool: a pilot study assessing five interventions for scalability. Public Health Res Pract. 2020;30:3022011.

Kwan BM, Luke DA, Adsul P, Koorts H, Morrato EH, Glasgow RE. Designing for Dissemination and Sustainability: Principles, Methods, and Frameworks for Ensuring Fit to Context. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health. Oxford University Press; 2023. https://doi.org/10.1093/oso/9780197660690.003.0027 .

Chapter Google Scholar

Hesketh KD, et al. Protocol for the Let’s Grow randomised controlled trial: examining efficacy, cost-effectiveness and scalability of a m-Health intervention for movement behaviours in toddlers. BMJ Open. 2022;12(3): e057521.

Sterman JD. Business Dynamics: Systems Thinking and Modeling for a Complex World. Boston: Irwin McGraw-Hill; 2000.

World Health Organization. WHO guidelines on physical activity and sedentary behaviour. Geneva: World Health Organization; 2020.

Biddle SJ, et al. Too much sitting and all-cause mortality: is there a causal link? BMC Public Health. 2016;16(1):635.

Owen N, et al. Sedentary behavior and public health: integrating the evidence and identifying potential solutions. Annu Rev Public Health. 2020;41:265–87.

Download references

Acknowledgements

The authors wish to acknowledge Narelle Robertson for contributing to the online database searches.

Additional files

JM is supported by a Deakin University Deans Strategic Fellowship. JS is supported by a National Health and Medical Research Council Leadership Level 2 Fellowship (APP 1176885). The funding bodies had no input into the study design, data collection or decision to publish.

Author information

Authors and affiliations.

School of Exercise and Nutrition Sciences, Deakin University, Institute for Physical Activity and Nutrition (IPAN), 221 Burwood Highway, Burwood Geelong, VIC, 3125, Australia

Harriet Koorts, Jiani Ma, Jo Salmon & Kristy A. Bolton

Cancer Epidemiology Division, Cancer Council Victoria, Melbourne, Australia

Christopher T. V. Swain

Department of Physiotherapy, Faculty of Medicine Dentistry and Health Sciences, Melbourne School of Health Sciences, University of Melbourne, Melbourne, Australia

Department of Social and Policy Sciences, University of Bath, Claverton Down, Bath, UK

Harry Rutter

You can also search for this author in PubMed Google Scholar

Contributions

HK conceptualised the manuscript, contributed to the design and execution of the study, reviewing literature and data extraction, analysis of data and led writing of the manuscript. JM, CS and KB contributed to the design and execution of the study, analysis of data and writing sections of the manuscript. JM and CS conducted the data screening, reviewing literature and data extraction, and KB and JM conducted quality appraisal ratings. JS and HR contributed to the design of the study and provided critical input into manuscript drafts. All authors revised the manuscript for intellectual content, and read and approved the final submitted version.

Corresponding author

Correspondence to Harriet Koorts .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

PRISMA Checklist.

Additional file 2.

Online database search strategy.

Additional file 3.

The Mixed Methods Appraisal Tool findings.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Koorts, H., Ma, J., Swain, C.T.V. et al. Systems approaches to scaling up: a systematic review and narrative synthesis of evidence for physical activity and other behavioural non-communicable disease risk factors. Int J Behav Nutr Phys Act 21 , 32 (2024). https://doi.org/10.1186/s12966-024-01579-6

Download citation

Received : 07 October 2023

Accepted : 24 February 2024

Published : 21 March 2024

DOI : https://doi.org/10.1186/s12966-024-01579-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Public health
Implementation
Non-communicable diseases

International Journal of Behavioral Nutrition and Physical Activity

ISSN: 1479-5868

Submission enquiries: Access here and click Contact Us
General enquiries: [email protected]

Strengths and limitations of early warning scores: A systematic review and narrative synthesis

Affiliations.

1 Leeds Institute of Biomedical & Clinical Sciences, Clinical Sciences Building St. James's University Hospital, University of Leeds, Leeds, LS9 7TF, United Kingdom. Electronic address: [email protected].
2 Leeds Institute of Biomedical & Clinical Sciences, Clinical Sciences Building St. James's University Hospital, University of Leeds, Leeds, LS9 7TF, United Kingdom.
3 School of Healthcare, Baines Wing, University of Leeds, Leeds LS2 9JT, United Kingdom.
4 Leeds Institute of Clinical Trials Research, Worsley Building, University of Leeds, Leeds LS2 9NL, United Kingdom.
PMID: 28950188
DOI: 10.1016/j.ijnurstu.2017.09.003

Background: Early warning scores are widely used to identify deteriorating patients. Whilst their ability to predict clinical outcomes has been extensively reviewed, there has been no attempt to summarise the overall strengths and limitations of these scores for patients, staff and systems. This review aims to address this gap in the literature to guide improvements for the optimization of patient safety.

Methods: A systematic review was conducted of MEDLINE ® , PubMed, CINAHL and The Cochrane Library in September 2016. The citations and reference lists of selected studies were reviewed for completeness. Studies were included if they evaluated vital signs monitoring in adult human subjects. Studies regarding the paediatric population were excluded, as were studies describing the development or validation of monitoring models. A narrative synthesis of qualitative, quantitative and mixed- methods studies was undertaken.

Findings: 232 studies met the inclusion criteria. Twelve themes were identified from synthesis of the data: Strengths of early warning scores included their prediction value, influence on clinical outcomes, cross-specialty application, international relevance, interaction with other variables, impact on communication and opportunity for automation. Limitations included their sensitivity, the need for practitioner engagement, the need for reaction to escalation and the need for clinical judgment, and the intermittent nature of recording. Early warning scores are known to have good predictive value for patient deterioration and have been shown to improve patient outcomes across a variety of specialties and international settings. This is partly due to their facilitation of communication between healthcare workers. There is evidence that the prediction value of generic early warning scores suffers in comparison to specialty-specific scores, and that their sensitivity can be improved by the addition of other variables. They are also prone to inaccurate recording and user error, which can be partly overcome by automation.

Conclusions: Early warning scores provide the right language and environment for the timely escalation of patient care. They are limited by their intermittent and user-dependent nature, which can be partially overcome by automation and new continuous monitoring technologies, although clinical judgment remains paramount.

Keywords: Early warning scores; Limitations; Strengths; Systematic review; Vital signs.

Publication types

Systematic Review
Patient Safety
Vital Signs*

Grants and funding

DRF-2016-09-037/DH_/Department of Health/United Kingdom
NIHR-RP-011-029/DH_/Department of Health/United Kingdom

- Google Chrome

Intended for healthcare professionals

Access provided by Google Indexer
My email alerts
BMA member login
Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

Advanced search
Search responses
Search blogs
Synthesis without meta...

Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline

Linked opinion.

Grasping the nettle of narrative synthesis

Related content
Peer review
Mhairi Campbell , research associate 1 ,
Joanne E McKenzie , associate professor 2 ,
Amanda Sowden , professor 3 ,
Srinivasa Vittal Katikireddi , clinical senior research fellow 1 ,
Sue E Brennan , research fellow 2 ,
Simon Ellis , associate director 4 ,
Jamie Hartmann-Boyce , senior researcher 5 ,
Rebecca Ryan , senior esearch fellow 6 ,
Sasha Shepperd , professor 7 ,
James Thomas , professor 8 ,
Vivian Welch , associate professor 9 ,
Hilary Thomson , senior research fellow 1
1 MRC/CSO Social and Public Health Sciences Unit, University of Glasgow, UK
2 School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
3 Centre for Reviews and Dissemination, University of York, York, UK
4 Centre for Guidelines, National Institute for Health and Care Excellence, London, UK
5 Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
6 School of Psychology and Public Health, La Trobe University, Melbourne, Australia
7 Nuffield Department of Population Health, University of Oxford, Oxford, UK
8 Evidence for Policy and Practice Information and Coordinating Centre, University College London, London, UK
9 Bruyere Research Institute, Ottawa, Canada
Correspondence to: M Campbell Mhairi.Campbell{at}glasgow.ac.uk
Accepted 8 October 2019

In systematic reviews that lack data amenable to meta-analysis, alternative synthesis methods are commonly used, but these methods are rarely reported. This lack of transparency in the methods can cast doubt on the validity of the review findings. The Synthesis Without Meta-analysis (SWiM) guideline has been developed to guide clear reporting in reviews of interventions in which alternative synthesis methods to meta-analysis of effect estimates are used. This article describes the development of the SWiM guideline for the synthesis of quantitative data of intervention effects and presents the nine SWiM reporting items with accompanying explanations and examples.

Summary points

Systematic reviews of health related interventions often use alternative methods of synthesis to meta-analysis of effect estimates, methods often described as “narrative synthesis”

Serious shortcomings in reviews that use “narrative synthesis” have been identified, including a lack of description of the methods used; unclear links between the included data, the synthesis, and the conclusions; and inadequate reporting of the limitations of the synthesis

The Synthesis Without Meta-analysis (SWiM) guideline is a nine item checklist to promote transparent reporting for reviews of interventions that use alternative synthesis methods

The SWiM items prompt users to report how studies are grouped, the standardised metric used for the synthesis, the synthesis method, how data are presented, a summary of the synthesis findings, and limitations of the synthesis

The SWiM guideline has been developed using a best practice approach, involving extensive consultation and formal consensus

Decision makers consider systematic reviews to be an essential source of evidence. 1 Complete and transparent reporting of the methods and results of reviews allows users to assess the validity of review findings. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA; http://www.prisma-statement.org/ ) statement, consisting of a 27 item checklist, was developed to facilitate improved reporting of systematic reviews. 2 Extensions are available for different approaches to conducting reviews (for example, scoping reviews 3 ), reviews with a particular focus (for example, harms 4 ), and reviews that use specific methods (for example, network meta-analysis. 5 ) However, PRISMA provides limited guidance on reporting certain aspects of the review, such as the methods for presentation and synthesis, and no reporting guideline exists for synthesis without meta-analysis of effect estimates. We estimate that 32% of health related systematic reviews of interventions do not do meta-analysis, 6 7 8 instead using alternative approaches to synthesis that typically rely on textual description of effects and are often referred to as narrative synthesis. 9 Recent work highlights serious shortcomings in the reporting of narrative synthesis, including a lack of description of the methods used, lack of transparent links between study level data and the text reporting the synthesis and its conclusions, and inadequate reporting of the limitations of the synthesis. 7 This suggests widespread lack of familiarity and misunderstanding around the requirements for transparent reporting of synthesis when meta-analysis is not used and indicates the need for a reporting guideline.

Scope of SWiM reporting guideline

This paper presents the Synthesis Without Meta-analysis (SWiM) reporting guideline. The SWiM guideline is intended for use in systematic reviews examining the quantitative effects of interventions for which meta-analysis of effect estimates is not possible, or not appropriate, for a least some outcomes. 10 Such situations may arise when effect estimates are incompletely reported or because characteristics of studies (such as study designs, intervention types, or outcomes) are too diverse to yield a meaningful summary estimate of effect. 11 In these reviews, alternative presentation and synthesis methods may be adopted, (for example, calculating summary statistics of intervention effect estimates, vote counting based on direction of effect, and combining P values), and SWiM provides guidance for reporting these methods and results. 11 Specifically, the SWiM guideline expands guidance on “synthesis of results” items currently available, such as PRISMA (items 14 and 21) and RAMESES (items 11, 14, and 15). 2 12 13 SWiM covers reporting of the key features of synthesis including how studies are grouped, synthesis methods used, presentation of data and summary text, and limitations of the synthesis.

SWiM is not intended for use in reviews that synthesise qualitative data, for which reporting guidelines are already available, including ENTREQ for qualitative evidence synthesis and eMERGe for meta-ethnography. 14 15

Development of SWiM reporting guideline

A protocol for the project is available, 10 and the guideline development was registered with the EQUATOR Network, after confirmation that no similar guideline was in development. All of the SWiM project team are experienced systematic reviewers, and one was a co-author on guidance on the conduct of narrative synthesis (AS). 9 A project advisory group was convened to provide greater diversity in expertise. The project advisory group included representatives from collaborating Cochrane review groups, the Campbell Collaboration, and the UK National Institute for Health and Care Excellence (see supplementary file 1).

The project was informed by recommendations for developing guidelines for reporting of health research. 16 We assessed current practice in reporting synthesis of effect estimates without meta-analysis and used the findings to devise an initial checklist of reporting items in consultation with the project advisory group. We invited 91 people, all systematic review methodologists or authors of reviews that synthesised results from studies without using meta-analysis, to participate in a three round Delphi exercise, with a response rate of 48% (n=44/91) in round one, 54% (n=37/68) in round two, and 82% (n=32/39) in round three. The results were discussed at a consensus meeting of an expert panel (the project advisory group plus one additional methodological expert) (see supplementary file 1). After the meeting, we piloted the revised guideline to assess ease of use and face validity. Eight systematic reviewers with varying levels of experience, who had not been involved in the Delphi exercise, were asked to read and apply the guideline. We conducted short interviews with the pilot participants to identify any clarification needed in the items or their explanations. We subsequently revised the items and circulated them for comment among the expert panel, before finalising them. Full methodological details of the SWiM guideline development process are provided in supplementary file 1.

Synthesis without meta-analysis reporting items

We identified nine items to guide the reporting of synthesis without meta-analysis. Table 1 shows these SWiM reporting items. An online version is available at www.equator-network.org/reporting-guidelines . An explanation and elaboration for each of the reporting items is provided below. Examples to illustrate the reporting items and explanations are provided in supplementary file 2.

Synthesis Without Meta-analysis (SWiM) items: SWiM is intended to complement and be used as an extension to PRISMA

View inline

Item 1: grouping studies for synthesis

1a) description.

Provide a description of, and rationale for, the groups used in the synthesis (for example, groupings of interventions, population, outcomes, study design).

1a) Explanation

Methodological and clinical or conceptual diversity may occur (for example, owing to inclusion of diverse study designs, outcomes, interventions, contexts, populations), and it is necessary to clearly report how these study characteristics are grouped for the synthesis, along with the rationale for the groups (see Cochrane Handbook Chapter 3 17 ). Although reporting the grouping of study characteristics in all reviews is important, it is particularly important in reviews without meta-analysis, as the groupings may be less evident than when meta-analysis is used.

Providing the rationale, or theory of change, for how the intervention is expected to work and affect the outcome(s) will inform authors’ and review users’ decisions about the appropriateness and usefulness of the groupings. A diagram, or logic model, 18 19 can be used to visually articulate the underlying theory of change used in the review. If the theory of change for the intervention is provided in full elsewhere (for example, in the protocol), this should be referenced. In Cochrane reviews, the rationale for the groups can be outlined in the section “How the intervention is expected to work.”

1b) Description

Detail and provide rationale for any changes made subsequent to the protocol in the groups used in the synthesis.

1b) Explanation

Decisions about the planned groups for the syntheses may need to be changed following study selection and data extraction. This may occur as a result of important variations in the population, intervention, comparison, and/or outcomes identified after the data are collected, or where limited data are available for the pre-specified groupings, and the groupings may need to be modified to facilitate synthesis (Cochrane Handbook Chapter 2 20 ). Reporting changes to the planned groups, and the reason(s) for these, is important for transparency, as this allows readers to assess whether the changes may have been influenced by study findings. Furthermore, grouping at a broader level of (any or multiple) intervention, population, or outcome will have implications for the interpretation of the synthesis findings (see item 8).

Item 2: describe the standardised metric and transformation method used

Description.

Describe the standardised metric for each outcome. Explain why the metric(s) was chosen, and describe any methods used to transform the intervention effects, as reported in the study, to the standardised metric, citing any methodological guidance used.

Explanation

The term “standardised metric” refers to the metric that is used to present intervention effects across the studies for the purpose of synthesis or interpretation, or both. Examples of standardised metrics include measures of intervention effect (for example, risk ratios, odds ratios, risk differences, mean differences, standardised mean differences, ratio of means), direction of effect, or P values. An example of a statistical method to convert an odds ratio to a standardised mean difference is that proposed by Chinn (2000). 21 For other methods and metrics, see Cochrane Handbook Chapter 6. 22

Item 3: describe the synthesis methods

Describe and justify the methods used to synthesise the effects for each outcome when it was not possible to undertake a meta-analysis of effect estimates.

For various reasons, it may not be possible to do a meta-analysis of effect estimates. In these circumstances, other synthesis methods need to be considered and specified. Examples include combining P values, calculating summary statistics of intervention effect estimates (for example, median, interquartile range) or vote counting based on direction of effect. See table 2 for a summary of possible synthesis methods (for further details, see McKenzie and Brennan 2019 11 ). Justification should be provided for the chosen synthesis method.

Questions answered according to types of synthesis methods and types of data used

Item 4: criteria used to prioritise results for summary and synthesis

Where applicable, provide the criteria used, with supporting justification, to select particular studies, or a particular study, for the main synthesis or to draw conclusions from the synthesis (for example, based on study design, risk of bias assessments, directness in relation to the review question).

Criteria may be used to prioritise the reporting of some study findings over others or to restrict the synthesis to a subset of studies. Examples of criteria include the type of study design (for example, only randomised trials), risk of bias assessment (for example, only studies at a low risk of bias), sample size, the relevance of the evidence (outcome, population/context, or intervention) pertaining to the review question, or the certainty of the evidence. Pre-specification of these criteria provides transparency as to why certain studies are prioritised and limits the risk of selective reporting of study findings.

Item 5: investigation of heterogeneity in reported effects

State the method(s) used to examine heterogeneity in reported effects when it is not possible to do a meta-analysis of effect estimates and its extensions to investigate heterogeneity.

Informal methods to investigate heterogeneity in the findings may be considered when a formal statistical investigation using methods such as subgroup analysis and meta-regression is not possible. Informal methods could involve ordering tables or structuring figures by hypothesised modifiers such as methodological characteristics (for example, study design), subpopulations (for example, sex, age), intervention components, and/or contextual/setting factors (see Cochrane Handbook Chapter 12 11 ). The methods used and justification for the chosen methods should be reported. Investigations of heterogeneity should be limited, as they are rarely definitive; this is more likely to be the case when informal methods are used. It should also be noted if the investigation of heterogeneity was not pre-specified.

Item 6: certainty of evidence

Describe the methods used to assess the certainty of the synthesis findings.

The assessment of the certainty of the evidence should aim to take into consideration the precision of the synthesis finding (confidence interval if available), the number of studies and participants, the consistency of effects across studies, the risk of bias of the studies, how directly the included studies address the planned question (directness), and the risk of publication bias. GRADE (Grading of Recommendations, Assessment, Development and Evaluations) is the most widely used framework for assessing certainty (Cochrane Handbook Chapter 14 23 ). However, depending on the synthesis method used, assessing some domains (for example, consistency of effects when vote counting is undertaken) may be difficult.

Item 7: data presentation methods

Describe the graphical and tabular methods used to present the effects (for example, tables, forest plots, harvest plots).

Specify key study characteristics (for example, study design, risk of bias) used to order the studies, in the text and any tables or graphs, clearly referencing the studies included

Study findings presented in tables or graphs should be ordered in the same way as the syntheses are reported in the narrative text to facilitate the comparison of findings from each included study. Key characteristics, such as study design, sample size, and risk of bias, which may affect interpretation of the data, should also be presented. Examples of visual displays include forest plots, 24 harvest plots, 25 effect direction plots, 26 albatross plots, 27 bubble plots, 28 and box and whisker plots. 29 McKenzie and Brennan (2019) provide a description of these plots, when they should be used, and their pros and cons. 11

Item 8: reporting results

For each comparison and outcome, provide a description of the synthesised findings and the certainty of the findings. Describe the result in language that is consistent with the question the synthesis addresses and indicate which studies contribute to the synthesis.

For each comparison and outcome, a description of the synthesis findings should be provided, making clear which studies contribute to each synthesis (for example, listing in the text or tabulated). In describing these findings, authors should be clear about the nature of the question(s) addressed (see table 2 , column 1), the metric and synthesis method used, the number of studies and participants, and the key characteristics of the included studies (population/settings, interventions, outcomes). When possible, the synthesis finding should be accompanied by a confidence interval.An assessment of the certainty of the effect should be reported.

Results of any investigation of heterogeneity should be described, noting if it was not pre-planned and avoiding over-interpretation of the findings.

If a pre-specified logic model was used, authors may report any changes made to the logic model during the review or as a result of the review findings. 30

Item 9: limitations of the synthesis

Report the limitations of the synthesis methods used and/or the groupings used in the synthesis and how these affect the conclusions that can be drawn in relation to the original review question.

When reporting limitations of the synthesis, factors to consider are the standardised metric(s) used, the synthesis method used, and any reconfiguration of the groups used to structure the synthesis (comparison, intervention, population, outcome).

The choice of metric and synthesis method will affect the question addressed (see table 2 ). For example, if the standardised metric is direction of effect, and vote counting is used, the question will ask “is there any evidence of an effect?” rather than “what is the average intervention effect?” had a random effects meta-analysis been used.

Limitations of the synthesis might arise from post-protocol changes in how the synthesis was structured and the synthesis method selected. These changes may occur because of limited evidence, or incompletely reported outcome or effect estimates, or if different effect measures are used across the included studies. These limitations may affect the ability of the synthesis to answer the planned review question—for example, when a meta-analysis of effect estimates was planned but was not possible.

The SWiM reporting guideline is intended to facilitate transparent reporting of the synthesis of effect estimates when meta-analysis is not used. The guideline relates specifically to transparently reporting synthesis and presentation methods and results, and it is likely to be of greatest relevance to reviews that incorporate diverse sources of data that are not amenable to meta-analysis. The SWiM guideline should be used in conjunction with other reporting guidelines that cover other aspects of the conduct of reviews, such as PRISMA. 31 We intend SWiM to be a resource for authors of reviews and to support journal editors and readers in assessing the conduct of a review and the validity of its findings.

The SWiM reporting items are intended to cover aspects of presentation and synthesis of study findings that are often left unreported when methods other than meta-analysis have been used. 7 These include reporting of the synthesis structure and comparison groupings (items 1, 4, 5, and 6), the standardised metric used for the synthesis (item 2), the synthesis method (items 3 and 9), presentation of data (item 7), and a summary of the synthesis findings that is clearly linked to supporting data (item 8). Although the SWiM items have been developed specifically for the many reviews that do not include meta-analysis, SWiM promotes the core principles needed for transparent reporting of all synthesis methods including meta-analysis. Therefore, the SWiM items are relevant when reporting synthesis of quantitative effect data regardless of the method used.

Reporting guidelines are sometimes interpreted as providing guidance on conduct or used to assess the quality of a study or review; this is not an appropriate application of a reporting guideline, and SWiM should not be used to guide the conduct of the synthesis. For guidance on how to conduct synthesis using the methods referred to in SWiM, we direct readers to the second edition of the Cochrane Handbook for Systematic Reviews of Interventions, specifically chapter 12. 11 Although an overlap inevitably exists between reporting and conduct, the SWiM reporting guideline is not intended to be prescriptive about choice of methods, and the level of detail for each item should be appropriate. For example, investigation of heterogeneity (item 5) may not always be necessary or useful. In relation to SWiM, we anticipate that the forthcoming update of PRISMA will include new items covering a broader range of synthesis methods, 32 but it will not provide detailed guidance and examples on synthesis without meta-analysis.

The SWiM reporting guideline emerged from a project aiming to improve the transparency and conduct of narrative synthesis (ICONS-Quant: Improving the CONduct and reporting of Narrative Synthesis). 10 Avoidance of the term “narrative synthesis” in SWiM is a deliberate move to promote clarity in the methods used in reviews in which the synthesis does not rely on meta-analysis. The use of narrative is ubiquitous across all research and can serve a valuable purpose in the development of a coherent story from diverse data. 33 34 However, within the field of evidence synthesis, narrative approaches to synthesis of quantitative effect estimates are characterised by a lack of transparency, making assessment of the validity of their findings difficult. 7 Together with the recently published guidance on conduct of alternative methods of synthesis, 11 the SWiM guideline aims to improve the transparency of, and subsequently trust in, the many reviews that synthesise quantitative data without meta-analysis, particularly for reviews of intervention effects.

Acknowledgments

We thank the participants of the Delphi survey and colleagues who informally piloted the guideline.

Contributors: All authors contributed to the development of SWiM. HT had the idea for the study. HT, SVK, AS, JEM, and MC designed the study methods. JT, JHB, RR, SB, SE, SS, and VW contributed to the consensus meeting and finalising the guideline items. MC prepared the first draft of the manuscript, and all authors critically reviewed and approved the final manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. HT is the guarantor.

Project advisory group members: Simon Ellis, Jamie Hartmann-Boyce, Mark Petticrew, Rebecca Ryan, Sasha Shepperd, James Thomas, Vivian Welch.

Expert panel members: Sue Brennan, Simon Ellis, Jamie Hartmann-Boyce, Rebecca Ryan, Sasha Shepperd, James Thomas, Vivian Welch.

Funding: This project was supported by funds provided by the Cochrane Methods Innovation Fund. MC, HT, and SVK receive funding from the UK Medical Research Council (MC_UU_12017-13 and MC_UU_12017-15) and the Scottish Government Chief Scientist Office (SPHSU13 and SPHSU15). SVK is supported by an NHS Research Scotland senior clinical fellowship (SCAF/15/02). JEM is supported by an NHMRC career development fellowship (1143429). RR’s position is funded by the NHMRC Cochrane Collaboration Funding Program (2017-2010). The views expressed in this article are those of the authors and not necessarily those of their employer/host organisations or of Cochrane or its registered entities, committees, or working groups.

Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: funding for the project as described above; HT is co-ordinating editor for Cochrane Public Health; SVK, SE, JHB, RR, and SS are Cochrane editors; JEM is co-convenor of the Cochrane Statistical Methods Group; JT is a senior editor of the second edition of the Cochrane Handbook; VW is editor in chief of the Campbell Collaboration and an associate scientific editor of the second edition of the Cochrane Handbook; SB is a research fellow at Cochrane Australia; no other relationships or activities that could appear to have influenced the submitted work.

Ethical approval: Ethical approval was obtained from the University of Glasgow College of Social Sciences Ethics Committee (reference number 400170060).

Transparency: The lead author affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

Patient and public involvement: This research was done without patient involvement. Patients were not invited to comment on the study design and were not consulted to develop outcomes or interpret the results.

Dissemination to participants and related patient and public communities: The authors plan to disseminate the research through peer reviewed publications, national and international conferences, webinars, and an online training module and by establishing an email discussion group.

Donnelly CA ,
Campbell P ,
Liberati A ,
Altman DG ,
Tetzlaff J ,
Tricco AC ,
Zorzela L ,
Ioannidis JP ,
PRISMAHarms Group
Salanti G ,
Caldwell DM ,
Shamseer L ,
Campbell M ,
Katikireddi SV ,
Katikireddi S ,
Roberts H ,
Higgins J ,
Chandler J ,
McKenzie J ,
Greenhalgh T ,
Westhorp G ,
Buckingham J ,
Flemming K ,
McInnes E ,
France EF ,
Cunningham M ,
Schulz KF ,
Higgins JPT ,
McKenzie JE ,
Brennan SE ,
Anderson LM ,
Petticrew M ,
Rehfuess E ,
Schünemann HJ ,
Ogilvie D ,
Thomson HJ ,
Harrison S ,
Martin RM ,
Higgins JPT
Schriger DL ,
Schroter S ,
Rehfuess EA ,
Brereton L ,
PRISMA Group
↵ Page M, McKenzie J, Bossuyt P, et al. Updating the PRISMA reporting guideline for systematic reviews and meta-analyses: study protocol. 2018. https://osf.io/xfg5n .
Melendez-Torres GJ ,
O’Mara-Eves A ,
Petticrew M

SYSTEMATIC REVIEW article

A systematic review and narrative synthesis of prevalence rates, risk and protective factors for suicidal behavior in international students.

1 Orygen, Parkville, VIC, Australia
2 Center for Youth Mental Health, The University of Melbourne, Parkville, VIC, Australia

International students are a crucial group for suicide prevention efforts. However, no comprehensive review of the prevalence, risk or protective factors for international student suicide has been conducted, complicating prevention efforts. The current systematic review addresses this issue by identifying the prevalence, risk, and protective factors for suicidal thoughts and behavior in international students enrolled in tertiary institutions. We searched CINAHL, EMBASE, ERIC, Medline, and PsycInfo from inception to November 2023, and 24 papers met the study inclusion criteria. Nineteen studies reported on the prevalence or intensity of suicidal ideation, six reported on the prevalence of suicide attempts, and seven on the prevalence of self-harm. No studies provided data on completed suicides. Studies indicated that international students experienced similar or lower levels of suicidal ideation and self-harm, but more frequent suicide attempts than domestic students. A narrative synthesis of risk and protective factors yielded mixed results, although social isolation, comorbid mental health conditions, and experiences of racism or discrimination were commonly linked to suicidal thoughts and behaviors. The mixed evidence quality, numerous measurement issues, and low number of studies in the literature point to a need for further research on suicide among international students.

Systematic review registration: https://www.crd.york.ac.uk/PROSPERO/ , identifier CRD42022307252.

Introduction

Suicide has been one of the leading causes of death worldwide in the recent years ( 1 – 3 ). Suicidal thoughts and behaviors are particularly prevalent among tertiary education students ( 4 ). A significant portion of students in higher education are international students who transition from their primary country of residence to pursue education abroad ( 5 , 6 ). International students face a unique combination of challenges such as language barriers, cultural adjustment, financial stress, and academic pressure ( 7 ). These factors can negatively impact mental health ( 8 , 9 ) and potentially increase the risk of suicidal thoughts and behaviors. Complicating matters, international students are less likely to seek help for mental health-related struggles compared to domestic students ( 10 ). Furthermore, recent investigations into international student suicides highlight that these students’ engagement with mental health services prior to their death is minimal ( 11 ).

Despite evidence suggesting that international students may experience multiple risk factors for suicide, there has been little focus on this population when it comes to suicide prevention research ( 12 ). A recent scoping review found that there is a lack of high-quality research on this topic, and that no suicide-specific interventions for international students currently exist ( 13 ). Encouragingly, broader suicide prevention activities delivered in tertiary education institutions have been shown to improve students’ knowledge and attitudes towards suicide and rates of engagement with mental health services ( 14 ). However, such programs may not be appropriate or adapted to international student needs. People from culturally and linguistically diverse backgrounds have distinct perspectives on mental health and suicide, and effective suicide prevention for these groups necessitates tailored strategies ( 13 , 15 ). Foundational to this is epidemiological research that identifies potential points for intervention. Currently, there is a gap in knowledge regarding the prevalence rates of, and risk factors for, suicidal thoughts and behaviors in international students. This limits our capacity to develop effective, evidence-based suicide prevention interventions for this population.

The present review aims to address these gaps by synthesizing the available literature on suicide and suicide related behaviors in international students by answering the following questions: a) What is the prevalence of suicidal ideation, suicide attempts and self-harm in international students, and b) What are the risk and protective factors that influence suicides, suicidal ideation, attempts, and self-harm in international students?

Materials and methods

The protocol for this review was registered on PROSPERO (CRD42022307252). The summary of prevalence rates of suicidal thoughts and behaviors was reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines ( 16 ). Because studies that examined risk and protective factors for suicidal thoughts and behaviors varied considerably in their methodologies and variables of interest, narrative synthesis was deemed most appropriate to summarize the findings. The narrative synthesis approach was guided by Rodgers et al. ( 17 ).

Eligibility criteria

The inclusion criteria comprised peer-reviewed empirical studies that a) identified rates of suicidal ideation, attempts, or self-harm, or b) examined risk or protective factors for suicide, suicidal ideation, attempts, or self-harm in international students.

International students were defined as tertiary education students of any age enrolled in an educational program in a different country to where they typically reside or hold citizenship. Studies were included if international student participants made up the entire sample or were identified as a subset of a broader sample with their data reported separately. Studies from any publication year and geographical region were included. Case studies and qualitative studies were excluded as the current review focused specifically on quantitative evidence. Sources that were not in English were excluded due to resource limitations.

Search strategy and data sources

A comprehensive search strategy was developed with support from a librarian to identify published articles relevant to the research questions. Two platforms (EBSCOhost & OVID) covering five databases (CINAHL, EMBASE, ERIC, Medline, & PsycINFO) were searched to cover a variety of disciplines (e.g., health, psychology, and education) relevant to the research topic. Search terms were adapted for each platform and database. Searches included studies from the database inception to 8 November, 2023. Reference lists from identified studies were checked for any additional relevant studies that may have been missed during the electronic search. The full search strategy is reported in Appendix A .

Reference management and screening

All search results were imported into Endnote, duplicates were checked and removed, and the full reference list was generated and uploaded to the systematic review screening tool Covidence ( 18 ). Additional duplicate checks were undertaken in Covidence. SM performed the database search, combined the records, and ran the duplicate removal processes. Titles and abstracts were independently screened for relevance to the research question and inclusion criteria by MV and SM. Full texts of included articles were then screened for more detail by the same two reviewers. All full-text studies that were excluded had their reasons for exclusion recorded. Discrepancies in screening decisions between the two reviewers were discussed until consensus was reached.

Data extraction and analysis

A tailored data extraction template was created ( Appendix B ) to extract incidence of suicide, prevalence rates of suicide attempts and self-harm, and prevalence rates and intensity levels of suicidal ideation. In this review, a risk or protective factor was conceptualized as any variable that was defined, measured, and tested for a statistically significant relationship with suicide, suicidal ideation, attempts, or self-harm, through any type of analysis (e.g., correlation or regression). SM and MV independently performed data extraction for each included article, and subsequently verified the extraction results for accuracy.

Quality assessment

Quality assessment was performed using the Checklist for Assessing the Quality of Quantitative Studies (CAQQS) ( 19 ) as this tool provides a comprehensive evaluation of the methodological quality of research. All studies were included in the final review regardless of how they scored on the CAQQS.

Selection of sources of evidence

The process of searching, retrieving, and screening studies is described in Figure 1 . The inter-rater reliability between the two reviewers was κ = 0.66 [substantial ( 20 )] for the abstract screening and κ = 0.85 (almost perfect agreement) for the full-text screening. Twenty-four studies were included in the final review, and their key characteristics and quality assessment scores are reported in Table 1 .

Figure 1 The PRISMA flowchart describing the process of systematic search and study selection.

Table 1 Summary of included studies and relevant key findings.

Characteristics of included studies

Of 24 studies, 21 were published between 2013 and 2023, and three prior to that ( 34 , 41 , 42 ). Fifteen studies were conducted in the USA; the remainder came from Asia (3), Australia (3), Europe (2), and Canada (1). There were 21 cross-sectional studies, one case series ( 24 ), one longitudinal ( 26 ), and one case-control study ( 40 ). The included studies scored between 63.6% and 95.5% on the CAQQS tool. Most studies were not guided by a suicide-specific theoretical framework, while four ( 32 , 33 , 36 , 39 ) referenced the Interpersonal Theory of Suicide ( 49 ).

Ten studies consisted solely of international students, and fourteen had an international student sub-sample and reported their demographic information separately. Because Yang and Clum’s two studies ( 41 , 42 ) used the same sample, from hereon it will be counted only once when reporting demographic data and suicide-related outcomes. The sizes of international student samples and sub-samples ranged from 46 to 22,385 with a median of 334.0. Where international students comprised a sub-sample, their proportion of the overall sample ranged from 4.6% to 75.0% ( x ¯ = 30.1%).

On average, international student samples had a slightly higher proportion of females (range = 27.7-68.0%, x ¯ = 56.1%) than males (32.0-72.3%, x ¯ = 43.9%). Twelve studies reported mean ages of international students (range = 20.2-28.5 years, x ¯ = 23.7). Six studies reported only age ranges (18-30+), and five did not report age. All samples consisted of college or university students at the undergraduate or postgraduate level. Most international students were from Asia, with China and India being the most common countries of origin.

Prevalence and intensity of suicidal thoughts and behaviors

Suicidal ideation.

Nineteen studies reported on suicidal ideation in international students using a diverse range of measures and timeframes. In terms of prevalence rates, one study found that 46.0% of students presenting at a psychiatric emergency clinic reported suicidal ideation in the past week, measured by the Columbia Suicide Severity Rating Scale (C-SSRS) ( 24 ). The proportions of students endorsing suicidal ideation in the past two weeks were 28.6% ( 22 ) and 23.4% ( 32 ) when measured by item 9 from the Patient Health Questionnaire-9, and 21.8% when measured by the Modified Scale for Suicide Ideation ( 41 , 42 ). According to one study, 6-month prevalence of suicidal ideation was 11.2%, measured by the C-SSRS ( 26 ). Five studies reported on prevalence in the past year, yielding a range of 5.4-18.2% ( 23 , 37 , 40 , 43 , 44 ). This was measured by various purpose-built questions or items borrowed from existing measures. Prevalence of suicidal ideation since commencing study abroad was 11.3% in females and 6.7% in males, as measured by a purpose-built question ( 34 ). Lifetime prevalence of suicidal ideation was 26.0% ( 26 ).

Five studies measured the intensity of suicidal ideation in the past week. The mean scores were 13.00 and 12.99 on the Suicidal Ideation Scale (full range 10-50) ( 33 , 39 ), 3.11 on the Beck Scale for Suicide Ideation (BSSI; full range 0-42) ( 27 ), 0.38 on the first five items of the BSSI (full range 0-10) ( 29 ), and 1.14 on a purpose-built question (full range 1-5) ( 21 ). The mean score for Jun and colleagues’ ( 25 ) sample on the BSSI was 11.30, but no timeframe was reported.

Five studies compared prevalence rates of suicidal ideation between international and domestic students. International students had significantly lower rates of suicidal ideation than domestic students in three of those studies ( 23 , 43 , 44 ) and similar rates to domestic students in the two remaining studies ( 24 , 26 ). There was an additional study where international students reported higher rates than domestic students, but it was not stated whether this difference was statistically significant ( 40 ).

Three studies compared the intensity of suicidal ideation between international and domestic students. International students had similar intensity levels to domestic students in two studies ( 21 , 36 ) and significantly higher levels than domestic students in one study ( 27 ).

Suicide attempts

Six studies provided data on prevalence rates of suicide attempts in international students. In one study, the rate of suicide attempts in the past week was 12.7% ( 24 ). In another study, 6-month prevalence was 3.5% ( 26 ). Four studies reported rates of attempts in the past year, ranging from 1.2% to 2.2% ( 23 , 37 , 40 , 43 ). As for lifetime suicide attempts, the prevalence was 7.5% ( 26 ), while in another study with students presenting to a psychiatric emergency clinic, lifetime prevalence of multiple attempts (2+) was 14.3% ( 24 ). Four studies assessed suicide attempts using purpose-built questions that were not part of existing measures, and the remaining two used the C-SSRS ( 24 , 26 ).

Five studies compared international and domestic students on the prevalence of suicide attempts, yielding mixed results. Rates over the past year were significantly higher in international than domestic students in two studies ( 23 , 43 ). There was a third study where international students also reported higher suicide attempt rates than domestic students, but this difference was not tested for significance ( 40 ). In one study, international and domestic students did not differ on lifetime prevalence of attempts at entry to college, but at 6-month follow-up international students were significantly more likely to have made an attempt than domestic students ( 26 ). Another study reported that while international and domestic students accessing psychiatric services did not differ on past-week attempt rates, international students had a higher lifetime prevalence of multiple attempts than domestic students ( 24 ).

No studies reported rates of completed suicide.

Six studies measured and reported self-harm rates in international students. Among students presenting at a psychiatric emergency clinic 12.7% reported engaging in self-harm in the past week ( 24 ), whereas 6-month prevalence of self-harm was 5.6% in a general university sample ( 26 ). Three studies reported on past-year rates of self-harm, which ranged from 4.8% to 17.2% ( 40 , 43 , 44 ). Prevalence of self-harm since commencing study abroad was 4.5% in females and 2.0% in males ( 34 ). Lifetime prevalence was 13.0% in one study ( 26 ), while 25.4% of international students presenting to psychiatric services reported a lifetime history of multiple self-harm instances (2+) ( 24 ). Five of the six studies used purpose-built questions as opposed to a standardized measure; the study by King et al. ( 26 ) used the C-SSRS.

Studies that compared the rates of self-harm in international and domestic students reported mixed results. Three studies found no difference in self-harm rates ( 24 , 26 , 43 ), while in one study international students had lower rates of self-harm than domestic students ( 44 ).

Narrative synthesis of risk and protective factors for suicidal thoughts and behaviors

Several factors related to social isolation emerged as risk factors for suicidal ideation. Three studies found an association between low social support and suicidal ideation ( 32 , 41 , 42 ). High social support was a protective factor, but this association was weaker for international than domestic students ( 32 ). Another factor related to suicidal ideation was low campus belongingness in two studies ( 29 , 36 ), while unmet interpersonal needs ( 37 ) and loneliness ( 29 ) were factors identified by single studies. Conversely, high family belongingness emerged as a potential risk factor, with international students reporting higher suicidal ideation at higher levels of family connectedness ( 36 ).

There were numerous intrapersonal factors relating to a person’s psychological functioning that emerged as risk factors. Three studies found a relationship between depression and suicidal ideation ( 25 , 39 , 41 ), while single studies identified the following as risk factors: generalized anxiety, social anxiety, eating concerns, hostility, maladaptive perfectionism ( 25 ); personal discrepancy between desired and actual educational performance ( 39 ); low problem-focused coping skills and hopelessness ( 41 ).

Several contextual factors related to an individual’s external setting emerged as risk factors. Suicidal ideation was associated with perceived discrimination in three studies ( 33 , 37 , 39 ), academic stress in two studies ( 25 , 33 ), life stress in two studies ( 29 , 41 ), and family discrepancy between desired and actual educational performance in two studies ( 25 , 39 ). Single studies identified a relationship of suicidal ideation with the following factors: cross-cultural loss ( 33 ); perceived public stigma ( 23 ); residing with family, perceived susceptibility to COVID-19, and perceived insufficiency with resources needed to prevent COVID-19 ( 21 ).

There was mixed evidence for several potential risk factors. There were no gender differences on suicidal ideation in two studies ( 21 , 36 ), but in Zhou and colleagues’ study ( 44 ) rates of suicidal ideation were significantly higher in females (9.9%) than males (7.0%). Similarly in King et al. ( 26 ), more females than males reported suicidal ideation at both entry to college (35.3% vs. 20.5%) and at 6-month follow-up (15.3% vs. 7.5%), but this difference was not tested for statistical significance. Of the three studies that assessed perceived burdensomeness, two reported a significant association ( 33 , 39 ) and one did not ( 36 ). Additionally, several factors correlated with suicidal ideation but did not predict it when using predictive models (e.g., regression analyses or structural equation modelling), namely thwarted belongingness ( 33 , 39 ), cultural stress, family conflict, feelings of entrapment, and perfectionism ( 37 ).

Two studies assessed risk or protective factors related to suicide attempts. Higher perceived public stigma was associated with greater incidence of attempts, and this association was stronger in Asian international than white students ( 23 ). Among international students, males had a higher prevalence of attempts than females, but whether this difference was significant was not reported ( 26 ). Being a Chinese versus non-Chinese international student was not associated with differing prevalence of attempts ( 26 ).

In a study that assessed suicidal ideation and attempts as a combined variable, being non-heterosexual significantly predicted suicidality ( 30 ).

No risk or protective factors for completed suicides were reported.

Two studies assessed risk and protective factors for self-harm behavior, while others focused on self-harm thoughts, or a combination of self-harm thoughts and behaviors. Female international students reported higher self-harm rates than males in two studies; 17.6% vs. 16.1% (statistically significant) ( 44 ) and 4.5% vs. 2.0% (not tested for significance) ( 34 ). Furthermore, significantly more female than male students endorsed an item asking if they have engaged in and/or thought about self-harm ( 28 ). One study found that international students who engaged in self-harm had significantly higher scores on depression, anxiety, stress, and experiences of abuse (and associated distress) than those who had not self-harmed ( 34 ). Loneliness and perceived discrimination were significantly associated with greater odds of self-harm thoughts, while the following covariates were not associated: gender, age, relationship status, mask-wearing, and being Asian ( 38 ).

This review examined the prevalence of, and risk and protective factors for, suicidal thoughts and behaviors in international students. A total of 24 studies were included with the majority published since 2013. Nineteen studies reported data on suicidal ideation, six reported on suicide attempts, and seven addressed self-harm. No studies reported on suicide deaths. In terms of quality appraisal, the studies scored moderate to high on the CAQQS, suggesting that the data quality is acceptable. However, the current review highlights shortcomings in research on suicide-related thoughts and behaviors among international students, including a lack of studies and diverse methodologies leading to inconsistent findings.

Key findings

The prevalence rates of suicidal ideation, attempts and self-harm varied depending on the instruments used, the measurement timeframe, and the nature of the sample. With the exception of international students presenting at a psychiatry clinic ( 24 ), rates of suicidal ideation (5.4-28.6%), suicide attempts (1.2%-7.5%), and self-harm (4.8%-17.2%) across timeframes, populations and measures were similar to rates from general college populations ( 50 ). Suicidal ideation intensity was typically low among international students, but the inclusion of cases without suicidal ideation may have positively skewed results.

Specific comparisons between international and domestic students produced mixed results. International students reported lower rates, but similar intensity of suicidal ideation compared to domestic students. In contrast, international students displayed a higher prevalence of suicide attempts than their domestic counterparts. Self-harm rates were similar in domestic and international students.

Similar to prior research on common stressors for international students ( 7 ), factors related to social isolation emerged as a common risk factor for suicidal ideation. This aligns with the Interpersonal Theory of Suicide ( 49 ), which proposes that thwarted belongingness, encompassing various elements related to social connectedness and the need to belong, is strongly related to suicidal thoughts and behaviors. Surprisingly, family connectedness was positively linked to increased suicidal ideation, contrasting with other social connectedness variables. This might be due to the distance from family networks during international studies, suggesting that factors related to social connectedness and isolation are not a homogenous group. More research is required to determine which factors relate to suicidal ideation and the time of their impact, with longitudinal studies needed to establish causality in these relationships.

Other significant risk factors for suicidal ideation encompass comorbid mental health conditions like depression and anxiety, contextual stressors such as experiences of racism and academic stress, along with inadequate coping abilities. These risk factors bear similarities to those observed in other student populations and provide potential opportunities for targeted intervention strategies. An emerging risk factor from the recent years was COVID-19 related stressors ( 21 ). International students are a population whose mental health was particularly impacted by COVID-19 restrictions ( 51 ), and the present review shows that COVID-19 related stressors are also specifically linked to suicidal thoughts and behaviors. As such, suicide prevention strategies for international students need to incorporate these common risk factors, including potential past and ongoing impacts of the COVID-19 or other pandemics.

The only significant risk factor for suicide attempts was high perceived public stigma. Given that suicidal ideation and attempts are closely related concepts, it may be that risk and protective factors for ideation are also applicable to attempts. However, more research is needed to confirm this. Risk factors for self-harm included identifying as female and common mental health symptoms like depression and anxiety. However, each of these were tested within individual studies so further evidence substantiating these associations is needed. Future studies should also differentiate self-harming behavior from thoughts of self-harm.

Measurement issues in the literature

Various measurement tools and timeframes were used for suicidal ideation, attempts, and self-harm, as well as risk and protective factors. This diversity introduced complexity in synthesizing and comparing research findings. Some studies also measured certain outcomes but did not report them. Furthermore, in several studies ( 28 , 30 , 35 ), the outcome measures were worded in a way that made it impossible to distinguish between different suicide or self-harm related outcomes, for example, using questions like “Have you considered or attempted suicide?” (Yes/No). As such, these studies could not be included in our synthesis of prevalence rates or risk and protective factors. Future studies should use validated assessment tools that clearly differentiate between suicidal ideation, suicide attempts, and self-harming behaviors ( 52 ).

When reporting the intensity of suicidal ideation, studies presented average scores on various instruments. These average scores were often low due to most participants in the sample reporting no ideation. Because the average score is skewed by the large number of zero scores, it is impossible to discern if the participants who scored above zero reported low or high levels of ideation. Future studies should consider reporting suicidal ideation data for only those who scored above zero to better capture intensity for those experiencing suicidal thoughts. Addressing these limitations would facilitate an improved understanding of suicidal thoughts and behaviors in international student cohorts.

Implications and future directions

This review showed consistent evidence for social connectedness as a protective factor against suicidal ideation. Promoting social connectedness is already part of university-wide suicide prevention frameworks ( 53 ). However, tailored approaches may be required for international students given that they are separated from their social support networks from their home country and may struggle to make new connections due to language barriers or cultural differences ( 54 ). Students’ sense of connectedness can be improved by socializing with co-nationals and other international students ( 54 ), and as such, universities should continue investing in social activities designed to help international students connect with peers.

Similarly, our review identified common mental health concerns like depression and anxiety as risk factors for suicidal thoughts and behaviors. Access to mental health care is imperative in reducing these symptoms and subsequently lowering the risk of suicide ( 55 ). However, it is well-documented that international students underutilize mental health services despite needing them ( 11 ). Evidence suggests there are multiple barriers to service access for international students, including low mental health literacy, a lack of culturally appropriate or accessible services, and high costs ( 13 ). These are all areas for improved prevention. For instance, international students are often unaware that mental health support is covered by their health insurance, which may be a barrier to help-seeking due to the assumed cost of services ( 56 ). International student offices and insurers should ensure that students stay adequately informed about the extent of their mental healthcare cover. Furthermore, mental healthcare pathways for international students should be streamlined across universities, community psychological services and emergency departments ( 12 ).

None of the studies included in this review reported on suicide deaths in international students. It is recommended that educational institutions adopt a specific reporting framework for international student suicides ( 57 ). Quantifying the rates of suicide mortality is imperative for tracking the scope of the issue on local and global scales, and guiding prevention efforts, as they can provide direct insight into potential causal factors in in such deaths ( 11 , 58 ).

Additionally, the literature rarely reported the length of time international students have spent in the host country. There is evidence that being new to the country is associated with elevated suicide risk; a coroner’s investigation into international student suicides in Victoria, Australia found that almost half of individuals that took their lives had been in the host country for less than two years ( 12 ). Therefore, future studies should investigate the effect of time spent in the host country on suicide-related outcomes.

Future research should explore suicide-related outcomes among diverse international student subgroups to enhance the generalizability of findings, as the current studies have limited representation of countries and ethnicities. International students are a heterogeneous population, and such comparisons are essential. There is preliminary evidence suggesting potential subgroup differences, particularly that Asian international students report higher rates of suicidal ideation, attempts, and self-harm than other international students ( 40 , 44 ).

There is currently no evidence on effective suicide prevention initiatives for the international student populations ( 13 ). However, based on the findings on risk and protective factors from this review, we suggest that emphasis should be placed on promoting social connectedness and improving student mental health via better access to psychology services. Combatting stigma and discrimination that international students experience and managing academic and general life stress are also areas for prevention. Generally, suicide prevention efforts will be most effective when undertaken collaboratively across educational institutions, mental health services, government, and other organizations relevant to international students ( 13 ). It is also essential to involve international students and those who work closely with them in designing and implementing suicide prevention initiatives ( 56 ).

Strengths and limitations

To authors’ knowledge, this is the first publication to summarize the rates of suicidal ideation, attempts and self-harm in international students across the existing body of literature. Furthermore, this review describes prominent risk and protective factors for suicidal thoughts and behaviors in international students and highlights gaps for future research in this area.

The main limitation of this review is that currently there is not enough available data to draw any major conclusions regarding prevalence rates or factors contributing to suicidality in this student group. It was also not possible to conduct a meta-analysis to quantify the available data due to the diverse range of measures in the primary studies. In line with this, the available literature had varying measures of suicidal thoughts and behaviors, and varying overall methodology, meaning that the quality of data may vary between studies. Finally, it is possible that some studies relevant to this review were not captured by our search strategy.

This review describes the prevalence rates, and risk and protective factors, for suicidal ideation, attempts, and self-harm in international students. Overall, there is little literature on this topic, limiting our ability to draw conclusions. However, in general, the literature suggests that international students may experience similar rates of suicidal ideation and self-harm, and higher rates of attempts compared to domestic students. Key risk factors include social isolation, intrapersonal factors (e.g., depression, anxiety), and contextual factors (e.g., perceived discrimination). More high-quality research featuring robust measurement is needed to adequately capture the prevalence of suicide-related outcomes in international students. Future research can also focus on subgroup comparisons within international students, and comparisons with domestic students. Risk and protective factors should be investigated further, particularly via longitudinal designs to establish causal relationships. Finally, this review provides directions for suicide prevention initiatives for international students, focusing on promoting social connectedness, improving student mental health, and investing in accessible and culturally sensitive mental health care. Given that international students comprise a significant portion of tertiary students worldwide and experience substantial levels of suicidal thoughts and behaviors, it is important that they receive targeted, appropriate, and effective support.

Data availability statement

The original contributions presented in the study are included in the article/ Supplementary Material . Further inquiries can be directed to the corresponding author.

Author contributions

MV: Formal analysis, Investigation, Writing – original draft, Writing – review & editing, Data curation. ML: Project administration, Resources, Writing – review & editing. JR: Conceptualization, Funding acquisition, Writing – review & editing. SM: Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Project administration, Writing – review & editing.

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2024.1358041/full#supplementary-material

1. World Health Organization. Suicide (2021). Available online at: https://www.who.int/news-room/fact-sheets/detail/suicide .

Google Scholar

2. Australian Institute of Health and Welfare. Suicide & self-harm monitoring (2022). Available online at: https://www.aihw.gov.au/suicide-self-harm-monitoring/data/suicide-self-harm-monitoring-data .

3. Centers for Disease Control and Prevention & National Center for Injury Prevention and Control. Centers for disease control and prevention. In: Suicide Data and Statistics (Atlanta, USA: Centers for Disease Control and Prevention) (2023). Available at: https://www.cdc.gov/suicide/suicide-data-statistics.html .

4. Mortier P, Auerbach RP, Alonso J, Bantjes J, Benjet C, Cuijpers P, et al. Suicidal thoughts and behaviors among first-year college students: results from the WMH-ICS project. J Am Acad Child Adolesc Psychiatry . (2018) 57:263–273.e1. doi: 10.1016/j.jaac.2018.01.018

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Organization for Economic Co-operation and Development. Education at a Glance 2020 . Paris, France: Organization for Economic Co-operation and Development (2020).

6. UNESCO Institute for Statistics. Internationally mobile students - Definition (2023). Available online at: https://uis.unesco.org/en/glossary-term/internationally-mobile-students .

7. Khanal J, Gaulee U. Challenges of international students from pre-departure to post-study: A literature review. J Int Students . (2019) 9:560–81. doi: 10.32674/jis.v9i2.673

CrossRef Full Text | Google Scholar

8. Zhang J, Goodson P. Predictors of international students’ psychosocial adjustment to life in the United States: A systematic review. Int J Intercultural Relations . (2011) 35:139–62. doi: 10.1016/j.ijintrel.2010.11.011

9. Minutillo S, Cleary M, Hills AP, Visentin D. Mental health considerations for international students. Issues Ment Health Nurs . (2020) 41:494–9. doi: 10.1080/01612840.2020.1716123

10. Xiong Y, Yang L. Asian international students’ help-seeking intentions and behavior in American Postsecondary Institutions. Int J Intercultural Relations . (2021) 80:170–85. doi: 10.1016/j.ijintrel.2020.11.007

11. McGregor S. Findings into Death of student AHT . Australia: Coroners Court of Victoria (2023).

12. Jamieson A. Findings into Death of Nguyen Pham Dinh Le . Victoria, Australia: Coroners Court of Victoria (2021). Available at: https://www.coronerscourt.vic.gov.au/sites/default/files/2021-01/Nguyen_186222.pdf .

13. McKay S, Veresova M, Bailey E, Lamblin M, Robinson J. Suicide prevention for international students: A scoping review. Int J Environ Res Public Health . (2023) 20:1500. doi: 10.3390/ijerph20021500

14. Black MH, Scott M, Baker-Young E, Thompson C, McGarry S, Hayden-Evans M, et al. Preventing suicide in post-secondary students: a scoping review of suicide prevention programs. Eur Child Adolesc Psychiatry . (2023) 32:735–71. doi: 10.1007/s00787-021-01858-8

15. Bowden M, McCoy A, Reavley N. Suicidality and suicide prevention in culturally and linguistically diverse (CALD) communities: A systematic review. Int J Ment Health . (2020) 49:293–320. doi: 10.1080/00207411.2019.1694204

16. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Rev . (2015) 4. doi: 10.1186/2046-4053-4-1

17. Rodgers M, Sowden A, Petticrew M, Arai L, Roberts H, Britten N. Testing methodological guidance on the conduct of narrative synthesis in systematic reviews: effectiveness of interventions to promote smoke alarm ownership and function. Evaluation . (2009) 15:47–71. doi: 10.1177/1356389008097871

18. Covidence Systematic Review Software. Veritas Health Innovation . Melbourne, Australia, Veritas Health Innovation (2023).

19. Kmet LM, Cook LS, Lee RC. HTA Initiative # 13: Standard Quality Assessment Criteria for Evaluating Primary Research Papers from a Variety of Fields, Alberta Heritage Foundation for Medical Research: Alberta, Canada . (Alberta, Canada: Alberta Heritage Foundation for Medical Research) (2004).

20. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med . (2012) 22:276–82. doi: 10.11613/issn.1846-7482

21. Ahorsu DK, Pramukti I, Strong C, Wang H-W, Griffiths MD, Lin C-Y, et al. COVID-19-related variables and its association with anxiety and suicidal ideation: differences between international and local university students in Taiwan. Psychol Res Behav Manage . (2021) 14:1857–66. doi: 10.2147/PRBM.S333226

22. Bi K, Yeoh D, Jiang Q, Wienk MNA, Chen S. Psychological distress and everyday discrimination among Chinese international students one year into COVID-19: a preregistered comparative study. Anxiety Stress Coping . (2022) 36(6):1–16. doi: 10.1080/10615806.2022.2130268

23. Goodwill JR, Zhou S. Association between perceived public stigma and suicidal behaviors among college students of color in the U.S. J Affect Disord . (2020) 262:1–7. doi: 10.1016/j.jad.2019.10.019

24. Hong V, Busby DR, O’Chel S, King CA. University students presenting for psychiatric emergency services: Socio-demographic and clinical factors related to service utilization and suicide risk. J Am Coll Health . (2022) 70:773–82. doi: 10.1080/07448481.2020.1764004

25. Jun HH, Wang KT, Suh HN, Yeung JG. Family profiles of maladaptive perfectionists among Asian international students. Couns Psychol . (2022) 50:649–73. doi: 10.1177/00110000221089643

26. King N, Rivera D, Cunningham S, Pickett W, Harkness K, McNevin SH, et al. Mental health and academic outcomes over the first year at university in international compared to domestic Canadian students. J Am Coll Health: J ACH . (2021) 71(9):1–10. doi: 10.1080/07448481.2021.1982950

27. Kivelä L, Mouthaan J, van der Does W, Antypa N. Student mental health during the COVID-19 pandemic: Are international students more affected? J Am Coll Health . (2022) 14:1–9. doi: 10.1080/07448481.2022.2037616

28. Li M, Su H, Liao Z, Qiu Y, Chen Y, Zhu J, et al. Gender differences in mental health disorder and substance abuse of chinese international college students during the COVID-19 pandemic. Front Psychiatry . (2021) 12:710878. doi: 10.3389/fpsyt.2021.710878

29. Low YS, Bhar SS, Chen WS. Moderators of suicide ideation in Asian international students studying in Australia. Aust Psychol . (2023) 58:169–78. doi: 10.1080/00050067.2022.2148514

30. Maffini CS. Campus safety experiences of Asian American and Asian international college students. Asian Am J Psychol . (2018) 9:98–107. doi: 10.1037/aap0000087

31. Nadareishvili I, Syunyakov T, Smirnova D, Sinauridze A, Tskitishvili A, Tskitishvili A, et al. University students’ mental health amidst the COVID-19 pandemic in Georgia. Int J Soc Psychiatry . (2022) 68:1036–46. doi: 10.1177/00207640221099420

32. Nguyen MH, Le TT, Nguyen HKT, Ho MT, Thanh Nguyen HT, Vuong QH. Alice in suicideland: Exploring the suicidal ideation mechanism through the sense of connectedness and help-seeking behaviors. Int J Environ Res Public Health . (2021) 18:3681. doi: 10.3390/ijerph18073681

33. Perez-Rojas AE, Choi N-Y, Yang M, Bartholomew TT, Perez GM. Suicidal ideation among international students: The role of cultural, academic, and interpersonal factors. Couns Psychol . (2021) 49:673–700. doi: 10.1177/00110000211002458

34. Rosenthal DA, Russell J, Thomson G. The health and wellbeing of international students at an Australian university. Higher Educ . (2008) 55:51–67. doi: 10.1007/s10734-006-9037-1

35. Sanci L, Williams I, Russell M, Chondros P, Duncan AM, Tarzia L, et al. Towards a health promoting university: descriptive findings on health, wellbeing and academic performance amongst university students in Australia. BMC Public Health . (2022) 22:2430. doi: 10.1186/s12889-022-14690-9

36. Servaty-Seib HL, Lockman J, Shemwell D, Reid Marks L. International and domestic students, perceived burdensomeness, belongingness, and suicidal ideation. Suicide Life-threatening Behav . (2016) 46:141–53. doi: 10.1111/sltb.12178

37. Taliaferro LA, Muehlenkamp JJ, Jeevanba SB. Factors associated with emotional distress and suicide ideation among international college students. J Am Coll Health: J ACH . (2020) 68:565–9. doi: 10.1080/07448481.2019.1583655

38. Um MY, Maleku A, Haran H, Kim YK, Yu M, Moon SS. Mask wearing and self-harming thoughts among international students in the United States during COVID-19: The moderating role of discrimination. J Am Coll Health: J ACH . (2022) 26:1–7. doi: 10.1080/07448481.2022.2103378

39. Wang KT, Wong YJ, Fu C-C. Moderation effects of perfectionism and discrimination on interpersonal factors and suicide ideation. J Couns Psychol . (2013) 60:367–78. doi: 10.1037/a0032551

40. Xiong Y, Pillay Y. A national study of the mental health status of Asian international students in the United States. J Multicultural Couns Dev . (2022) 51:82–91. doi: 10.1002/jmcd.12238

41. Yang B, Clum GA. Life stress, social support, and problem-solving skills predictive of depressive symptoms, hopelessness, and suicide ideation in an Asian student population: A test of a model. Suicide Life-Threatening Behav . (1994) 24:127–39. doi: 10.1111/j.1943-278X.1994.tb00797.x

42. Yang B, Clum GA. Measures of life stress and social support specific to an Asian student population. J Psychopathol Behav Assess . (1995) 17:51–67. doi: 10.1007/BF02229203

43. Yeung TS, Hyun S, Zhang E, Wong F, Stevens C, Liu CH, et al. Prevalence and correlates of mental health symptoms and disorders among US international college students. J Am Coll Health: J ACH . (2022) 70:2470–5. doi: 10.1080/07448481.2020.1865980

44. Zhou X, Zhou AQ, Sun X. Prevalence of common mental concerns and service utilization among international students studying in the U.S. Counselling Psychol Q . (2022) 35:483–502. doi: 10.1080/09515070.2021.1875400

45. Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: The PHQ primary care study. Primary care evaluation of mental disorders. Patient Health Questionnaire. JAMA . (1999) 282:1737–44. doi: 10.1001/jama.282.18.1737

46. Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, et al. The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry . (2011) 168:1266–77. doi: 10.1176/appi.ajp.2011.10111704

47. Beck AT, Steer RA, Ranieri WF. Scale for Suicide Ideation: Psychometric properties of a self-report version. J Clin Psychol . (1988) 44:499–505. doi: 10.1002/(ISSN)1097-4679

48. Rudd MD. The prevalence of suicidal ideation among college students. Suicide Life-Threatening Behav . (1989) 19:173–83. doi: 10.1111/j.1943-278X.1989.tb01031.x

49. Joiner TE. Why People Die By Suicide . Cambridge, MA: Harvard University Press (2005).

50. Mortier P, Cuijpers P, Kiekens G, Auerbach RP, Demyttenaere K, Green JG, et al. The prevalence of suicidal thoughts and behaviors among college students: a meta-analysis. psychol Med . (2018) 48:554–65. doi: 10.1017/S0033291717002215

51. Riboldi I, Capogrosso CA, Piacenti S, Calabrese A, Lucini Paioni S, Bartoli F, et al. Mental health and COVID-19 in university students: findings from a qualitative, comparative study in Italy and the UK. Int J Environ Res Public Health . (2023) 20:4071. doi: 10.3390/ijerph20054071

52. Batterham PJ, Ftanou M, Pirkis J, Brewer JL, Mackinnon AJ, Beautrais A, et al. A systematic review and evaluation of measures for suicidal ideation and behaviors in population-based research. psychol Assess . (2015) 27:501. doi: 10.1037/pas0000053

53. MacPhee J, Modi K, Gorman S, Roy N, Riba E, Cusumano D, et al. Strengthening safety nets: A comprehensive approach to mental health promotion and suicide prevention for colleges and universities. NAM Perspect . (2021). doi: 10.31478/namperspectives

54. Smith RA, Khawaja NG. A review of the acculturation experiences of international students. Int J Intercultural Relations . (2011) 35:699–713. doi: 10.1016/j.ijintrel.2011.08.004

55. Bear HA, Edbrooke-Childs J, Norton S, Krause KR, Wolpert M. Systematic review and meta-analysis: outcomes of routine specialist mental health care for young people with depression and/or anxiety. J Am Acad Child Adolesc Psychiatry . (2020) 59:810–41. doi: 10.1016/j.jaac.2019.12.002

56. Orygen. International Students and Their Mental and Physical Safety . Victoria, Australia: Orygen (2020).

57. Council of International Students Australia. Council of International Student Australia’s Productivity Commission Recommendations (2019). Available online at: https://www.pc.gov.au/:data/assets/pdf_file/0015/251142/sub893-mental-health.pdf .

58. Hill NT, Witt K, Rajaram G, McGorry PD, Robinson J. Suicide by young Australians 2006–2015: a cross-sectional analysis of national coronial data. Med J Aust . (2021) 214:133–9. doi: 10.5694/mja2.50876

Keywords: international students, suicide, suicide prevention, self-harm, risk factors, protective factors, systematic review

Citation: Veresova M, Lamblin M, Robinson J and McKay S (2024) A systematic review and narrative synthesis of prevalence rates, risk and protective factors for suicidal behavior in international students. Front. Psychiatry 15:1358041. doi: 10.3389/fpsyt.2024.1358041

Received: 20 December 2023; Accepted: 28 February 2024; Published: 14 March 2024.

Reviewed by:

Copyright © 2024 Veresova, Lamblin, Robinson and McKay. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Maria Veresova, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Open access
Published: 29 November 2021

A systematic review of the effects of psychiatric medications on social cognition

Zoë Haime ORCID: orcid.org/0000-0001-6599-2176 1 ,
Andrew J. Watson 2 ,
Nadia Crellin 1 ,
Louise Marston 3 ,
Eileen Joyce 2 &
Joanna Moncrieff 1

BMC Psychiatry volume 21 , Article number: 597 ( 2021 ) Cite this article

4403 Accesses

10 Citations

23 Altmetric

Metrics details

Introduction

Social cognition is an important area of mental functioning relevant to psychiatric disorders and social functioning, that may be affected by psychiatric drug treatments. The aim of this review was to investigate the effects of medications with sedative properties, on social cognition.

This systematic review included experimental and neuroimaging studies investigating drug effects on social cognition. Data quality was assessed using a modified Downs and Black checklist (Trac et al. CMAJ 188: E120-E129, 2016). The review used narrative synthesis to analyse the data.

40 papers were identified for inclusion, 11 papers investigating benzodiazepine effects, and 29 investigating antipsychotic effects, on social cognition.

Narrative synthesis showed that diazepam impairs healthy volunteer’s emotion recognition, with supporting neuroimaging studies showing benzodiazepines attenuate amygdala activity. Studies of antipsychotic effects on social cognition gave variable results. However, many of these studies were in patients already taking medication, and potential practice effects were identified due to short-term follow-ups.

Healthy volunteer studies suggest that diazepam reduces emotional processing ability. The effects of benzodiazepines on other aspects of social cognition, as well as the effects of antipsychotics, remain unclear. Interpretations of the papers in this review were limited by variability in measures, small sample sizes, and lack of randomisation. More robust studies are necessary to evaluate the impact of these medications on social cognition.

Peer Review reports

Introduction (narrative synthesis element 1: theory development)

What is social cognition.

Social cognition is defined as the mental processes which underlie the ability to understand and act on the thoughts, intentions, and behaviours of others [ 1 ]. Deficits in social cognition have been found in psychiatric disorders including depression, schizophrenia and bipolar disorder [ 2 , 3 , 4 ] and can lead to the faulty interpretation of others’ intentions and thinking, as well as inaccuracies in identifying others’ emotions [ 5 ].

Social cognition can be separated into individual testable domains. However, many of these domains overlap, and there is no consistent agreement between cognitive scientists as to which are the most important. In psychiatry research the domains most frequently studied tend to reflect those identified by the National Institute of Mental Health (NIMH) at their meeting to define social cognition in schizophrenia in 2006 [ 6 , 7 ]. These domains can be seen in Fig. 1 , and include: Theory of mind (ToM) - the ability to ‘infer intentions, dispositions, and beliefs of others’ [ 8 ] ; emotion processing - the ability to perceive emotions and interpret them appropriately [ 9 ] ; social perception - the ability to process social cues and context to decipher social situations [ 10 ]; attributional bias - how people interpret the causes of events, which may be positive or negative in nature [ 11 ]; and social knowledge – how mental schemas of social situations guide behaviour [ 12 ]. Additional domains of social cognition tested in research include emotional intelligence, prejudice and stereotyping, and empathy [ 6 ].

Social Cognition Domains identified by the NIMH

Social functioning deficits are a core feature of most psychiatric diagnoses and considered an integral treatment target for many conditions, in order to promote recovery [ 13 , 14 ]. Social cognition deficits have been associated with poor social functioning in several psychiatric disorders including schizophrenia, bipolar disorder, anorexia, Alzheimer’s disorder, and depression [ 2 , 3 , 15 , 16 , 17 ]. Additionally, in schizophrenia better social cognitive ability has been linked to better social functioning outcomes [ 18 ]. This makes social cognition a potential target for treatment interventions across psychiatric care.

Psychiatric medications

The existing studies showing social cognition deficits in psychiatric disorders often involve people who are currently taking psychiatric medication [ 19 ]. However, it is unclear how these medications might affect social cognition. Many psychiatric medications have sedative effects, including benzodiazepines, antipsychotics (to varying degrees) and drugs used as mood stabilisers (22), and evidence suggests these drugs impair neurocognitive functioning in volunteers [ 20 , 21 , 22 , 23 ]. Antipsychotics also impair cognitive functioning in people with Alzheimer’s disease [ 24 ], but evidence on the effects of antipsychotics in people diagnosed with schizophrenia is inconclusive. Some evidence suggests that antipsychotics improve neurocognitive functioning [ 25 , 26 ] and some that they impair it [ 27 , 28 ].

Along with sedative effects, psychiatric medications affect emotion and motivation. Antipsychotics, for example, reduce motivation and suppress emotions in volunteers [ 29 ], effects which are also reported by patients [ 30 ], and these effects may impact on social cognition. On the other hand, psychiatric drugs may improve social cognition by alleviating symptoms that impair social interaction, such as psychotic symptoms and anxiety. Moreover, different agents within the same class may have different effects on social cognition, depending on their sedative profile and other effects [ 31 ].

Therefore, there is good reason to believe that psychiatric medications may influence social cognition, especially those with sedative actions that are known to impair neurocognitive functioning in volunteers. Clarifying these effects is important in order to understand the nature of social cognitive deficits in psychiatric disorders, and to evaluate the effects of treatment on social cognition and associated outcomes, such as social functioning. A previous review highlighted the paucity of evidence on the effects of antipsychotic treatment, but it did not explore the use of other medications or effects in volunteers [ 31 ]. Volunteer studies help to distinguish those effects that occur in the absence of symptoms of psychiatric disorders from those that are related to the disorder itself, or to the interaction of the treatment with the disorder. They can help with the interpretation of studies with patients who are taking medication, and ultimately improve our understanding of this complex area.

Neuroimaging

The realisation that social disability may be linked to cognitive dysfunction has led to the employment of neuroimaging techniques to study this phenomenon in psychiatric populations, including the use of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI). EEG can identify temporal changes in brain activity in response to specific tasks via event-related potentials (ERP), and fMRI is used to detect the location of changes in brain activity via variations in blood-oxygen-level-dependence (BOLD) [ 32 , 33 ]. ERPs typically associated with social cognitive emotional stimuli are the P300-P400 potentials, where the brain shows activations 300-400 ms post-stimulus [ 34 ]. In fMRI, a social cognitive brain network has been identified and includes consistent activation of regions, including the temporo-parietal junction (TPJ), anterior cingulate cortex (ACC), superior temporal sulcus (STS), ventral and dorsal medial prefrontal cortex (VMPFC and DMPFC), precuneus, and inferior frontal gyrus (IFG) [ 35 ]. Neuroimaging studies investigating the effects of sedative medications on social cognition will help to identify any temporal or spatial neural changes in social cognitive brain regions, as a result of medication effects. This research is important in allowing researchers to assess the biological impact of psychiatric pharmaceutical treatments. In studies where patients have been using psychiatric medications with sedative effects long-term, permanent changes to structural and functional brain systems may inhibit the identification of medication effects on social cognition, Therefore, healthy volunteer and drug-naïve patient studies will be integral to our understanding of medication effects on social cognition in neuroimaging studies.

Despite evidence of effects on neurocognitive functioning, there has been little consideration of how psychiatric medications affect social cognition. We hypothesised that psychiatric medications that produce sedative effects might affect social cognition, and we conducted a systematic review of the literature in this area. We included research on healthy volunteers as well as research conducted with patients with diagnosed psychiatric disorders.

An additional aim of this review was to explore any temporal or spatial brain differences between healthy volunteers and clinical populations with psychiatric diagnoses conducting social cognition tasks after administration of psychiatric medication using neuroimaging technology. Notable differences in brain activity may reflect the effects of medication on social cognitive processing.

The review will help to clarify the nature of any underlying deficits in social cognition in people diagnosed with psychiatric disorders, and this will help in the development of targeted treatments for social cognition, which may also improve social functioning and general outcomes [ 28 ].

Protocol and registration

This review follows the PRISMA guidelines for reporting systematic reviews [ 36 ]. The review protocol is available on the PROSPERO registry, ID: CRD42018092883.

Narrative synthesis

The scope of our narrative synthesis was to examine the effects of sedative medications on social cognition. Following guidance from Popay et al. [ 37 ] we used four iterative elements shown in Fig. 2 . As the first point suggests, we conducted an initial scoping of the literature to summarise the current research in the field, and in order to construct our search strategy. To address point two, we reported our findings in the results section and summarised relevant data from the included papers in a table (Table 1 ). In our discussion we critically explored relationships between the reported studies and went on to discuss the strengths and limitations of the current review, to address points three and four.

The four iterative elements of narrative synthesis [ 37 ]

Search strategy

We searched the following major databases: MEDLINE (OViD), Embase, Psychinfo, Web of Science, Lilacs, and Scopus as well as grey literature through greylit.org and opengrey.eu. Database-specific search terms included the keywords ‘social cognition’, ‘mental disorder’, ‘neuroleptic agents’, ‘sedatives’, and ‘tranquilisers’ with intervention-specific terms (including names of drug classes, and individual agents in classes that were not included as a whole, e.g., some sedative antidepressants), diagnosis-specific terms, outcome-specific subtypes and synonyms (see Additional file 1 : Appendix A for full list of search terms and search strategy). An attempt to find additional studies was made through a backward reference search and contacting experts in the field.

Inclusion/exclusion criteria

a We included all antipsychotics, benzodiazepines, Z-drugs, and barbiturates. Tricyclic antidepressants, mirtazapine and trazadone were also included, and pregabalin. Drugs that are prescribed for mental disorders but predominantly used for physical health complaints, such as gabapentin and beta-blockers, were excluded

Citations were imported to Mendeley and all duplicates were removed. ZH independently screened all citation titles for their applicability [ 77 ]. Titles that did not meet eligibility criteria were removed. Full-text papers were then screened and any uncertainties about inclusion were discussed with a second reviewer (AJW).

Quality of assessment of studies

Study quality was evaluated using the Downs and Black checklist [ 78 ] as it allows for assessment of both randomised and non-randomised studies. The checklist evaluates papers on reporting, external validity, and internal validity (bias and confounding). It consists of 27 items scored with 0 points for ‘no/unable to determine’, or 1 point for a ‘yes’ response. Item 5 is scored differently with 0 points for a ‘no’ response, 1 point for a ‘partially’ response, or 2 points for a ‘yes’ response. The last item on the checklist regarding power was altered in concordance with a previous review conducted by Trac et al. [ 79 ] to rate whether a power analysis was calculated (1 point), or not (0 points). The maximum score for the checklist was 28, with the scoring ranges being (< 14) poor quality, (15–19) fair quality, (20–25) good quality, and (26–28) excellent quality.

Results (narrative synthesis element 2: developing a preliminary synthesis)

Search results.

The search identified 2931 papers fitting the eligibility criteria, with 2681 remaining after de-duplication. The abstracts and titles of those records were then screened and 2511 were excluded due to not meeting the inclusion criteria. This resulted in 170 papers for full-paper screening. A further 130 papers were excluded during this stage, for reasons shown in Fig. 3 . The remaining 40 full-text papers were used in the narrative synthesis. Data from these papers including study design, sample, medication (name, dosage), pre-intervention treatment, placebo (yes/no), social cognitive domains tested, social cognition measures, follow-up timepoints, key findings, and study limitations, were extracted and can be viewed in Table 1 . Notably there were no studies of mood stabilisers, barbiturates, pregabalin or any sedative antidepressants.

PRISMA flow diagram [ 36 ]

Data quality was rated by ZH on all 40 full-text papers and AJW on 20% randomly selected papers. An interrater reliability score Cohens Kappa Coefficient of 0.85 was calculated [ 80 ], indicating good agreement between authors. Of the 40 included papers, 11 were rated poor and 20 were rated fair. Only 9 total papers were rated good, and none were rated excellent. Scores for each paper are reported in Table 1 .

Benzodiazepine studies

Four benzodiazepine studies conducted in healthy volunteers showed significant impairments in emotion recognition social cognition tasks following diazepam administration [ 39 , 41 , 44 , 81 ], suggesting that sedative medication at a therapeutic dose impairs emotion processing. One of these papers also incidentally investigated the effects of Metropolol, a beta-blocker with mild sedative effects, and found no significant effect of the drug on emotion recognition [ 41 ]. A further study [ 42 ] showed a selective effect of diazepam on recognition of angry expressions only. This result may be due to using a small dose in comparison to other studies. One benzodiazepine study using oxazepam showed no effect on a measure of empathy [ 38 ].

One study was conducted in patients with schizophrenia, which looked at the effects of benzodiazepine withdrawal. Patients who were in the process of withdrawing from benzodiazepines were significantly impaired in recognising negative emotions compared to healthy volunteers, in contrast to patients who had already withdrawn, who were not. However, all patients were likely to have been on other medications [ 81 ].

Neuroimaging studies of benzodiazepines and social cognition

All neuroimaging studies compared social cognition before and immediately after administration of the experimental drug. Del-Ben et al. [ 43 ] showed that a single dose of diazepam in healthy volunteers resulted in attenuated activation of the right amygdala when responding to fearful faces, although no evidence was found for this interaction when participants viewed angry faces. In another healthy volunteer study, Paulus et al. [ 82 ] showed that lorazepam attenuated activation in the amygdala and insula, and that the activation was significantly lower after 1 mg compared to 0.25 mg, suggesting a dose-dependent reaction in emotional processing regions. However, a study by Olofsson et al. [ 45 ] found no interaction between benzodiazepines and EEG activity during response to an affective processing task.

A study investigating benzodiazepine effects on patients with ‘catatonic’ schizophrenia and patients with bipolar disorder found that lorazepam induced BOLD signal decreases in the occipital cortex and medial prefrontal cortex (MPFC) in patients with schizophrenia when undertaking a negative emotion recognition task. This resulted in BOLD patterns resembling those of healthy volunteers taking a placebo drug during the same emotion recognition tasks [ 46 ]. However, at the time of the fMRI task all patients were taking either antipsychotic or antidepressant medications in addition to the administered lorazepam .

Antipsychotic studies

Healthy volunteers.

Only two studies tested the effects of antipsychotics on social cognition in healthy volunteers. A small crossover study by Lawrence et al. [ 47 ] ( N = 14) found that recognition of angry facial expressions was reduced in participants taking sulpiride, but recognition of other expressions was not affected. In addition, a larger randomised parallel group trial of quetiapine versus placebo by Rock et al. [ 48 ] ( N = 27) found no effect of the medication on facial expression recognition, though dropout rates were high (25%) in the quetiapine arm, which may have obscured an effect.

Patient studies

All studies comparing patients with schizophrenia and healthy volunteers found patients performed less well on social cognition tasks whether or not they were taking antipsychotics at baseline [ 49 , 50 , 51 , 52 , 53 , 54 , 56 , 61 , 63 , 68 ]. This included one study with patients who were drug naïve [ 61 ], two studies with patients who were drug-free at study commencement [ 53 , 68 ], studies including participants with a mixture of drug-naïve, drug-free, and previous treatment for under 4 weeks [ 50 , 51 , 52 , 54 , 57 , 72 ], and one study with patients stable on an antipsychotic [ 58 ]. Most longitudinal studies involving people with schizophrenia taking antipsychotics showed improvements in performance on social cognition tasks at follow-up compared to baseline [ 50 , 51 , 52 , 55 , 56 , 59 , 61 , 63 , 64 , 66 , 70 ], although some found no effect [ 49 , 53 , 54 , 58 , 60 , 62 , 65 , 67 ] and one showed a decline [ 68 ].

When studies were classified by the prior medication status of participants, two longitudinal studies involved patients who were previously drug naïve. One of these studies detected improvements on an emotional processing task at follow-up [ 61 ], the other study involved an attributional style task, and found no effects of the medication [ 49 ]. Studies that involved patients who had a prior drug-free period, mostly found improvements in emotion processing tasks [ 50 , 52 , 53 , 70 ], and one in a theory of mind task [ 56 ]. Some studies specified that participants were taking an antipsychotic at baseline prior to switching to another [ 51 , 54 , 55 , 58 , 59 , 60 , 62 , 63 , 64 , 65 , 67 , 68 ] and one did not describe the prior treatment status of participants [ 66 ]. In studies in which people were already taking antipsychotic treatment, results reflect effects of changing the type of antipsychotic rather than starting treatment.

One study tested healthy volunteers at baseline and follow-up to control for practice effects [ 50 ]. It found that patients with schizophrenia treated with antipsychotics (a mixture of people who were previously drug naïve ( n = 11) or drug free ( n = 12)) showed significant improvements in emotion recognition at 6 months compared to healthy volunteers.

One longitudinal study involving 29 people with schizophrenia and 28 with bipolar disorder explored dose-response relationships [ 68 ]. Findings showed that patients with schizophrenia who were taking higher doses of antipsychotic medication had more difficulty recognising sad and neutral facial expressions compared to those taking lower doses at follow-up. In bipolar patients, antipsychotic dose was unrelated to the accuracy of performance in judging emotions.

Studies comparing different antipsychotics produced inconsistent results. Some found that patients treated with second-generation drugs did better than those taking first-generation antipsychotics [ 55 , 59 , 66 , 69 , 70 ], but there was no consistent pattern to the results. Others found no difference between different agents or types of agent [ 57 , 60 , 62 , 65 , 67 , 72 ]. The largest study by Penn et al. [ 64 ] showed improvements in all treatment groups (except for ziprasidone) on an emotion processing task, with no difference between individual second-generation drugs or between first- and second-generation drugs.

One study involving participants with Huntingdon’s disease showed poorer performance on facial recognition tests in those taking antipsychotics compared to those who were not, after controlling for the stage of the disease [ 71 ].

In this review, several studies were conducted by authors who received funding from pharmaceutical companies for research purposes or consulting. One study had a pharmaceutical company provide the medication for the research [ 65 ]. Studies that were conducted by authors who received pharmaceutical company funding found either improvements in social cognition after antipsychotic administration [ 55 , 64 , 73 ], or no effect of the drug on performance [ 60 , 63 , 65 ]. However, improvements were also shown in studies that did not rely on pharmaceutical funding [ 50 , 51 , 52 , 56 , 61 , 66 ].

Neuroimaging studies of patients and social cognition

A study by Sumiyoshi et al. [ 73 ] investigated the effect of the antipsychotic, perospirone, on social perception in schizophrenia patients. They found an increase in the P300 ERP activation in the left pre-frontal cortex (PFC), as well as improvements in the social cognitive script task, after 6 months treatment compared to baseline.

A study investigating the effect of sultopride on emotion processing in healthy volunteers found decreased BOLD responses in the amygdala when viewing negatively valenced stimuli compared to before sultopride administration [ 74 ]. There was also increased activation in the PFC identified during positron emission tomography (PET) scans. However, behaviourally they found minimal changes to performance on social cognition tasks. Additionally, a crossover EEG study by Franken et al. [ 75 ] with healthy volunteers, found that both the dopamine agonist bromocriptine, and antipsychotic haloperidol produced no significant difference in emotion-related ERPs (P300-P400) compared to before drug administration. This study used low doses of medication, however, and some participants were also prescribed domperidone to treat nausea.

Discussion (narrative synthesis element 3: exploring the relationships within and between the studies)

Clarifying the effects of prescribed medication on social cognitive ability is important since social cognition appears to be impaired in people across psychiatric diagnoses, and this impairment may be related to deficits in social functioning that represent a significant disability. Hypothetically, psychiatric drugs may impair social cognition due to their sedative effects, or may, through improving psychiatric symptoms, benefit social cognition.

The findings suggest that psychiatric drugs with sedative properties, such as benzodiazepines, can impair emotion recognition in healthy volunteers [ 39 , 41 , 42 , 44 , 81 ]. Findings were most consistent for emotion processing following the use of diazepam, however few studies were conducted using other benzodiazepines or measures of social cognition. Two neuroimaging studies investigating lorazepam found decreased activation in the social cognitive neural network during emotion processing [ 46 , 82 ]. These findings suggest sedative effects of lorazepam may be altering brain processes required for emotion recognition, although neither study used a behavioural measure to confirm the effects on social cognitive ability. In contrast effects of antipsychotics on healthy volunteers were inconsistent, but only two studies were identified. As antipsychotics have different pharmacological profiles and cause varying levels of sedation, different agents may have different effects. Further research is required to clarify effects of antipsychotics on social cognition in volunteers, especially considering the evidence that antipsychotics impair neurocognitive performance and their reported effects on emotional reactivity.

Results of studies with patient populations found that antipsychotic treatment improves or has no effect on social cognition in patients with schizophrenia. The studies suffered from several important methodological limitations, however. First, practice effects in cognitive tasks are common [ 76 ], and as most studies in this review had short follow-up time windows (averaging 3.2 months) it is expected that improvements would be caused by task memory from earlier sessions. Only one study controlled for practice effects by including a healthy volunteer control group. The study identified practice effects, but also showed an additional improvement in social cognition that was independent of practice effects [ 51 ]. Second, studies did not distinguish the effects of the medication from the effects of changes in symptoms. Symptom improvement may occur as a result of antipsychotic treatment but may also occur spontaneously. One of the present studies detected a correlation between psychotic symptoms and social cognition [ 53 ], but ultimately, placebo-controlled comparisons are needed to reliably detect treatment-specific effects.

In contrast to studies showing improvement in social cognition with antipsychotics, one study on emotion processing identified a negative effect with a dose-response relationship, such that higher doses of antipsychotics related to higher levels of social cognitive impairment in patients with schizophrenia, but this was only found in people diagnosed with schizophrenia and not with bipolar disorder. The study with patients with Huntingdon’s disease also found worse facial recognition performance associated with antipsychotic use, even after adjusting for disease severity [ 71 ]. This is consistent with the evidence of reduced neurocognitive functioning in people with Alzheimer’s following antipsychotic use, but further studies are required to clarify the effects of drugs in other psychiatric conditions.

Patients who experience psychiatric disorders are likely to experience neurocognitive deficits such as poor attention and decision-making skills due to the nature of their symptoms [ 83 ], which may directly prevent successful social cognitive ability [ 27 ]. In addition to this, some research has found that patients with a mental health diagnosis are more likely to have poorer intellectual abilities than the healthy population [ 84 ], which could result in difficulties with language and communication skills. These difficulties may make individuals less experienced or confident in a social environment and have a negative influence on social cognition as a result. The studies examined here confirmed that there is an impairment of social cognition in people experiencing a psychotic episode, even before drug treatment is started. However, the research base is currently not adequate to unravel whether there are additional positive or negative effects associated with the use of psychiatric drugs.

Neuroimaging findings suggest that medication may be affecting brain processes that have been found to be associated with social cognitive ability. Sumiyoshi et al. [ 73 ] found an increase in the P300 ERP during a social perception task in patients with schizophrenia after antipsychotic administration, which was positively correlated with their task performance. However, only 7 of 20 participants started the study drug-free, and 8 participants dropped out after the baseline assessment, making it difficult to make firm conclusions. Takahashi et al’s [ 74 ] study on an affective processing task showed decreased BOLD responses in the amygdala and greater activations in the PFC following antipsychotic administration in healthy volunteers. This was noteworthy as the PFC is known to attenuate amygdala activation during emotional processing [ 85 ]. Therefore, it is possible antipsychotic medication is working directly on the PFC, and decreased amygdala signals are secondary to this.

Strengths and limitations (narrative synthesis element 4: assessment of the robustness of the synthesis)

One of the most important strengths of this review was establishing the current literature on the effects of sedative psychiatric medications on social cognition using a rigorous search strategy of published and unpublished work.

We included all psychiatric populations, and healthy volunteer studies in our search. However, in our review we found the research was largely limited to studies of benzodiazepine effects in healthy volunteers, and studies of antipsychotics in patients with schizophrenia with one study of patients with the neuropsychiatric disorder, Huntingdon’s disease [ 86 ]. Research on neurocognitive function suggests that antipsychotics, in particular, may have specifically detrimental effects in people with psychiatric disorders, such as Alzheimer’s, and further research on their effects on social cognition in people with these disorders would be valuable.

We made efforts to also include all prescribed psychiatric medications with sedative effects in our search, but we may have omitted some medications that are not commonly used. We also excluded drugs that are prescribed for mental disorders but are predominantly used for non-psychiatric indications, such as gabapentin or beta-blockers. We also did not include drugs with sedative effects that are routinely used for physical disorders, such as opiate anaesthetics, for example, and we also did not include recreational sedatives such as alcohol or heroin in our search. The review focused on prescribed psychiatric medication in order to clarify the effects of these medications in people with diagnoses of mental disorders, but recreational drugs are commonly used amongst patients with a mental health diagnosis [ 87 ], and their sedative effects may also influence social cognitive ability. Therefore, this should be an important consideration for future research in the area.

An integral strength of our search for this review was the inclusion of all known terminology for social cognitive domains and measures. However, this was difficult due to the use of interchangeable terms for similar items, exposing a feature of the social cognition field that needs to be addressed.

A limitation of the current review was the poor quality of available studies. Our data quality analysis tool allowed us to identify several deficiencies with current papers available in the field, such as small sample sizes, non-randomised designs, and few adherence to medication measures. Only four of the studies found conducted power analysis to qualify their sample size. This resulted in many of the included studies having small numbers of recruits, undermining the internal and external validity of the research findings. During quality analysis, researchers also found only three of the longitudinal studies included were recording medication adherence. In addition to this, very few studies considered the influence of practice effects, which have an important influence on the results of longitudinal studies of cognitive performance, and there were few randomised placebo-controlled studies that would allow conclusions about whether changes in patients taking antipsychotics were attributable to specific medication effects, or whether they were the result of unrelated symptom improvement or of practice effects. Additionally, only one study of antipsychotics, and no benzodiazepine studies, looked at dose-dependent effects. This variability in studies also restricted analysis of the papers included, making a meaningful meta-analysis impossible to conduct.

One other major limitation of this review was that 80% of the included studies explored emotion processing tasks, leaving the other domains of social cognition largely ignored in the literature. Resultingly, our review is more of an insight into the effects of sedative medications on emotion processing, rather than the broader area of social cognition as a whole.

Finally, benzodiazepines have reasonably consistent effects, but antipsychotics vary widely in their receptor targets, pharmacological actions and sedative profiles [ 47 , 48 ]. The studies examined did not enable a comprehensive comparison of the effects of different agents within any class of drugs. In addition, no studies were found that assessed effects of other prescribed psychiatric drugs with sedative properties, such as mood stabilisers, sedative antidepressants or pregabalin.

Further research

We suggest that further research of higher quality is needed to clarify the effects of sedative medications on social cognition in healthy volunteers and patients with psychiatric diagnoses.

Further studies conducted with neuroimaging techniques will allow better insight into structural or functional brain changes resulting from administration of psychiatric medication with sedative effects. Conducting these studies with a behavioural performance element will also allow researchers to identify if brain changes are consistent with changes in social cognitive ability.

Further studies also need to control for practice effects, and studies involving patients should include placebo or no treatment control groups in order to distinguish the effects of medication from the natural evolution of psychiatric symptoms. Studies should be conducted across a range of social cognition domains, to ensure we are getting an accurate picture of complete social cognitive ability. Additionally, studies should be conducted across a range of psychiatric medications with sedative properties, to ensure we are able to identify any significant differences between drugs, and in different psychiatric diagnoses to clarify the effects of medication across conditions.

Notably, recent research in the field of social cognition and psychiatry has focused on the potential benefits of non- pharmacological interventions, such as social cognition and interaction training (SCIT), metacognitive reflection and insight therapy (MRIT), and metacognitive training. A review in 2009 by Choi and colleagues [ 88 ] found that five intervention studies showed promising results for social cognitive improvements in patients with schizophrenia, and a comprehensive review by Kurtz et al. [ 89 ] showed large effect sizes for training on facial affect recognition, moderate effect sizes on theory of mind, and small effect sizes on attribution bias for patients with schizophrenia. Although some studies included in these reviews used control groups, the majority of studies failed to control for potential medication effects in participants with schizophrenia. Psychiatric medication use alongside a social cognitive training intervention may cause improvements or deficits in participant outcomes, which we consider an important clinical implication in treatment implementation. Therefore, we suggest future research in this area accounts for psychiatric medication use in the analysis of intervention effectiveness.

Deficits in social cognition have been identified in people with psychiatric diagnoses, and are associated with impaired social functioning, yet we remain uncertain to what extent these are attributable to the effects of the disorder or the effects of its treatment. A number of healthy volunteer studies suggest that diazepam and lorazepam can impair emotion processing abilities. Studies on antipsychotics were inconclusive and suffered from methodological limitations. There were no studies on any other drugs with recognised sedative properties, and studies focused mainly on the emotion processing domain of social cognition. Better data on the ability of drugs to affect social cognition will help to improve our understanding of the nature of social cognitive deficits in mental disorders, and the effects of treatment. Optimising the treatment of social cognition could potentially lead to better social functioning outcomes.

Availability of data and materials

Not Applicable.

Ostrom T. In: Wyer Jr RS, Srull TK, editors. The sovereignity of social cognition, in Handbook of social cognition. Hillsdale: Erlbaum; 1984. p. 1–38.

Google Scholar

Ospina LH, Nitzburg GC, Shanahan M, Perez-Rodriguez MM, Larsen E, Latifoglu A, et al. Social cognition moderates the relationship between neurocognition and community functioning in bipolar disorder. J Affect Disord. 2018;235:7–14. https://doi.org/10.1016/j.jad.2018.03.013 .

Weightman MJ, Air TM, Baune BT. A review of the role of social cognition in major depressive disorder. Front Psychiatry. 2014;5:179.

Article Google Scholar

Green MF, Horan WP, Lee J. Social cognition in schizophrenia. Nat Rev Neurosci. 2015;16(10):620–31. https://doi.org/10.1038/nrn4005 .

Article CAS PubMed Google Scholar

Augoustinos M, Walker I, Donaghue N. Social cognition: an integrated introduction. 2nd ed. London: Sage; 2006.

Green MF, Penn DL, Bentall R, Carpenter WT, Gaebel W, Gur RC, et al. Social cognition in schizophrenia: an NIMH workshop on definitions, assessment, and research opportunities. Schizophr Bull. 2008;34(6):1211–20. https://doi.org/10.1093/schbul/sbm145 .

Taylor SE, Fiske S. Social Cognition, from Brains to Culture. 3rd ed. London: SAGE; 2017.

Baron-Cohen S. Theory of mind in normal development and autism. Prisme. 2001;34:174–83.

Lemerise EA, Arsenio WF. An integrated model of emotion processes and cognition in social information processing. Child Dev. 2000;71(1):107–18. https://doi.org/10.1111/1467-8624.00124 .

Hamilton DL, Sherman SJ. Perceiving persons and groups. Psychol Rev. 1996;103(2):336–55. https://doi.org/10.1037/0033-295X.103.2.336 .

Garety PA, Freeman D. Cognitive approaches to delusions: a critical review of theories and evidence. Br J Clin Psychol. 1999;38(2):113–54. https://doi.org/10.1348/014466599162700 .

Spelke ES, Bernier EP, Skerry AE. In: Banaji MR, Gelman SA, editors. Core social Cognition, in navigating the social world: what infants, children, and other species can teach us. New York: Oxford University Press; 2013.

Cohen S. Social relationships and health. Am Psychol. 2004;59(8):676–84. https://doi.org/10.1037/0003-066X.59.8.676 .

Article PubMed Google Scholar

Hannigan B, Bartlett H, Clilverd A. Improving health and social functioning: perspectives of mental health service users. J Ment Health. 1997;6(6):613–20. https://doi.org/10.1080/09638239718473 .

Penn DL, Sanna LJ, Roberts DL. Social cognition in schizophrenia: an overview. Schizophr Bull. 2008;34(3):408–11. https://doi.org/10.1093/schbul/sbn014 .

Article PubMed PubMed Central Google Scholar

Kessels RPC, Waanders-Oude Elferink M, van Tilborg I. Social cognition and social functioning in patients with amnestic mild cognitive impairment or Alzheimer’s dementia. J Neuropsychol. 2020;15(2):186-203.

Adenzato M, Todisco P, Ardito RB. Social cognition in anorexia nervosa: evidence of preserved theory of mind and impaired emotional functioning. PLoS One. 2012;7(8):e44414. https://doi.org/10.1371/journal.pone.0044414 .

Article CAS PubMed PubMed Central Google Scholar

Schmidt SJ, Mueller DR, Roder V. Social cognition as a mediator variable between neurocognition and functional outcome in schizophrenia: empirical review and new results by structural equation modeling. Schizophr Bull. 2011;37(Suppl 2):S41–54. https://doi.org/10.1093/schbul/sbr079 .

Derntl B, Habel U. Deficits in social cognition: a marker for psychiatric disorders? Eur Arch Psychiatry Clin Neurosci. 2011;261(Suppl 2):S145–9. https://doi.org/10.1007/s00406-011-0244-0 .

Deakin JB, Aitken MRF, Dowson JH, Robbins TW, Sahakian BJ. Diazepam produces disinhibitory cognitive effects in male volunteers. Psychopharmacology. 2004;173(1–2):88–97. https://doi.org/10.1007/s00213-003-1695-4 .

Ramaekers JG, Louwerens JW, Muntjewerff ND, Milius H, de Bie A, Rosenzweig P, et al. Psychomotor, cognitive, extrapyramidal, and affective functions of healthy volunteers during treatment with an atypical (amisulpride) and a classic (haloperidol) antipsychotic. J Clin Psychopharmacol. 1999;19(3):209–21. https://doi.org/10.1097/00004714-199906000-00003 .

Veselinovic T, et al. Effects of antipsychotic treatment on cognition in healthy subjects. J Psychopharmacol. 2013;27(4):374–85. https://doi.org/10.1177/0269881112466183 .

Kleykamp BA, Griffiths RR, Mintzer MZ. Dose effects of triazolam and alcohol on cognitive performance in healthy volunteers. Exp Clin Psychopharmacol. 2010;18(1):1–16. https://doi.org/10.1037/a0018407 .

Vigen CL, et al. Cognitive effects of atypical antipsychotic medications in patients with Alzheimer's disease: outcomes from CATIE-AD. Am J Psychiatry. 2011;168(8):831–9. https://doi.org/10.1176/appi.ajp.2011.08121844 .

Keefe RS, et al. Neurocognitive effects of antipsychotic medications in patients with chronic schizophrenia in the CATIE trial. Arch Gen Psychiatry. 2007;64(6):633–47. https://doi.org/10.1001/archpsyc.64.6.633 .

Weickert TW, Goldberg TE, Marenco S, Bigelow LB, Egan MF, Weinberger DR. Comparison of cognitive performances during a placebo period and an atypical antipsychotic treatment period in schizophrenia: critical examination of confounds. Neuropsychopharmacology. 2003;28(8):1491–500. https://doi.org/10.1038/sj.npp.1300216 .

Faber G, Smid HGOM, van Gool AR, Wiersma D, van den Bosch RJ. The effects of guided discontinuation of antipsychotics on neurocognition in first onset psychosis. Eur Psychiatry. 2012;27(4):275–80. https://doi.org/10.1016/j.eurpsy.2011.02.003 .

Husa AP, Moilanen J, Murray GK, Marttila R, Haapea M, Rannikko I, et al. Lifetime antipsychotic medication and cognitive performance in schizophrenia at age 43 years in a general population birth cohort. Psychiatry Res. 2017;247:130–8. https://doi.org/10.1016/j.psychres.2016.10.085 .

Healy D, Farquhar G. Immediate effects of droperidol. Hum Psychopharmacol. 1998;13(2):113–20. https://doi.org/10.1002/(SICI)1099-1077(199803)13:2<113::AID-HUP958>3.0.CO;2-N .

Moncrieff J. The myth of the magic antipsychotic. Ment Health Today. 2008;22:18–9.

Kucharska-Pietura K, Mortimer A. Can antipsychotics improve social cognition in patients with schizophrenia? CNS Drugs. 2013;27(5):335–43. https://doi.org/10.1007/s40263-013-0047-0 .

Teplan M. Fundamentals of EEG measurement. Meas Sci Rev. 2002;2(2):1–11.

Logothetis NK, Wandell BA. Interpreting the BOLD signal. Annu Rev Physiol. 2004;66(1):735–69. https://doi.org/10.1146/annurev.physiol.66.082602.092845 .

Amodio DM, Bartholow BD, Ito TA. Tracking the dynamics of the social brain: ERP approaches for social cognitive and affective neuroscience. Soc Cogn Affect Neurosci. 2014;9(3):385–93. https://doi.org/10.1093/scan/nst177 .

Redcay E, Moraczewski D. Social cognition in context: a naturalistic imaging approach. Neuroimage. 2020;216:116392. https://doi.org/10.1016/j.neuroimage.2019.116392 .

Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009;62(10):1006–12. https://doi.org/10.1016/j.jclinepi.2009.06.005 .

Popay J, et al. Guidance on the conduct of narrative synthesis in systematic reviews. London: Institute for health research; 2006.

Nilsonne G, Tamm S, Golkar A, Sörman K, Howner K, Kristiansson M, et al. Effects of 25mg oxazepam on emotional mimicry and empathy for pain: a randomized controlled experiment. R Soc Open Sci. 2018;4(3). https://doi.org/10.1098/rsos.160607 .

Pringle A, Warren M, Gottwald J, Cowen PJ, Harmer CJ. Cognitive mechanisms of diazepam administration: a healthy volunteer model of emotional processing. Psychopharmacology. 2016;233(12):2221–8. https://doi.org/10.1007/s00213-016-4269-y .

Murphy SE, Downham C, Cowen PJ, Harmer CJ. Direct effects of diazepam on emotional processing in healthy volunteers. Psychopharmacology. 2008;199(4):503–13. https://doi.org/10.1007/s00213-008-1082-2 .

Zangara A, Blair RJ, Curran HV. A comparison of the effects of a beta-adrenergic blocker and a benzodiazepine upon the recognition of human facial expressions. Psychopharmacology. 2002;163(1):36–41. https://doi.org/10.1007/s00213-002-1120-4 .

Blair RJ, Curran HV. Selective impairment in the recognition of anger induced by diazepam. Psychopharmacology. 1999;147(3):335–8. https://doi.org/10.1007/s002130051177 .

Del-Ben CM, et al. Effects of diazepam on BOLD activation during the processing of aversive faces. J Psychopharmacol. 2012;26(4):443–51. https://doi.org/10.1177/0269881110389092 .

Coupland NJ, Singh AJ, Sustrik RA, Ting P, Blair R. Effects of diazepam on facial emotion recognition. J Psychiatry Neurosci. 2003;28(6):452–63.

PubMed PubMed Central Google Scholar

Olofsson JK, Gospic K, Petrovic P, Ingvar M, Wiens S. Effects of oxazepam on affective perception, recognition, and event-related potentials. Psychopharmacology. 2011;215(2):301–9. https://doi.org/10.1007/s00213-010-2141-z .

Richter A, Grimm S, Northoff G. Lorazepam modulates orbitofrontal signal changes during emotional processing in catatonia. Hum Psychopharmacol. 2010;25(1):55–62. https://doi.org/10.1002/hup.1084 .

Lawrence AD, Calder AJ, McGowan SW, Grasby PM. Selective disruption of the recognition of facial expressions of anger. Neuroreport. 2002;13(6):881–4. https://doi.org/10.1097/00001756-200205070-00029 .

Rock PL, Goodwin GM, Wulff K, McTavish SFB, Harmer CJ. Effects of short-term quetiapine treatment on emotional processing, sleep and circadian rhythms. J Psychopharmacol. 2016;30(3):273–82. https://doi.org/10.1177/0269881115626336 .

Mizrahi R, Addington J, Remington G, Kapur S. Attribution style as a factor in psychosis and symptom resolution. Schizophr Res. 2008;104(1–3):220–7. https://doi.org/10.1016/j.schres.2008.05.003 .

Gaebel W, Wolwer W. Facial expression and emotional face recognition in schizophrenia and depression. Eur Arch Psychiatry Clin Neurosci. 1992;242(1):46–52. https://doi.org/10.1007/BF02190342 .

Olivier MR, Killian S, Chiliza B, Asmal L, Schoeman R, Oosthuizen PP, et al. Cognitive performance during the first year of treatment in first-episode schizophrenia: a case-control study. Psychol Med. 2015;45(13):2873–83. https://doi.org/10.1017/S0033291715000860 .

Zhou Z, Zhu Y, Wang J, Zhu H. Risperidone improves interpersonal perception and executive function in patients with schizophrenia. Neuropsychiatr Dis Treat. 2017;13:101–7. https://doi.org/10.2147/NDT.S120843 .

Lewis SF, Garver DL. Treatment and diagnostic subtype in facial affect recognition in schizophrenia. J Psychiatr Res. 1995;29(1):5–11. https://doi.org/10.1016/0022-3956(94)00033-N .

Herbener ES, Hill SK, Marvin RW, Sweeney JA. Effects of antipsychotic treatment on emotion perception deficits in first-episode schizophrenia. Am J Psychiatry. 2005;162(9):1746–8. https://doi.org/10.1176/appi.ajp.162.9.1746 .

Roberts DL, Penn DL, Corrigan P, Lipkovich I, Kinon B, Black RA. Antipsychotic medication and social cue recognition in chronic schizophrenia. Psychiatry Res. 2010;178(1):46–50. https://doi.org/10.1016/j.psychres.2010.04.006 .

Mizrahi R, et al. The effect of antipsychotic treatment on theory of mind. Psychol Med. 2007;37(4):595–601. https://doi.org/10.1017/S0033291706009342 .

Kucharska-Pietura K, Mortimer A, Tylec A, Czernikiewicz A. Social cognition and visual perception in schizophrenia inpatients treated with first-and second-generation antipsychotic drugs. Clin Schizophr Relat Psychoses. 2012;6(1):14–20. https://doi.org/10.3371/CSRP.6.1.2 .

Wolwer W, et al. Facial affect recognition in the course of schizophrenia. Eur Arch Psychiatry Clin Neurosci. 1996;246(3):165–70. https://doi.org/10.1007/BF02189118 .

Gultekin G, Yuksek E, Kalelioglu T, Bas A, Ocek Bas T, Duran A. Differential effects of clozapine and risperidone on facial emotion recognition ability in patients with treatment-resistant schizophrenia. Psychiatry Clin Psychopharmacol. 2017;27(1):19–23. https://doi.org/10.1080/24750573.2017.1293242 .

Article CAS Google Scholar

Maat A, Cahn W, Gijsman HJ, Hovens JE, Kahn RS, Aleman A. Open, randomized trial of the effects of aripiprazole versus risperidone on social cognition in schizophrenia. Eur Neuropsychopharmacol. 2014;24(4):575–84. https://doi.org/10.1016/j.euroneuro.2013.12.009 .

Behere RV, Venkatasubramanian G, Arasappa R, Reddy N, Gangadhar BN. Effect of risperidone on emotion recognition deficits in antipsychotic-naive schizophrenia: a short-term follow-up study. Schizophr Res. 2009;113(1):72–6. https://doi.org/10.1016/j.schres.2009.05.026 .

Harvey PD, et al. Improvement in social competence with short-term atypical antipsychotic treatment: a randomized, double-blind comparison of quetiapine versus risperidone for social competence, social cognition, and neuropsychological functioning. Am J Psychiatry. 2006;163(11):1918–25. https://doi.org/10.1176/ajp.2006.163.11.1918 .

Shi C, Yao Shuqiao S, Xu Y, Shi J, Xu X, Zhang C, et al. Improvement in social and cognitive functioning associated with paliperidone extended-release treatment in patients with schizophrenia: a 24-week, single arm, open-label study. Neuropsychiatr Dis Treat. 2016;12:2095–104. https://doi.org/10.2147/NDT.S112542 .

Penn DL, Keefe RSE, Davis SM, Meyer PS, Perkins DO, Losardo D, et al. The effects of antipsychotic medications on emotion perception in patients with chronic schizophrenia in the CATIE trial. Schizophr Res. 2009;115(1):17–23. https://doi.org/10.1016/j.schres.2009.08.016 .

Sergi MJ, et al. Social cognition [corrected] and neurocognition: effects of risperidone, olanzapine, and haloperidol. Am J Psychiatry. 2007;164(10):1585–92. https://doi.org/10.1176/appi.ajp.2007.06091515 .

Kee KS, Kern RS, Marshall BD Jr, Green MF. Risperidone versus haloperidol for perception of emotion in treatment-resistant schizophrenia: preliminary findings. Schizophr Res. 1998;31(2–3):159–65. https://doi.org/10.1016/S0920-9964(98)00026-7 .

Koshikawa Y, Takekita Y, Kato M, Sakai S, Onohara A, Sunada N, et al. The comparative effects of risperidone long-acting injection and Paliperidone palmitate on social functioning in schizophrenia: a 6-month, open-label, Randomized Controlled Pilot Trial. Neuropsychobiology. 2016;73(1):35–42. https://doi.org/10.1159/000442209 .

Daros AR, Ruocco AC, Reilly JL, Harris MSH, Sweeney JA. Facial emotion recognition in first-episode schizophrenia and bipolar disorder with psychosis. Schizophr Res. 2014;153(1–3):32–7. https://doi.org/10.1016/j.schres.2014.01.009 .

Savina I, Beninger RJ. Schizophrenic patients treated with clozapine or olanzapine perform better on theory of mind tasks than those treated with risperidone or typical antipsychotic medications. Schizophr Res. 2007;94(1–3):128–38. https://doi.org/10.1016/j.schres.2007.04.010 .

Fakra E, Salgado-Pineda P, Besnier N, Azorin JM, Blin O. Risperidone versus haloperidol for facial affect recognition in schizophrenia: findings from a randomised study. World J Biol Psychiatry. 2009;10(4 Pt 3):719–28. https://doi.org/10.1080/15622970701432536 .

Labuschagne I, Jones R, Callaghan J, Whitehead D, Dumas EM, Say MJ, et al. Emotional face recognition deficits and medication effects in pre-manifest through stage-II Huntington's disease. Psychiatry Res. 2013;207(1–2):118–26. https://doi.org/10.1016/j.psychres.2012.09.022 .

Kucharska-Pietura K, Tylec A, Czernikiewicz A, Mortimer A. Attentional and emotional functioning in schizophrenia patients treated with conventional and atypical antipsychotic drugs. Med Sci Monit. 2012;18(1):CR44–9. https://doi.org/10.12659/msm.882202 .

Sumiyoshi T, Higuchi Y, Itoh T, Matsui M, Arai H, Suzuki M, et al. Effect of perospirone on P300 electrophysiological activity and social cognition in schizophrenia: a three-dimensional analysis with sloreta. Psychiatry Res. 2009;172(3):180–3. https://doi.org/10.1016/j.pscychresns.2008.07.005 .

Takahashi H, Yahata N, Koeda M, Takano A, Asai K, Suhara T, et al. Effects of dopaminergic and serotonergic manipulation on emotional processing: a pharmacological fMRI study. Neuroimage. 2005;27(4):991–1001. https://doi.org/10.1016/j.neuroimage.2005.05.039 .

Franken IH, Nijs I, Pepplinkhuizen L. Effects of dopaminergic modulation on electrophysiological brain response to affective stimuli. Psychopharmacology. 2008;195(4):537–46. https://doi.org/10.1007/s00213-007-0941-6 .

Wesnes K, Pincock C. Practice effects on cognitive tasks: a major problem? Lancet Neurol. 2002;1(8):473. https://doi.org/10.1016/S1474-4422(02)00236-3 .

Foeckler, P., V. Henning, and J. Reichelt, Mendeley. 2008: London, UK

Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998;52(6):377–84. https://doi.org/10.1136/jech.52.6.377 .

Trac MH, McArthur E, Jandoc R, Dixon SN, Nash DM, Hackam DG, et al. Macrolide antibiotics and the risk of ventricular arrhythmia in older adults. CMAJ. 2016;188(7):E120–9. https://doi.org/10.1503/cmaj.150901 .

McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276–82. https://doi.org/10.11613/BM.2012.031 .

Zurowska N, et al. Recognition of emotional facial expressions in benzodiazepine dependence and detoxification. Cogn Neuropsychiatry. 2018;23(2):74–87. https://doi.org/10.1080/13546805.2018.1426448 .

Paulus MP, Feinstein JS, Castillo G, Simmons AN, Stein MB. Dose-dependent decrease of activation in bilateral amygdala and insula by lorazepam during emotion processing. Arch Gen Psychiatry. 2005;62(3):282–8. https://doi.org/10.1001/archpsyc.62.3.282 .

Fujii DE, Wylie AM, Nathan JH. Neurocognition and long-term prediction of quality of life in outpatients with severe and persistent mental illness. Schizophr Res. 2004;69(1):67–73. https://doi.org/10.1016/S0920-9964(03)00122-1 .

Hudson C, Chan J. Individuals with intellectual disability and mental illness: a literature review. Aust J Soc Issues. 2016;37(1):31–49. https://doi.org/10.1002/j.1839-4655.2002.tb01109.x .

Banks SJ, Eddy KT, Angstadt M, Nathan PJ, Phan KL. Amygdala-frontal connectivity during emotion regulation. Soc Cogn Affect Neurosci. 2007;2(4):303–12. https://doi.org/10.1093/scan/nsm029 .

Anderson KE, Marder KS. An overview of psychiatric symptoms in Huntington's disease. Curr Psychiatry Rep. 2001;3(5):379–88. https://doi.org/10.1007/s11920-996-0030-2 .

McKee SA. Concurrent substance use disorders and mental illness: bridging the gap between research and treatment. Can Psychol. 2017;58(1):50–7. https://doi.org/10.1037/cap0000093 .

Choi JH, Kim JH, Lee J, Green MF. Social Cognition training for individuals with schizophrenia: a review of targeted interventions. Clin Psychopharmacol Neurosci. 2009;7(2):29–38.

Kurtz MM, Gagen E, Rocha NBF, Machado S, Penn DL. Comprehensive treatments for social cognitive deficits in schizophrenia: a critical review and effect-size analysis of controlled studies. Clin Psychol Rev. 2016;43:80–9. https://doi.org/10.1016/j.cpr.2015.09.003 .

Download references

Acknowledgements

EMJ is supported by the UCLH/UCL NIHR Biomedical Research Centre.

Code availability

This research was funded by North East London Foundation Trust supporting the PhD study of first author ZH.

Author information

Authors and affiliations.

Psychiatry Department, University College London, London, UK

Zoë Haime, Nadia Crellin & Joanna Moncrieff

Institute of Neurology, UCL, London, UK

Andrew J. Watson & Eileen Joyce

Department of Primary Care and Population Health, UCL, London, UK

Louise Marston

You can also search for this author in PubMed Google Scholar

Contributions

ZH, AJW, NC and JM contributed to the design of this review. ZH developed the protocol with supervision from NC and JM. ZH and AJW were involved in conducting the literature search and data extraction. ZH consulted LM on analysis for the systematic review. ZH, AJW, NC, and JM defined the themes of the narrative synthesis. ZH took the lead on writing the manuscript, with comments and support from all authors. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Zoë Haime .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

JM is chief investigator for the NIHR-funded RADAR study of antipsychotic reduction. All other authors do not have any conflict of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: appendix a.

. Literature Search Strategy.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Haime, Z., Watson, A.J., Crellin, N. et al. A systematic review of the effects of psychiatric medications on social cognition. BMC Psychiatry 21 , 597 (2021). https://doi.org/10.1186/s12888-021-03545-z

Download citation

Received : 23 June 2021

Accepted : 14 October 2021

Published : 29 November 2021

DOI : https://doi.org/10.1186/s12888-021-03545-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Social cognition
Psychopharmacology
Schizophrenia
Antipsychotic
Benzodiazepine

BMC Psychiatry

ISSN: 1471-244X

Submission enquiries: [email protected]
General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

Advanced Search
Journal List
Lippincott Williams & Wilkins Open Access

A Systematic Review and Narrative Synthesis: Determinants of the Effectiveness and Sustainability of Measurement-Focused Quality Improvement Trainings

Associated data.

Supplemental Digital Content is Available in the Text.

Introduction:

The ability of health care professionals to measure change is critical for successful quality improvement (QI) efforts. Currently, there are no systematic reviews focusing on continuing education for health care professionals in data skills for QI. The purpose of this systematic review is to define effectiveness and sustainability of QI programs for health care professionals containing a measurement skills component and to identify barriers and facilitators to effectiveness and sustainability.

The systematic review involved study identification, screening, full text review, and data extraction. Four electronics databases and grey literature sources were searched to identify studies published between 2009 and 2019 (11 years). A customized data extraction form was developed. Mixed methods appraisal tool was used for quality assessment and a thematic analysis was conducted for narrative synthesis.

Fifty-three studies from 11 countries were included. Most study designs were quantitative descriptive (17/53) and used a blended learning approach (25/53) combining face-to face and distance learning modes. The programs included basic, intermediate, and advanced data skills concepts. Overall, studies reported positive outcomes for participant reaction, learning, and behavior, but reported variable success in sustainability and spread of QI.

Discussion:

Studies discussed measurement as a key competency for clinical QI. Effectiveness definitions focused on the short-term impact of the programs, whereas sustainability definitions emphasized maintenance of outcomes and skills in the long-term. Factors that influenced effectiveness and sustainability of the included studies were strategic approach to QI, organizational support, intervention design, communication, accountability, leadership support, and learning networks.

Health care organizations worldwide continue to test new systems and ways to enhance health care quality and patient safety. 1 Organizations are using continuing education programs in quality improvement (QI) methodologies to transform care and improve patient safety, reduce variations in care outcomes, and deliver sustainable changes in the health care system. 2 The use of such programs to improve health care has also gained considerable popularity in the health care system. 3 However, the health care system is complex and professional knowledge alone is not enough to engage in QI work to bring about change. 4 Numerous QI training programs have been developed to train health care staff in QI methodology and application.

QI training can improve processes, staff knowledge, and health outcomes. 5 Measurement is an important construct in all QI efforts because unless we measure, it is impossible to demonstrate whether the change has resulted in an improvement or not. 6 For health care staff today, collecting, processing, and understanding data is a part of routine practice. 7 This makes a strong case to train health care staff in quality measurement and to develop their expertise in the use of data. 8

Although there are several systematic reviews evaluating QI training and curricula, 9 – 11 none have focused on the evaluation of measurement for improvement training components. This systematic review differs from previous reviews by focusing on QI curricula and training programs containing a significant data skills component. The concepts of effectiveness and sustainability are critical to assessing the impact of teaching measurement skills to health care staff, but these concepts are underexplored in the QI literature. Effectiveness is a micro concept and refers to the assessment of the usefulness of an output at a certain point, with little reference to context. On the other hand, sustainability is a macro concept which extends over a longer period as the new ways of working or improved outcomes become the norm, with context being an essential element. 12 The purpose of this systematic review is to address this gap in literature and define effectiveness and sustainability of QI programs for health care professionals that have a data for improvement component and to identify the associated barriers and enablers.

Protocol and Registration

Review protocol for this systematic review is registered on PROSPERO (ID: CRD42019122997). This study was approved by the IRB of our institution.

Eligibility Criteria

Studies were included if:

Conducted in health care setting
Intervention was QI-based training and included a measurement component.
Study was about development, evaluation, or implementation of the program
Population was health care staff or postgraduate students
Based on primary research

Studies were excluded if:

There was no measurement for improvement component in intervention
Conference proceedings
Population was undergraduate students

Information Sources

Systematic review protocols were scanned in Prospero and Cochrane library to ensure novelty of the review question. A scoping search of databases was conducted to inform the development of the search strategy. Databases were purposively selected to include health care and education sources. The databases were: PubMed, CINAHL Plus, ERIC (via Pro-Quest), and Web of Science. Grey literature sources included two databases: OAIster and OpenGrey along with websites of leading organizations (see Supplemental File 1, Supplemental Digital Content 1, http://links.lww.com/JCEHP/A103 ). The reference lists of eligible studies were scanned to identify additional papers.

The search strategy (see Supplemental File 2, Supplemental Digital Content 2, http://links.lww.com/JCEHP/A104 ) was optimized toward sensitivity rather than specificity because the scoping search revealed that measurement for improvement was integrated into QI studies rather than being delivered as a standalone training. 13 The authors finalized the search strategy and databases iteratively. The systematic search of the literature was conducted in January 2019 and updated in June 2020. The search was restricted to papers published in last 11 years (Search date: January 1, 2009–December 31, 2019). Foreign language papers with English abstracts were considered at the initial stage but only included in full text review if a complete translation was available.

Study Selection

The systematic review consisted of four stages: study identification, title and abstract screening, full text review, and data extraction. Study screening was completed using Covidence tool. 14 Two reviewers independently conducted title and abstract screening. The reviewers met regularly to resolve disputes. The full text review was also conducted independently by two reviewers and discrepancies resolved via discussion. The two reviewers consulted a third reviewer to assist in making the decision on one paper at the full text review stage. Because the studies were heterogenous, a narrative synthesis was performed.

The database search returned 6184 articles, which were imported into Covidence. The 2499 duplicates were removed, leaving 3685 studies eligible for screening. After screening, 110 studies were shortlisted for full text review. A total of 53 studies were included in the review. The PRISMA flow diagram is presented in Figure Figure1 1 and the checklist is attached in Supplemental Digital Content 3 (see Supplemental File 3, http://links.lww.com/JCEHP/A105 ). 15

An external file that holds a picture, illustration, etc.
Object name is ceh-41-210-g001.jpg

PRISMA Statement. An overview of the study selection process. The 6184 records identified through database searching were screened by 2 reviewers. Nineteen records were identified and included from grey literature sources. Exclusion reasons at each stage are shown.

Assessment of Methodological Quality

The Mixed Methods Appraisal Tool (MMAT) was used to evaluate methodological quality. 16 The validity and reliability of the MMAT has been established and is suitable for appraising mixed method studies. 17 Two reviewers assessed quality independently and results were compared. Studies meeting the screening questions of the MMAT on clarity of research questions and appropriateness of collected data were considered appropriate quality for inclusion. All 53 studies met these criteria and were included in the review. The quality assessment is presented in Table Table1 1 .

Assessment of Methodological Quality Using MMAT *

Data Extraction

Two reviewers completed the data extraction independently. A customized data extraction form was developed (see Supplemental File 4, Supplemental Digital Content 4, http://links.lww.com/JCEHP/A106 ). One reviewer compared the data extraction forms and discrepancies were resolved through discussion between reviewers.

The 53 included studies were published between 2009 and 2019 and set in 11 countries. Most studies (35/53) were based in the United States. Most Study designs were quantitative descriptive (17/53) followed by mixed methods studies (16/53). The population varied widely, ranging from frontline staff, clinical and nonclinical staff, and leaders. A summary of studies is presented in Supplemental File 5, Supplemental Digital Content 5, http://links.lww.com/JCEHP/A131 .

Training Description

Less than half (14/53) of the studies were based on a collaborative approach. Duration of the collaboratives was variable, ranging from 2 months to 72 months. Half of the studies used a blended learning approach (25/53) combining face-to face and distance learning modes, whereas 21 studies relied solely on face-to-face learning modes. Four trainings were delivered online, whereas three studies did not state training modality used. Interventions included multiple training methods; the most common (39/53) one being face-to-face learning sessions. Other methods included teleconferencing (12/53), online modules (10/53), workshops (9/53), webinars (6/53), and emails (6/53).

Curriculum Description

The curricula were summarized into categories of basic, intermediate, and advanced data skills based on complexity of data concepts taught. Figure Figure2 2 summarizes the three categories and highlights the data concepts part of the training and curricula in the included studies. Basic data skills include concepts of measurement and QI knowledge, which are important for all health care staff. Intermediate data skills concepts are useful for staff working in improvement teams, whereas advanced data skills concepts are useful for improvement team leads and advisors.

An external file that holds a picture, illustration, etc.
Object name is ceh-41-210-g002.jpg

Measurement for Improvement Concepts. Summary of basic, intermediate, and advanced measurement and data skills taught in the QI programs.

Study Outcomes

Study outcomes are categorized as participant reaction, participant learning, participant behavior, sustainability, spread, and course design elements (see Supplemental File 6, Supplemental Digital Content 6, http://links.lww.com/JCEHP/A107 ). All studies measuring participation reaction to training and improved learning reported positive outcomes regardless of the study design. Studies measuring participant behavior also reported positive results except two quantitative descriptive studies. 63 , 65 Two randomized control trial studies 24 , 53 and a controlled interrupted time series study 50 reported not achieving the clinical outcomes being measured. A cohort study also reported not achieving the outcome of developing a culture of QI. 42 In spread, one quantitative descriptive study 49 reported no spread of QI methods. For sustainability outcomes, Glasgow et al 37 (Interrupted time series), Doyle et al 30 (Quantitative descriptive), and Cranley et al 26 (Mixed methods) reported a lack of sustainability of QI.

The Role of Measurement

The included studies reported on the role of measurement in QI for tracking progress toward goals and offering a systematic way to test changes to close performance gaps. 20 , 61 Measurement was identified as a key competency for clinical QI 21 to understand variation and improve the design health care. 4 Measurement was used to view data over time and draw conclusions regarding variations. 54 Measurement also played a role in implementation of QI methods 19 and contributed to the success of QI. 49

Knowledge about statistics and statistical process control 4 and additional support for measurement skills was reported by studies as critical. 2 , 33 , 39 Defining clear aims and measuring progress toward them was described as essential for QI. 62 Measurement was used to demonstrate patient outcomes to the host organizations 30 and provide guidance to decision makers. 49 Continuous measurement followed the processes through the project period into daily operations. 4 One study labelled measurement as one of the crucial elements of strategy for QI spread and sustainability. 31 Timely data and measurement are important for assessing progress and evaluation. 30

Challenges in identifying, collecting, and displaying appropriate measures of care impact QI program success. 4 , 18 , 63 Studies cited measurement challenges such as difficulty in obtaining measurable data 34 , 64 and presenting data in run/control chart formats. 35 Many participating hospitals were not equipped for systematic data collection. 31 Data collection and measurement was valued, 29 but perceived to be time consuming by participants. 52 , 53

Defining Effectiveness and Sustainability

The purpose of the review is to define effectiveness and sustainability and identify the barriers and enablers to success, in the context of QI programs with a focus on data and measurement for improvement. There is variability in how effectiveness and sustainability is defined in the studies. Another related concept that emerged was that of spread. It is therefore important to distinguish between effectiveness, spread, and sustainability. Effectiveness and sustainability definitions were extracted as part of the data extraction tool and summarized in Supplemental Digital Content 7 (see Supplemental file 7, http://links.lww.com/JCEHP/A108 ). The aspects addressed by these extracted definitions were then used to synthesize definitions. This was completed via consultation between three reviewers.

Effectiveness definitions focused on the short-term impact of the QI programs and were measured using participant reaction to the program, improved knowledge and skill application of participants, program participation and completion of QI projects by participants, and improvement in clinical outcomes at the end of the intervention period. Sustainability on the other hand, is defined not only as long-term outcomes beyond the intervention period, but also as a continuous process. Spread definitions focused on the diffusion of QI methods, processes, and skills from the intervention setting to nonintervention settings. We synthesized the following definitions of effectiveness, sustainability, and spread for measurement for improvement programs:

Effectiveness

Demonstrating improvement in key process, outcome, or quality measures being tracked, accompanied with an improvement in measurement knowledge, skills, and behaviors of learners during the intervention period.

Sustainability

Ongoing measurement, and development of processes and policies to maintain and improve the achieved gains in outcomes and participant skills and integration of measurement practices into routine after the intervention period, without further support from the trainers.

Active and passive diffusion of measurement skills and practices to areas and staff within and outside the organization that were not exposed to the training intervention.

Barriers and Enablers

A six-phased thematic analysis methodology (familiarization with data, initial coding, identifying themes, reviewing themes, naming themes, and reporting) was used to identify the barriers and enablers of sustainability 66 (see Supplemental file 8, Supplemental Digital Content 8, http://links.lww.com/JCEHP/A109 ). The definitions of effectiveness and sustainability synthesized were used as reference. The coding process was done manually by one reviewer and final themes were discussed and agreed with two other reviewers.

The four themes that emerged in enablers to effectiveness were intervention design, staff engagement, supportive leadership, and organizational support. Intervention design was the most important factor in the effectiveness of the program. Customizing training allows teaching of skills relevant to participant's role. 3 , 31 Considering the implementation context 3 , 39 , 52 , 53 and the challenges and opportunities of the setting 33 , 43 , 62 leads to targeted skill building. 19 A good starting point is to assess the prior knowledge and experience of participants 21 to determine training needs 31 and design a suitable range of resources 57 , 65 corresponding to diversity of experiences and knowledge levels. 64 Offering online modules 44 and online resources 53 also helps bridge this gap.

Intervention effectiveness can be enhanced using multiple learning strategies 21 and evidence-based curricula. 28 An effective intervention is responsive to participant learning styles 65 and improves the training based on feedback. 28 , 64 The best way to learn is by doing 21 and incorporating experiential learning principles 23 through demonstration projects 29 and case studies 65 builds capability. Another aspect of customized content is developing an interdisciplinary and team-based course 46 , 60 , 65 as working in teams prevents participants from becoming overburdened with measurement. 64 Having a participatory, data-driven approach contributes to effectiveness. 19 , 53 Focusing on real-time data increases 20 the program's value as participants can identify gaps in current processes. 23 , 55 Teaching practical data gathering, 21 statistical control charts, 4 data analysis, and comparison contribute to effectiveness. Feedback from fellow participants allows them to learn from each other and adds to effectiveness. 36 , 64 Similarly, informing participants about other team's progress creates healthy competition and prevents redundancy of efforts. 64

Effective coaching also plays an important role. Customized coaching experience through just-in-time coaching 28 and direct onsite, in-person support 33 improves effectiveness. Coaching is more effective when trainers can respond directly to participant concerns. 40 The ability of coaches to provide measurement support 22 in creating data collection processes 27 and data quality troubleshooting 33 adds to effectiveness. Practice facilitation 33 is also an enabler because providing private coaching between learning sessions, 44 ongoing mentorship 21 and tools and resources 39 are valued by participants. Coaches can provide customized feedback and assistance. 29 When participants perceive the training organization to be credible and have a sense of affiliation with it, they consider the training to be more effective. 23

Guiding participants in indicator selection by focusing on establishing clear, realistic, mutually agreed, 18 , 22 and clinically meaningful goals 43 , 47 , 50 is a successful strategy. Encouraging participants to focus on simple solutions 18 , 25 and making small changes 26 , 62 leads to effectiveness. In-person workshops 21 are an effective mode of training as face-to-face contact 33 is preferred by participants. Using technology for designing easy to access, self-paced and self-initiated interventions 57 improves effectiveness.

Successfully engaging health care staff is another important theme in effectiveness. Clinical staff feel empowered when they can identify and address gaps 25 and select relevant QI topics. 41 , 52 Providing dedicated time to participants to attend training sessions 34 , 41 , 42 , 45 , 53 also adds to effectiveness. Demonstrating the value of competency in QI skills 64 and offering maintenance of certification credit 59 also helps in creating enthusiasm among staff. Supportive policies of the organization such as assuring time release recognizes the training as a valued activity. 21 Leadership support is an important factor in the success of such programs. 21

There are four themes in barriers to effectiveness: incompatible intervention design, lack of staff engagement, lack of organizational support, and lack of strategic approach. Fast pace of collaboratives 18 and didactic instruction 30 which did not correspond to learning needs of all participants, especially those in support roles 3 were perceived as barriers. The number of concepts covered in the training made it difficult for participants to keep up and the terminology used was sometimes difficult to understand. 29 A single day of classroom training was an insufficient dose 50 and scheduling a full day training workshop is tiring for participants. 63 When training programs that did not incorporate advice on implementation 53 and leading change, 36 it proved to be a barrier to effectiveness.

Lack of organizational support was visible in cases where participants were not provided protected time and struggled to attend the sessions. 3 , 64 Poor data infrastructure impeded data collection 31 and obtaining baseline measures. 34 , 44 Lack of staff engagement and a negative perception about QI work and training because of previous negative experiences 18 , 45 dampened effectiveness. Some programs failed to incorporate appropriate reward systems to motivate behavior 36 and the lack of interest among participants resulted in low attendance 35 , 50 and in some cases, staff disliked new tools and processes that required learning new methods. 39 Staff struggled with learning measurement skills such as presenting data as run charts/control charts 35 , 36 which decreased collaborative effectiveness. Some did not see any value in investing time in such collaboratives 63 and believed the burdens outweighed the benefits. 29 , 36 Another barrier was the lack of a strategic approach and the participants selected projects that were incompatible with the goals of their institutions. 36 , 51

The themes observed in enablers to sustainability were taking a strategic approach, accountability, communication, learning networks, staff engagement, organizational support, intervention design, and supportive leadership. Taking a strategic approach requires connecting the program to organizational and national priorities, 21 strategic goals, 2 , 27 and teaming up with other departments 64 and organizations 53 with similar agendas. 29 As organizations prioritize and implement QI, 57 they move from sporadic efforts toward performance management systems, 5 which sustains learning. Incorporating strategies to address psychology 20 of change improves sustainability. Using a standard approach to QI ensures a common and clear improvement language. 44

Another aspect of sustainability is to recruit the right people in the project team. 4 , 22 A purposeful participant selection strategy 46 ensures inclusion of individuals who are interested in improvement work. Scale-up plans 18 with a goal of institution-wide diffusion 31 add to sustainability. It is important to integrate QI into programs and services 42 , 53 through updated job descriptions, 33 building QI responsibility into operational responsibilities 47 and continually reinforcing skills. 49 Engaging all stakeholders from an early stage 2 , 18 , 19 , 21 , 53 is also an enabler. In addition, while planning evaluations, it is important to assess learner involvement and QI project outcomes beyond completion of the programme. 52 A strategic approach requires taking a system-level view 20 , 21 of improvements with a blameless culture focusing on systems rather than individuals, 24 , 53 which considers challenges as system issues rather than staff issues. 28

Supportive organizational practices encourage QI by removing barriers, 26 investing in workforce capacity and culture change 28 and providing a conducive environment for teamwork. 38 , 45 It also commits resources 3 , 33 , 46 , 56 and provides opportunities to practice the skills learned. 19 , 22 , 27 Accountability is an important enabler for sustainability. A clear definition of responsibilities, 34 tasks 39 and individual roles 62 is key. Establishing time-bound targets 20 and regular meetings to follow through on action 62 ensures accountability. It is also beneficial to establish measurement guidelines to follow the process through the project period into daily operations. 4 This continuous sharing of numbers leads 53 to motivation and boosts sustainability. 4 The training organization can also provide external accountability 33 and ensure participants see projects to completion. 41

Focus on capacity building also improves sustainability. This includes training staff for specialized QI roles such as QI champion, 28 process coach, 31 and QI advisor. 33 A mentorship framework to support those interested in developing QI skills and encouraging permanent staff to develop coaching skills improves sustainability. 47 Effective communication contributes to sustainability. Recognizing the efforts of QI teams 26 , 34 by showcasing success stories 28 through ongoing promotional activities 56 is a rewarding strategy. Senior leader communication through board letters 31 also supports sustainability. Formal and informal dissemination are vital to communication and sustainability. Formal dissemination can include internal dissemination, 31 dissemination to local, national, and international audiences 2 and toolkits. 55 Informal dissemination can include enthusiastic employees 53 and other informal contacts. 31 Similarly, visual display of data and progress helps in disseminating the message of improvement. 59

Learning networks are an important enabler. 53 Learning from peers by sharing ideas 18 , 21 and building relationships creates a strong learning community for idea exchange. 33 These learning platforms serve as venues for knowledge transfer 57 and repositories for QI. 27 Development of collaborations between organizations leads to networking 56 and solution sharing. 33 Another area in staff engagement is generating awareness about QI 18 beyond the project team 2 , 26 , 42 and its impact on career. 22 Extended support from coaches for implementation sequencing 48 improves sustainability.

Support from leaders is crucial to sustainability. 53 This involves improving leaders' QI skills so they can develop infrastructure for QI in their organizations 28 such as establishing QI teams. 53 Senior leadership support 22 including board executives and chief of the medical staff provide legitimacy to QI. 31 A strong leadership structure championing QI on a daily basis 34 sends a message for sustainability. Leadership support allows staff to try new ideas in a safe environment that does not punish risk-taking. 56 Organizational support plays a role through various strategies such as incentivizing diffusion 48 and providing resources and autonomy to innovate. 3

Themes in barriers to sustainability include lack of accountability, poor communication, lack of leadership support, lack of staff engagement, lack of organizational support, absence of learning networks, and not having a strategic approach. When timelines, roles, and responsibilities are not established, the plan of actions can evaporate leading to slippage in agreed timeframes and a loss of momentum. 50 Because of poor institutional communication, staff lack a shared perception of problems 51 and often lack institutional knowledge to approach the relevant individuals for QI work. 60 Lack of leadership support manifests in the form of a lack of interest from top management 4 and variations in the readiness of senior leaders to engage in QI. 46

Learning networks play a vital role in sustainability; however, they are challenging to establish because few practices reach out to others to learn from them 62 and may also face difficulty in learning from practices with dissimilar QI capacity and patients. 29 Lack of organizational support is a major barrier to sustainability 53 because it represents a culture that is not conducive to making or sustaining change. 33 Presence of administrative red tape 3 , 64 can inhibit innovation and indicate that QI is not a priority for the organization. 55

Poor data infrastructure, 4 , 33 , 53 data quality, and access to data 4 decrease sustainability. Repeated data collection can be cumbersome and labor intensive in the long run. 52 Lack of resource availability 3 , 26 , 46 for QI projects is another barrier. Programs that lack ongoing organizational support are likely to be unsuccessful. 65 Health care staff have competing demands on their time, 46 , 62 , 63 , 63 which interferes with team's ability to meet and work. 3 Because QI teams are a disparate group of staff, 50 a lack of dedicated time for QI work 57 can be a barrier. Failing to engage staff, and leadership effectively and not focusing on motivation and behavior change can be a barrier to sustainability. 53 It is also important to account for the high levels of stress and emotional demands experienced by front-line staff. 1

The purpose of the systematic review is to define effectiveness and sustainability of QI programs with a significant data skills component and to identify the relevant barriers and enablers. Fifty-three studies were included in the review. There was heterogeneity in the content, teaching methods, and program design in the included studies and variability in the way effectiveness, sustainability, and spread were defined and measured in the context of QI programs. The review also highlighted variation in the ability of the programs to achieve desired outcomes. These inconsistencies in program success were attributed to various barriers and enablers to effectiveness and sustainability.

The lack of staff engagement, lack of a strategic approach, and lack of organizational support are barriers common between effectiveness and sustainability, which implies that these factors have implications for the short-term and long-term success of the programs. Poor intervention design affects the effectiveness of the program while poor communication, lack of accountability, and lack of leadership support can plague the ability to sustain the skills and results in the long-term. In enablers, intervention design, supportive leadership, engaged staff, and organizational support can affect positively on both effectiveness and sustainability of programs. Enablers that are relevant to sustainability are learning networks, communication, accountability, and a strategic approach to QI. The barriers and enablers highlight the importance of organizational, 39 learner, teacher, curricular, 35 and contextual factors 3 in the success of QI programs.

The definitions derived for effectiveness and sustainability highlight the importance of measurement. Studies reported measurement as a key competency for clinical QI. 21 Continuous measuring and remeasuring play an important role in maintaining 62 and operationalizing improvements in the long run. Selecting appropriate measures, 18 , 22 data collection 34 and using charts to display data 35 are essential to show effective change. 64 QI programs therefore need to focus on training staff in QI methods and how to measure care and use data to drive change. 62 There is an increasing expectation from health care professionals to measure, report, and continually improve the quality of care. 62 This indicates the need for a cultural shift from traditional academic-focused programs toward programs focusing on measurement and results to develop the capability of health professionals in leading improvement. 44

The findings of this systematic review also advocate for program evaluation to consider impact on participant behavior, patient outcomes, and supporting downstream learning beyond the direct participants of the programme. 43 Instead of solely relying on measuring quantitative outcomes, evaluators should also use qualitative data to assess whether program outcomes are achieved. 29

Measurement emerged as a critical element of QI training programs, which enables health care professionals and organizations to demonstrate effectiveness of improvement efforts and sustain improvements in the long run. Training health care professionals in data skills can have implications for improving health systems. However, health care systems are complex and various actors such as the health care authorities, training organizations, trainers, trainees, and trainee's organization have a collaborative role to play in ensuring effectiveness and sustainability of QI programs. Outputs of the thematic analysis in the form of effectiveness and sustainability barriers and enablers were broken down into inputs, outputs, and short- and long-term outcomes, which were then mapped onto a logic model. This was completed via consultation between three reviewers and presented in Figure Figure3 3 .

An external file that holds a picture, illustration, etc.
Object name is ceh-41-210-g003.jpg

Intervention success factors. Summary of QI program inputs contributing to achievement of short-term and long-term outcomes.

Limitations

A limitation of this review is that there were no stand-alone measurement for improvement training studies. The reviewers overcame this by establishing the presence of measurement component in the QI programs as an inclusion criterion. Because no search strategy is perfect, there is a risk of missing relevant studies; however, we mitigated this risk using a search strategy focused on sensitivity and iteratively testing the search strategy in selected databases.

The review highlighted that measuring the improvement in outcomes and participant knowledge establishes effectiveness while remeasuring continuously helps in sustaining outcomes in the long-term for QI programs with a significant measurement skills component. The review identified staff engagement, strategic approach to QI, organizational support, intervention design, communication, accountability, leadership support, and learning networks as factors that affect effectiveness and sustainability. The review expands current knowledge about the importance of measurement in QI training programs. Ensuring effectiveness and sustainability of measurement for improvement programs requires a collective effort from trainers, trainees, the organizations in which the interventions are implemented and policy makers.

Lessons for Practice

■ Measurement has a central role in demonstrating improvements and maintaining desired improvement outcomes of QI programs in the short- and long-term.
■ Staff engagement, strategic approach to QI, organizational support, intervention design, communication, accountability, leadership support, and learning networks influence effectiveness and sustainability of QI programs.
■ Effectiveness, sustainability, and spread of QI programs with a measurement component requires a collective effort from trainers, trainees, the organizations in which the interventions are implemented, and policy makers.

Supplementary Material

The corresponding author receives a PhD funding from the Health Service Executive Ireland (Project reference 57399). The study is also supported by the Irish Health Research Board (RL-2015-1588).

Disclosures: The authors declare no conflict of interest.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's Web site ( www.jcehp.org ).

Open access
Published: 19 March 2024

Interventions, methods and outcome measures used in teaching evidence-based practice to healthcare students: an overview of systematic reviews

Lea D. Nielsen 1 ,
Mette M. Løwe 2 ,
Francisco Mansilla 3 ,
Rene B. Jørgensen 4 ,
Asviny Ramachandran 5 ,
Bodil B. Noe 6 &
Heidi K. Egebæk 7

BMC Medical Education volume 24 , Article number: 306 ( 2024 ) Cite this article

218 Accesses

Metrics details

To fully implement the internationally acknowledged requirements for teaching in evidence-based practice, and support the student’s development of core competencies in evidence-based practice, educators at professional bachelor degree programs in healthcare need a systematic overview of evidence-based teaching and learning interventions. The purpose of this overview of systematic reviews was to summarize and synthesize the current evidence from systematic reviews on educational interventions being used by educators to teach evidence-based practice to professional bachelor-degree healthcare students and to identify the evidence-based practice-related learning outcomes used.

An overview of systematic reviews. Four databases (PubMed/Medline, CINAHL, ERIC and the Cochrane library) were searched from May 2013 to January 25th, 2024. Additional sources were checked for unpublished or ongoing systematic reviews. Eligibility criteria included systematic reviews of studies among undergraduate nursing, physiotherapist, occupational therapist, midwife, nutrition and health, and biomedical laboratory science students, evaluating educational interventions aimed at teaching evidence-based practice in classroom or clinical practice setting, or a combination. Two authors independently performed initial eligibility screening of title/abstracts. Four authors independently performed full-text screening and assessed the quality of selected systematic reviews using standardized instruments. Data was extracted and synthesized using a narrative approach.

A total of 524 references were retrieved, and 6 systematic reviews (with a total of 39 primary studies) were included. Overlap between the systematic reviews was minimal. All the systematic reviews were of low methodological quality. Synthesis and analysis revealed a variety of teaching modalities and approaches. The outcomes were to some extent assessed in accordance with the Sicily group`s categories; “skills”, “attitude” and “knowledge”. Whereas “behaviors”, “reaction to educational experience”, “self-efficacy” and “benefits for the patient” were rarely used.

Conclusions

Teaching evidence-based practice is widely used in undergraduate healthcare students and a variety of interventions are used and recognized. Not all categories of outcomes suggested by the Sicily group are used to evaluate outcomes of evidence-based practice teaching. There is a need for studies measuring the effect on outcomes in all the Sicily group categories, to enhance sustainability and transition of evidence-based practice competencies to the context of healthcare practice.

Peer Review reports

Evidence-based practice (EBP) enhances the quality of healthcare, reduces the cost, improves patient outcomes, empowers clinicians, and is recognized as a problem-solving approach [ 1 ] that integrates the best available evidence with clinical expertise and patient preferences and values [ 2 ]. A recent scoping review of EBP and patient outcomes indicates that EBPs improve patient outcomes and yield a positive return of investment for hospitals and healthcare systems. The top outcomes measured were length of stay, mortality, patient compliance/adherence, readmissions, pneumonia and other infections, falls, morbidity, patient satisfaction, patient anxiety/ depression, patient complications and pain. The authors conclude that healthcare professionals have a professional and ethical responsibility to provide expert care which requires an evidence-based approach. Furthermore, educators must become competent in EBP methodology [ 3 ].

According to the Sicily statement group, teaching and practicing EBP requires a 5-step approach: 1) pose an answerable clinical question (Ask), 2) search and retrieve relevant evidence (Search), 3) critically appraise the evidence for validity and clinical importance (Appraise), 4) applicate the results in practice by integrating the evidence with clinical expertise, patient preferences and values to make a clinical decision (Integrate), and 5) evaluate the change or outcome (Evaluate /Assess) [ 4 , 5 ]. Thus, according to the World Health Organization, educators, e.g., within undergraduate healthcare education, play a vital role by “integrating evidence-based teaching and learning processes, and helping learners interpret and apply evidence in their clinical learning experiences” [ 6 ].

A scoping review by Larsen et al. of 81 studies on interventions for teaching EBP within Professional bachelor-degree healthcare programs (PBHP) (in English undergraduate/ bachelor) shows that the majority of EBP teaching interventions include the first four steps, but the fifth step “evaluate/assess” is less often applied [ 5 ]. PBHP include bachelor-degree programs characterized by combined theoretical education and clinical training within nursing, physiotherapy, occupational therapy, radiography, and biomedical laboratory students., Furthermore, an overview of systematic reviews focusing on practicing healthcare professionals EBP competencies testifies that although graduates may have moderate to high level of self-reported EBP knowledge, skills, attitudes, and beliefs, this does not translate into their subsequent EBP implementation [ 7 ]. Although this cannot be seen as direct evidence of inadequate EBP teaching during undergraduate education, it is irrefutable that insufficient EBP competencies among clinicians across healthcare disciplines impedes their efforts to attain highest care quality and improved patient outcomes in clinical practice after graduation.

Research shows that teaching about EBP includes different types of modalities. An overview of systematic reviews, published by Young et al. in 2014 [ 8 ] and updated by Bala et al. in 2021 [ 9 ], synthesizes the effects of EBP teaching interventions including under- and post graduate health care professionals, the majority being medical students. They find that multifaceted interventions with a combination of lectures, computer lab sessions, small group discussion, journal clubs, use of current clinical issues, portfolios and assignments lead to improvement in students’ EBP knowledge, skills, attitudes, and behaviors compared to single interventions or no interventions [ 8 , 9 ]. Larsen et al. find that within PBHP, collaboration with clinical practice is the second most frequently used intervention for teaching EBP and most often involves four or all five steps of the EBP teaching approach [ 5 ]. The use of clinically integrated teaching in EBP is only sparsely identified in the overviews by Young et al. and Bala et al. [ 8 , 9 ]. Therefore, the evidence obtained within Bachelor of Medicine which is a theoretical education [ 10 ], may not be directly transferable for use in PBHP which combines theoretical and mandatory clinical education [ 11 ].

Since the overview by Young et al. [ 8 ], several reviews of interventions for teaching EBP used within PBHP have been published [ 5 , 12 , 13 , 14 ].

We therefore wanted to explore the newest evidence for teaching EBP focusing on PBHP as these programs are characterized by a large proportion of clinical teaching. These healthcare professions are certified through a PBHP at a level corresponding to a University Bachelor Degree, but with strong focus on professional practice by combining theoretical studies with mandatory clinical teaching. In Denmark, almost half of PBHP take place in clinical practice. These applied science programs qualify “the students to independently analyze, evaluate and reflect on problems in order to carry out practice-based, complex, and development-oriented job functions" [ 11 ]. Thus, both the purpose of these PBHP and the amount of clinical practice included in the educations contrast with for example medicine.

Thus, this overview, identifies the newest evidence for teaching EBP specifically within PBHP and by including reviews using quantitative and/or qualitative methods.

We believe that such an overview is important knowledge for educators to be able to take the EBP teaching for healthcare professions to a higher level. Also reviewing and describing EBP-related learning outcomes, categorizing them according to the seven assessment categories developed by the Sicily group [ 2 ], will be useful knowledge to educators in healthcare professions. These seven assessment categories for EBP learning including: Reaction to the educational experience, attitudes, self-efficacy, knowledge, skills, behaviors and benefits to patients, can be linked to the five-step EBP approach. E.g., reactions to the educational experience: did the educators teaching style enhance learners’ enthusiasm for asking questions? (Ask), self-efficacy: how well do learners think they critically appraise evidence? (Appraise), skills: can learners come to a reasonable interpretation of how to apply the evidence? (Integrate) [ 2 ]. Thus, this set of categories can be seen as a basic set of EBP-related learning outcomes to classify the impact from EBP educational interventions.

Purpose and review questions

A systematic overview of which evidence-based teaching interventions and which EBP-related learning outcomes that are used will give teachers access to important knowledge on what to implement and how to evaluate EBP teaching.

Thus, the purpose of this overview is to synthesize the latest evidence from systematic reviews about EBP teaching interventions in PBHP. This overview adds to the existing evidence by focusing on systematic reviews that a) include qualitative and/ or quantitative studies regardless of design, b) are conducted among PBHP within nursing, physiotherapy, occupational therapy, midwifery, nutrition and health and biomedical laboratory science, and c) incorporate the Sicily group's 5-step approach and seven assessment categories when analyzing the EBP teaching interventions and EBP-related learning outcomes.

The questions of this overview of systematic reviews are:

Which educational interventions are described and used by educators to teach EBP to Professional Bachelor-degree healthcare students?

What EBP-related learning outcomes have been used to evaluate teaching interventions?

The study protocol was guided by the Cochrane Handbook on Overviews of Reviews [ 15 ] and the review process was reported in accordance with The Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement [ 16 ] when this was consistent with the Cochrane Handbook.

Inclusion criteria

Eligible reviews fulfilled the inclusion criteria for publication type, population, intervention, and context (see Table 1 ). Failing a single inclusion criterion implied exclusion.

Search strategy

On January 25th 2024 a systematic search was conducted in; PubMed/Medline, CINAHL (EBSCOhost), ERIC (EBSCOhost) and the Cochrane library from May 2013 to January 25th, 2024 to identify systematic reviews published after the overview by Young et al. [ 8 ]. In collaboration with a research librarian, a search strategy of controlled vocabulary and free text terms related to systematic reviews, the student population, teaching interventions, teaching context, and evidence-based practice was developed (see Additional file 1 ). For each database, the search strategy was peer reviewed, revised, modified and subsequently pilot tested. No language restrictions were imposed.

To identify further eligible reviews, the following methods were used: Setting email alerts from the databases to provide weekly updates on new publications; backward and forward citation searching based on the included reviews by screening of reference lists and using the “cited by” and “similar results” function in PubMed and CINAHL; broad searching in Google Scholar (Advanced search), Prospero, JBI Evidence Synthesis and the OPEN Grey database; contacting experts in the field via email to first authors of included reviews, and by making queries via Twitter and Research Gate on any information on unpublished or ongoing reviews of relevance.

Selection and quality appraisal process

Database search results were merged, duplicate records were removed, and title/abstract were initially screened via Covidence [ 17 ]. The assessment process was pilot tested by four authors independently assessing eligibility and methodological quality of one potential review followed by joint discussion to reach a common understanding of the criteria used. Two authors independently screened each title/abstract for compliance with the predefined eligibility criteria. Disagreements were resolved by a third author. Four authors were paired for full text screening, and each pair assessed independently 50% of the potentially relevant reviews for eligibility and methodological quality.

For quality appraisal, two independent authors used the AMSTAR-2 (A MeaSurement Tool to Assess systematic Reviews) for reviews including intervention studies [ 18 ] and the Joanna Briggs Institute Checklist for systematic reviews and research Synthesis (JBI checklist) [ 19 ] for reviews including both quantitative and qualitative or only qualitative studies. Uncertainties in assessments were resolved by requesting clarifying information from first authors of reviews and/or discussion with co-author to the present overview.

Overall methodological quality for included reviews was assessed using the overall confidence criteria of AMSTAR 2 based on scorings in seven critical domains [ 18 ] appraised as high (none or one non-critical flaw), moderate (more than one non-critical flaw), low (one critical weakness) or critically low (more than one critical weakness) [ 18 ]. For systematic reviews of qualitative studies [ 13 , 20 , 21 ] the critical domains of the AMSTAR 2, not specified in the JBI checklist, were added.

Data extraction and synthesis process

Data were initially extracted by the first author, confirmed or rejected by the last author and finally discussed with the whole author group until consensus was reached.

Data extraction included 1) Information about the search and selection process according to the PRISMA statement [ 16 , 22 ], 2) Characteristics of the systematic reviews inspired by a standard in the Cochrane Handbook (15), 3) A citation index inspired by Young et al. [ 8 ] used to illustrate overlap of primary studies in the included systematic reviews, and to ensure that data from each primary study were extracted only once [ 15 ], 4) Data on EBP teaching interventions and EBP-related outcomes. These data were extracted, reformatted (categorized inductively into two categories: “Collaboration interventions” and “ Educational interventions ”) and presented as narrative summaries [ 15 ]. Data on outcome were categorized according to the seven assessment categories, defined by the Sicily group, to classify the impact from EBP educational interventions: Reaction to the educational experience, attitudes, self-efficacy, knowledge, skills, behaviors and benefits to patients [ 2 ]. When information under points 3 and 4 was missing, data from the abstracts of the primary study articles were reviewed.

Results of the search

The database search yielded 691 references after duplicates were removed. Title and abstract screening deemed 525 references irrelevant. Searching via other methods yielded two additional references. Out of 28 study reports assessed for eligibility 22 were excluded, leaving a total of six systematic reviews. Screening resulted in 100% agreement among the authors. Figure 1 details the search and selection process. Reviews that might seem relevant but did not meet the eligibility criteria [ 15 ], are listed in Additional file 2 . One protocol for a potentially relevant review was identified as ongoing [ 23 ].

PRISMA flow diagram on search and selection of systematic reviews

Characteristics of included systematic reviews and overlap between them

The six systematic reviews originated from the Middle East, Asia, North America, Europe, Scandinavia, and Australia. Two out of six reviews did not identify themselves as systematic reviews but did fulfill this eligibility criteria [ 12 , 20 ]. All six represented a total of 64 primary studies and a total population of 6649 students (see Table 2 ). However, five of the six systematic reviews contained a total of 17 primary studies not eligible to our overview focus (e.g., postgraduate students) (see Additional file 3 ). Results from these primary studies were not extracted. Of the remaining primary studies, six were included in two, and one was included in three systematic reviews. Data from these studies were extracted only once to avoid double-counting. Thus, the six systematic reviews represented a total of 39 primary studies and a total population of 3394 students. Nursing students represented 3280 of these. One sample of 58 nutrition and health students and one sample of 56 mixed nursing and midwife students were included but none from physiotherapy, occupational therapy, or biomedical laboratory scientists. The majority ( n = 28) of the 39 primary studies had a quantitative design whereof 18 were quasi-experimental (see Additional file 4 ).

Quality of systematic review

All the included systematic reviews were assessed as having critically low quality with 100% concordance between the two designed authors (see Fig. 2 ) [ 18 ]. The main reasons for the low quality of the reviews were a) not demonstrating a registered protocol prior to the review [ 13 , 20 , 24 , 25 ], b) not providing a list of excluded studies with justification for exclusion [ 12 , 13 , 21 , 24 , 25 ] and c) not accounting for the quality of the individual studies when interpreting the result of the review [ 12 , 20 , 21 , 25 ].

Overall methodological quality assessment for systematic reviews. Quantitative studies [ 12 , 24 , 25 ] were assessed following the AMSTAR 2 critical domain guidelines. Qualitative studies [ 13 , 20 , 21 ] were assessed following the JBI checklist. For overall classification, qualitative studies were also assessed with the following critical AMSTAR 2 domains not specified in the JBI checklist (item 2. is the protocol registered before commencement of the review, item 7. justification for excluding individual studies and item 13. consideration of risk of bias when interpreting the results of the review)

Missing reporting of sources of funding for primary studies and not describing the included studies in adequate detail were, most often, the two non-critical items of the AMSTAR 2 and the JBI checklist, not met.

Most of the included reviews did report research questions including components of PICO, performed study selection and data extraction in duplicate, used appropriate methods for combining studies and used satisfactory techniques for assessing risk of bias (see Fig. 2 ).

Main findings from the systematic reviews

As illustrated in Table 2 , this overview synthesizes evidence on a variety of approaches to promote EBP teaching in both classroom and clinical settings. The systematic reviews describe various interventions used for teaching in EBP, which can be summarized into two themes: Collaboration Interventions and Educational Interventions.

Collaboration interventions to teach EBP

In general, the reviews point that interdisciplinary collaboration among health professionals and/or others e.g., librarian and professionals within information technologies is relevant when planning and teaching in EBP [ 13 , 20 ].

Interdisciplinary collaboration was described as relevant when planning teaching in EBP [ 13 , 20 ]. Specifically, regarding literature search Wakibi et al. found that collaboration between librarians, computer laboratory technicians and nurse educators enhanced students’ skills [ 13 ]. Also, in terms of creating transfer between EBP teaching and clinical practice, collaboration between faculty, library, clinical institutions, and teaching institutions was used [ 13 , 20 ].

Regarding collaboration with clinical practice, Ghaffari et al. found that teaching EBP integrated in clinical education could promote students’ knowledge and skills [ 25 ]. Horntvedt et al. found that during a six-week course in clinical practice, students obtained better skills in reading research articles and orally presenting the findings to staff and fellow students [ 20 ]. Participation in clinical research projects combined with instructions in analyzing and discussing research findings also “led to a positive approach and EBP knowledge” [ 20 ]. Moreover, reading research articles during the clinical practice period enhances the students critical thinking skills. Furthermore, Horntvedt et al. mention, that students found it meaningful to conduct a “mini” – research project in clinical settings, as the identified evidence became relevant [ 20 ].

Educational interventions

Educational interventions can be described as “Framing Interventions” understood as different ways to set up a framework for teaching EBP, and “ Teaching methods ” understood as specific methods used when teaching EBP.

Various educational interventions were described in most reviews [ 12 , 13 , 20 , 21 ]. According to Patelarou et al., no specific educational intervention regardless of framing and methods was in favor to “ increase knowledge, skills and competency as well as improve the beliefs, attitudes and behaviors of nursing students” [ 12 ].

Framing interventions

The approaches used to set up a framework for teaching EBP were labelled in different ways: programs, interactive teaching strategies, educational programs, courses etc. Approaches of various durations from hours to months were described as well as stepwise interventions [ 12 , 13 , 20 , 21 , 24 , 25 ].

Some frameworks [ 13 , 20 , 21 , 24 ] were based on the assessments categories described by the Sicily group [ 2 ] or based on theory [ 21 ] or as mentioned above clinically integrated [ 20 ]. Wakibi et al. identified interventions used to foster a spirit of inquiry and EBP culture reflecting the “5-step approach” of the Sicily group [ 4 ], asking PICOT questions, searching for best evidence, critical appraisal, integrating evidence with clinical expertise and patient preferences to make clinical decisions, evaluating outcomes of EBP practice, and disseminating outcomes useful [ 13 ]. Ramis et al. found that teaching interventions based on theory like Banduras self-efficacy or Roger’s theory of diffusion led to positive effects on students EBP knowledge and attitudes [ 21 ].

Teaching methods

A variety of teaching methods were used such as, lectures [ 12 , 13 , 20 ], problem-based learning [ 12 , 20 , 25 ], group work, discussions [ 12 , 13 ], and presentations [ 20 ] (see Table 2 ). The most effective method to achieve the skills required to practice EBP as described in the “5-step approach” by the Sicely group is a combination of different teaching methods like lectures, assignments, discussions, group works, and exams/tests.

Four systematic reviews identified such combinations or multifaceted approaches [ 12 , 13 , 20 , 21 ]. Patelarou et al. states that “EBP education approaches should be blended” [ 12 ]. Thus, combining the use of video, voice-over, PowerPoint, problem-based learning, lectures, team-based learning, projects, and small groups were found in different studies. This combination had shown “to be effective” [ 12 ]. Similarly, Horntvedt et al. found that nursing students reported that various teaching methods improved their EBP knowledge and skills [ 20 ].

According to Ghaffari et al., including problem-based learning in teaching plans “improved the clinical care and performance of the students”, while the problem-solving approach “promoted student knowledge” [ 25 ]. Other teaching methods identified, e.g., flipped classroom [ 20 ] and virtual simulation [ 12 , 20 ] were also characterized as useful interactive teaching interventions. Furthermore, face-to-face approaches seem “more effective” than online teaching interventions to enhance students’ research and appraisal skills and journal clubs enhance the students critically appraisal-skills [ 12 ].

As the reviews included in this overview primarily are based on qualitative, mixed methods as well as quasi-experimental studies and to a minor extent on randomized controlled trials (see Table 2 ) it is not possible to conclude of the most effective methods. However, a combination of methods and an innovative collaboration between librarians, information technology professionals and healthcare professionals seem the most effective approach to achieve EBP required skills.

EBP-related outcomes

Most of the systematic reviews presented a wide array of outcome assessments applied in EBP research (See Table 3 ). Analyzing the outcomes according to the Sicily group’s assessment categories revealed that assessing “knowledge” (used in 19 out of 39 primary studies), “skills” (used in 18 out of 39 primary studies) and “attitude” (used in 17 out of 39) were by far the most frequently used assessment categories, whereas outcomes within the category of “behaviors” (used in eight studies) “reaction to educational experience” (in five studies), “self-efficacy” (in two studies), and “benefits for the patient” (in one study), were used to a far lesser extent. Additionally, outcomes, that we were not able to categorize within the seven assessment categories, were “future use” and “Global EBP competence”.

The purpose of this overview of systematic reviews was to collect and summarize evidence of the diversity of EBP teaching interventions and outcomes measured among professional bachelor- degree healthcare students.

Our results give an overview of “the state of the art” of using and measuring EBP in PBHP education. However, the quality of included systematic reviews was rated critically low. Thus, the result cannot support guidelines of best practice.

The analysis of the interventions and outcomes described in the 39 primary studies included in this overview, reveals a wide variety of teaching methods and interventions being used and described in the scientific literature on EBP teaching of PBHP students. The results show some evidence of the five step EBP approach in accordance with the inclusion criteria “interventions aimed at teaching one or more of the five EBP steps; Ask, Search, Appraise, Integrate, Assess/evaluate”. Most authors state, that the students´ EBP skills, attitudes and knowledge improved by almost any of the described methods and interventions. However, descriptions of how the improvements were measured were less frequent.

We evaluated the described outcome measures and assessments according to the seven categories proposed by the Sicily group and found that most assessments were on “attitudes”, “skills” and “knowledge”, sometimes on “behaviors” and very seldom on” reaction to educational experience”, “self-efficacy” and “benefits to the patients”. To our knowledge no systematic review or overview has made this evaluation on outcome categories before, but Bala et al. [ 9 ] also stated that knowledge, skills, and attitudes are the most common evaluated effects.

Comparing the outcomes measured between mainly medical [ 9 ] and nursing students, the most prevalent outcomes in both groups are knowledge, skills and attitudes around EBP. In contrast, measuring on the students´ patient care or on the impact of the EBP teaching on benefits for the patients is less prevalent. In contrast Wu et al.’s systematic review shows that among clinical nurses, educational interventions supporting implementation of EBP projects can change patient outcomes positively. However, they also conclude that direct causal evidence of the educational interventions is difficult to measure because of the diversity of EBP projects implemented [ 26 ]. Regarding EBP behavior the Sicily group recommend this category to be assessed by monitoring the frequency of the five step EBP approach, e.g., ASK questions about patients, APPRAISE evidence related to patient care, EVALUATE their EBP behavior and identified areas for improvement [ 2 ]. The results also showed evidence of student-clinician transition. “Future use” was identified in two systematic reviews [ 12 , 13 ] and categorized as “others”. This outcome is not included in the seven Sicily categories. However, a systematic review of predictive modelling studies shows, that future use or the intention to use EBP after graduation are influenced by the students EBP familiarity, EBP capability beliefs, EBP attitudes and academic and clinical support [ 27 ].

Teaching and evaluating EBP needs to move beyond aiming at changes in knowledge, skills, and attitudes, but also start focusing on changing and assessing behavior, self-efficacy and benefit to the patients. We recommend doing this using validated tools for the assessment of outcomes and in prospective studies with longer follow-up periods, preferably evaluating the adoption of EBP in clinical settings bearing in mind, that best teaching practice happens across sectors and settings supported and supervised by multiple professions.

Based on a systematic review and international Delphi survey, a set of interprofessional EBP core competencies that details the competence content of each of the five steps has been published to inform curriculum development and benchmark EBP standards [ 28 ]. This consensus statement may be used by educators as a reference for both learning objectives and EBP content descriptions in future intervention research. The collaboration with clinical institutions and integration of EBP teaching components such as EBP assignments or participating in clinical research projects are important results. Specifically, in the light of the dialectic between theoretical and clinical education as a core characteristic of Professional bachelor-degree healthcare educations.

Our study has some limitations that need consideration when interpreting the results. A search in the EMBASE and Scopus databases was not added in the search strategy, although it might have been able to bring additional sources. Most of the 22 excluded reviews included primary studies among other levels/ healthcare groups of students or had not critically appraised their primary studies. This constitutes insufficient adherence to methodological guidelines for systematic reviews and limits the completeness of the reviews identified. Often, the result sections of the included reviews were poorly reported and made it necessary to extract some, but not always sufficient, information from the primary study abstracts. As the present study is an overview and not a new systematic review, we did not extract information from the result section in the primary studies. Thus, the comprehensiveness and applicability of the results of this overview are limited by the methodological limitations in the six included systematic reviews.

The existing evidence is based on different types of study designs. This heterogeneity is seen in all the included reviews. Thus, the present overview only conveys trends around the comparative effectiveness of the different ways to frame, or the methods used for teaching EBP. This can be seen as a weakness for the clarity and applicability of the overview results. Also, our protocol is unpublished, which may weaken the transparency of the overview approach, however our search strategies are available as additional material (see Additional file 1 ). In addition, the validity of data extraction can be discussed. We extracted data consecutively by the first and last author and if needed consensus was reached by discussion with the entire research group. This method might have been strengthened by using two blinded reviewers to extract data and present data with supporting kappa values.

The generalizability of the results of this overview is limited to undergraduate nursing students. Although, we consider it a strength that the results represent a broad international perspective on framing EBP teaching, as well as teaching methods and outcomes used among educators in EBP. Primary studies exist among occupational therapy and physiotherapy students [ 5 , 29 ] but have not been systematically synthesized. However, the evidence is almost non-existent among midwife, nutrition and health and biomedical laboratory science students. This has implications for further research efforts because evidence from within these student populations is paramount for future proofing the quality assurance of clinical evidence-based healthcare practice.

Another implication is the need to compare how to frame the EBP teaching, and the methods used both inter-and mono professionally among these professional bachelor-degree students. Lastly, we support the recommendations of Bala et al. of using validated tools to increase the focus on measuring behavior change in clinical practice and patient outcomes, and to report in accordance with the GREET guidelines for educational intervention studies [ 9 ].

This overview demonstrates a variety of approaches to promote EBP teaching among professional bachelor-degree healthcare students. Teaching EBP is based on collaboration with clinical practice and the use of different approaches to frame the teaching as well as different teaching methods. Furthermore, this overview has elucidated, that interventions often are evaluated according to changes in the student’s skills, knowledge and attitudes towards EBP, but very rarely on self-efficacy, behaviors, benefits to the patients or reaction to the educational experience as suggested by the Sicily group. This might indicate that educators need to move on to measure the effect of EBP on outcomes comprising all categories, which are important to enhance sustainable behavior and transition of knowledge into the context of practices where better healthcare education should have an impact. In our perspective these gaps in the EBP teaching are best met by focusing on more collaboration with clinical practice which is the context where the final endpoint of teaching EBP should be anchored and evaluated.

Availability of data and materials

The datasets used an/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Evidence-Based Practice

Professional bachelor-degree healthcare programs

Mazurek Melnyk B, Fineout-Overholt E. Making the Case for Evidence-Based Practice and Cultivalting a Spirit of Inquiry. I: Mazurek Melnyk B, Fineout-Overholt E, redaktører. Evidence-Based Practice in Nursing and Healthcare A Guide to Best Practice. 4. ed. Wolters Kluwer; 2019. p. 7–32.

Tilson JK, Kaplan SL, Harris JL, Hutchinson A, Ilic D, Niederman R, et al. Sicily statement on classification and development of evidence-based practice learning assessment tools. BMC Med Educ. 2011;11(78):1–10.

Google Scholar

Connor L, Dean J, McNett M, Tydings DM, Shrout A, Gorsuch PF, et al. Evidence-based practice improves patient outcomes and healthcare system return on investment: Findings from a scoping review. Worldviews Evid Based Nurs. 2023;20(1):6–15.

Article PubMed Google Scholar

Dawes M, Summerskill W, Glasziou P, Cartabellotta N, Martin J, Hopayian K, et al. Sicily statement on evidence-based practice. BMC Med Educ. 2005;5(1):1–7.

Article PubMed PubMed Central Google Scholar

Larsen CM, Terkelsen AS, Carlsen AF, Kristensen HK. Methods for teaching evidence-based practice: a scoping review. BMC Med Educ. 2019;19(1):1–33.

Article CAS Google Scholar

World Health Organization. Nurse educator core competencies. 2016 https://apps.who.int/iris/handle/10665/258713 Accessed 21 Mar 2023.

Saunders H, Gallagher-Ford L, Kvist T, Vehviläinen-Julkunen K. Practicing healthcare professionals’ evidence-based practice competencies: an overview of systematic reviews. Worldviews Evid Based Nurs. 2019;16(3):176–85.

Young T, Rohwer A, Volmink J, Clarke M. What Are the Effects of Teaching Evidence-Based Health Care (EBHC)? Overview of Systematic Reviews PLoS ONE. 2014;9(1):1–13.

Bala MM, Poklepović Peričić T, Zajac J, Rohwer A, Klugarova J, Välimäki M, et al. What are the effects of teaching Evidence-Based Health Care (EBHC) at different levels of health professions education? An updated overview of systematic reviews. PLoS ONE. 2021;16(7):1–28.

Article Google Scholar

Copenhagen University. Bachelor in medicine. 2024 https://studier.ku.dk/bachelor/medicin/undervisning-og-opbygning/ Accessed 31 Jan 2024.

Ministery of Higher Education and Science. Professional bachelor programmes. 2022 https://ufm.dk/en/education/higher-education/university-colleges/university-college-educations Accessed 31 Jan 2024.

Patelarou AE, Mechili EA, Ruzafa-Martinez M, Dolezel J, Gotlib J, Skela-Savič B, et al. Educational Interventions for Teaching Evidence-Based Practice to Undergraduate Nursing Students: A Scoping Review. Int J Env Res Public Health. 2020;17(17):1–24.

Wakibi S, Ferguson L, Berry L, Leidl D, Belton S. Teaching evidence-based nursing practice: a systematic review and convergent qualitative synthesis. J Prof Nurs. 2021;37(1):135–48.

Fiset VJ, Graham ID, Davies BL. Evidence-Based Practice in Clinical Nursing Education: A Scoping Review. J Nurs Educ. 2017;56(9):534–41.

Pollock M, Fernandes R, Becker L, Pieper D, Hartling L. Chapter V: Overviews of Reviews. I: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions version 62. 2021 https://training.cochrane.org/handbook Accessed 31 Jan 2024.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, m.fl. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:1-9

Covidence. Covidence - Better systematic review management. https://www.covidence.org/ Accessed 31 Jan 2024.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;21(358):1–9.

Joanna Briggs Institute. Critical Appraisal Tools. https://jbi.global/critical-appraisal-tools Accessed 31 Jan 2024.

Horntvedt MT, Nordsteien A, Fermann T, Severinsson E. Strategies for teaching evidence-based practice in nursing education: a thematic literature review. BMC Med Educ. 2018;18(1):1–11.

Ramis M-A, Chang A, Conway A, Lim D, Munday J, Nissen L. Theory-based strategies for teaching evidence-based practice to undergraduate health students: a systematic review. BMC Med Educ. 2019;19(1):1–13.

Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021;10(1):1–19.

Song CE, Jang A. Simulation design for improvement of undergraduate nursing students’ experience of evidence-based practice: a scoping-review protocol. PLoS ONE. 2021;16(11):1–6.

Cui C, Li Y, Geng D, Zhang H, Jin C. The effectiveness of evidence-based nursing on development of nursing students’ critical thinking: A meta-analysis. Nurse Educ Today. 2018;65:46–53.

Ghaffari R, Shapoori S, Binazir MB, Heidari F, Behshid M. Effectiveness of teaching evidence-based nursing to undergraduate nursing students in Iran: a systematic review. Res Dev Med Educ. 2018;7(1):8–13.

Wu Y, Brettle A, Zhou C, Ou J, Wang Y, Wang S. Do educational interventions aimed at nurses to support the implementation of evidence-based practice improve patient outcomes? A systematic review. Nurse Educ Today. 2018;70:109–14.

Ramis MA, Chang A, Nissen L. Undergraduate health students’ intention to use evidence-based practice after graduation: a systematic review of predictive modeling studies. Worldviews Evid Based Nurs. 2018;15(2):140–8.

Albarqouni L, Hoffmann T, Straus S, Olsen NR, Young T, Ilic D, et al. Core competencies in evidence-based practice for health professionals: consensus statement based on a systematic review and Delphi survey. JAMA Netw Open. 2018;1(2):1–12.

Hitch D, Nicola-Richmond K. Instructional practices for evidence-based practice with pre-registration allied health students: a review of recent research and developments. Adv Health Sci Educ Theory Pr. 2017;22(4):1031–45.

Download references

Acknowledgements

The authors would like to acknowledge research librarian Rasmus Sand for competent support in the development of literature search strategies.

This work was supported by the University College of South Denmark, which was not involved in the conduct of this study.

Author information

Authors and affiliations.

Nursing Education & Department for Applied Health Science, University College South Denmark, Degnevej 17, 6705, Esbjerg Ø, Denmark

Lea D. Nielsen

Department of Oncology, Hospital of Lillebaelt, Beriderbakken 4, 7100, Vejle, Denmark

Mette M. Løwe

Biomedical Laboratory Science & Department for Applied Health Science, University College South Denmark, Degnevej 17, 6705, Esbjerg Ø, Denmark

Francisco Mansilla

Physiotherapy Education & Department for Applied Health Science, University College South Denmark, Degnevej 17, 6705, Esbjerg Ø, Denmark

Rene B. Jørgensen

Occupational Therapy Education & Department for Applied Health Science, University College South Denmark, Degnevej 17, 6705, Esbjerg Ø, Denmark

Asviny Ramachandran

Department for Applied Health Science, University College South Denmark, Degnevej 17, 6705, Esbjerg Ø, Denmark

Bodil B. Noe

Centre for Clinical Research and Prevention, Section for Health Promotion and Prevention, Bispebjerg and Frederiksberg Hospital, Nordre Fasanvej 57, 2000, Frederiksberg, Denmark

Heidi K. Egebæk

You can also search for this author in PubMed Google Scholar

Contributions

All authors have made substantial contributions to the conception and design of the study, acquisition of data, analysis, and interpretation of data, writing the main manuscript, preparing figures and tables and revising the manuscripts.

Corresponding author

Correspondence to Lea D. Nielsen .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., supplementary material 4., rights and permissions.

Reprints and permissions

About this article

Cite this article.

Nielsen, L.D., Løwe, M.M., Mansilla, F. et al. Interventions, methods and outcome measures used in teaching evidence-based practice to healthcare students: an overview of systematic reviews. BMC Med Educ 24 , 306 (2024). https://doi.org/10.1186/s12909-024-05259-8

Download citation

Received : 29 May 2023

Accepted : 04 March 2024

Published : 19 March 2024

DOI : https://doi.org/10.1186/s12909-024-05259-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

MH "Students, Health occupations+"
MH "Students, occupational therapy"
MH "Students, physical therapy"
MH "Students, Midwifery"
“Students, Nursing"[Mesh]
“Teaching"[Mesh]
MH "Teaching methods+"
"Evidence-based practice"[Mesh]

BMC Medical Education

ISSN: 1472-6920

Submission enquiries: [email protected]
General enquiries: [email protected]

IMAGES

10 Steps to Write a Systematic Literature Review Paper in 2023
Guidance on the conduct of narrative synthesis in systematic Reviews. A
Systematic Review Appraisal.docx
Systematic Review Appraisal.docx
(PDF) Staff understanding of recovery-orientated practice A systematic
(PDF) Guidance on the conduct of narrative synthesis in systematic reviews

VIDEO

SYSTEMATIC AND LITERATURE REVIEWS
The Art of VIDEO EDITING: Navigating Narrative Structure
Writing Systematic and Narrative Review
Summary, Conclusions and Recommendations
Review article types
Rhetorical Synthesis-Narrative Prompt

COMMENTS

PDF Guidance on the Conduct of Narrative Synthesis in Systematic Reviews
bridged. Telling a trustworthy story is at the heart of narrative synthesis. 1.2 Narrative synthesis, narrative reviews and evidence synthesis 'Narrative' synthesis' refers to an approach to the systematic review and synthesis of findings from multiple studies that relies primarily on the use of words and text to summarise and explain the
How to Write a Systematic Review: A Narrative Review
A systematic review, as its name suggests, is a systematic way of collecting, evaluating, integrating, and presenting findings from several studies on a specific question or topic. [ 1] A systematic review is a research that, by identifying and combining evidence, is tailored to and answers the research question, based on an assessment of all ...
PDF STARTING A NARRATIVE SYNTHESIS
This document aims to assist authors in planning their narrative analysis at protocol stage, and to highlight some issues for authors to consider at review stage. Narrative forms of synthesis are an area of emerging research, and so advice is likely to be adapted as methods develop. This document sits alongside the RevMan templates for ...
Full article: Narrative approaches to systematic review and synthesis
Narrative methods of synthesis can be used to synthesise both quantitative and qualitative studies and have been used when the experimental and quasi-experimental studies included in a systematic review are not sufficiently similar for a meta-analysis to be appropriate (Mays et al. Citation 2005a). Narrative synthesis is used in different ways.
PDF Guidance on the conduct of narrative synthesis in systematic review
meta-analysis as part of a previous Cochrane review which investigated the effects of interventions for promoting smoke alarm ownership and function.1. The reviewers carrying out the new narrative synthesis were blinded to the findings of the original Cochrane review. We then compared the results and conclusions of the two different approaches.
How to Conduct a Systematic Review: A Narrative Literature Review
Writing a research question is the first step in conducting a systematic review and is of paramount importance as it outlines both the need and validity of systematic reviews (Nguyen, et al., unpublished data). It also increases the efficiency of the review by limiting the time and cost of identifying and obtaining relevant literature [ 11 ].
Narrative Reviews: Flexible, Rigorous, and Practical
Introduction. Narrative reviews are a type of knowledge synthesis grounded in a distinct research tradition. They are often framed as non-systematic, which implies that there is a hierarchy of evidence placing narrative reviews below other review forms. 1 However, narrative reviews are highly useful to medical educators and researchers. While a systematic review often focuses on a narrow ...
How to Do a Systematic Review: A Best Practice Guide for Conducting and
The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information.
Guidelines for writing a systematic review
It is important to understand that the synthesis and the type of systematic review are different for example, a quantitative SR can be conducted and use a narrative synthesis this would mean that the SR only included quantitative studies but did not conduct statistical analysis on the results and instead provided a narrative of the included ...
Improving Conduct and Reporting of Narrative Synthesis of Quantitative
Narrative synthesis of quantitative data (NS) is a method commonly used in systematic reviews where it may not be appropriate, or possible, to meta-analyse estimates of intervention effects. A common criticism of NS is that it is opaque and subject to author interpretation, casting doubt on the trustworthiness of a review's conclusions.
Narrative Synthesis
Narrative' synthesis' refers to an approach to the systematic review and synthesis of findings from multiple studies that relies primarily on the use of words and text to summarise and explain the findings of the synthesis. ... Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme ...
PDF Testing Methodological Guidance on the Conduct of Narrative Synthesis
Rodgers et al.: Narrative synthesis in systematic Reviews 49 Objectives The aim of this article is to demonstrate the way in which narrative synthesis guidance can be used in the context of a review of effectiveness, and to evaluate what the guidance might add (or otherwise) to the ﬁ ndings of a systematic review.
Testing Methodological Guidance on the Conduct of Narrative Synthesis
The objective was to assess the impact of new guidance on the conduct of narrative synthesis in systematic reviews of effectiveness, by means of a blinded comparison of guidance-led narrative synthesis against a meta-analysis of the same study data.The conclusions of the two syntheses were broadly similar.
(PDF) Narrative Synthesis: Considerations and challenges
This is a systematic approach for undertaking a review where statistical methods or pooling of data is neither possible nor appropriate (Lisy & Porritt, 2016). Narrative synthesis enables ...
(PDF) Guidance on the conduct of narrative synthesis in systematic
'Narrative' synthesis' refers to an approach to the systematic review and synthesi s of findings from multiple studies that relies primarily on the use of words and text to summarise a nd ...
A systematic literature review and narrative synthesis on the risks of
Synthesis. Then a narrative synthesis was performed to synthesize the findings of the different studies. Because of the range of very different studies that were included in this systematic review, we have decided that a narrative synthesis constitutes the best instrument to synthesise the findings of the studies.
Systems approaches to scaling up: a systematic review and narrative
Conclusion. In physical activity research, and NCD prevention more broadly, the use of systems approaches in scale up remains in its infancy. ... Koorts, H., Ma, J., Swain, C.T.V. et al. Systems approaches to scaling up: a systematic review and narrative synthesis of evidence for physical activity and other behavioural non-communicable disease ...
Strengths and limitations of early warning scores: A systematic review
Strengths and limitations of early warning scores: A systematic review and narrative synthesis Int J Nurs Stud. 2017 Nov:76:106-119. doi: 10.1016/j.ijnurstu.2017.09.003. ... Conclusions: Early warning scores provide the right language and environment for the timely escalation of patient care. They are limited by their intermittent and user ...
Synthesis without meta-analysis (SWiM) in systematic reviews ...
In systematic reviews that lack data amenable to meta-analysis, alternative synthesis methods are commonly used, but these methods are rarely reported. This lack of transparency in the methods can cast doubt on the validity of the review findings. The Synthesis Without Meta-analysis (SWiM) guideline has been developed to guide clear reporting in reviews of interventions in which alternative ...
An overview of methodological approaches in systematic reviews
1. INTRODUCTION. Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the "gold standard" of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search ...
Exploring the use of social network interventions for adults with
This systematic review and narrative synthesis aimed to identify and synthesise current evidence pertaining to the use of social network interventions for people with mental health difficulties, with a view to understanding their effectiveness, and the conditions in which these interventions might work best.
Frontiers
Conclusion. This review describes the prevalence rates, and risk and protective factors, for suicidal ideation, attempts, and self-harm in international students. ... A systematic review and narrative synthesis of prevalence rates, risk and protective factors for suicidal behavior in international students. Front. Psychiatry 15:1358041. doi: 10 ...
A systematic review of the effects of psychiatric medications on social
Strengths and limitations (narrative synthesis element 4: assessment of the robustness of the synthesis) One of the most important strengths of this review was establishing the current literature on the effects of sedative psychiatric medications on social cognition using a rigorous search strategy of published and unpublished work.
Clinical skills development for healthcare practitioners working with
Conclusions. The review findings demonstrate that developing healthcare practitioners' communicative behaviours led to increased confidence and self-efficacy when working with PPS, which facilitated improved consultations and improvements on some patient outcomes. ... (PPS) in healthcare settings: a systematic review and narrative synthesis ...
N350 Week 6 Systematic Review Appraisal
Systematic Review Appraisal appendix appraisal guide conclusions of systematic review with narrative synthesis citation: redmond, grimes, mcdonnell, boland,
Systematic Review PICOT
N/A appendix appraisal guide: conclusions of systematic review with narrative synthesis citation: inanlou, bahmani, farhoudian, rafiee, (2020). addiction
Neuroticism facets and mortality risk in adulthood: A systematic review
Conclusions: Various facets related to neuroticism are associated with an increased or decreased mortality risk. Encompassing all facets in a broad trait likely masks very important personality-health relations, which later impact longevity. ... Neuroticism facets and mortality risk in adulthood: A systematic review and narrative synthesis ...
A Systematic Review and Narrative Synthesis: Determinants of the
The full text review was also conducted independently by two reviewers and discrepancies resolved via discussion. The two reviewers consulted a third reviewer to assist in making the decision on one paper at the full text review stage. Because the studies were heterogenous, a narrative synthesis was performed.
Interventions, methods and outcome measures used in teaching evidence
Background To fully implement the internationally acknowledged requirements for teaching in evidence-based practice, and support the student's development of core competencies in evidence-based practice, educators at professional bachelor degree programs in healthcare need a systematic overview of evidence-based teaching and learning interventions. The purpose of this overview of systematic ...
Full article: Factors Associated with PrEP Stigma Among Gay, Bisexual
This systematic review aimed to explore experiences of PrEP stigma and to identify factors associated with this. Four databases were searched for papers including terms relating to (i) gbMSM, (ii) PrEP, and (iii) stigma, with narrative synthesis used to analyze results. After screening, 70 studies were included in the final analysis.

Log in using your username and password

You are here

Statistics from Altmetric.com

Strengths and limitations of this study

Introduction

ICONS-Quant

Methods and analysis

Project Advisory Group

Work Programme One: assessment of current reporting and conduct of NS in Cochrane reviews

Work Programme Two: Delphi survey

Consensus meeting

Work Programme Three: enhancement of existing guidance on NS methods

Work Programme Four: development of learning materials on implementation of NS

Ethics and dissemination

Read the full text or download the PDF:

Which review is that? A guide to review types.

Narrative Synthesis

Further Reading/Resources

A systematic literature review and narrative synthesis on the risks of medical discharge letters for patients’ safety

Conclusions

Search strategy

Eligibility criteria

Data extraction

Risk factors

Delayed delivery

Quality, lack of information

Writing of medical discharge letter is missing in medical education

Strengths and limitations

Abbreviations

Acknowledgements

Availability of data and materials

Author information

Contributions

Corresponding author

Ethics declarations

Publisher’s Note

Rights and permissions

About this article

Share this article

BMC Health Services Research

Systems approaches to scaling up: a systematic review and narrative synthesis of evidence for physical activity and other behavioural non-communicable disease risk factors

Trial registration

Introduction

Definition of a systems approach to scaling up

Eligibility criteria

Information sources and search strategy

Study selection

Quality appraisal

Data extraction

Data synthesis

Study sample characteristics

Approaches and strategies to scaling up

Barriers and facilitators to scaling up

Facilitator 1: Committed stakeholder engagement

Facilitator 2: Capacity building in local workforce

Facilitator 3: Flexibility in delivery and implementation

Facilitator 4: Ongoing programme feedback and improvement

Barrier 1: Competing interests and priorities

Barrier 2: Insufficient time and resources

Use of systems approaches to scaling up

High use of a systems approach to scaling up

Conceptualisation and use of ‘systems’ and a systems approach

Methods, theoretical frameworks and/or principles adopted to study scale up processes or outcomes

Moderate use of a systems approach to scaling up

No use of a systems approach to scaling up

Quality appraisal findings

Strengths and limitations

Availability of data and materials

Abbreviations

Acknowledgements

Additional files

Author information

Contributions

Corresponding author

Ethics declarations

Consent for publication

Additional information

Supplementary Information

Additional file 2.

Additional file 3.