empirical research in healthcare

• Systematic review
• Open access
• Published: 16 April 2020

A systematic review of empirical studies examining mechanisms of implementation in health

• Cara C. Lewis 1 , 2 , 3 ,
• Meredith R. Boyd 4 ,
• Callie Walsh-Bailey 1 , 5 ,
• Aaron R. Lyon 3 ,
• Rinad Beidas 6 ,
• Brian Mittman 7 ,
• Gregory A. Aarons 8 ,
• Bryan J. Weiner 9 &
• David A. Chambers 10  

Implementation Science volume  15 , Article number:  21 ( 2020 ) Cite this article

20k Accesses

144 Citations

61 Altmetric

Metrics details

Understanding the mechanisms of implementation strategies (i.e., the processes by which strategies produce desired effects) is important for research to understand why a strategy did or did not achieve its intended effect, and it is important for practice to ensure strategies are designed and selected to directly target determinants or barriers. This study is a systematic review to characterize how mechanisms are conceptualized and measured, how they are studied and evaluated, and how much evidence exists for specific mechanisms.

We systematically searched PubMed and CINAHL Plus for implementation studies published between January 1990 and August 2018 that included the terms “mechanism,” “mediator,” or “moderator.” Two authors independently reviewed title and abstracts and then full texts for fit with our inclusion criteria of empirical studies of implementation in health care contexts. Authors extracted data regarding general study information, methods, results, and study design and mechanisms-specific information. Authors used the Mixed Methods Appraisal Tool to assess study quality.

Search strategies produced 2277 articles, of which 183 were included for full text review. From these we included for data extraction 39 articles plus an additional seven articles were hand-entered from only other review of implementation mechanisms (total = 46 included articles). Most included studies employed quantitative methods (73.9%), while 10.9% were qualitative and 15.2% were mixed methods. Nine unique versions of models testing mechanisms emerged. Fifty-three percent of the studies met half or fewer of the quality indicators. The majority of studies (84.8%) only met three or fewer of the seven criteria stipulated for establishing mechanisms.

Conclusions

Researchers have undertaken a multitude of approaches to pursue mechanistic implementation research, but our review revealed substantive conceptual, methodological, and measurement issues that must be addressed in order to advance this critical research agenda. To move the field forward, there is need for greater precision to achieve conceptual clarity, attempts to generate testable hypotheses about how and why variables are related, and use of concrete behavioral indicators of proximal outcomes in the case of quantitative research and more directed inquiry in the case of qualitative research.

Peer Review reports

Contributions to the literature statement

This is the first systematic review of implementation mechanisms across health that assesses the quality of studies and the extent to which they offer evidence in support of establishing mechanisms of implementation.

We summarize nine examples of models for evaluating mechanisms.

We offer conceptual, theoretical, and methodological guidance for the field to contribute to the study of implementation mechanisms.

Implementation research is the scientific evaluation of strategies or methods used to support the integration of evidence-based practices or programs (EBPs) into healthcare settings to enhance the quality and effectiveness of services [ 1 ]. There is mounting evidence that multi-faceted or blended implementation strategies are necessary (i.e., a discrete strategy is insufficient) [ 2 , 3 ], but we have a poor understanding of how and why these strategies work. Mechanistic research in implementation science is in an early phase of development. As of 2016, there were only nine studies included in one systematic review of implementation mediators Footnote 1 specific to the field of mental health. Mediators are an intervening variable that may statistically account for the relation between an implementation strategy and outcome. We define the term mechanism as a process or event through which an implementation strategy operates to affect one or more implementation outcomes (see Table 1 for key terms and definitions used throughout this manuscript). Mechanisms offer causal pathways explaining how strategies operate to achieve desired outcomes, like changes in care delivery. Some researchers conflate moderators, mediators, and mechanisms [ 6 ], using the terms interchangeably [ 7 ]. Mediators and moderators can point toward mechanisms, but they are not all mechanisms as they typically are insufficient to explain exactly how change came about.

In addition to these linguistic inconsistencies and lack of conceptual clarity, there is little attention paid to the criteria for establishing a mechanistic relation. Originally, Bradford-Hill [ 8 ], and more recently Kazdin offers [ 4 ] at least seven criteria for establishing mechanisms of psychosocial treatments that are equally relevant to implementation strategies: strong association, specificity, consistency, experimental manipulation, timeline, gradient, plausibility, or coherence (see Table 2 for definitions). Taken together, these criteria can guide study designs for building the case for mechanisms over time. In lieu of such criteria, disparate models and approaches for investigating mechanisms are likely to exist that make synthesizing findings across studies quite challenging. Consequently, the assumption that more strategies will achieve better results is likely to remain, driving costly and imprecise approaches to implementation.

Understanding the mechanisms of implementation strategies, defined as the processes by which strategies produce desired effects [ 4 , 8 ], is important for both research and practice. For research, it is important to specify and examine mechanisms of implementation strategies, especially in the case of null studies, in order to understand why a strategy did or did not achieve its intended effect. For practice, it is crucial to understand mechanisms so that strategies are designed and selected to directly target implementation determinants or barriers. In the absence of this kind of intentional, a priori matching (i.e., strategy targets determinant), it is possible that the “wrong” (or perhaps less potent) strategy will be deployed. This phenomenon of mismatched strategies and determinants was quite prevalent among the 22 tailored improvement intervention studies included in Bosch et al.’s [ 9 ] multiple case study analysis. Upon examining the timing of determinant identification and the degree to which included studies informed tailoring of the type versus the content of the strategies using determinant information, they discovered frequent determinant-strategy mismatch across levels of analysis (e.g., clinician-level strategies were used to address barriers that were at the organizational level) [ 9 ]. Perhaps what is missing is a clear articulation of implementation mechanisms to inform determinant-strategy matching. We argue that, ultimately, knowledge of mechanisms would help to create a more rational, efficient bundle of implementation strategies that fit specific contextual challenges.

Via a systematic review, we sought to understand how mechanisms are conceptualized and measured, how they are studied (by characterizing the wide array of models and designs used to evaluate mechanisms) and evaluated (by applying Kazdin’s seven criteria), and how much evidence exists for specific mechanisms. In doing so, we offer a rich characterization of the current state of the evidence. In reflecting on this evidence, we provide recommendations for future research to optimize their contributions to mechanistic implementation science.

Search protocol

The databases, PubMed and CINAHL Plus, were chosen because of their extensive collection of over 32 million combined citations of medical, nursing and allied health, and life science journals, as well as inclusiveness of international publications. We searched both databases in August 2018 for empirical studies published between January 1990 and August 2018 testing candidate mechanisms of implementation strategies. This starting date was selected given that the concept of evidence-based practice/evidence-based treatment/evidence-based medicine first gained prominence in the 1990’s with the field of implementation science following in response to a growing consciousness of the research to practice gap [ 10 , 11 ]. The search terms were based on input from all authors who represent a variety of methodological and content expertise related to implementation science and reviewed by a librarian; see Table 3 for all search terms. The search string consisted of three levels with terms reflecting (1) implementation science, (2) evidence-based practice (EBP), and (3) mechanism. We adopted Kazdin’s [ 4 ] definition of mechanisms, which he indicates are the basis of an effect. Due to the diversity of definitions that exist in the literature, the term “mechanism” was supplemented with the terms “mediator” and “moderator” to ensure all relevant studies were collected.

Study inclusion and exclusion criteria

Studies were included if they were considered an empirical implementation study (i.e., original data collection) and statistically tested or qualitatively explored mechanisms, mediators, or moderators. We did not include dissemination studies given the likely substantive differences between strategies, mechanisms, and outcomes. Specifically, we align with the distinction made between dissemination and implementation put forth by the National Institutes of Health program announcement for Dissemination and Implementation Research in Health that describes dissemination as involving distribution of evidence to a target audience (i.e., communication of evidence) and implementation as involving use of strategies to integrate evidence into target settings (i.e., use of evidence in practice) [ 12 ]. However, the word “dissemination” was included in our search terms because of the tendency of some researchers to use “implementation” and “dissemination” interchangeably. Studies were excluded if they were not an implementation study, used the terms “mediator,” “moderator,” or “mechanism” in a different context (i.e., conflict mediator), did not involve the implementation of an EBP, or were a review, concept paper, or opinion piece rather than original research. All study designs were considered. Only studies in English were assessed. See Additional File 1 for exclusion criteria and definitions. We strategically cast a wide net and limited our exclusions so as to characterize the broad range of empirical studies of implementation mechanisms.

Citations generated from the search of PubMed and CINAHL were loaded into EPPI Reviewer 4, an online software program used for conducting literature reviews [ 13 ]. Duplicate citations were identified for removal via the duplicate checking function in EPPI and via manual searching. Two independent reviewers (MRB, CWB) screened the first ten citations on title and abstract for inclusion. They then met to clarify inclusion and exclusion criteria with the authorship team, as well as add additional criteria if necessary, and clarify nuances of the inclusion/exclusion coding system (see Additional File 1 for exclusion criteria and definitions). The reviewers met once a week to compare codes and resolve discrepancies through discussion. If discrepancies could not be easily resolved through discussion among the two reviewers, the first author (CCL) made a final determination. During full text review, additional exclusion coding was applied for criteria that could not be discerned from the abstract; articles were excluded at this phase if they only mentioned the study of mechanisms in the discussion or future directions. Seven studies from the previous systematic review of implementation mechanisms [ 14 ] were added to our study for data extraction; these studies likely did not appear in our review due to differences in the search strategy in that the review undertaken by Williams hand searched published reviews of implementation strategies in mental health.

Study quality assessment

The methodological quality of included studies was assessed using the Mixed Methods Appraisal Tool (MMAT-version 2018) [ 15 ]. This tool has been utilized in over three dozen systematic reviews in the health sciences. The MMAT includes two initial screening criteria that assess for the articulation of a clear research question/objective and for the appropriateness of the data collected to address the research question. Studies must receive a “yes” in order to be included. The tool contains a subset of questions to assess for quality for each study type—qualitative, quantitative, and mixed methods. Table 4 summarizes the questions by which studies were evaluated, such as participant recruitment and relevance and quality of measures. Per the established approach to MMAT application, a series of four questions specific to each study design type are assigned a dichotomous “yes” or “no” answer. Studies receive 25 percentage points for each “yes” response. Higher percentages reflect higher quality, with 100% indicating all quality criteria were met. The MMAT was applied by the third author (CWB). The first author (CCL) checked the first 15% of included studies and, based on reaching 100% agreement on the application of the rating criteria, the primary reviewer then applied the tool independently to the remaining studies.

Data extraction and synthesis

Data extraction focused on several categories: study information/ background (i.e., country, setting, and sample), methods (i.e., theories that informed study, measures used, study design, analyses used, proposed mediation model), results (i.e., statistical relations between proposed variables of the mediation model tested), and criteria for establishing mechanisms (based on the seven listed in Table 2 [ 4 ];). All authors contributed to the development of data extraction categories that were applied to the full text of included studies. One reviewer (MRB) independently extracted relevant data and the other reviewer (CWB) checked the results for accuracy, with the first author (CCL) addressing any discrepancies or questions, consistent with the approach of other systematic reviews [ 61 ]. Extracted text demonstrating evidence of study meeting (or not meeting) each criterion for establishing a mechanism was further independently coded as “1” reflecting “criterion met” or “0” reflecting “criterion not met” by MRB and checked by CWB. Again, discrepancies and questions were resolved by the first author (CCL). Technically, mechanisms were considered “established” if all criteria were met. See Additional File 2 for PRISMA checklist for this study.

The search of PubMed and CINAHL Plus yielded 2277 studies for title and abstract screening, of which 447 were duplicates, and 183 moved on to full-text review for eligibility. Excluded studies were most frequently eliminated due to the use of mechanism in a different context (i.e., to refer to a process, technique, or system for achieving results of something other than implementation strategies). After full article review, 39 studies were deemed suitable for inclusion in this review. Two of the included studies appeared in the only other systematic review of implementation mechanisms in mental health settings [ 14 ]. For consistency and comprehensiveness, the remaining seven studies from the previously published review were added to the current systematic review for a total of 46 studies. Footnote 2 See Fig. 1 for a PRISMA Flowchart of the screening process and results.

Mechanisms of Implementation Systematic Review PRISMA Flowchart

Study characteristics

Setting, sampling, and interventions.

Table 5 illustrates the characteristics of the 46 included studies. Twenty-five studies (54.3%) were completed in the USA, while 21 studies were conducted in other countries (e.g., Australia, Canada, Netherlands, UK). Settings were widely variable; studies occurred in behavioral health (e.g., community mental health, residential facilities) or substance abuse facilities most frequently (21.7%), followed by hospitals (15.2%), multiple sites across a health care system (15.2%), schools (15.2%), primary care clinics (10.9%), and Veteran’s Affairs facilities (8.7%). Sampling occurred at multiple ecological levels, including patients (17.4%), providers (65.2%), and organizations (43.5%). Seventeen (40.0%) studies examined the implementation of a complex psychosocial intervention (e.g., Cognitive behavioral therapy [ 42 , 56 ];, multisystemic therapy [ 25 , 26 , 58 ]).

Study design

Our review included six qualitative (10.9%), seven mixed methods (15.2%), and 34 quantitative studies (73.9%). The most common study design was quantitative non-randomized/observational (21 studies; 45.7%), of which 11 were cross-sectional. There were 13 (28.3%) randomized studies included in this review. Twenty-nine studies (63.0%) were longitudinal (i.e., included more than one data collection time point for the sample).

Study quality

Table 4 shows the results of the MMAT quality assessment. Scores for the included studies ranged from 25 to 100%. Six studies (13.0%) received a 25% rating based on the MMAT criteria [ 15 ], 17 studies (40.0%) received 50%, 21 studies (45.7%) received 75%, and only three studies (6.5%) scored 100%. The most frequent weaknesses were the lack of discussion on researcher influence in qualitative and mixed methods studies, lack of clear description of randomization approach utilized in the randomized quantitative studies, and subthreshold rates for acceptable response or follow-up in non-randomized quantitative studies.

Study design and evaluation of mechanisms theories, models, and frameworks

Twenty-seven (58.7%) of the studies articulated their plan to evaluate mechanisms, mediators, or moderators in their research aims or hypotheses; the remaining studies included this as a secondary analysis. Thirty-five studies (76.1%) cited a theory, framework, or model as the basis or rationale for their evaluation. The diffusion of innovations theory [ 63 , 64 ] was most frequently cited, appearing in nine studies (19.6%), followed by the theory of planned behavior [ 65 ], appearing in seven studies (15.2%). The most commonly cited frameworks were the theoretical domains framework (five studies; 10.9%) [ 66 ] and Promoting Action on Research in Health Services (PARiHS) [ 67 ] (three studies; 6.5%).

Ecological levels

Four studies (8.7%) incorporated theories or frameworks that focused exclusively on a single ecological level; two focusing on leadership, one at the organizational level, and one at the systems level. There was some discordance between the theories that purportedly informed studies and the potential mechanisms of interest, as 67.4% of candidate mechanisms or mediators were at the intrapersonal level, while 30.4% were at the interpersonal level, and 21.7% at the organizational level. There were no proposed mechanisms at the systems or policy level. Although 12 studies (26.1%) examined mechanisms or mediators across multiple ecological levels, few explicitly examined multilevel relationships (e.g., multiple single-level mediation models were tested in one study).

Measurement and analysis

The vast majority of studies (38, 82.6%) utilized self-report measures as the primary means of assessing the mechanism, and 13 of these studies (28.3%) utilized focus groups and/or interviews as a primary measure, often in combination with other self-report measures such as surveys. Multiple regression constituted the most common analytic approach for assessing mediators or moderators, utilized by 25 studies (54.3%), albeit this was applied in a variety of ways. Twelve studies (26.1%) utilized hierarchical linear modeling (HLM) and six studies (13.0%) utilized structural equation modeling (SEM); see Table 6 for a complete breakdown. Studies that explicitly tested mediators employed diverse approaches including Baron and Kenny’s ( N = 8, 17.4 causal steps approach [ 78 ], Preacher and Hayes’ ( N = 3, 6.5%) approach to conducting bias-corrected bootstrapping to estimate the significance of a mediated effect (i.e., computing significance for the product of coefficients) [ 95 , 126 ], and Sobel’s ( N = 4, 8.9%) approach to estimating standard error for the product of coefficients often using structural equation modeling [ 79 ]. Only one study tested a potential moderator, citing Raudenbush’s [ 80 , 82 ]. Two other studies included a potential moderator in their conceptual frameworks, but did not explicitly test moderation.

Emergent mechanism models

There was substantial variation in the models that emerged from the studies included in this review. Table 7 represents variables considered in mediating or moderating models across studies (or identified as candidate mediators, moderators, or mechanisms in the case of qualitative studies). Additional file 3 depicts the unique versions of models tested and their associated studies. We attempted to categorize variables as either (a) an independent variable ( X ) impacting a dependent variable; (b) a dependent variable ( Y ), typically the outcome of interest for a study; or (c) an intervening variable ( M ), a putative mediator in most cases, though three studies tested potential moderators. We further specified variables as representing a strategy, determinant, and outcome; see Table 1 for definitions. Footnote 3

Common model types

The most common model type (29; 63.0%) was one in which X was a determinant, M was also a determinant, and Y was an implementation outcome variable (determinant ➔ determinant ➔ implementation outcome). For example, Beenstock et al. [ 36 ] tested a model in which propensity to act (determinant) was evaluated as a mediator explaining the relation between main place of work (determinant) and referral to smoking cessation services (outcome). Just less than half the studies (22; 47.8%) included an implementation strategy in their model, of which 16 (34.8%) evaluated a mediation model in which an implementation strategy was X , a determinant was the candidate M , and an implementation outcome was Y (strategy ➔ determinant ➔ implementation outcome); ten of these studies experimentally manipulated the relation between the implementation strategy and determinant. An example of this more traditional mediation model is a study by Atkins and colleagues [ 21 ] which evaluated key opinion leader support and mental health practitioner support (determinants) as potential mediators of the relation between training and consultation (strategy) and adoption of the EBP (implementation outcome). Five studies included a mediation model in which X was an implementation strategy, Y was a clinical outcome, and M was an implementation outcome (strategy ➔ implementation outcome ➔ clinical outcome) [ 25 , 26 , 28 , 29 , 31 ].

Notable exceptions to model types

While the majority of quantitative studies tested a three-variable model, there were some notable exceptions. Several studies tested multiple three variable models that held the independent variable and mediator constant but tested the relation among several dependent variables. Several studies tested multiple three variable models that held the independent variable and dependent variables constant but tested several mediators.

Qualitative studies

Five studies included in this review utilized qualitative methods to explore potential mechanisms or mediators of change, though only one explicitly stated this goal in their aims [ 17 ]. Three studies utilized a comparative case study design incorporating a combination of interviews, focus groups, observation, and document review, whereas two studies employed a cross-sectional descriptive design. Although three of the five studies reported their analytic design was informed by a theory or previously established model, only one study included an interview guide in which items were explicitly linked to theory [ 19 ]. All qualitative studies explored relations between multiple ecological levels, drawing connections between intra and interpersonal behavioral constructs and organization or system level change.

Criteria for establishing mechanisms of change

Finally, with respect to the seven criteria for establishing mechanisms of change, the plausibility/coherence (i.e., a logical explanation of how the mechanism operates that incorporates relevant research findings) was the most frequently fulfilled requirement, met by 42 studies (91.3%). Although 20 studies (43.5%), of which 18 were quantitative, provided statistical evidence of a strong association between the dependent and independent variables, only 13 (28.2%) studies experimentally manipulated an implementation strategy or the proposed mediator or mechanism. Further, there was only one study that attempted to demonstrate a dose-response relation between mediators and outcomes. Most included studies (39; 84.8%) fulfilled three or fewer criteria, and only one study fulfilled six of the seven requirements for demonstrating a mechanism of change; see Table 8 .

Observations regarding mechanistic research in implementation science

Mechanism-focused implementation research is in an early phase of development, with only 46 studies identified in our systematic review across health disciplines broadly. Consistent with the field of implementation science, no single discipline is driving the conduct of mechanistic research, and a diverse array of methods (quantitative, qualitative, mixed methods) and designs (e.g., cross-sectional survey, longitudinal non-randomized, longitudinal randomized, etc.) have been used to examine mechanisms. Just over one-third of studies ( N = 16; 34.8%) evaluated a mediation model with the implementation strategy as the independent variable, determinant as a putative mediator, and implementation outcome as the dependent variable. Although this was the most commonly reported model, we would expect a much higher proportion of studies testing mechanisms of implementation strategies given the ultimate goal of precise selection of strategies targeting key mechanisms of change. Studies sometimes evaluated models in which the determinant was the independent variable, another determinant was the putative mediator, and an implementation outcome was the dependent variable ( N = 11; 23.9%). These models suggest an interest in understanding the cascading effect of changes in context on key outcomes, but without manipulating or evaluating an implementation strategy as the driver of observed change. Less common (only 5, 10.9%) were more complex models in which multiple mediators and outcomes and different levels of analyses were tested (e.g., [ 37 , 39 ]), despite that this level of complexity is likely to characterize the reality of typical implementation contexts. Although there were several quantitative studies that did observe significant relations pointing toward a mediator, none met all criteria for establishing a mechanism.

Less than one-third of the studies experimentally manipulated the strategy-mechanism linkage. As the field progresses, we anticipate many more tests of this nature, which will allow us to discern how strategies exert their effect on outcomes of interest. However, implementation science will continue to be challenged by the costly nature of the type of experimental studies that would be needed to establish this type of evidence. Fortunately, methodological innovations that capitalize on recently funded implementation trials to engage in multilevel mediation modeling hold promise for the next iteration of mechanistic implementation research [ 14 , 127 ] As this work unfolds, a number of scenarios are possible. For example, it is likely the case that multiple strategies can target the same mechanism; that a single strategy can target multiple mechanisms; and that mechanisms across multiple levels of analysis must be engaged for a given strategy to influence an outcome of interest. Accordingly, we expect great variability in model testing will continue and that more narrowly focused efforts will remain important contributions so long as shared conceptualization of mechanisms and related variables is embraced, articulated, and rigorously tested. As with other fields, we observed great variability in the degree to which mechanisms (and related variables of interest) were appropriately specified, operationalized, and measured. This misspecification coupled with the overall lack of high-quality studies (only three met 100% of the quality criteria), and the diversity in study methods, strategies tested, and mediating or moderating variables under consideration, we were unable to synthesize the findings across studies to point toward promising mechanisms.

The need for greater conceptual clarity and methodological advancements

Despite the important advances that the studies included in this review represent, there are clear conceptual and methodological issues that need to be addressed to allow future research to more systematically establish mechanisms. Table 1 offers a list of key terms and definitions for the field to consider. We suggest the term “mechanism” be used to reflect a process or event through which an implementation strategy operates to affect desired implementation outcomes . Consistent with existing criteria [ 4 ], mechanisms can only be confidently established via carefully designed (i.e., longitudinal; experimentally manipulated) empirical studies demonstrating a strong association, and ideally a dose-response relation, between an intervening variable and outcome (e.g., via qualitative data or mediation or moderator analyses) that are supported by very specific theoretical propositions observed consistently across multiple studies. We found the term “mediator” to be most frequently used in this systematic review, which can point toward a mechanism, but without consideration of these full criteria, detection of a mediator reflects a missed opportunity to contribute more meaningfully to the mechanisms literature.

Interestingly, the nearly half of studies (43.5%) treated a variable that many would conceptualize as a “determinant” as the independent variable in at least one proposed or tested mediation pathway. Presumably, if researchers are exploring the impact of a determinant on another determinant and then on an outcome, there must be a strategy (or action) that caused the change in the initial determinant. Or, it is possible that researchers are simply interested in the natural associations among these determinants to identify promising points of leverage. This is a prime example where the variable or overlapping use of concepts (i.e., calling all factors of interest “determinants”) becomes particularly problematic and undermines the capacity of the field to accumulate knowledge across studies in the service of establishing mechanisms. We contend that it is important to differentiate among concepts to use more meaningful terms like preconditions, putative mechanisms, proximal and distal outcomes, all of which were under-specified in the majority of the included studies. Several authors from our team have articulated an approach to building causal pathway diagrams [ 128 ] that clarifies that preconditions are necessary factors for a mechanism to be activated and proximal outcomes are the immediate result of a strategy that is realized only because the specific mechanism was activated. We conceptualize distal outcomes as the eight implementation outcomes articulated by Proctor and colleagues [ 129 ]. Disentangling these concepts can help characterize why strategies fail to exert an impact on an outcome of interest. Examples of each follow in the section below.

Conceptual and methodological recommendations for future research

Hypothesis generation.

With greater precision among these concepts, the field can also generate and test more specific hypotheses about how and why key variables are related. This begins with laying out mechanistic research questions (e.g., How does a network intervention, like a learning collaborative, influence provider attitudes?) and generating theory-driven hypotheses. For instance, a testable hypothesis may be that learning collaboratives [strategy] operate through sharing [mechanism] of positive experiences with a new practice to influence provider attitudes [outcome]. As another example, clinical decision support [strategy] may act through helping the provider to remember [mechanism] to administer a screener [proximal outcome] and flagging this practice before an encounter may not allow the mechanism to be activated [precondition]. Finally, organizational strategy development [strategy] may have an effect because it means prioritizing competing demands [mechanism] to generate a positive implementation climate [proximal outcome]. Research questions that allow for specific mechanism-focused hypotheses have the potential to expedite the rate at which effective implementation strategies are identified.

Implementation theory

Ultimately, theory is necessary to drive hypotheses, explain implementation processes, and effectively inform implementation practice by providing guidance about when and in what contexts specific implementation strategies should or should not be used. Implementation theories can offer mechanisms that extend across levels of analysis (e.g., intrapersonal, interpersonal, organizational, community, macro policy [ 130 ]). However, there is a preponderance of frameworks and process models, with few theories in existence. Given that implementation is a process of behavior change at its core, in lieu of implementation-specific theories, many researchers draw upon classic theories from psychology, decision science, and organizational literatures, for instance. Because of this, the majority of the identified studies explored intrapersonal-level mechanisms, driven by their testing of social psychological theories such as the theory of planned behavior [ 65 ] and social cognitive theory [ 76 , 77 , 99 ]. Nine studies cited the diffusion of innovations [ 63 , 64 ] as a theory guiding their mechanism investigation, which does extend beyond intrapersonal to emphasize interpersonal, and to some degree community level mechanisms, although we did not see this materialize in the included study analyses [ 63 , 64 , 65 , 76 , 77 ]. Moving forward, developing and testing theory is critical for advancing the study of implementation mechanisms because theories (implicitly or explicitly) tend to identify putative mechanisms instead of immutable determinants.

Measurement

Inadequate measurement has the potential to undermine our ability to advance this area of research. Our coding indicated that mechanisms were assessed almost exclusively via self-report (questionnaire, interview, focus group) suggesting that researchers conceptualize the diverse array of mechanisms to be latent constructs and not directly observable. This may indeed be appropriate, given that mechanisms are typically processes like learning and reflecting that occur within an individual and it is their proximal outcomes that are directly observable (e.g., knowledge acquisition, confidence, perceived control). However, conceptual, theoretical, and empirical work is needed to (a) articulate the theorized mechanisms for the 70+ strategies and proximal outcomes [ 128 ], (b) identify measures of implementation mechanisms and evaluate their psychometric evidence base [ 131 ] and pragmatic qualities [ 132 ], and (c) attempt to identify and rate or develop objective measures of proximal outcomes for use in real-time experimental manipulations of mechanism-outcome pairings.

Quantitative analytic approaches

The multilevel interrelations of factors implicated in an implementation process also call for sophisticated quantitative and qualitative methods to uncover mechanisms. With respect to quantitative methods, it was surprising that the Baron and Kenny [ 78 ] approach to mediation testing remains most prevalent despite that most studies are statistically underpowered to use this approach, and the other most common approach (i.e., the Sobel test [ 79 ]) relies on an assumption that the sampling distribution of the mediation effect is normal [ 14 , 133 ], neither of which were reported on in any of the 12 included studies that used these methods. Williams [ 14 ] suggests the product of coefficients approach [ 134 , 135 ] is more appropriate for mediation analysis because it is a highly general approach to both single and multi-level mediation models that minimizes type I error rates, maximizes statistical power, and enhances accuracy of confidence intervals [ 14 ]. The application of moderated mediation models and mediated moderator models will allow for a nuanced understanding of the complex interrelations among factors implicated in an implementation process.

Qualitative analytic approaches

Because this was the first review of implementation mechanisms across health disciplines, we believed it was important to be inclusive with respect to methods employed. Qualitative studies are important to advancing research on implementation mechanisms in part because they offer a data collection method in lieu of having an established measure to assess mechanisms quantitatively. Qualitative research is important for informing measure development work, but also for theory development given the richness of the data that can be gleaned. Qualitative inquiry can be more directive by developing hypotheses and generating interview guides to directly test mechanisms. Diagramming and tracing causal linkages can be informed by qualitative inquiry in a structured way that is explicit with regard to how the data informs our understanding of mechanisms. This kind of directed qualitative research is called for in the United Kingdom’s MRC Guidance for Process Evaluation [ 136 ]. We encourage researchers internationally to adopt this approach as it would importantly advance us beyond the descriptive studies that currently dominate the field.

Limitations

There are several limitations to this study. First, we took an efficient approach to coding for study quality when applying the MMAT. Although it was a strength that we evaluated study quality, the majority of studies were assessed only by one research specialist. Second, we may have overlooked relevant process evaluations conducted in the UK where MRC Guidance stipulates inclusion of mechanisms that may have been described using terms not included in our search string. Third, although we identified several realist reviews, we did not include them in our systematic review because they conceptualize mechanisms differently than how they are treated in this review [ 137 ]. That is, realist synthesis posits that interventions are theories and that they imply specific mechanisms of action instead of separating mechanisms from the implementation strategies/interventions themselves [ 138 ]. Thus, including the realist operationalization would have further confused an already disharmonized literature with respect to mechanisms terminology but ultimately synthesizing findings from realist reviews with standard implementation mechanism evaluations will be important. Fourth, our characterization of the models tested in the identified studies may not reflect those intended by researchers given our attempt to offer conceptual consistency across studies, although we did reach out to corresponding authors for whom we wished to seek clarification on their study. Finally, because of the diversity of study designs and methods, and the inconsistent use of relevant terms, we are unable to synthesize across the studies and report on any robustly established mechanisms.

This study represents the first systematic review of implementation mechanisms in health. Our inclusive approach yielded 46 qualitative, quantitative, and mixed methods studies, none of which met all seven criteria (i.e., strong association, specificity, consistency, experimental manipulation, timeline, gradient, plausibility or coherence) that are deemed critical for empirically establishing mechanisms. We found nine unique versions of models that attempted to uncover mechanisms, with only six exploring mediators of implementation strategies. The results of this review indicated inconsistent use of relevant terms (e.g., mechanisms, determinants) for which we offer guidance to achieve precision and encourage greater specificity in articulating research questions and hypotheses that allow for careful testing of causal relations among variables of interest. Implementation science will benefit from both quantitative and qualitative research that is more explicit in their attempt to uncover mechanisms. In doing so, our research will allow us to test the idea that more is better and move toward parsimony both for standardized and tailored approaches to implementation.

A mediator can point toward a mechanism as it is an intervening variable that may account (statistically) for the relation between the independent variable (strategy) and the dependent variable (implementation outcome), revealing one possible causal pathway for the observed effect [ 4 ]. Compared to mediators, mechanisms are conceptualized as more precise in their description of the operations underlying causal processes [ 5 ].

Key differences in Williams’ [ 14 ] search method are important to note. Williams first conducted a broad search for randomized controlled trials concerning implementation or dissemination of evidence-based therapies. Only after screening references for these criteria, did Williams narrow the search to studies that specifically addressed mediators. Conversely, the present method included mediators/moderators/mechanisms as terms in the initial search string. Additionally, Williams hand searched references included in four previous reviews of implementation strategies in mental health.

We refer to variables in the ways the study authors did, even if we might have a different way in which we would approach their conceptualization.

Abbreviations

Evidence-based practice

Mixed methods appraisal tool

Promoting Action on Research in Health Services

Hierarchical linear modeling

Structural equation modeling

Eccles MP, Mittman BS. Welcome to implementation science. Implement Sci. 2006;1(1):1.

Article   PubMed Central   Google Scholar  

Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012;69(2):123–57.

Article   PubMed   Google Scholar  

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21.

Article   PubMed   PubMed Central   Google Scholar  

Kazdin AE. Mediators and mechanisms of change in psychotherapy research. Annu Rev Clin Psychol. 2007;3(1):1–27.

Kraemer HC, Wilson GT, Fairburn CG, Agras WS. Mediators and moderators of treatment effects in randomized clinical trials. Arch Gen Psychiatry. 2002;59(10):877–83.

Gerring J. Social science methodology: a criterial framework. Cambridge: Cambridge University Press; 2001.

Book   Google Scholar  

Frazier PA, Tix AP, Barron KE. Testing moderator and mediator effects in counseling psychology research. US: American Psychological Association; 2004. p. 115–34.

Google Scholar  

Hill AB. The Environment and Disease: Association or Causation? Proc R Soc Med. 1965;58:295–300.

CAS   PubMed   PubMed Central   Google Scholar  

Bosch M, van der Weijden T, Wensing M, Grol R. Tailoring quality improvement interventions to identified barriers: a multiple case analysis. J Eval Clin Pract. 2007;13(2):161–8.

Claridge JA, Fabian TC. History and development of evidence-based medicine. World J Surg. 2005;29(5):547–53.

Cook SC, Schwartz AC, Kaslow NJ. Evidence-Based psychotherapy: advantages and challenges. Neurotherapeutics. 2017;14(3):537–45.

Dissemination and Implementation Research in Health (R01 Clinical Trial Optional). National Institutes of Health (NIH); 2019. https://grants.nih.gov/grants/guide/pa-files/PAR-19-274.html .

Thomas J, Brunton J, Graziosi S. EPPI-Reviewer 4: software for research synthesis. EPPI-Centre Software. London: Social Science Research Unit, UCL Institute of Education; 2010.

Williams NJ. Multilevel mechanisms of implementation strategies in mental health: integrating theory, research, and practice. Adm Policy Ment Health. 2016;43(5):783–98.

Hong QN, Pluye P, Fabregues S, Bartlett G, Boardman F, Cargo M, et al. Mixed Methods Appraisal Tool (MMAT) Montreal, Canada: McGill University; 2018 [Available from: http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/127916259/MMAT_2018_criteria-manual_2018-08-01_ENG.pdf .

Bardosh KL, Murray M, Khaemba AM, Smillie K, Lester R. Operationalizing mHealth to improve patient care: a qualitative implementation science evaluation of the WelTel texting intervention in Canada and Kenya. Global Health. 2017;13(1):87.

Brewster AL, Curry LA, Cherlin EJ, Talbert-Slagle K, Horwitz LI, Bradley EH. Integrating new practices: a qualitative study of how hospital innovations become routine. Implement Sci. 2015;10:168.

Carrera PM, Lambooij MS. Implementation of out-of-office blood pressure monitoring in the netherlands: from clinical guidelines to patients’ adoption of innovation. Medicine. 2015;94(43):e1813.

Frykman M, Hasson H, Muntlin Athlin Å, von Thiele Schwarz U. Functions of behavior change interventions when implementing multi-professional teamwork at an emergency department: a comparative case study. BMC Health Serv Res. 2014;14:218.

Wiener-Ogilvie S, Huby G, Pinnock H, Gillies J, Sheikh A. Practice organisational characteristics can impact on compliance with the BTS/SIGN asthma guideline: qualitative comparative case study in primary care. BMC Fam Pract. 2008;9:32.

Atkins MS, Frazier SL, Leathers SJ, Graczyk PA, Talbott E, Jakobsons L, et al. Teacher key opinion leaders and mental health consultation in low-income urban schools. J Consult Clin Psychol. 2008;76(5):905–8.

Baer JS, Wells EA, Rosengren DB, Hartzler B, Beadnell B, Dunn C. Agency context and tailored training in technology transfer: a pilot evaluation of motivational interviewing training for community counselors. J Subst Abuse Treat. 2009;37(2):191–202.

Bonetti D, Eccles M, Johnston M, Steen N, Grimshaw J, Baker R, et al. Guiding the design and selection of interventions to influence the implementation of evidence-based practice: an experimental simulation of a complex intervention trial. Soc Sci Med. 2005;60(9):2135–47.

Garner BR, Godley SH, Bair CML. The impact of pay-for-performance on therapists’ intentions to deliver high quality treatment. J Subst Abuse Treat. 2011;41(1):97–103.

Glisson C, Schoenwald SK, Hemmelgarn A, Green P, Dukes D, Armstrong KS, et al. Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. J Consult Clin Psychol. 2010;78(4):537–50.

Holth P, Torsheim T, Sheidow AJ, Ogden T, Henggeler SW. Intensive quality assurance of therapist adherence to behavioral interventions for adolescent substance use problems. J Child Adolesc Subst Abuse. 2011;20(4):289–313.

Lee H, Hall A, Nathan N, Reilly KL, Seward K, Williams CM, et al. Mechanisms of implementing public health interventions: a pooled causal mediation analysis of randomised trials. Implement Sci. 2018;13(1):42.

Lochman JE, Boxmeyer C, Powell N, Qu L, Wells K, Windle M. Dissemination of the coping power program: importance of intensity of counselor training. J Consult Clin Psychol. 2009;77(3):397–409.

Rapkin BD, Weiss E, Lounsbury D, Michel T, Gordon A, Erb-Downward J, et al. Reducing Disparities in cancer screening and prevention through community-based participatory research partnerships with local libraries: a comprehensive dynamic trial. Am J Community Psychol. 2017;60(1-2):145–59.

Rohrbach LA, Graham JW, Hansen WB. Diffusion of a school-based substance abuse prevention program: predictors of program implementation. Prev Med. 1993;22(2):237–60.

Article   CAS   PubMed   Google Scholar  

Seys D, Bruyneel L, Sermeus W, Lodewijckx C, Decramer M, Deneckere S, et al. Teamwork and adherence to recommendations explain the effect of a care pathway on reduced 30-day readmission for patients with a COPD exacerbation. COPD. 2018;15(2):157–64.

Williams NJG, C. The role of organizational culture and climate in the dissemination and implementation of empirically-supported treatments for youth. Dissemination and implementation of evidence based practices in child and adolescent mental health. New York: Oxford University Press; 2014. p. 61-81.

Williams NJ, Glisson C, Hemmelgarn A, Green P. Mechanisms of change in the ARC Organizational strategy: increasing mental health clinicians' EBP adoption through improved organizational culture and capacity. Adm Policy Ment Health. 2017;44(2):269–83.

Aarons GA, Sommerfeld DH, Walrath-Greene CM. Evidence-based practice implementation: the impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implement Sci. 2009;4:83.

Becker SJ, Squires DD, Strong DR, Barnett NP, Monti PM, Petry NM. Training opioid addiction treatment providers to adopt contingency management: a prospective pilot trial of a comprehensive implementation science approach. Subst Abus. 2016;37(1):134–40.

Beenstock J, Sniehotta FF, White M, Bell R, Milne EMG, Araujo-Soares V. What helps and hinders midwives in engaging with pregnant women about stopping smoking? A cross-sectional survey of perceived implementation difficulties among midwives in the North East of England. Implement Sci. 2012;7:36.

Beets MW, Flay BR, Vuchinich S, Acock AC, Li KK, Allred C. School climate and teachers' beliefs and attitudes associated with implementation of the positive action program: a diffusion of innovations model. Prev Sci. 2008;9(4):264–75.

Bonetti D, Johnston M, Clarkson J, Turner S. Applying multiple models to predict clinicians' behavioural intention and objective behaviour when managing children's teeth. Psychol Health. 2009;24(7):843–60.

Chou AF, Vaughn TE, McCoy KD, Doebbeling BN. Implementation of evidence-based practices: applying a goal commitment framework. Health Care Manage Rev. 2011;36(1):4–17.

Chambers D, Simpson L, Neta G, UvT S, Percy-Laurry A, Aarons GA, et al. Proceedings from the 9th annual conference on the science of dissemination and implementation. Implementation Sci. 2017;12(1):48.

Article   Google Scholar  

David P, Schiff M. Self-efficacy as a mediator in bottom-up dissemination of a Research-supported intervention for young, traumatized children and their families. J Evid Inf Soc Work. 2017;14(2):53–69.

Edmunds JM, Read KL, Ringle VA, Brodman DM, Kendall PC, Beidas RS. Sustaining clinician penetration, attitudes and knowledge in cognitive-behavioral therapy for youth anxiety. Implement Sci. 2014;9.

Gnich W, Sherriff A, Bonetti D, Conway DI, Macpherson LMD. The effect of introducing a financial incentive to promote application of fluoride varnish in dental practice in Scotland: a natural experiment. Implement Sci. 2018;13(1):95.

Guerrero EG, Frimpong J, Kong Y, Fenwick K. Aarons GA. Health Care Manage Rev: Advancing theory on the multilevel role of leadership in the implementation of evidence-based health care practices; 2018.

Huis A, Holleman G, van Achterberg T, Grol R, Schoonhoven L, Hulscher M. Explaining the effects of two different strategies for promoting hand hygiene in hospital nurses: a process evaluation alongside a cluster randomised controlled trial. Implement Sci. 2013;8:41.

Little MA, Pokhrel P, Sussman S, Rohrbach LA. The process of adoption of evidence-based tobacco use prevention programs in California schools. Prev Sci. 2015;16(1):80–9.

Llasus L, Angosta AD, Clark M. Graduating baccalaureate students' evidence-based practice knowledge, readiness, and implementation. J Nurs Educ. 2014;53(Suppl 9):S82–9.

Nelson TD, Steele RG. Predictors of practitioner self-reported use of evidence-based practices: practitioner training, clinical setting, and attitudes toward research. Adm Policy Ment Health. 2007;34(4):319–30.

Potthoff S, Presseau J, Sniehotta FF, Johnston M, Elovainio M, Avery L. Planning to be routine: habit as a mediator of the planning-behaviour relationship in healthcare professionals. Implement Sci. 2017;12(1):24.

Presseau J, Grimshaw JM, Tetroe JM, Eccles MP, Francis JJ, Godin G, et al. A theory-based process evaluation alongside a randomised controlled trial of printed educational messages to increase primary care physicians' prescription of thiazide diuretics for hypertension [ISRCTN72772651]. Implement Sci. 2016;11(1):121.

Simmonds MJ, Derghazarian T, Vlaeyen JW. Physiotherapists' knowledge, attitudes, and intolerance of uncertainty influence decision making in low back pain. Clin J Pain. 2012;28(6):467–74.

Stockdale SE, Rose D, Darling JE, Meredith LS, Helfrich CD, Dresselhaus TR, et al. Communication among team members within the patient-centered medical home and patient satisfaction with providers: the mediating role of patient-provider communication. Med Care. 2018;56(6):491–6.

Wanless SB, Rimm-Kaufman SE, Abry T, Larsen RA, Patton CL. Engagement in training as a mechanism to understanding fidelity of implementation of the responsive classroom approach. Prev Sci. 2015;16(8):1107–16.

Armson H, Roder S, Elmslie T, Khan S, Straus SE. How do clinicians use implementation tools to apply breast cancer screening guidelines to practice? Implement Sci. 2018;13(1):79.

Birken SA, Lee S-YD, Weiner BJ, Chin MH, Chiu M, Schaefer CT. From strategy to action: how top managers’ support increases middle managers’ commitment to innovation implementation in healthcare organizations. Health Care Manage Rev. 2015;40(2):159–68.

Kauth MR, Sullivan G, Blevins D, Cully JA, Landes RD, Said Q, et al. Employing external facilitation to implement cognitive behavioral therapy in VA clinics: a pilot study. Implement Sci. 2010;5(1):75.

Lukas CV, Mohr DC, Meterko M. Team effectiveness and organizational context in the implementation of a clinical innovation. Qual Manag Health Care. 2009;18(1):25–39.

Panzano PC, Sweeney HA, Seffrin B, Massatti R, Knudsen KJ. The assimilation of evidence-based healthcare innovations: a management-based perspective. J Behav Health Serv Res. 2012;39(4):397–416.

Rangachari P, Madaio M, Rethemeyer RK, Wagner P, Hall L, Roy S, et al. The evolution of knowledge exchanges enabling successful practice change in two intensive care units. Health Care Manage Rev. 2015;40(1):65–78.

Shrubsole K, Worrall L, Power E, O'Connor DA. The acute aphasia implementation study (AAIMS): a pilot cluster randomized controlled trial. Int J Lang Commun Disord. 2018;53(5):1021–56.

Scott SD, Albrecht L, O'Leary K, Ball GD, Hartling L, Hofmeyer A, et al. Systematic review of knowledge translation strategies in the allied health professions. Implement Sci. 2012;7:70.

Yamada J, Squires JE, Estabrooks CA, Victor C, Stevens B, Pain CTiCs. The role of organizational context in moderating the effect of research use on pain outcomes in hospitalized children: a cross sectional study. BMC Health Serv Res. 2017;17(1):68.

Rogers E. Diffusion of innovations. 4th ed. New York: Free Press; 1995.

Rogers E. Diffusion of Innovations. 3rd ed. New York: Free Press; 1983.

Ajzen I. The theory of planned behavior. Organ Behav Hum Decis Process. 1991;50(2):179–211.

Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A, et al. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care. 2005;14(1):26–33.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care. 1998;7(3):149–58.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

Klein KJ, Sorra JS. The challenge of innovation implementation. Acad Manage Rev. 1996;21(4):1055–80.

Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly. 1989;13(3):319–40.

Thompson RS, Higgins CA, Howell JM. Personal computing: toward a conceptual model of utilization. MIS Quarterly. 1991;15(1):125–43.

Braksick LW. Unlock behavior, unleash profits: developing leadership behavior that drives profitability in your organization. New York, NY: McGraw-Hill; 2007.

Johnson J, Dakens L, Edwards P, Morse N. SwitchPoints: culture change on the fast track to business success. Hoboken, NJ: John Wiley & Sons; 2008.

Hedeker D, Gibbons RD. Longitudinal data analysis. New York, NY: Wiley; 2006.

Krull JL, MacKinnon DP. Multilevel modeling of individual and group level mediated effects. Multivariate Behav Res. 2001;36(2):249–77.

Bandura A. Self-efficacy: the exercise of control. New York: Macmillan; 1997.

Bandura A. Exercise of human agency through collective efficacy. Curr Dir Psychol Sci. 2000;9(3):75–8.

Baron RM, Kenny DA. The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J Pers Soc Psychol. 1986;51(6):1173–82.

Sobel ME. Asymptotic confidence intervals for indirect effects in structural equation models. In: Leinhart S, editor. Sociological Methodology. San Francisco: Jossey-Bass; 1982.

Raudenbush SW, Bryk AS, Cheong YF, Congdon RT. HLM7: hierarchical linear and nonlinear modeling. Chicago: Scientific Software International; 2004.

Hosmer DW, Lemeshow S. Applied logistic regression. New York, NY: John Wiley & Sons; 1989.

Raudenbush SW, Bryk A, Congdon RT. HLM 6. Scientific Software International: Lincolnwood, IL; 2005.

Singer JD, Willet JB. Applied longitudinal data analysis: modeling change and event occurrence. New York, NY: Oxford University Press; 2003.

Cane J, O'Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:37.

Imai K, Keele L, Tingley D. A general approach to causal mediation analysis. Psychol Methods. 2010;15(4):309–34.

van Buuren SG-O, K. Mice: multivariate imputation by chained equations in R. J Stat Softw. 2010:1–68.

Rogers EM. Diffusion of innovations. 5th ed. New York, NY: Free Press; 2003.

Raudenbush SW, Liu X. Statistical power and optimal design for multisite randomized trials. Psychol Methods. 2000;5(2):199–213.

Allison PD. Event history analysis. Thousand Oaks, CA: SAGE Publications; 1984.

Yuk Fai C, Randall PF, Stephen WR. Efficiency and robustness of alternative estimators for two- and three-level models: the case of NAEP. J Educ Behav Stat. 2001;26(4):411–29.

Hox JJ, Maas CJM. The accuracy of multilevel structural equation modeling with pseudobalanced groups and small samples. Struct Equ Model. 2001;8(2):157–74.

Zhang Z, Zyphur MJ, Preacher KJ. Testing multilevel mediation using hierarchical linear models: problems and solutions. Organizational Research Methods. 2009;12(4):695–719.

Scott WR. Institutions and Organizations. Thousand Oaks, CA: Sage; 2001.

Eisenberger R, Huntington R, Hutchison S, Sowa D. Perceived organizational support. Journal of Applied Psychology. 1986;71:500–7.

Preacher KJ, Hayes AF. SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behav Res Methods Instrum Comput. 2004;36(4):717–31.

Chen HT. Theory-driven evaluations. In: Reynolds HJ, Walber HJ, editors. Advances in educational productivity: evaluation research for educational productivity. 7th ed. Bingley, UK: Emerald Group Publishing Limited; 1998.

Marsh HW, Hau KT, Balla JR, Grayson D. Is More Ever Too Much? The Number of indicators per factor in confirmatory factor analysis. Multivariate Behav Res. 1998;33(2):181–220.

Bandalos DL, Finney SJ. Item parceling issues in structural equation modeling. In: Marcoulides GA, editor. New developments and techniques in structural equation modeling. Mahwah, NJ: Erlbaum; 2001. p. 269–96.

Bandura A. Health promotion from the perspective of social cognitive theory. Psychol Health. 1998;13(4):623–49.

Blackman D. Operant conditioning: an experimental analysis of behaviour. London, UK: Methuen; 1974.

Gollwitzer PM. Implementation intentions: strong effects of simple plans. Am Psychol. 1999;54:493–503.

Leventhal H, Nerenz D, Steele DJ. Illness representations and coping with health threats. In: Baum A, Taylor SE, Singer JE, editors. Handbook of psychology and health, volume 4: social psychological aspects of health. Hillsdale, NJ: Lawrence Erlbaum; 1984. p. 219–51.

Weinstein N. The precaution adoption process. Health Psychol. 1988;7:355–86.

Prochaska JO, DiClemente CC. Stages and processes of self-change of smoking: toward an integrative model of change. J Consult Clin Psychol. 1983;51(3):390–5.

Landy FJ, Becker W. Motivation theory reconsidered. In: Cumming LL, Staw BM, editors. Research in organizational behavior. Greenwich, CT: JAI Press; 1987.

Locke EA, Latham GP. Building a practically useful theory of goal setting and task motivation: a 35-year odyssey. Am Psychol. 2002;57(9):705–17.

Kennedy P. A guide to econometrics. Cambridge, MA: MIT Press; 2003.

Joreskog KGS, D. LISRELR 8: User’s reference guide. Lincolnwood, IL: Scientific Software International; 1996.

Valente TW. Social network thresholds in the diffusion of innovations. Social Networks. 1996;18:69–89.

Hayes AF. Beyond Baron and Kenny: Statistical mediation analysis in the new millennium. Communication Monographs. 2009;76:408–20.

Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38(1):4–23.

Raudenbush SW, Bryk AS. Hierarchical linear models. Thousand Oaks: Sage; 2002.

Bryk AS, Raudenbush SW. Hierarchical linear models. Newbury Park, CA: Sage; 1992.

Muthén LK, Muthén BO. Mplus user's guide Los Angeles, CA: Muthén & Muthén 2012 [Seventh Edition:[Available from: https://www.statmodel.com/download/usersguide/Mplus%20user%20guide%20Ver_7_r3_web.pdf .

Bentler PM. On tests and indices for evaluating structural models. Personal Individ Differ. 2007;42(5):825–9.

MacKinnon DP, Fairchild AJ, Fritz MS. Mediation analysis. Annu Rev Psychol. 2007;58:593–614.

Graham I, Logan J, Harrison M, Straus S, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26.

Epstein S. Cognitive-experiential self-theory. In: Pervin LA, editor. Handbook of personality: theory and research. New York: Guilford; 1990. p. 165–92.

Karlson KB, Holm A, Breen R. Comparing Regression coefficients between same-sample Nested models using logit and probit: a new method. Sociological Methodology. 2012;42(1):274–301.

Rycroft-Malone J, Kitson A, Harvey G, McCormack B, Seers K, Titchen A, et al. Ingredients for change: revisiting a conceptual framework. BMJ Qual Saf. 2002;11(2):174–80.

Article   CAS   Google Scholar  

Yukl G, Gordon A, Taber T. A hierarchical taxonomy of leadership behavior: integrating a half century of behavior research. J Leadersh Organ Stud. 2002;9(1):15–32.

Shrout PE, Bolger N. Mediation in experimental and nonexperimental studies: new procedures and recommendations. Psychol Methods. 2002;7(4):422–45.

Fixsen DL, Naoom SF, Blase KA, Friedman RM. Implementation research: a synthesis of the literature; 2005.

Frambach R. An integrated model of organizational adoption and diffusion of innovations. Eur J Mark. 1993;27(5):22–41.

Institute of Medicine (IOM). Crossing the quality chasm: a new health system for the 21st century. Washington, DC: Institute of Medicine, National Academy Press; 2001.

Preacher KJ, Hayes AF. Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behav Res Methods. 2008;40(3):879–91.

Stahmer AC, Suhrheinrich J, Schetter PL, McGee HE. Exploring multi-level system factors facilitating educator training and implementation of evidence-based practices (EBP): a study protocol. Implement Sci. 2018;13(1):3.

Lewis CC, Klasnja P, Powell BJ, Lyon AR, Tuzzio L, Jones S, et al. From classification to causality: advancing understanding of mechanisms of change in implementation science. Front Public Health. 2018;6:136.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76.

Weiner BJ, Lewis MA, Clauser SB, Stitzenberg KB. In search of synergy: strategies for combining interventions at multiple levels. J Natl Cancer Inst Monogr. 2012;2012(44):34–41.

Lewis CC, Weiner BJ, Stanick C, Fischer SM. Advancing implementation science through measure development and evaluation: a study protocol. Implement Sci. 2015;10:102.

Powell BJ, Stanick CF, Halko HM, Dorsey CN, Weiner BJ, Barwick MA, et al. Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping. Implement Sci. 2017;12(1):118.

Wu AD, Zumbo BD. Understanding and using mediators and moderators. Soc Indic Res. 2007;87(3):367.

MacKinnon DP, Lockwood CM, Hoffman JM, West SG, Sheets V. A comparison of methods to test mediation and other intervening variable effects. Psychol Methods. 2002;7(1):83.

Pituch KA, Murphy DL, Tate RL. Three-level models for indirect effects in school- and class-randomized experiments in education. J Exp Educ. 2009;78(1):60–95.

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ : British Medical Journal. 2015;350:h1258.

Pawson R, Manzano-Santaella A. A realist diagnostic workshop. Evaluation. 2012;18(2):176–91.

Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist synthesis: an introduction. Manchester, UK: ESRC Research Methods Programme, University of Manchester; 2004.

Download references

Acknowledgments

Not applicable.

Availability of data and material

The authors are willing to share the raw data tables that informed the summary tables included in this manuscript.

This project was supported by grant number R13HS025632 from the Agency for Healthcare Research and Quality. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.

Author information

Authors and affiliations.

Kaiser Permanente Washington Health Research Institute, 1730 Minor Avenue, Suite 1600, Seattle, WA, 98101, USA

Cara C. Lewis & Callie Walsh-Bailey

Department of Psychological and Brain Sciences, Indiana University, 1101 E 10th Street, Bloomington, IN, 47405, USA

Cara C. Lewis

Department of Psychiatry and Behavioral Sciences, School of Medicine, University of Washington, 1959 NE Pacific Avenue, Seattle, WA, 98195, USA

Cara C. Lewis & Aaron R. Lyon

Department of Psychology, University of California Los Angeles, 1177 Franz Hall, 502 Portola Plaza, Los Angeles, CA, 90095, USA

Meredith R. Boyd

Brown School, Washington University in St. Louis, 1 Brookings Drive, St. Louis, MO, 63130, USA

Callie Walsh-Bailey

Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, 3535 Market Street, Philadelphia, PA, 19104, USA

Rinad Beidas

Department of Research and Evaluation, Kaiser Permanente Southern California, 100 S Los Robles Avenue, Pasadena, CA, 91101, USA

Brian Mittman

Department of Psychiatry, School of Medicine, University of California San Diego, 9500 Gilman Drive, La Jolla, CA, 92093, USA

Gregory A. Aarons

Department of Health Services, University of Washington, 1959 NE Pacific Street, Seattle, WA, 98195, USA

Bryan J. Weiner

Division of Cancer Control and Population Science, National Cancer Institute, 9609 Medical Center Drive, Rockville, MD, 20850, USA

David A. Chambers

You can also search for this author in PubMed   Google Scholar

Contributions

CCL conceptualized the larger study and articulated the research questions with all coauthors. CCL, MRB, and CWB designed the approach with feedback from all coauthors. MRB and CWB executed the systematic search with oversight and checking by CCL. MRB led the data extraction and CWB led the study appraisal. All authors contributed to the discussion and reviewed and approved the manuscript.

Corresponding author

Correspondence to Cara C. Lewis .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1: figure s1..

Inclusion and Exclusion Criteria and Definitions.

Additional file 2.

PRISMA 2009 Checklist.

Additional file 3.

Emergent Mechanism Models.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Lewis, C.C., Boyd, M.R., Walsh-Bailey, C. et al. A systematic review of empirical studies examining mechanisms of implementation in health. Implementation Sci 15 , 21 (2020). https://doi.org/10.1186/s13012-020-00983-3

Download citation

Received : 12 November 2019

Accepted : 12 March 2020

Published : 16 April 2020

DOI : https://doi.org/10.1186/s13012-020-00983-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Determinant
• Implementation
• Causal model

Implementation Science

ISSN: 1748-5908

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Organizational transformation: a systematic review of empirical research in health care and other industries

Affiliation.

• 1 University of Michigan, Ann Arbor, MI, USA. [email protected]
• PMID: 22955697
• DOI: 10.1177/1077558712458539

Health care organization leaders and policy makers seeking ways to reform the delivery of health care have become increasingly interested in transformational change. To foster understanding of how organizational transformation occurs and to stimulate further research, we report findings from a systematic review of empirical research on transformational change in the health care and non-health care literature, with a focus on the antecedents, processes (or paths), and outcomes of transformational change. Fifty-six studies, of which 13 were in health care, met our selection criteria. With one exception, all were published since 1990, indicating the recent upsurge of interest in this area. Limited differences were found between health care and non-health care studies. Available research documents the multiplicity of factors affecting change and the complexity of their interactions, but less information is available about the processes of transformational change than about its antecedents and consequences. Research and practice implications are discussed.

Publication types

• Research Support, Non-U.S. Gov't
• Systematic Review
• Delivery of Health Care / organization & administration*
• Empirical Research
• Health Services Research*
• Industry / organization & administration
• Organizational Innovation*

• Open access
• Published: 07 August 2023

The rise of resilient healthcare research during COVID-19: scoping review of empirical research

• Louise A Ellis 1 ,
• Maree Saba 1 ,
• Janet C Long 1 ,
• Hilda Bø Lyng 2 ,
• Cecilie Haraldseid-Driftland 2 ,
• Kate Churruca 1 ,
• Siri Wiig 2 ,
• Elizabeth Austin 1 ,
• Robyn Clay-Williams 1 ,
• Ann Carrigan 1 &
• Jeffrey Braithwaite 1  

BMC Health Services Research volume  23 , Article number:  833 ( 2023 ) Cite this article

1605 Accesses

2 Citations

5 Altmetric

Metrics details

The COVID-19 pandemic has presented many multi-faceted challenges to the maintenance of service quality and safety, highlighting the need for resilient and responsive healthcare systems more than ever before. This review examined empirical investigations of Resilient Health Care (RHC) in response to the COVID-19 pandemic with the aim to: identify key areas of research; synthesise findings on capacities that develop RHC across system levels (micro, meso, macro); and identify reported adverse consequences of the effort of maintaining system performance on system agents (healthcare workers, patients).

Three academic databases were searched (Medline, EMBASE, Scopus) from 1st January 2020 to 30th August 2022 using keywords pertaining to: systems resilience and related concepts; healthcare and healthcare settings; and COVID-19. Capacities that developed and enhanced systems resilience were synthesised using a hybrid inductive-deductive thematic analysis.

Fifty publications were included in this review. Consistent with previous research, studies from high-income countries and the use of qualitative methods within the context of hospitals, dominated the included studies. However, promising developments have been made, with an emergence of studies conducted at the macro-system level, including the development of quantitative tools and indicator-based modelling approaches, and the increased involvement of low- and middle-income countries in research (LMIC). Concordant with previous research, eight key resilience capacities were identified that can support, develop or enhance resilient performance, namely: structure, alignment, coordination, learning, involvement, risk awareness, leadership, and communication. The need for healthcare workers to constantly learn and make adaptations, however, had potentially adverse physical and emotional consequences for healthcare workers, in addition to adverse effects on routine patient care.

Conclusions

This review identified an upsurge in new empirical studies on health system resilience associated with COVID-19. The pandemic provided a unique opportunity to examine RHC in practice, and uncovered emerging new evidence on RHC theory and system factors that contribute to resilient performance at micro, meso and macro levels. These findings will enable leaders and other stakeholders to strengthen health system resilience when responding to future challenges and unexpected events.

Peer Review reports

Resilient Health Care (RHC) is defined as the ability of a system to adjust its functioning prior to, during, or following changes and disturbances, so that it can sustain required operations under both expected and unexpected conditions [ 1 ]. The COVID-19 pandemic presented challenges that healthcare systems must address to maintain service quality and safety, highlighting the need for resilient and responsive healthcare systems more than ever before [ 2 ]. Healthcare practitioners, managers, and policy makers had to suddenly, and dramatically, adapt in order to absorb the shock of the pandemic and coordinate the capacities needed to deal with its impact. Since the onset of the pandemic, ‘health systems resilience’ has emerged as a key concept in global public health with the World Health Organization (WHO) publishing several papers [ 3 , 4 , 5 , 6 , 7 ] on the importance of building and strengthening health emergency preparedness and responsiveness to future epidemics and shocks.

The application of resilience thinking to healthcare is however not new, with RHC being first proposed by Eric Hollnagel in 2011 [ 8 ] to describe the application of resilience engineering [ 9 ] and disaster resilience [ 10 , 11 ] to healthcare. RHC acknowledges the complex adaptive nature of healthcare, recognising the adaptive and transformative capabilities that enable healthcare systems to continue to perform their functions in the face of challenges [ 12 , 13 ]. Despite its conceptual appeal, there have been challenges in translating the principles of RHC into concrete improvements, with compelling examples remaining scarce [ 14 ].

The importance of RHC is reflected in the growing number of reviews on the topic [ 13 , 15 , 16 ]. Although these reviews identified that the RHC literature has been predominantly conceptual, rather than empirical [ 13 , 15 , 16 ], empirical applications of RHC have increased. A systematic review conducted prior to the pandemic identified 71 empirical studies on health system resilience from 2008 to 2019, with 62% of these published in the last two years of the review (i.e., from 2017 to 2019) [ 15 ]. However, much of this existing empirical literature has focused on clinical microsystems at the ‘sharp end’ and how frontline healthcare professionals within hospital settings collectively adapt, ‘work around’, or enable things to go well [ 2 , 13 ], with a lack of empirical studies particularly at the meso and macro-levels (i.e., government, national, international) [ 14 ]. Qualitative research methods have also predominated in the empirical studies [ 13 , 15 ], reflecting that priorities have been placed on gaining in-depth understanding of everyday clinical work at the micro-level.

Another noteworthy gap in the RHC literature is the limited discussions on how ‘individual agents’ (e.g., doctors, nurses) [ 17 ] within the health system may be personally affected by their efforts to maintain system resilience [ 18 ]. However, the time appears ripe for this issue to be explored in the context of RHC, particularly in light of the COVID-19 pandemic, which has caused major disruptions across all system levels and created a need for ongoing adaptation by healthcare workers, which many suggest has resulted in widespread mental health issues and burnout amongst these workers [ 19 , 20 ].

The present study

Interest in RHC has accelerated since the onset of the COVID-19 pandemic, as indicated by the sharp increase in the number of publications in ‘health systems resilience’ since 2020 (Fig.  1 ). With the growth in empirical contributions in this field, it is timely to examine the published empirical research to determine the status of the field and identify whether there is any further evidence on how to generate or strengthen resilient performance to manage future pandemics and emergencies. Understanding factors that develop or enhance RHC is critical to developing strategies and tools for strengthening their resilience [ 12 ]. For this review, we defined an empirical study as one that reports primary or secondary data gathered by means of a specific methodological approach [ 21 ]. The objective of this study was to conduct a scoping review of empirical investigations of RHC in response to the COVID-19 pandemic with four key aims:

Map out the empirical research within the resilient healthcare domain across all system levels (micro, meso, macro).

Identify the key areas of research, including study designs and research methods that have been employed.

Synthesise findings on factors (capacities, actions, or strategies) that developed or enhanced resilient performance.

Identify any reported findings on consequences of maintaining system performance on system agents (healthcare workers, patients).

Increased publications in PubMed using the search term “health systems resilience” in titles or abstracts

The review followed a pre-determined protocol, developed in accordance with the Preferred Reporting Items of Systematic Review and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) [ 22 , 23 ] (also see PRISMA-ScR in Supplementary File 1 ). A scoping review method was used; a method which is used to examine the extent, range and nature of work on this topic and to identify gaps and provide suggestions to improve future directions for RHC research [ 24 ]. Quality assessments were not undertaken, as the aim was to examine the full breadth of the empirical literature, consistent with general aims and methodology of scoping reviews [ 25 ].

Search strategy

Three academic databases (Medline, EMBASE, Scopus) were searched from 1st January 2020 to 30th August 2022. The search strategy consisted of terms pertaining to: systems resilience (e.g., resilient healthcare) and related concepts (e.g., Safety-II); healthcare (e.g., health care) and healthcare settings (e.g., primary care, hospital); and COVID-19. The search strategy was adapted for each database as necessary (see Supplementary File 2 for the complete search strategy, using Ovid MEDLINE as an example). The search strategy was developed in consultation with an academic research librarian and was reviewed by all authors prior to execution.

Inclusion and exclusion criteria

Articles were included if they were: (a) published between the onset of COVID-19 (from 1st January 2020) and 30 August 2022, (b) in the English language, (c) peer-reviewed publications, (d) had an explicit focus on healthcare or health systems resilience in the context of COVID-19, and (e) were empirical studies. Studies that only mentioned “resilience” briefly, were concerned with individual or psychological resilience (e.g., the psychological wellbeing of healthcare workers) rather than systems-resilience or were not conducted in the context of COVID-19 were excluded. Study protocols, review papers, journal commentaries, and editorials were also excluded, as were studies not in English.

Eligibility screening

Reference details (including abstracts) were downloaded into the reference management software Endnote X9 and then exported to Rayyan QCRI for title and abstract screening. Seven reviewers (LAE, MS, JCL, KC, EA, LT, DT) screened the title/abstracts to determine their inclusion against the criteria, with 5% of the retrieved publications being independently screened by the entire review team to ensure consistent inclusion. Any discrepancies among reviewers’ judgements were reviewed by two authors (LAE and MS) with JB available for consultation if and as needed.

Data extraction

Data from included studies meeting inclusion criteria were extracted into a custom workbook in Microsoft Excel. Full-text screening was conducted initially by two independent reviewers (LT, DT), with LAE and MS subsequently duplicating the full-text review process, with any discrepancies being discussed and resolved in consultation with JB. The extraction workbook included data items on: [ 1 ] publication details (paper title, year, output type); [ 2 ] study context (e.g., hospital, primary care); [ 3 ] system level (micro: healthcare practitioner; meso: management, organisation; and/or macro: government, national, international); [ 4 ] study design (quantitative, qualitative, mixed methods); [ 5 ] study data type (primary or secondary); [ 6 ] data collection method/s (quantitative, qualitative, mixed-methods); [ 7 ] conceptual framework, model, or theory used; [ 8 ] resilience measure or tool used; [ 9 ] factors (capacities, actions, or strategies) that developed and enhanced systems resilience; and [ 10 ] reported negative consequences of resilient performance on system agents (healthcare workers, patients).

Data synthesis and analysis

A data-based convergent synthesis was employed [ 26 ]; where quantitative data were transformed into categories or themes, and summarised through narrative techniques [ 27 ]. Country of the corresponding author was coded by income classification based on World Bank definitions of gross national income per capita. The three categories were low (< US$1085), middle (US$1086–13,205), and high income (> US$13,205) [ 28 ]. Data collection methods were categorised as qualitative, quantitative or mixed methods, with specific data collection methods (e.g., interviews, surveys) also extracted and examined.

The factors that supported, developed or enhanced systems resilience were initially identified through an inductive thematic approach [ 29 ] by two authors (LAE, MS). Themes and sub-themes were then discussed and agreed by the researchers using an iterative process. Upon further analysis and reflection of the themes, it was clear that a number of the themes aligned with the ‘capacities’ for resilience outlined by Lyng et al. [ 30 ]. Therefore, in the next phase, a deductive approach was taken where the themes and sub-themes were mapped to eight of the resilience ‘capacities’. Some minor amendments were made based upon differences in themes identified in the literature included in the present review compared with the capacities. Two of the ‘capacities’ outlined by Lyng et al. [ 30 ], namely ‘competence’ and ‘facilitators’, were not included owing to the lack of data mapping to these themes, as identified from the initial inductive analysis. Themes and subthemes were cross-referenced across all studies to ensure that the revised thematic map captured the meaning across all the included studies. The last phase involved defining the themes (see Table  1 for definitions as applied in this study). Consequences of maintaining resilient performance were similarly identified using an inductive thematic approach [ 29 ] by two authors (LAE, MS).

Overview of included studies

The initial search retrieved a total of 5844 publications. After removing duplicates, 4634 remained for title and/or abstract review. Following title and/or abstract screening, 4404 publications were discarded as they did not meet the inclusion criteria. Based on the full-text assessment, a further 184 publications did not meet the inclusion criteria, resulting in 50 publications included in this review (see Supplementary File 3 for included articles). Figure  2 demonstrates the inclusion and exclusion of papers at each stage of the screening process.

PRISMA flow diagram for study selection process

Summary characteristics of the included studies

A summary of the key characteristics of the included papers is provided in Table  2 . The 50 studies were spread widely, across 45 different journals, with Safety Science (n = 3, 6.0%) and the International Journal of Health Policy and Management (n = 3, 6.0%) being the most popular. The source location was also spread widely, across 25 different countries, with most corresponding authors from the United Kingdom (n = 8, 16.0%), followed by the United States (n = 6, 12.0%). Although most studies were restricted to high-income countries (n = 34, 68%), a notable number of corresponding authors were identified from low- and middle-income countries (LMIC) (n = 16, 32.0%), and with four (8%) of these being from Brazil.

Close to half (n = 20, 40%) of the studies were conducted in the context of hospitals, which generally involved hospital healthcare workers and/or hospital leaders as participants. Four studies (8%) [ 31 , 32 , 33 , 34 ] were specifically focused on supply chain issues related to medical supply availability in the context of system adaptability and resilience, and its impact on the healthcare system more broadly. Of the studies conducted in the context of community and specialised care (n = 15, 30%), a number were focused on the resilient performance of aged care services [ 35 , 36 , 37 ] or community mental health services [ 38 , 39 , 40 ]. Primary care was a setting in seven studies (14%), with a focus on the perspectives of primary care providers in relation to healthcare system resilience [ 38 , 41 , 42 , 43 , 44 , 45 , 46 ]. Over half of the studies were classified as being at the meso level (n = 29, 58%) of the healthcare system, with fewer studies being at the micro level (n = 17, 34%) or macro level (n = 18, 36%). Notably, eleven (61%) of these macro-level studies, incorporated data from multiple countries, such as a comparison study of health system resilience across six European countries, a comparison study of government actions and their relation to systems resilience between Canada and Australia, and an indicator-based analysis of risk and resilience that incorporated ‘big data’ from 11 countries.

Three-quarters of the studies were qualitative (n = 39, 78%), seven were mixed-methods (14%) and four were quantitative (8%). Although most studies utilised primary data alone (n = 39, 78%), seven studies relied on secondary datasets (14%), such as existing big data sources [ 47 ] and questionnaire data [ 48 , 49 ], and a smaller number used both primary and secondary datasets (n = 4, 8%).

Data collection methods and tools to assess RHC

Most of the studies collected data from direct sources (i.e., where participants directly express their experience of how work takes place in practice) [ 16 ], and included interviews (n = 32, 64%), surveys (n = 15, 30%) or focus groups (n = 3, 6%). A smaller number of studies included indirect sources, such as document analysis (n = 9, 18%), observations (n = 4, 8%), and/or simulation (n = 2, 4%). One-third of studies developed and/or used tools to study RHC (n = 17, 34%); of these, over half employed researcher-developed questionnaires to assess or understand resilient performance (n = 11, 65%), three adopted a ‘big data’ indicator-based approach to assess systems resilience for emergency preparedness, two studies drew on the more commonly regarded Functional Resonance Analysis Method (FRAM) [ 50 ], and one study used observation tools based on the “Mayo high performance team scale” [ 51 ] and the “Scrub Practitioners List of Intra-operative Non-Technical Skills (SPLINTS)” [ 52 ].

Over half the researcher-developed questionnaires (n = 7, 64%) were based on a conceptual framework, including Hollnagel’s [ 53 ] ‘four cornerstones of resilience’ [ 54 ], Anderson et al.’s [ 55 ] Integrated Resilience Attributes Framework [ 56 ], Bueno et al.’s [ 57 ] guidelines for coping with complexity [ 58 ], Macrae and Wiig’s [ 59 ] resilience framework [ 35 ], the WHO’s [ 60 ] fundamental ‘building blocks’ of health systems [ 61 , 62 ] and the WHO’s hospital readiness checklist [ 63 , 64 ]. Three additional survey studies lacking a conceptual framework collected predominantly open-ended questionnaire data on how everyday clinical work is being performed during the pandemic (i.e., work-as-done), via the perceptions and experiences of healthcare workers [ 32 , 43 ], using inductive content analysis, and to confirm or corroborate any emerging themes identified from interview data [ 65 ]. One final questionnaire tool was developed to assess hospital inventory management, including the impact of COVID-19 on the availability of supply and the processes established to enhance supply chain resilience [ 31 ].

Capacities that developed and enhanced resilient performance

Based on the analysis of the included studies, eight key factors or capacities were identified at different system levels to develop or enhance resilient performance, as outlined in the following section. In this section, the eight resilience capacities have been discussed sequentially from the capacity that occurred most prevalently within the included studies to the capacity that occurred least prevalently, namely: structure, alignment, coordination, learning, involvement, risk awareness, leadership, and communication. Figure  3 provides a visual summary of the eight factors and their sub-themes (also see Supplementary File 4 giving examples for each subtheme).

Resilience capacities and related sub-themes

Structure as a capacity for resilience was identified in more than four-fifths of included studies (n = 37, 74%) and referred to the structures that support work and practice within healthcare organisations. Across the included studies in this review, five sub-themes contributed to structural capacity, including: technology, physical equipment, workforce, governance systems and financial resources.

The most prevalent among the subthemes, technology (n = 27, 54%), concerned how software and hardware were utilised during the pandemic to support the continued delivery of regular healthcare services, as well as COVID-specific responses. Several studies highlighted a spike in the use of different technologies to enable the provision of patient care in different settings [ 41 , 44 , 66 , 67 ]. For example, Gifford et al. [ 66 ] reported the way in which wards and outpatient clinics rapidly converted to “digital” wards involving e-health, video and phone consultations. Alternatively, in one study from Canada [ 68 ], a lack of appropriate technology impeded resilient performance, with the rapid but “piecemeal” adoption of multiple virtual care technologies during COVID-19 resulting in systems that duplicated administrative work for healthcare professionals.

Access to physical equipment (n = 18, 36%), such as personal protective equipment (PPE), or flexible workspaces, was another prevalent subtheme across the studies. In many instances it was the lack of availability of this equipment, particularly during the early stages of the pandemic, that impeded the COVID response [ 36 , 46 , 69 ]. However, several studies reported the way in which organisations rapidly responded by adapting equipment levels, including how and where they sourced physical equipment, as well as their novel repurposing of in-house equipment [ 35 ] and wards to create additional capacity [ 66 ].

Workforce (n = 11, 22%) involved access to staff, workforce stability, and the designation of roles and responsibilities. Some of these studies highlighted challenges in recruitment, and how understaffing affected resilient performance [ 39 , 69 ], as there was both increased demand for healthcare and staff shortages due to workers contracting COVID-19. Organisational adaptations to promote resilience and address this issue included the reassignment of staff to other parts of the hospital [ 56 ] and expanding their reach in hiring new staff, which included the provision of financial incentives [ 39 ] and the re-employment of recently retired staff [ 66 ].

Governance systems and protocols (n = 19, 38%) involved the development of new policies, or modification of existing ones, to support the many changes in work practices during the pandemic. In some instances, these policies were devised at a macro-level [ 39 ], while in others they were more locally developed [ 70 ]. Along with this, financial resources (n = 5, 10%), involved funding changes wrought by the pandemic, including the allocation of funding to support COVID care delivery [ 71 ], as well as the financial implications of the pandemic in lost revenue due to a reduction in consultations, particularly identified for small healthcare providers [ 41 ].

Alignment as a capacity for resilient performance referred to the adaptation of practices in response to the ever-changing problems posed by the COVID-19 pandemic [ 30 ]. Identified in over half of the included studies (n = 30, 60%), the alignment capacity included three subthemes: role evolution; micro-level workarounds and trade-offs; and meso- to macro-level re-structuring, rescaling and compensation strategies.

Role evolution (n = 13, 26%) concerned how roles and responsibilities of healthcare workers and leaders changed or expanded in response to the ongoing challenges of the pandemic. Healthcare managers and leaders were asked to step into different functions; for example, in crisis management, communications and crisis responses [ 66 ]. Clinical staff also needed to expand their responsibilities, extend their working hours, and were redeployed to other wards to fulfill staff shortages and meet patient demands [ 66 ]. A smaller number of staff were redeployed to special COVID-19 teams, providing direct care to infected patients [ 56 , 66 , 72 ] and healthcare leaders worked from home [ 56 ], to limit further staff exposure to the virus. The change in workspace and role, as well as the pressing needs of COVID-19 infected patients, meant that staff had to be trained in new procedures and practices; for instance, redeployed physiotherapists into intensive care units and research staff into clinical roles [ 71 ]. Although redeployment sometimes caused stress and uncertainty, with the additional challenge of unfamiliar workspaces and colleagues, redeployment was also perceived as an opportunity for positive career development and empowerment [ 65 ].

The COVID-19 pandemic introduced a need for healthcare workers to improvise and develop solutions to unexpected and frequent problems, introducing workarounds and trade-offs (n = 19, 38%) at the micro-system level. Several studies highlighted how healthcare workers developed unique and creative workarounds at the front-line to help them cope with ongoing challenges [ 35 , 41 , 66 , 70 ]. For example, workarounds intended to ease the impact of the pandemic on patients and their families included: decorating PPE masks, using dance as a greeting instead of hugging, and providing outdoor concerts for patients [ 35 , 70 ]. Additionally, some studies described staff changes in prioritization, also known in the RHC literature as trade-offs, directing their capacity to where it was needed most. This meant that scheduled surgeries and regular care were scaled down to increase capacity such as in intensive care units (ICUs) and emergency departments [ 66 ]. The risk of infection also introduced trade-offs for community health workers, as home visits were no longer allowed; instead, community health workers began to take on administrative tasks at health clinics [ 43 ].

The COVID-19 pandemic also led to alignment strategies at the meso- and macro-levels, as COVID-19 provided exceptional demands for all parts of the health system. Re-organisation , rescaling and compensation (n = 19, 38%) strategies at the organizational level included arranging for COVID-19 treatment areas, wards, assessment clinics, COVID-19 teams, and new types of administration [ 71 ]. Furthermore, new emergency plans, policies, and safety standards, such as providing separate entrances and exits at nursing homes [ 35 ], were initiated to limit spread of the virus [ 69 ]. Unlike their traditional way of working, strategies for restructuring, rescaling, and compensation often had to be created “on the go” due to the unpredictability and unfamiliarity of the situation [ 39 ]. However, two studies highlighted [ 58 , 66 ] that healthcare systems can cope more effectively with future crises by factoring in “slack resources” at an organizational level and collective level (i.e., network or national), thereby ensuring the continued availability of critical medical supplies, equipment, and human resources. Likewise, supply chain resilience studies described the adoption of “buffering” and “bridging” strategies [ 34 ], along with “strategic purchasing” [ 33 ], to ensure continued healthcare supply and equipment availability across the healthcare system.

Coordination

Coordination as a capacity for resilience referred to how teams facilitated and organised work within and between teams and organisations. Identified in over half (n = 28, 56%) of studies in this review, coordination included the following five subthemes: team cohesion; multidisciplinary teamwork; team communication; inter-organisational coordination; and intra-organisational coordination. In terms of team cohesion (n = 10, 20%), building a supportive and cohesive team was regarded as an important factor in developing and sustaining resilient performance, particularly at the clinical micro-systems of care. Several studies expressed increased “connection” [ 72 ], “collaboration” [ 39 , 70 , 71 , 72 ] and a “sense of camaraderie” [ 70 ] among teams during the pandemic as they “rallied together” [ 40 ] and “worked together toward a common goal” [ 70 ]. Traditional clinical hierarchies were also reported as less important during delivery of care [ 72 ], leading to enhanced team dynamics and coordination [ 73 ]. Three studies also highlighted the role of “peer support” [ 56 , 65 , 69 ] as co-workers provided reassurance and supported staff wellbeing.

Multidisciplinary teamwork (n = 10, 20%) was also emphasised as critical in developing and sustaining resilient performance during the pandemic. Multidisciplinary teamwork was often initially made more difficult (e.g., in cases where teams were physically divided, or fewer staff on site), however, healthcare workers adapted [ 70 ] and used creative solutions to make multidisciplinary care more accessible [ 44 , 56 , 70 , 74 ]. Hodgins et al. [ 71 ] described the “breaking down of silos”, with staff from different disciplines “coming together” to support each other and sustain resilience. Ensuring that team communication (n = 5, 10%) remained open within and between teams was also critical to ensure teams remained connected and up to date with the ever-changing situation, as well as helping to facilitate the support process [ 39 , 42 , 72 , 75 ].

Along with evolving processes and workflows, inter-organisational coordination (n = 15, 30%) and teamwork evolved throughout the pandemic. Several studies outlined the establishment of multidisciplinary teams being formed at the hospital throughout various stages of the pandemic (e.g., COVID-19-management teams, emergency response teams, specialist care teams) [ 40 , 63 , 66 , 72 , 74 ] to enable rapid response and care to changing situations. Resilient performance was fostered by experienced teams and inter-organisational collaborations who adapted and worked together, with tenacity and creativity, in ways that previously had not been required [ 36 , 67 , 70 ]. Intra-organisational coordination (n = 7, 14%) was also described as critical during the pandemic, providing a buffer to combat resource shortages (e.g., workforce, equipment, knowledge). Services were reported as drawing on both new and pre-existing relationships to overcome barriers to care [ 34 , 36 , 74 ].

Learning as a capacity for resilient performance described the facilitation of knowledge acquisition, through the provision of learning activities and opportunities [ 30 ]. Learning was identified in just under half of the included studies (n = 21, 42%), and consisted of three subthemes: on-the-job learning, training, and simulation.

On-the-job learning (n = 9, 18%) became particularly important during the COVID-19 pandemic. Exposure to new situations, equipment, and regulations, forced healthcare personnel to continuously adjust and learn during everyday work; for example, the appropriate use of protective equipment [ 35 ] or the prompt need to develop decision-making and communication skills [ 69 ]. The novelty of the situation, with lack of standardized treatment plans often brought a trial-and-error approach whereby healthcare personnel became prepared through on-going daily training sessions [ 72 ], and through shared knowledge and experience [ 65 , 69 , 72 ].

Training ( n = 15, 30%) referred to more planned and scheduled efforts to increase knowledge and preparedness through organised learning efforts, such as courses, simulations, e-learning, and workshops [ 56 ]. These training efforts had different aims than those before the pandemic, ranging from technical skill development, such as medical equipment [ 69 ], to non-technical skills such as management skills [ 66 , 70 ]. The training sessions often took place at in-house-learning arenas such as simulation centres or labs, but also online learning resources were applied to reach a boarder audience and avoid spread of the virus [ 70 ].

Simulation (n = 3, 6%) as a novel training approach was identified in a small number of studies to increase preparedness to the COVID- 19 situation. Simulations allowed for interdisciplinary teams to train together and become confident in their technical and non-technical skills [ 75 ]. New simulation teams were created, and schedules developed to run consecutive training sessions, allowing for a large part of the healthcare personnel to be involved in the training [ 71 ].

Involvement

Involvement, as a key capacity for resilience in healthcare, referred to how the organisation involved and supported effective interactions between different system actors such as family, patients, and other stakeholders [ 35 ]. Meaningful involvement was evident in over one-third (n = 18, 36%) of the included studies and identified through two subthemes: communication with patients and families, and meeting patients’ needs.

Technology and roles were leveraged as a means for communication with patients and families (n = 14, 28%) and ensured patients and families continued to be engaged with care delivery during the COVID-19 pandemic. Changes to protocols and policy intending to reduce the transmission of COVID-19 (e.g., physical distancing, reduced capacity) required healthcare personnel to adjust how patients and families were meaningfully involved in care from primarily face-to-face to remote platforms. For example, teleconsultation technology was used to facilitate patient access to care services including a 24-hour helpline [ 76 ], and new systems to provide care services with the means to monitor and support patients remotely [ 41 ]. Technology was also used during the ‘no visitor policy’ to allow COVID-19 patients to connect with their family and medical staff when in isolation [ 66 ]. Volunteer networks and patient navigators were also used to extend services and connect healthcare providers with families [ 70 , 77 ], with posters and flyers on public noticeboards also used to share important health related information with families with limited literacy [ 70 ].

Practices and processes were adapted to ensure the health system was meeting patients’ needs (n = 10, 20%) during the pandemic. Changes to practices and processes were intended to mitigate unintended consequences of reduced or remote interaction service delivery methods to manage COVID-19 (e.g., postponing care, contagion fear) and ensure care delivery strategies had the capacity to address the needs of patients and that patient access to care was maintained [ 38 ]. For example, nursing specific care delivery processes were adapted to overcome difficulties in involving patients and family members to meet the immediate needs of patients [ 72 ] and practices were reorganised to comply with hygienic guidelines, thus enabling patients with acute non-COVID-19 needs to access care [ 41 ].

Risk awareness

Risk awareness as a capacity for resilient performance, enhances a system’s resilience when understanding and responding to potential adverse events [ 30 ]. Identified in over one-third of included studies (n = 18, 36%), risk awareness comprised two subthemes: emergency preparedness; and proactive responses.

From the early stages of the pandemic, emergency preparedness (n = 10, 20%) to COVID-19 was fundamental in planning and arranging strategies to meet the constant demands on the health system [ 72 ]. The development and continued “fine-tuning” of emergency preparedness plans [ 39 , 41 , 42 , 61 , 78 ] has been described as both important and necessary [ 39 ]. Emergency plans were attuned to strengthen other resilience capacities, such as streamlining communication systems [ 42 , 78 ], governance structures (78) and decision-making structures, to ensure the “continued, effective operation of the health system” [ 42 ]. One study also highlighted that the knowledge and experience gained from COVID-19 has led to ongoing conversations at a leadership level around emergency preparedness for any future crises [ 39 ].

Monitoring and proactive response (n = 16, 32%) referred to the understanding of situational risks to allow for proactive responses at all healthcare levels [ 30 ]. Early responses to the pandemic were often described as “ad-hoc”, but as the pandemic progressed, indicators and responses were monitored internationally [ 36 , 72 , 79 ] to assess risk, enabling proactive rather than reactive responses to problems [ 36 , 72 , 79 ]. Several studies outlined the implementation of an emergency taskforce [ 36 , 61 , 72 ] which met daily to evaluate emerging evidence [ 36 ], or devised new prevention strategies [ 61 ] or digital healthcare supply chain strategy [ 78 ]. Other studies discussed organisational infrastructure to prepare for the future risk of an outbreak, such as tracking COVID-19 positive individuals within hospitals, monitoring PPE levels [ 71 ] and developing plans for housing patients at alternative locations [ 39 ].

Leadership (n = 16, 32%) as a resilient capacity demonstrated the important contribution of leaders to both their employees and the broader healthcare organisation. Four subthemes were identified that contributed to the leadership capacity: transparent and open communication; visibility at the frontlines of care; supportive and empowering; and decisive leadership.

Transparent and open communication (n = 4, 8%) from leaders was noted as crucial in dealing with the pandemic. Leaders were required to distribute a continuous flow of information from national and regional authorities to the front-line staff through various channels [ 35 ], providing updates as new information became known. In general, frontline staff found this information to be both useful and supportive [ 72 ].

Increased visibility of leaders at the frontlines of care (n = 8, 16%) was also identified as important. For example, Lyng et al. [ 35 ] reported that leaders at Norwegian nursing homes heavily affected by the pandemic altered their daily work schedules so they could be present at the frontlines of care. On the other hand, where staff expressed an absence of effective and visible leadership, there was a sense of “mistrust in leaders”, generating a negative environment [ 65 ].

Resilient performance was also associated with leaders who were s upportive and empowering (n = 8, 16%). Along with visibility at the frontlines, leaders were reported as providing logistical support, expressing “appreciation of hard work”, offering “motivations and rewards” to continue, and “empowerment” to adapt to the changed conditions [ 69 ]. At one large healthcare organisation, leaders were reported as showing genuine concern for their staff’s mental and physical wellbeing [ 39 ], and at others, as providing reassurance to “frightened and exhausted” staff [ 36 ].

The value of decisive leadership (n = 10, 20%) in enabling resilient performance during the pandemic was reported in several studies. The ongoing changing nature of the pandemic required leaders to make rapid decisions [ 36 ], be flexible yet decisive [ 39 ], take proactive steps, and adopt a more hierarchical “military” style of command [ 80 ]. For example, with the constant stream of new updates and information comings to leaders, they needed to adopt a “learning mindset” to respond effectively and be willing to change course if warranted by the new information [ 66 ].

Communication

In almost one-third of included studies (n = 15, 30%), communication was identified as a key capacity for resilient performance and included the systems of communication used to translate information within and between teams and organisations. Two main systems of communication were identified: formal communication, such as information communication technology [ 72 ] and policies sent via email [ 70 ]; and informal communication, such as social media apps [ 56 , 65 , 70 ].

Several studies reported the utilisation of formal communication systems (n = 10, 20%) during the COVID-19 pandemic. It was widely accepted that the pandemic necessitated the rapid upskilling and education of staff and patients, and it was crucial that information was accurately resourced and disseminated [ 71 ]. For example, rapidly changing information from national and regional authorities was circulated, and healthcare executives provided daily COVID-19 updates via several communication platforms, such as the staff intranet and emails [ 35 , 70 , 71 , 80 ]. Providers also received regular policy and procedural updates (e.g., infection control) as more information from regulatory bodies became available [ 72 ]. However, some communication gaps were also identified; for example, a lack of communication aligned with rapidly changing protocols that increased the difficulty of remaining informed [ 56 ]. Challenges included a lack of intra-and inter professional communication between other units [ 56 ], a lack of access to technology and inconsistent information [ 81 ].

Informal communication (n = 10, 20%) was also reported among many of the included studies, commonly involving the development of group chats via social media apps, such as WhatsApp. These communication tools facilitated the sharing of information, such as policy and procedural change, and helped to provide emotional support and load sharing at the start of the pandemic among teams [ 35 , 56 , 65 , 70 , 76 ].

Consequences on system agents

It was clear from the included studies that navigating the challenges of the COVID-19 pandemic, which came with the need to constantly learn and make adaptations in response to unexpected variation and changes, came at a personal cost to healthcare workers, particularly to those at the frontlines of care. Nine (18%) of the included studies reported that the increased workload and strenuous work conditions had negative physical consequences on healthcare workers [ 54 , 56 , 61 , 67 , 68 , 69 , 79 , 81 , 82 ]. For example, nurses reported increased “tiredness”, “exhaustion”, “muscle weakness” and “loss of appetite”, during the pandemic as a result of working longer shifts, often without breaks, while being “weighed down by PPE equipment” [ 67 , 69 ].

The pandemic also exposed staff to stressful situations, which had considerable emotional consequences on staff, a theme identified in one-third of studies (n = 17, 34%). During the early stages of the pandemic, COVID-19 created an environment of uncertainty and fear among the population as a whole, but especially among front line workers [ 43 ], who expressed fear of dying from COVID-19, depression, worry, and frustration, among other psychological complaints [ 69 ]. Leaders were no different, with one study reporting that COVID-19 had also been emotionally demanding for staff in administrative and clinical leadership roles, with “constant exposure to vicarious trauma seeping into their personal and family time outside of work” [ 39 ]. Facing simultaneous pressures of physical and emotional demands, resulted in increased incidence of severe stress, emotional exhaustion, and burnout amongst healthcare workers [ 69 ]. One study further identified the cyclical nature of the problem, with burnt out healthcare workers on stress-leave causing greater staff shortages and increased workload for those remaining at work [ 56 ].

Several studies also identified that despite the healthcare system demonstrating several capacities to exhibit resilient performance in response to COVID-19, negative “spillover effects” were exhibited on routine patient care [ 44 ]. For example, Lotta et al. noted that the physical distancing requirements and mandatory use of PPE undermined everyday clinical work, with healthcare workers not being able to maintain contact with families [ 43 ]. Additionally, Akinyemi et al. [ 80 ] detailed that the COVID-19 pandemic negatively impacted service delivery in the healthcare system, for example, through disruptions to the appointment system and emergency and routine care services, which affected patient access to healthcare.

RHC broadly refers as a system’s capacity to maintain or restore its functions despite disruptions caused by external factors [ 59 ]. RHC does not focus on an individual’s coping and resilience capacity but rather on the factors and tools that enable the workers, teams, department and organisation to adapt and cope effectively in different situations [ 16 ]. RHC is a theoretically attractive concept, with its positive focus on how ‘things go right’ rather than wrong, and as evidenced by the number of reviews that have appeared on the topic in recent years [ 10 , 13 , 16 ].

Despite signs that RHC is maturing and formalising as a research paradigm [ 13 , 16 , 59 ], there have been calls for continued developments to strengthen RHC theory and research [ 13 ]. As evidenced by this review, the COVID-19 pandemic presented a unique opportunity to research and critically advance our understanding of RHC, and in particular, created a shift in focus from theoretical conceptualisations to identifying how we might understand factors or capacities that foster resilience across the health system [ 83 ]. Previously, empirical studies on RHC were rare and skewed towards the clinical microsystems of care, however, the surge of literature on RHC during the pandemic provided a unique opportunity to take stock of the empirical landscape [ 83 ]. Indeed, since the previous review by Iflaifel et al. [ 16 ], which found 71 empirical studies on RHC over an 18-year period, the present scoping review identified a further 50 studies, highlighting the unprecedented growth of empirical applications within the RHC field over the past three years.

Consistent with previous reviews [ 13 , 16 ], qualitative methods dominated the included studies, with interviews typically being used to capture healthcare workers’ perceptions and experiences during the pandemic. Although the extensive use of qualitative methods has been cited as one of the strengths of RHC [ 13 ], this review saw the application of existing tools (e.g., FRAM, SPLINTS) along with the emergence of new quantitative assessments and indicator-based modelling approaches that could have fruitful implications, particularly in terms of enhancing system preparedness and advancing measurement and monitoring of resilient performance over time. We also identified the development of new questionnaires to assess RHC; many of which were based on a conceptual framework (e.g., such as Hollnagel’s [ 53 ] ‘four cornerstones of resilience’ and Anderson et al.’s [ 55 ] Integrated Resilience Attributes Framework). In addition, we saw an increased number of studies examining RHC in LMICs. For example, the two studies of Karamaji et al. [ 48 , 49 ] presented an approach to assessing and monitoring health systems functionality in developing African countries, with a set of indicators that combine into a “resilience index”, each with varying levels of “transformation capacity”. While RHC theorists have historically resisted establishing indicators and measurement in this field, some people are expressing a need to advance our understanding of system resilience beyond the conventional health system building blocks of the WHO published 15 years ago [ 60 ]; thus, including measurement and monitoring is increasingly pressing.

A previous criticism has been that a preponderance of studies of RHC at micro and meso levels is “not sufficient to understand systems resilience” [ 84 ], and thus it was promising to see the emergence of macro level studies in this review. The macro-level study by Smaggus et al. [ 14 ], for example, examined government responses to the pandemic, by way of a document analysis of media releases, in two countries, Canada and Australia, expanding the scope of RHC research to different system levels, and incorporating a cross-country comparison [ 84 ]. Furthermore, Smaggus et al. [ 14 ] integrated several resilience theoretical frameworks to guide their study, illustrating how theory can inform research design and analysis. However, this study also highlighted some of the difficulties of researching RHC, particularly at the macro level, and that a mixed-methods approach (e.g., including interviews and observations alongside document analysis) would be likely to provide a more complex understanding on how government actions affect health system resilience, and build a better understanding of the links between actions at the macro level and other system levels.

What was clear was that the included studies reported varying degrees of preparedness and adaptive capacity across the different healthcare services. For example, a number of studies reported how well organisations or the people who work in them “evolved” to make things work [ 39 , 54 , 81 ], while others reported extreme physical and emotional demands, leading to stress and burnout amongst healthcare workers and poor clinical care [ 37 , 39 , 43 , 65 , 69 , 73 ]. This discrepancy between resilient performance and physical and emotional burnout could be explained by the extensive use of short-term adaptations, rather than long-term innovation and system change [ 35 ]. This tradeoff between short and long term adaptations can also be expressed as a tradeoff between “specified” and “general” resilience [ 85 ]. Healthcare personnel initiating short term adaptations and workarounds, such as taking on extra responsibility, working longer shifts, often without breaks to compensate for systems deficiencies, such as workforce shortages, may only have a short-term ‘firefighting’ effect on the specific situation [ 86 ]. Without long-term, general adaptations that foster organisational and system change, short term adaptations could potentially end up as a barrier for systemic resilient performance instead of a capacity [ 55 , 87 , 88 ].

This issue also reminds us of Woods [ 89 ] notion that all systems have an “envelope of performance”; a range of how much they can adapt, due to finite resources and the inherent variation in the system. When a system is pushed to the edge of its envelope, the system can either adapt and expand its performance further into “graceful extensibility” or become “brittle” and potentially lead to system collapse. Wear and Hettinger [ 90 ] also pointed to circumstances where local adaptations may become too extensive (the “tragedy of adaptability”). In the case of COVID-19, the continuous need for short-term adaptations placed the responsibility of the system’s ability for resilient performance on the sharp-end agents rather than the system itself, who over time became physically and emotionally exhausted. Although RHC has not often considered an individual’s coping and resilience capacity, how individual-level resilience interacts with team-, organizational- and broader systems resilience is a key area for future research.

An important contribution of this study is the recognition of eight key factors or capacities in the existing literature that potentially develop and enhance resilient performance. Recognising that healthcare is highly complex and unpredictable, and understanding that these factors were identified from studies in the context of COVID-19, these findings are highly concordant with the “capacities for resilient performance” identified in the qualitative study by Lyng et al. [ 30 ]. It is hoped that the capacities identified in this study can be facilitated and supported through the development of tools and interventions [ 91 ]. As identified by Lyng et al. [ 30 ] there were obvious interdependencies between the capacities; for example, between structure and leadership, given that leaders often facilitated the implementation and adherence to different structural features such as technology, guidelines or learning arenas; and between coordination and learning given that the greatest number of learning efforts related to team training and coordinating efforts to tackle the challenges related to COVID-19.

One noticeable difference, however, between our findings and those reported by Lyng et al., [ 30 ] was the emphasis placed of the the need for teamwork and collaboration during COVID-19. While Lyng et al. [ 30 ] suggested that different capacities require different levels of collaboration, higher levels of collaboration may have been required across all eight capacities during the pandemic. Again, this may reflect that many of the adaptations reported were largely reactive efforts focused on system recovery and restoring its equilibrium, particularly during the early stages of the pandemic, thus requiring short-term workarounds or solutions particularly at the front lines of care; but which are noble and important responses to handle peak activity situations [ 87 ]. Furthermore, COVID-19 prompted higher levels of collaboration, with the need to ‘rally together’ as they faced the same issues or ‘enemy’ across contexts and system levels. In the same way, two capacities presented by Lyng et al., namely ‘facilitators’ by way of champions and ‘competence’ by way of experience and knowledge, were less prominent in the present study. This is not to say that Lyng et al.’s capacities of competence and facilitators are not important for resilient performance, but rather, in the context of the pandemic, that the collaborative efforts needed to adapt to their joint challenges, may have made individual competencies and facilitators less important, or they were not reported in our included studies. Future studies should continue advancing this theoretical framework in order to integrate factors from different countries and settings and under different situations (stress, crisis, ordinary). Arguably, three of the most important capacities in advancing systems from reactive short-term adaptations at the micro-system level to longer-term “graceful extensibility” are effective leadership, communication and learning [ 92 ]. Indeed, examples of interventions promoting these three capacities are appearing in the literature [ 92 , 93 , 94 ]. For example, ‘tiered team huddles’ to enable sharing of ideas and issues from health workers at the ‘sharp end’ with middle and senior leadership, enabling communication across boundaries and enabling organizational learning [ 92 ]. A ‘learning health system’ [ 95 , 96 ], cultivated through innovative interventions like tiered team huddles, could improve communication across boundaries and facilitate long-term lasting change. Leaders also need to consider the negative impacts of short-term adaptations and workarounds on staff mental health.

The importance of system “slack” (or “buffer”) at an organizational level and collective level (i.e., network or national), was also highlighted in the study findings, to ensure that the healthcare system is prepared and enables organizational flexibility to deploy equipment and staff rapidly and effectively to where they are needed most [ 97 ]. The provision of a margin of manoeuvrability may also reduce the resulting negative effects of continuous micro-adaptations and increased staff workloads; thereby serving as a protective [ 98 ] mechanism.

Implications for research, policy and practice

Despite that the literature confirms that resilience-based efforts and analysis need to occur across system levels (i.e., micro, meso, macro), there is still relatively little understanding – both conceptually and empirically – about how the system levels interact with each other. Although the pandemic affected all system levels, presenting the perfect opportunity to study “cross-level interactions”, most of our included studies focused on one level of analysis. Yet as our review showed, there can be a “dark side or downside of resilience” [ 29 ]. What started out as resilient short-term adaptations were exhausting for the people working in the system, resulting in stress and burnout. Considerations for how individual-level resilience factors affect resilience factors at the team and organization-level is an important area for future research.

Of course, identifying the interactions between system levels is challenging, given the non-linear nature of such interactions and the time over which they may occur. Again, this issue points to the need for mixed methods (quantitative and qualitative) approaches, the dual consideration of both positive consequences (e.g., performance, efficiency, safety outcomes), and negative consequences (e.g., by including measures of stress, job satisfaction and burnout) of systems resilience, as well as the need to collect data longitudinally to increase our understanding of causal processes between the various system levels. Although quantitative resilience tools are emerging in the literature, more work is needed to establish theory driven and well validated tools for application at the various system levels.

In this study, the resilience capacities developed by Lyng et al. [ 30 ] proved to be an applicable and useful framework. Further empirical research building on this framework would be valuable, such as clarifying the degree of interrelatedness between the capacities, as well as designing and testing interventions around the capacities. One issue remains to be resolved, however; clarification is needed as to whether resilience should be studied as an “outcome, mediator, or determinant of a system’s performance” [ 83 ]. Some previous studies use these interchangeably: with resilience described as an underlying potential required to achieve a given outcome, while at the same time concluding that the system “was” or “proved” to be resilient. The capacity approach that we have taken here suggests that resilience is an underlying potential of the system, at its various levels, to adapt or restore its functions in response to disruption. We also call on researchers to be specific about whether they are referring to reactive adaptations focused on recovery or proactive efforts to minimise brittleness, with Woods’ [ 99 ] four conceptions of resilience potentially serving as a useful framework in this regard.

The results of this study, in combination with the Lyng et al.’s [ 30 ] capacities for resilient performance framework, can be used to guide interventions to support, develop or strengthen resilience. Understanding factors that develop or enhance RHC is critical to developing interventions and tools for strengthening their resilience [ 100 ]. This study thereby contributes to this work with key insights for intervention development that can be employed to enhance resilience performance.

Strengths and limitations

Data analysis and synthesis built on and strengthened the work of Lyng et al.’s [ 30 ] capacities for resilient performance framework; this framework can be further used as a basis to guide the next wave of research on RHC. The limitations of this review are primarily methodological. Due to our search strategy, we may have not identified valuable findings published in books, research reports and white papers. Future reviews of empirical studies in this field would benefit from by-hand searching particularly of books, where much of the foundational RHC literature has been identified [ 13 ]. Although we identified a relatively high proportion of articles from medium-income countries, our restriction to records in English and published works may have underestimated the true amount of literature emerging from LMIC. Our data extraction was also restricted to what was reported and discussed in the included studies. As a result, we may have under identified some important capacities and negative consequences. Using a data-based convergent synthesis approach, we transformed data from quantitative studies into categories or themes and did not analyse or report the results separately for different study types. Future research involving innovative methods for combining systematic review, concept analysis and bibliometric analysis could be used to summarise qualitative, quantitative and mixed methods RHC studies [ 101 ].

Our review identified an explosion of new empirical studies on health system resilience associated with COVID-19. The pandemic provided a unique ‘natural experiment’ and unprecedented opportunity to examine RHC theory in practice, and uncovered emerging new evidence on RHC theory and system factors that contribute to resilient performance at micro, meso and macro levels. Additionally, we identified potential unintended consequences of short-term responses to improve resilience without due consideration of the longer-term effects. These findings will facilitate strengthening of health system performance and resilience in responding to challenges and other unexpected events in the future.

Data Availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Functional Resonance Analysis Method

Preferred Reporting Items of Systematic Review and Meta-Analyses Extension for Scoping Reviews

Resilient Health Care

Scrub Practitioners List of Intra-operative Non-Technical Skills

World Health Organisation

Low- and middle-income countries

Hollnagel E, Braithwaite J, Wears RL. Preface: on the need for resilience in health care. Resilient health care Farnham: Ashgate Publishing. 2013:2–3.

Wiig S, O’Hara JK. Resilient and responsive healthcare services and systems: challenges and opportunities in a changing world. BMC Health Serv Res. 2021;21(1):1–5.

Article   Google Scholar  

World Health Organization. Health systems resilience toolkit: a WHO global public health good to support building and strengthening of sustainable health systems resilience in countries with various contexts. 2022.

World Health Organization. Building Health Systems Resilience for Universal Health Coverage and Health Security during the COVID-19 Pandemic and beyond: WHO Position Paper.; 2021.

World Health Organization. Strengthening health emergency preparedness and response in the WHO South-East Asia Region building upon lessons learnt from COVID-19. World Health Organization. Regional Office for South-East Asia; 2022.

World Health Organization. Outbreak preparedness and resilience. 2020.

Ghebreyesus TA, Jakab Z, Ryan MJ, Mahjour J, Dalil S, Chungong S, et al. WHO recommendations for resilient health systems. Bull World Health Organ. 2022;100(4):240.

Article   PubMed   PubMed Central   Google Scholar  

Hollnagel E. About The Resilient Health Care Net [cited 2023 Jan 31] Available from: https://resilienthealthcare.net/about/ .

Woods DD, Hollnagel E, Prologue. Resilience engineering concepts. Resilience engineering: CRC Press; 2017. pp. 1–6.

Google Scholar  

Fridell M, Edwin S, Von Schreeb J, Saulnier DD. Health system resilience: what are we talking about? A scoping review mapping characteristics and keywords. Int J Health Policy Manage. 2020;9(1):6.

Tiernan A, Drennan L, Nalau J, Onyango E, Morrissey L, Mackey B. A review of themes in disaster resilience literature and international practice since 2012. Policy Des Pract. 2019;2(1):53–74.

Barasa E, Mbau R, Gilson L. What is resilience and how can it be nurtured? A systematic review of empirical literature on organizational resilience. Int J Health Policy Manage. 2018;7(6):491.

Ellis LA, Churruca K, Clay-Williams R, Pomare C, Austin EE, Long JC, et al. Patterns of resilience: a scoping review and bibliometric analysis of resilient health care. Saf Sci. 2019;118:241–57.

Smaggus A, Long JC, Ellis LA, Clay-Williams R, Braithwaite J. Government actions and their relation to resilience in healthcare during the COVID-19 pandemic in New South Wales, Australia and Ontario, Canada. Int J Health Policy Manage. 2022;11(9):1682–94.

Biddle L, Wahedi K, Bozorgmehr K. Health system resilience: a literature review of empirical research. Health Policy Plann. 2020;35(8):1084–109.

Iflaifel M, Lim RH, Ryan K, Crowley C. Resilient health care: a systematic review of conceptualisations, study methods and factors that develop resilience. BMC Health Serv Res. 2020;20(1):1–21.

Braithwaite J, Churruca K, Ellis LA, Long J, Clay-Williams R, Damen N, et al. Complexity science in healthcare. Sydney: Australian Institute of Health Innovation, Macquarie University.; 2017.

Nyssen A-S, Bérastégui P. Is system resilience maintained at the expense of individual resilience. Resilient health care III: reconciling work-as-imagined and work-as-done. 2016:37–47.

Chirico F, Ferrari G, Nucera G, Szarpak L, Crescenzo P, Ilesanmi O. Prevalence of anxiety, depression, burnout syndrome, and mental health disorders among healthcare workers during the COVID-19 pandemic: a rapid umbrella review of systematic reviews. J Health Soc Sci. 2021;6(2):209–20.

Vizheh M, Qorbani M, Arzaghi SM, Muhidin S, Javanmard Z, Esmaeili M. The mental health of healthcare workers in the COVID-19 pandemic: a systematic review. J Diabetes Metabolic Disorders. 2020;19(2):1967–78.

Article   CAS   Google Scholar  

Chen W, Hirschheim R. A paradigmatic and methodological examination of information systems research from 1991 to 2001. Inform Syst J. 2004;14(3):197–235.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med [Internet]. 2018; 169(7):[467 – 73 pp.]. Available from: https://www.ncbi.nlm.nih.gov/pubmed/30178033 .

Peters MDJ, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L et al. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth [Internet]. 2020; 18(10):[2119-26 pp.]. Available from: https://www.ncbi.nlm.nih.gov/pubmed/33038124 .

Paré G, Trudel M-C, Jaana M, Kitsiou S. Synthesizing information systems knowledge: a typology of literature reviews. Inf Manag [Internet]. 2015; 52(2):[183 – 99 pp.].

Pham MT, Rajic A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods [Internet]. 2014 Dec PMC4491356]; 5(4):[371 – 85 pp.]. Available from: https://www.ncbi.nlm.nih.gov/pubmed/26052958 .

Hong QN, Pluye P, Bujold M, Wassef M. Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence. Syst Reviews. 2017;6(1):1–14.

Sandelowski M. Telling stories: narrative approaches in qualitative research. Image: The Journal of Nursing Scholarship. 1991;23(3):161–6.

CAS   PubMed   Google Scholar  

The World Bank. World Bank Country and Lending Groups: The World Bank. ; 2021 [Available from: https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups .

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8(1):1–10.

Lyng HB, Macrae C, Guise V, Haraldseid-Driftland C, Fagerdal B, Schibevaag L, et al. Capacities for resilience in healthcare; a qualitative study across different healthcare contexts. BMC Health Serv Res. 2022;22(1):1–14.

Alajmi A, Adlan N, Lahyani R. Assessment of Supply Chain Management Resilience within Saudi Medical Laboratories during Covid-19 pandemic. Procedia Cirp. 2021;103:32–6.

Smith EM, Hernandez MLT, Ebuenyi I, Syurina EV, Barbareschi G, Best KL et al. Assistive technology use and provision during COVID-19: results from a rapid global survey. Int J Health Policy Manage. 2020.

Montás MC, Klasa K, van Ginneken E, Greer SL. Strategic purchasing and health systems resilience: Lessons from COVID-19 in selected european countries. Health Policy. 2022.

Spieske A, Gebhardt M, Kopyto M, Birkel H. Improving resilience of the healthcare supply chain in a pandemic: evidence from Europe during the COVID-19 crisis. J Purchasing Supply Manage. 2022:100748.

Lyng HB, Ree E, Wibe T, Wiig S. Healthcare leaders’ use of innovative solutions to ensure resilience in healthcare during the Covid-19 pandemic: a qualitative study in norwegian nursing homes and home care services. BMC Health Serv Res. 2021;21(1):1–11.

Marshall F, Gordon A, Gladman JR, Bishop S. Care homes, their communities, and resilience in the face of the COVID-19 pandemic: interim findings from a qualitative study. BMC Geriatr. 2021;21(1):1–10.

Martin A, Hatzidimitriadou E. Optimising health system capacity: a case study of community care staff’s role transition in response to the coronavirus pandemic. Health Soc Care Commun. 2022;30(5):e2147–e56.

Cannedy S, Bergman A, Medich M, Rose DE, Stockdale SE, editors. Health System Resiliency and the COVID-19 pandemic: a case study of a New Nationwide Contingency staffing program. Healthcare: MDPI; 2022.

Brenner AB, Knaub M, Robinson K, Lotspeich M, Eisen J. Building Resilience in the Face of Crisis: Lessons learned from a community behavioral Healthcare Organization. J Behav Health Serv Res. 2022:1–8.

van Gool F, Bongers I, Bierbooms J, Janssen R. Whether and how top management create flexibility in mental healthcare organizations: COVID-19 as a test case. J Health Organ Manag. 2022.

Danhieux K, Buffel V, Pairon A, Benkheil A, Remmen R, Wouters E, et al. The impact of COVID-19 on chronic care according to providers: a qualitative study among primary care practices in Belgium. BMC Fam Pract. 2020;21(1):1–6.

Leslie M, Fadaak R, Pinto N, Davies J, Green L, Seidel J, et al. Achieving resilience in primary care during the COVID-19 pandemic: competing visions and lessons from Alberta. Healthc Policy. 2021;17(2):54.

PubMed   PubMed Central   Google Scholar  

Lotta G, Fernandez M, Corrêa M. The vulnerabilities of the brazilian health workforce during health emergencies: analysing personal feelings, access to resources and work dynamics during the COVID-19 pandemic. Int J Health Plann Manag. 2021;36(S1):42–57.

Yoon S, Goh H, Chan A, Malhotra R, Visaria A, Matchar D, et al. Spillover effects of COVID-19 on essential chronic care and ways to foster health system resilience to support vulnerable non-COVID patients: a multistakeholder study. J Am Med Dir Assoc. 2022;23(1):7–14.

Article   PubMed   Google Scholar  

Juárez-Ramírez C, Reyes-Morales H, Gutiérrez-Alba G, Reartes-Peñafiel DL, Flores-Hernández S, Muños-Hernández JA, et al. Local health systems resilience in managing the COVID-19 pandemic: lessons from Mexico. Health Policy Plann. 2022;37(10):1278–94.

Stengel S, Roth C, Breckner A, Cordes L, Weber S, Ullrich C, et al. Resilience of the primary health care system–german primary care practitioners’ perspectives during the early COVID-19 pandemic. BMC Prim Care. 2022;23(1):1–13.

Jovanović A, Klimek P, Renn O, Schneider R, Øien K, Brown J, et al. Assessing resilience of healthcare infrastructure exposed to COVID-19: emerging risks, resilience indicators, interdependencies and international standards. Environ Syst Decisions. 2020;40(2):252–86.

Karamagi HC, Titi-Ofei R, Kipruto HK, Seydi AB-W, Droti B, Talisuna A, et al. On the resilience of health systems: a methodological exploration across countries in the WHO African Region. PLoS ONE. 2022;17(2):e0261904.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Karamagi HC, Tumusiime P, Titi-Ofei R, Droti B, Kipruto H, Nabyonga-Orem J, et al. Towards universal health coverage in the WHO African Region: assessing health system functionality, incorporating lessons from COVID-19. BMJ Global Health. 2021;6(3):e004618.

Hollnagel E. FRAM: the functional resonance analysis method: modelling complex socio-technical systems. Crc Press; 2017.

Malec JF, Torsher LC, Dunn WF, Wiegmann DA, Arnold JJ, Brown DA, et al. The mayo high performance teamwork scale: reliability and validity for evaluating key crew resource management skills. Simul Healthc. 2007;2(1):4–10.

Mitchell L. Scrub practitioners’ list of intra-operative non-technical skills–SPLINTS. Safer surgery: CRC Press; 2017. pp. 67–81.

Hollnagel E. The four cornerstones of resilience engineering. Resilience Engineering Perspectives. Volume 2. CRC Press; 2016. pp. 139–56.

Corbaz-Kurth S, Juvet TM, Benzakour L, Cereghetti S, Fournier C-A, Moullec G, et al. How things changed during the COVID-19 pandemic’s first year: a longitudinal, mixed-methods study of organisational resilience processes among healthcare workers. Saf Sci. 2022;155:105879.

Anderson J, Ross A, Macrae C, Wiig S. Defining adaptive capacity in healthcare: a new framework for researching resilient performance. Appl Ergon. 2020;87:103111.

Article   CAS   PubMed   Google Scholar  

Juvet TM, Corbaz-Kurth S, Roos P, Benzakour L, Cereghetti S, Moullec G, et al. Adapting to the unexpected: problematic work situations and resilience strategies in healthcare institutions during the COVID-19 pandemic’s first wave. Saf Sci. 2021;139:105277.

Bueno WP, Saurin TA, Wachs P, Kuchenbecker R, Braithwaite J. Coping with complexity in intensive care units: a systematic literature review of improvement interventions. Saf Sci. 2019;118:814–25.

Saurin TA, Wachs P, Bueno WP, de Souza Kuchenbecker R, Boniatti MM, Zani CM, et al. Coping with complexity in the COVID pandemic: an exploratory study of intensive care units. Hum Factors Ergon Manuf Serv Ind. 2022;32(3):301–18.

Macrae C, Wiig S. Resilience: from practice to theory and Back again. Exploring resilience a scientific journey from practice to theory Cham: Springer Open. 2019:121-8.

World Health Organization. Everybody’s business–strengthening health systems to improve health outcomes: WHO’s framework for action. 2007.

Seruwagi G, Nakidde C, Otieno F, Kayiwa J, Luswata B, Lugada E, et al. Healthworker preparedness for COVID-19 management and implementation experiences: a mixed methods study in Uganda’s refugee-hosting districts. Confl Health. 2021;15(1):1–22.

Balqis-Ali NZ, Fun WH, Ismail M, Ng RJ, Jaaffar FSA, Low LL. Addressing gaps for Health Systems strengthening: a public perspective on Health Systems’ response towards COVID-19. Int J Environ Res Public Health. 2021;18(17):9047.

Khalil M, Mataria A, Ravaghi H. Building resilient hospitals in the Eastern Mediterranean Region: lessons from the COVID-19 pandemic. BMJ Global Health. 2022;7(Suppl 3):e008754.

World Health Organization. Hospital readiness checklist for COVID-19 interim version February 24 2020. World Health Organization. Regional Office for Europe; 2020.

Ballantyne H, Achour N. The challenges of nurse redeployment and opportunities for leadership during COVID-19 pandemic. Disaster Med Pub Health Prep. 2022:1–7.

Gifford R, Fleuren B, van de Baan F, Ruwaard D, Poesen L, Zijlstra F, et al. To uncertainty and Beyond: identifying the Capabilities needed by hospitals to function in dynamic environments. Med Care Res Rev. 2022;79(4):549–61.

Parikh N, Chaudhuri A, Syam SB, Singh P. Fostering resilient Health Systems in India: Providing Care for PLHIV under the Shadow of COVID-19. Front Public Health. 2022:1581.

Shah A, Guessi M, Wali S, Ware P, McDonald M, O’Sullivan M, et al. The resilience of cardiac care through virtualized services during the COVID-19 pandemic: case study of a heart function clinic. JMIR cardio. 2021;5(1):e25277.

Abu Mansour SI, Abu Shosha GM. Experiences of first-line nurse managers during COVID‐19: a jordanian qualitative study. J Nurs Adm Manag. 2022;30(2):384–92.

Graetz DE, Sniderman E, Villegas CA, Kaye EC, Ragab I, Laptsevich A, et al. Resilient health care in global pediatric oncology during the COVID-19 pandemic. Cancer. 2022;128(4):797–807.

Hodgins M, Van Leeuwen D, Braithwaite J, Hanefeld J, Wolfe I, Lau C, et al. The COVID-19 system shock framework: capturing health system innovation during the COVID-19 pandemic. Int J Health Policy Manage. 2022;11(10):2155–65.

Aliyu S, Norful AA, Schroeder K, Odlum M, Glica B, Travers JL. The powder keg: Lessons learned about clinical staff preparedness during the early phase of the COVID-19 pandemic. Am J Infect Control. 2021;49(4):478–83.

Britton CR, Hayman G, Stroud N. Awareness of human factors in the operating theatres during the COVID-19 pandemic. J Perioper Pract. 2021;31(1–2):44–50.

PubMed   Google Scholar  

Dunleavy L, Preston N, Bajwah S, Bradshaw A, Cripps R, Fraser LK, et al. Necessity is the mother of invention’: specialist palliative care service innovation and practice change in response to COVID-19. Results from a multinational survey (CovPall). Palliat Med. 2021;35(5):814–29.

Lakissian Z, Sabouneh R, Zeineddine R, Fayad J, Banat R, Sharara-Chami R. In-situ simulations for COVID-19: a safety II approach towards resilient performance. Adv Simul. 2020;5(1):1–10.

Nair S, Kannan P, Mehta K, Raju A, Mathew J, Ramachandran P. The COVID-19 pandemic and its impact on mental health services: the provider perspective. J Public Health. 2021;43(Supplement2):ii51–ii6.

Graetz DE, Sniderman E, Villegas C, Ragab I, Laptsevich A, Maliti B, et al. Utilizing multilingual methods and Rapid Analysis for Global qualitative research during a pandemic. Global Qualitative Nursing Research. 2022;9:23333936221080969.

Snowdon A, Wright A, editors. Digitally enabled supply chain as a strategic asset for the COVID-19 response in Alberta. Healthcare Management Forum; 2022: SAGE Publications Sage CA: Los Angeles, CA.

Peat G, Olaniyan J, Fylan B, Breen L, Grindey C, Hague I, et al. Mapping the resilience performance of community pharmacy to maintain patient safety during the Covid-19 pandemic. Research in Social and Administrative Pharmacy; 2022.

Akinyemi OO, Popoola OA, Fowotade A, Adekanmbi O, Cadmus EO, Adebayo A. Qualitative exploration of health system response to COVID-19 pandemic applying the WHO health systems framework: case study of a nigerian state. Sci Afr. 2021;13:e00945.

CAS   PubMed   PubMed Central   Google Scholar  

McCollum R, Zaizay Z, Dean L, Watson V, Frith L, Alhassan Y, et al. Qualitative study exploring lessons from Liberia and the UK for building a people-centred resilient health systems response to COVID-19. BMJ Open. 2022;12(8):e058626.

Pilevari N, Shiva MV. Country-wide resilience model for the health system: a case study on iran, under coronavirus outbreak. Iran J Public Health. 2021;50(4):806.

Bozorgmehr K, Zick A, Hecker T. Resilience of Health Systems: understanding uncertainty uses, intersecting crises and cross-level interactions; comment on “Government actions and their relation to Resilience in Healthcare during the COVID-19 pandemic in New South Wales, Australia and Ontario, Canada. Int J Health Policy Manage. 2022;11(9):1956–9.

Anderson JE, Theorising Health System Resilience and the Role of Government Policy-Challenges and Future Directions. Comment on “Government actions and their relation to Resilience in Healthcare during the COVID-19 pandemic in New South Wales, Australia and Ontario, Canada. Int J Health Policy Manage. 2022;11(9):1960–3.

Walker B, Westley F. Perspectives on resilience to disasters across sectors and cultures. Ecol Soc. 2011;16(2).

Löf A. Exploring adaptability through learning layers and learning loops. Environ Educ Res. 2010;16(5–6):529–43.

Lyng HB, Macrae C, Guise V, Haraldseid-Driftland C, Fagerdal B, Schibevaag L, et al. Balancing adaptation and innovation for resilience in healthcare–a metasynthesis of narratives. BMC Health Serv Res. 2021;21:1–13.

Glette MK, Wiig S. The role of organizational factors in how efficiency-thoroughness trade-offs potentially affect clinical quality dimensions–a review of the literature. Int J Health Gov. 2021;26(3):250–65.

Woods DD. Resilience as Graceful Extensibility to Overcome Brittleness i. An edited collection of authored pieces comparing, contrasting, and integrating risk and resilience with an emphasis on ways to measure resilience. 2016:258.

Wears RL, Hettinger AZ. The tragedy of adaptability. Ann Emerg Med. 2014;63(3):338–9.

Ree E, Haraldseid-Driftland C. How can Digital Learning Tools be used to promote resilience in Healthcare? Resilience in a Digital Age: Global Challenges in Organisations and Society. Springer; 2022. pp. 231–45.

Rangachari P, Woods L. Preserving organizational resilience, patient safety, and staff retention during COVID-19 requires a holistic consideration of the psychological safety of healthcare workers. Int J Environ Res Public Health. 2020;17(12):4267.

Wahl K, Stenmarker M, Ros A. Experience of learning from everyday work in daily safety huddles—a multi-method study. BMC Health Serv Res. 2022;22(1):1–14.

Ree E, Ellis LA, Wiig S. Managers’ role in supporting resilience in healthcare: a proposed model of how managers contribute to a healthcare system’s overall resilience. Int J Health Gov. 2021;26(3):266–80.

Ellis LA, Sarkies M, Churruca K, Dammery G, Meulenbroeks I, Smith CL, et al. The science of learning health systems: scoping review of empirical research. JMIR Med Inf. 2022;10(2):e34907.

Pomare C, Mahmoud Z, Vedovi A, Ellis LA, Knaggs G, Smith CL, et al. Learning health systems: a review of key topic areas and bibliometric trends. Learn Health Syst. 2022;6(1):e10265.

McHugh JP, Cross DA. The application of organizational slack to hospital system responsiveness during the COVID-19 pandemic. J Hosp Manage Health Policy. 2021;5:14.

Saurin TA, editor. Removing waste while preserving slack: the lean and complexity perspectives. LC3 2017 Volume II–Proceedings of the 25th Annual Conference of the International Group for Lean, Heraklion, Greece; 2017.

Woods DD. Four concepts for resilience and the implications for the future of resilience engineering. Reliab Eng Syst Saf. 2015;141:5–9.

Wiig S, O’Hara JK. Resilient and responsive healthcare services and systems: challenges and opportunities in a changing world. BMC Health Serv Res. 2021;21:1–5.

Gartner J-B, Abasse KS, Bergeron F, Landa P, Lemaire C, Côté A. Definition and conceptualization of the patient-centered care pathway, a proposed integrative framework for consensus: a Concept analysis and systematic review. BMC Health Serv Res. 2022;22(1):1–24.

Download references

Acknowledgements

The authors would like to thank and acknowledge Dylan Thomas (DT) and Lillian Tricker (LT) for their assistance with the title/abstract and full-text screening and Mr Jeremy Cullis for his help with devising the search strategy.

This work was supported by funded from NHMRC Partnership Centre in Health System Sustainability (Grant ID 9100002) and NHMRC Investigator Grant (Grant ID 1176620). HBL, CHD and SW receiving funding from the Research Council of Norway from the FRIPRO TOPPFORSK program (Grant ID 275367) to support their time on this project. These funding bodies had no role in the conception, design, data collection, analysis or decision to publish.

Author information

Authors and affiliations.

Centre for Healthcare Resilience and Implementation Science, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia

Louise A Ellis, Maree Saba, Janet C Long, Kate Churruca, Elizabeth Austin, Robyn Clay-Williams, Ann Carrigan & Jeffrey Braithwaite

Centre Faculty of Health Sciences, SHARE - Centre for Resilience in Healthcare, University of Stavanger, Stavanger, Norway

Hilda Bø Lyng, Cecilie Haraldseid-Driftland & Siri Wiig

You can also search for this author in PubMed   Google Scholar

Contributions

This study was originally conceived by LAE. Data extraction and screening was conducted by LAE, MS, JCL, KC, EA, with assistance from HBL, CHD, SW, AC, RCW and JB. First draft of the results section was written by LAE and MS. All authors provided critical feedback and helped shape the final manuscript.

Corresponding author

Correspondence to Louise A Ellis .

Ethics declarations

Ethics approval and consent to participate.

Not Applicable since it is a scoping review.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, supplementary material 3, supplementary material 4, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Ellis, L.A., Saba, M., Long, J.C. et al. The rise of resilient healthcare research during COVID-19: scoping review of empirical research. BMC Health Serv Res 23 , 833 (2023). https://doi.org/10.1186/s12913-023-09839-0

Download citation

Received : 22 February 2023

Accepted : 22 July 2023

Published : 07 August 2023

DOI : https://doi.org/10.1186/s12913-023-09839-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Healthcare research
• Health systems
• Resilience in healthcare
• Resilient healthcare

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Open access
• Published: 17 May 2016

Which health research gets used and why? An empirical analysis of 30 cases

• Maarten Olivier Kok 1 , 2 ,
• John Owusu Gyapong 3 ,
• Ivan Wolffers 4 ,
• David Ofori-Adjei 5 &
• Joost Ruitenberg 2  

Health Research Policy and Systems volume  14 , Article number:  36 ( 2016 ) Cite this article

7189 Accesses

61 Citations

100 Altmetric

Metrics details

While health research is considered essential for improving health worldwide, it remains unclear how it is best organized to contribute to health. This study examined research that was part of a Ghanaian-Dutch research program that aimed to increase the likelihood that results would be used by funding research that focused on national research priorities and was led by local researchers. The aim of this study was to map the contribution of this research to action and examine which features of research and translation processes were associated with the use of the results.

Using Contribution Mapping, we systematically examined how 30 studies evolved and how results were used to contribute to action. We combined interviews with 113 purposively selected key informants, document analysis and triangulation to map how research and translation processes evolved and contributions to action were realized. After each case was analysed separately, a cross-case analysis was conducted to identify patterns in the association between features of research processes and the use of research.

The results of 20 of the 30 studies were used to contribute to action within 12 months. The priority setting and proposal selection process led to the funding of studies which were from the outset closely aligned with health sector priorities. Research was most likely to be used when it was initiated and conducted by people who were in a position to use their results in their own work. The results of 17 out of 18 of these user-initiated studies were translated into action. Other features of research that appeared to contribute to its use were involving potential key users in formulating proposals and developing recommendations.

Conclusions

Our study underlines the importance of supporting research that meets locally-expressed needs and that is led by people embedded in the contexts in which results can be used. Supporting the involvement of health sector professionals in the design, conduct and interpretation of research appears to be an especially worthwhile investment.

Peer Review reports

One of the most common laments heard in research policy circles is that the results of even the best studies are rarely translated into action [ 1 – 3 ]. This is especially distressing in the context of health-related research in lower-income countries, where new knowledge, well used, has the potential to save lives and improve welfare [ 4 ].

The traditional response to this apparent under-use of research is to encourage researchers to communicate their results more effectively. While it may help, better communication tends to be insufficient for improving the use of research. Communication is not ad-hoc, but requires ongoing interaction and trust, as well as relevant infrastructure [ 5 , 6 ]. In addition, local capacities are required for translating generic knowledge claims to the specific local situation in which they could be useful [ 7 – 9 ]. An additional challenge that has long hampered research uptake in low-income countries is its limited local utility. As early as 1990, the prominent Commission on Health Research for Development reported that conventional health research contributed little to health and development in poorer countries because it was dominated by foreign scholars instead of locally embedded researchers, and met international rather than local information needs [ 10 ].

To align research more closely with national needs, local policymakers, health professionals and community representatives were encouraged to join in with NGOs, academics and others to set national health research agendas [ 11 ]. The idea was that this would lead to research driven by the demands of local stakeholders, which was more likely to be used than research driven by supply from foreign academics.

It is, however, difficult to ascertain how these various efforts influence the likelihood that research results will be used. Studies of the use of research tend to start with finalized results or evidence-based recommendations, and trace their use in action [ 12 – 16 ]. Most of these studies indicate that the use of research increases as potential users consider research pertinent, as research coincides with the users’ needs, as the users’ attitude is to give credibility to research and as results reach users at the right time [ 17 – 20 ].

In line with these observations, research funders and others have tried to foster interaction between the producers and users of research. Initially, this interaction was focused on the joint interpretation of research results and the development of recommendations. More recently, interesting methods have been designed that encourage researchers and others to think how results might be used, and engage potential users from the time research is planned and throughout research processes [ 21 – 24 ].

While approaches such as priority setting and involving potential users are increasingly promoted, there are few systematic studies that examine how they influence the eventual use of research [ 25 – 27 ]. Such studies need to examine what happens throughout research processes and relate that to the use of results [ 28 ].

Our work aimed to fill this gap, using a newly-developed method known as Contribution Mapping, to systematically assess 30 studies conducted in Ghana between 2001 and 2008 [ 29 ]. These studies were part of a program jointly developed by the governments of the Netherlands and Ghana that aimed to increase the use of research by ensuring that it was locally relevant and locally led [ 30 ]. Beginning with a national research agenda-setting process, the Ghanaian Dutch Health Research for Development Program supported research-use efforts at various points in the research process. Ghanaian professionals from three groups identified as representing the health research constituency were invited to submit research proposals that would fit the research priority agenda. These three groups comprised academia, policymakers at all levels and end-users of health research: the health workers and the communities that were to benefit from efforts to improve their health. NGOs were asked to represent the communities, and especially the more marginalized groups that were poorly reached by the regular health system. The Ghanaians leading the studies could invite Dutch researchers to collaborate with them. At the end of each study, the researchers had to submit a detailed report which contained a policy brief and specific recommendations, which were disseminated to potential key users. The research program started in 2001 and funded 79 research projects through five annual rounds of priority setting, proposal selection, funding and support.

Our study aims to map the contribution of these research projects to action and examine which features of research and translation processes were associated with the use of the results. To our knowledge, this is the first study to try to systematically analyse the relation between features of research processes and the eventual use of research across the spectrum of health research processes in a low-income country, using a substantial number of case studies.

We used Contribution Mapping to assess how 30 research projects evolved and the results were translated into action. Contribution Mapping, which is described more fully elsewhere [ 29 ], is ground in social studies of science. Contribution Mapping recognizes that determining and attributing the ultimate ‘impact’ of research is often unrealistic and practically impossible. A true ‘impact’ perspective neglects the active role of users, who combine research outcomes with existing knowledge and use it for their own purposes in an evolving world full of ongoing processes. To take into account the active role of users and contexts, the translation of knowledge into action is better viewed as a collective process in which the agency is distributed.

A key feature of Contribution Mapping is that it contains a specific perspective on how research outcomes are integrated with existing knowledge and translated into action. This 'actor-scenario' perspective begins with the idea that those who try to translate knowledge into action put forward a more or less explicit story about a future in which they assign roles and responsibilities to a variety of ‘actors’ such as people, organizations, technologies, budgets, microbes and artefacts (e.g. these findings mean that this organization should do this, those professionals should do that, these medicine should do this, and that funder is responsible for that, etc.). Knowledge can be brought into such an ‘actor-scenario’ to confirm, support or strengthen it or introduce new elements. Knowledge can also be used to undermine the actor-scenarios of others (e.g. these findings show that they should stop funding because that policy will not work). When users bring research outcomes into such a scenario, they combine these outcomes with existing knowledge and formulate what that knowledge means for a specific aim in a specific situation. The actor-scenario perspective thus recognizes that research outcomes do not have a fixed meaning that is somehow imposed upon a passive user. While research outcomes can play a role, the perspective recognizes that such outcomes can be assigned different meanings by different actors in different situations. Regardless of the role that knowledge plays, its use can always be analyzed in terms of evolving and interacting actor-scenarios, and attempts to realize them.

Instead of trying to attribute ultimate ‘impacts’, Contribution Mapping focusses on how research and translation processes evolve and contributions to action come about, by tracing the actions of actors that are involved in, or interact with, a research project and the most likely influential users amongst them, which are referred to as potential key-users. The method follows a structured, iterative approach in which key informant interviews and document analysis are combined to develop a narrative of how processes evolved and contributions to action were realized.

Determining whether a study was used

The outcome measure of our study was whether the results of research were used to contribute to action. A contribution to action can be described as a process in which knowledge plays a meaningful role in action for health. For the purpose of this study, we made a somewhat crude distinction between studies that were used and studies that were not. We considered a study as ‘used’ when at least one person described that produced knowledge had played a meaningful role in action for health, this was corroborated by someone else and/or documentary evidence, and the translation process seemed plausible to the external analyst. We focused on the contributions to action that could be identified between 6 and 12 months after a study was finalized. We chose this relatively fixed timeframe to allow us to compare cases.

Case selection

For this multiple case study, we selected the first 30 research projects of the Ghanaian Dutch Health Research for Development Program that were finalized. These 30 research projects were funded between 2002 and 2004 and are described in Table  1 . These research projects were all led by a Ghanaian principal investigator (PI) and included one or more co-investigators. Most research projects were completed less than 2 years after funding was provided. The research projects had budgets varying from US$ 10,000 to 20,000, excluding the salaries of the involved investigators. Until at least 6 months after a study was finalized, those involved in research and translation processes were not made aware that the use of the results would be assessed.

The research program aimed to fund research that was oriented towards the national research agenda in Ghana. The research agenda was set in four steps: (1) reviewing existing research information, (2) consulting the health sector, policymakers and NGOs about research needs, (3) interviewing community members and (4) holding a workshop to prioritize issues based upon the existence of a problem, relevance, urgency, or whether research was needed to solve the problem. The research agenda was widely disseminated and public and private research institutes, NGOs and other interested groups were invited to submit a letter of intent that fell within the research priorities. The best letters of intent were selected and research teams invited to submit a full proposal. Each proposal had to contain a section about the societal relevance/utility of the proposed research. An external, Ghanaian-Dutch scientific review committee reviewed the full proposals for scientific merit, societal relevance/utility of the research, feasibility within time, budgetary and methodological framework, and ethical considerations. Final selection of proposals was done by the Joint Program Committee based on the comments of the reviewers.

Organization of data collection

Data collection started in March 2005 with the creation of an overview of the background and development of the research program. The assessment of each case started with reading available documentation, such as research proposals, mid-term reviews and final reports, and making a timeline-based process map. The timeline was divided into three phases: (1) formulation phase, (2) knowledge production phase and (3) the knowledge extension phase (e.g. dissemination and utilization). For each phase, the main actors, activities and interactions were mapped.

The first interview was held between 12 and 18 months after the investigators had established their results and were ready to disseminate them. The mapping process started with interviewing the principal investigators of a research project, developing a first version of the three-phase process map and identifying potential key-users and translation processes. Next, potential key-users and other informants were interviewed to trace, explore and triangulate possible contributions. In the third stage, process summaries were shared with key informants for feedback and validation. After inconsistencies were clarified, the process maps and description of contributions were finalized.

Interviewing

One hundred and thirteen purposively selected participants were interviewed face-to-face in four rounds of data collection (2005–2008) by four different researchers; 18 participants were interviewed about several research projects and 36 participants were involved in the studies as PI or co-investigator. The others were selected as potential key-user or interviewed to further explore, triangulate or elaborate descriptions of translation processes and contributions to action. Most potential key-users had a leading role at the Ministry of Health, the Ghana Health Service or other health-related organizations.

Following the steps of Contribution Mapping, interviewees were asked to describe how the process of formulating a study proposal and conducting research had evolved, and how produced knowledge claims were disseminated and translated into action. Interviewees were encouraged to be specific about processes and interactions, to provide detailed examples and share documents that supported their claims and to provide further insight into how research and translation processes evolved. Examples of such documents are texts related to specific meetings, policy briefs, reports and presentation sheets. Emerging descriptions of translations and contributions to action were triangulated with subsequent interviewees, who could also put forward new stories of contributions and other documents that supported their claims. Interviews were audio recorded, except in five cases in which equipment failed or interviewees did not want to be recorded, and a detailed summary was made directly afterwards.

Data management and analysis

Directly after each interview, a detailed summary was prepared. By listening to the interviews, all relevant parts were identified and transcribed verbatim. Data analysis was done in two steps: (1) a detailed qualitative within-case analysis and (2) cross-case analysis. Data analysis for each case started after the first interview and continued during the whole data collection time [ 28 ]. Interview summaries, documents and transcripts were used to iteratively develop the three-phase process maps, and the contributions to action.

To identify which features of research and translation processes were associated with the use of research, we first analyzed the individual process maps and developed a set of open codes. Examples of codes are ‘involving potential key-users in the formulation of research’ and ‘targeted dissemination of written results’. Using a constant comparative method of analysis and a manual coding system, two researchers and a research assistant then developed a more specific set of codes for those process features that seemed to matter the most [ 31 ].

We then conducted a second systematic cross-case analysis, in which, for each case, we analysed the presence and role of the selected process features and described them in a table. For each of the process features, a specific summary was developed. Our analysis was recursive, constantly moving from the specific cases, to the more general, with the aim of identifying commonalities and patterns across the variety of cases.

This study did not require ethics approval in Ghana. Under Dutch law, ethics approval in the Netherlands was also not required. Even though formal approval was not required, we followed regular ethically responsible qualitative research practice to ensure that substantive ethical issues would be dealt with in an appropriate way. Informed consent to participate in the study, record the interviews, use quotations and publish the results was obtained from all study participants. A report with the preliminary results was shared with participants in 2008. Based upon comments, two small adaptations were made in how the data was presented. The preliminary results were presented and discussed at a meeting with participants in Ghana in 2008 and at a meeting in the Netherlands in 2009. Those involved in the discussions confirmed the presented results.

We start this section with an overview of some of the studies and how they contributed to changes in health policy and practice. Next, we describe which features of research and translation processes were related to the contribution of research to action. In the last part, we further examine how translation processes evolved. In Table  1 , the 30 research projects and the most prominent translation process and contributions to action are described.

The identified contributions to action

In 20 of the 30 studies, we identified a contribution to action between 6 and 12 months after the studies were finalized. We refer to these 20 studies as the ‘used’ studies.

The produced knowledge was used in many different ways. Several studies provided new knowledge about the nature and scope of health problems. This new knowledge was often used by investigators with a formal position in the health system. An example is case 1, a study into factors associated with treatment default among tuberculosis patients [ 32 ]. The PI of the study was in charge of the regional tuberculosis program and said that he initiated the study with the aim of improving tuberculosis treatment. “ For a long time I was concerned about treatment default, we talked about what to do. […] This study created an opportunity to do something about it, to better understand the problems and improve treatment success. ” The PI translated his results in several actions: “ This study showed that financial constraint was the main reason for patients for defaulting. Distance was one of the main issues, because they had to board vehicles to the hospital every day. When we noticed that, one of the things I have done is I have opened five new treatment locations to bring access to TB treatment. Previously there was only one treatment center in the whole district. We have also arranged for transportation money to the treatment and a daily meal during the intensive phase .”

Another study revealed unexpected problems with the functioning and implementation of the immunization program, such as illegal charges and the sale of food supplements by health workers alongside the vaccination (case 10). Poor mothers who could not afford these extra charges and food supplements felt stigmatized and were less likely to have their children vaccinated. One of the co-investigators of the study was a district director for the Ghana Health Service, who aligned the research proposal with his concerns about the immunization program in his district and his aim to improve it. Towards the end of his study, he was promoted to the position of regional director in the Ghana Health Service. In this new function, he used the results in designing and implementing a new communication policy, a policy on abolishing illegal charges and the sale of food supplements at vaccination sites, and a new way for supervising the immunization process. “ When I started I was at district level, so I saw the need to do something to EPI [immunization program] . Being at regional level was a great opportunity. I met all the districts of the region and showed them the issues of immunization in Techiman, what I thought was not so different from other areas […] actually showing what went wrong was important for making those changes. ”

In many cases, the produced knowledge was first used in the research context and subsequently elsewhere. The co-investigator of the previous case said that he continued to use his results after he was transferred to a new region. In his new region, he informed the staff of the health districts about his study findings, encouraged them to look out for similar problems and implement the proposed policies. A district director confirmed this translation process: “ He has informed us in one of the EPI meetings […]. He studied the performance of the district and how to increase the performance. So he showed us the figures before the study, the difficulties they were having and after the study, the input they put in and the figures they were having. Since he came, we put everything in place. ”

In several studies, new practices, protocols and methods were developed and tested, which were first implemented locally and subsequently used elsewhere. An example is case 12, in which quality indicators were developed and teams were trained to improve the quality of care in a district. After the research project, the use of the developed indicators and quality teams was continued: “ The quality assurance has been institutionalized. Some of the district wide quality parameters that were proposed are being used already. Some others are still being reviewed for use ” (district director Ghana Health Service). A different interviewee linked to the study in case 12 described a second translation process: “ there were constraints between regular and enrolled nurses. This had been ongoing for years and came out again during the focus groups. Before the report was even finished they have changed the rule. Now they are wearing the same uniform to lower this rivalry. ” Two more examples are cases 13 and 19, in which results were used to develop a training program and support package for implementing the Community-based Health Planning and Services Initiative. The program and support package was used by different people involved in implementing this initiative throughout the country.

Interviewees also described a range of unanticipated ways in which the conduct of the research itself contributed to changes in health service practices. Case 9 provides two examples. According to one investigator: “ When we conducted the study we noticed very sharp shortages [in consumables for preventing maternal mortality] and linked up with the medical stores. When we found out that the stores were not there at all, we immediately reported to the regional director and made sure the situation was addressed. So indirectly that will enhance service delivery. And also the filing system: we had difficulties retrieving data. Some patients went out and with their cards. So the records were not complete. When we discovered that, we had to correct the system. So it facilitated the documentation system. ”

In several cases, results were used by different actors in different translation processes. An illustrative example is case 17, a study into the prevalence of infectious diseases among prisoners and guards in Ghana. For years, there had been anecdotes and occasional media reports about the poor health status of prisoners. After being contacted by a concerned prison officer, a university-based researcher initiated a disease surveillance study showing a significant outbreak of HIV and hepatitis C among prisoners and guards and a lot of risk behaviour among prisoners such as illegal drugs use, unprotected sex and tattooing with shared needles [ 33 ]. Counselling and treatment were provided and a peer education program was set up following the research in the prisons in which the studies were conducted. The prison service used the results to encourage the Ministry of Health to provide better health services to prisoners. Interviewees described how, after several discussions, the results played a role in the decision to include prisoners in the National Health Insurance Scheme. Other interviewees described how the results played a role in the lobby, and eventual decision to close Usher Fort prison, which was housed in an 17th century Dutch colonial fortress. Interviewees also described how the results inspired USAID to provide anti-retroviral treatment at a clinic next to Nsawam prison.

Another example in which results were translated in diverse actions is case 3: a study into resistance to anti-microbial drugs in Ghana. The study, which was initiated by a microbiology professor from a medical school, showed an alarming resistance to commonly used antimicrobials, such as tetracycline (82%), ampicillin (76%) and chloramphenicol (75%), and widespread multi-drug resistance [ 34 ]. The researchers provided several recommendations, such as training laboratory technicians, re-evaluating criteria for the use of antibiotics, enforcing laws on the sale of antibiotics and educating the public about their use. While participants described several plans that were inspired by the findings, most were shelved due to a lack of resources. The head of the Reference Laboratory described a plan to train laboratory technicians, but soon after this he retired. His successor was aware of the results, but did not mention any training plans and pointed to the lack of funding for such initiatives. The head of the Quality Assurance Unit described ideas to encourage laboratory testing before prescribing antibiotics, but had not taken any action. After additional interviews, two translation processes were identified. In response to feedback from the study, hospitals had taken the initiative to start a training for laboratory technicians. A policymaker pointed out that the results also played a role in discussions and decision-making about the list of essential medicines at the Ghana Health Service. This claim was confirmed by second interviewee who attended the same meeting.

Participants reported that the results of seven studies (cases 4, 5, 7, 23, 25, 27, 29) contributed, in diverse ways, to the design and implementation of the National Health Insurance Scheme. The development of the health insurance law was a lengthy, complex and sometimes highly contested process in which numerous actors were involved who negotiated about different proposals and plans, which slowly converged into the law that was eventually passed by Parliament in 2003 [ 35 ]. During this process, countless ideas, recommendations and plans were put forward in which all kinds of knowledge claims, experiences and interests played a role. Participants described how results of two studies were used in this process to support new proposals and challenge existing plans that were being developed. Case 5 showed that citizens wanted to be able to opt out of the insurance and districts needed additional funding to start-up the health insurance, which were both taken into account in the eventual policy. Participants described how the results of case 5, together with those of case 4, supported the choice for a district wide organization of the health insurance and provided insights in how these could be implemented. Case 4 also provided a method for identifying the poorest of the poor, which was adapted and then used in practice.

Other studies were used in the implementation of the National Health Insurance Scheme. The results of case 27 were used to successfully advocate for a flat fee system for reimbursing hospitals. Unit-cost data that were developed during this study were used by the Ghana Health Service to fund hospitals. The results of cases 7, 23 and 25 were used to improve the implementation of the insurance at district level. Case 7 showed local policymakers which groups were less likely to enroll in the insurance, after which a targeted enrolment campaign was organized. The results of both cases 7 and 25 helped to identify existing structures and networks through which the insurance could better reach target groups and collect premiums.

Process features that were associated with the use of research

Below, we describe which features of research and translation processes were associated with the use of the produced knowledge. We start with the ones directly linked to the research program.

Fit with the national research agenda

The national health research priority strategy, which was a key component of the research program, helped to attune the research projects to the health sector priorities. The priority setting process, which interviewees described as useful, resulted in a research agenda with four priority themes (Box 1). These four themes matched with the health policy priorities that were described in the 2001–2006 Ghanaian Health Sector Programme of Work. The research agenda clearly influenced the formulation of research proposals. Some researchers said that they took the priorities as starting point for formulating a proposal. Others adapted their existing ideas and proposals to make them fit with the research agenda. Of the 30 assessed studies, 28 were clearly in line with the research agenda. This is unsurprising, since alignment with national needs was an important consideration in the selection of studies for funding.

Initiation by potential key users

Eighteen studies were initiated by people who were primary decision-makers or held influential positions in the health system (Table  2 ). These people defined research questions that arose from the programs they ran or advised and were thus themselves a potential key user. As one PI described: “ The proposal grew out of observations as a district director that there is a problem with the functioning of this level in the health system. From years of problems. All kinds of problems. Then you realize that, because when you talk with your other colleague district directors, and they all say yes, we also have this problem. So then it is like, instead of investigating this felt need in my little district, why don’t I look at it beyond. So it was a national scale study. ”

Examples of these ‘user-investigators’ include the head of the regional tuberculosis program who initiated a study into therapy adherence, the district director who aimed to better implement the vaccination program, and a member of a health financing committee who studied ways to fund hospitals. “ It very much influenced how the proposal was structured, because I realized there was a gap that needed some kind of investigation, some kind of evidence, to be able to present, if I should say, a paper for policy decision to be taken. ”

User-investigators were a striking feature of studies that were utilized: 17 of the 18 studies with a user-investigator were translated into action.

Involving potential key users during the formulation of a research proposal

In addition to the potential key users who were part of the study team (as user-investigators), studies could also involve external potential key users during the formulation of the research proposal (Table  3 ). Participants described different reasons for consulting these external key users. Some were consulted to inform them about the proposal, ask for input, or increase the likelihood of use. Others mentioned that these potential key users had to be consulted in order to access to study populations, clinics or hospital administrations.

These consultations often led to adaptations of research proposals. The proposal of the study into maternal mortality was adapted after discussing it with the regional director: “ It was his idea that I should refocus on the service delivery factors, because that is what we have immediate control over. I had to remodel the framework a bit. I was going for a broader investigation” . In the prison health study, the director of the Prison Service asked the researchers to include not only inmates, but also prison officers in the disease surveillance study.

User-initiators also discussed their proposals with other potential key users. One of them said: “ What changed the proposal? For example, comments like, well, because it is possible, we are going to look at this to inform policy in the whole health sector. The Ghana Health Service, which has over two hundred hospitals and over a thousand clinics. Can you expend the sample size? I think to about two of each type, across the country, about eight or so. Try to cover all types, locations? So that influenced the design and also the sample size. ”

External potential key users were involved in the formulation of eight of the 18 user-initiated studies and in four of the other 12 studies, of which two were used. In a further four studies, none of them were user-initiated; external key users were informed about research proposals, but were not involved in shaping them.

Introducing new practices as part of research

Activities that were part of the implementation of the research itself could also contribute directly to action, and make it easier to use the results. In several cases, investigators and others provided examples of direct contributions that resulted from research activities, such as training health workers to follow a protocol, reorganizing administrative or logistical procedures, or teaching community members about HIV or the right to exemptions during interviews. While these direct contributions were of limited scope, interviewees said that they often remained after a study ended, and facilitated the use of the results. Case 12 provides a clear example. For the purpose of the study, new quality indicators were developed and teams were trained to use them to monitor quality of services in local clinics. After the study showed that this quality improvement strategy was beneficial, the use of these indicators was institutionalized in the involved clinics.

Involving potential key-users in developing recommendations

In almost all used studies, potential key users were engaged in interpreting the meaning of the results and developing recommendations for action (Table  4 ). In 15 of the 18 user-initiated studies and five of the 12 other studies, external potential key users were involved in developing recommendations. A user-investigator said: “ First we sat down together in the region and pooled the study findings. We came out with an operational document. What is the job description of a sub district head? What support must be given to them? How should they relate? We then send it out, everybody has commented on it. We then said ok, let’s start working with this. ”

Targeted distribution of printed results

The results of almost all studies were distributed in printed form beyond the scientific domain. In three cases, this dissemination was organized by the secretariat of the research program. In the other cases, the researchers had themselves taken the initiative to disseminate their results. Investigators who tried to mobilize others to use their results more often said that they adapted texts and prints to their target audience and sent it specifically to them: “ I send it to the Director of Human Resources and the Director General. What I did, I send not a research report, sometimes when people are busy, they don’t want a research report, but rather a memo. ”

Another investigator, who seemed very keen on the use of their results, described: “ When we did the final report. The Health Summit, you know the annual health summit. It was going on. We couldn’t get a slot to present the report, but were allowed to give people copies. So we carried copies of the report there and gave everybody a copy. We budgeted to print the report so that it looked attractive. ”

While the distribution of printed results may have supported translation processes, it was never described as playing an influential role in the use of research.

The translation of results into action: examining the process

As the preceding paragraphs show, many researchers made concerted efforts to involve potential users in interpreting study results, and to make sure users were aware of those results. On further examination, we found that the translation of results into action involved a complex interplay between different actors with different ideas about the meaning of the results, actual change efforts in which results were used and evolving dynamics and structures in the context. Here, we describe some of these processes in more detail.

Envisioning what should be done and who should do what

Researchers were themselves the first to shape the meaning of their study results. In four of the cases we studied, the investigators said that their results had no immediate implications for action. These investigators argued that their results confirmed existing knowledge or that further research was required. One of them explained: “ My findings and recommendations are not new things to the people in policy. They are things they already know. If there is anything at all, the presentation would only be to reinforce, to tell them that what they are doing is in the right direction. ” Not surprisingly, the results of these studies were not used to contribute to changes in policy or practice.

In the other 26 cases, the investigators said that their results should be translated into action, and they had ideas about how that should happen, and who should be involved. To achieve the changes they envisioned, actors put forward more or less explicit stories about a desired future, in which they assigned roles and responsibilities to a variety of actors and described what they should be doing. Depending on the forces at play and the situation in which these ‘actor-scenarios’ were put forward, research knowledge was assigned a role in them.

A technical advisor who aimed to use the results of the antimicrobial resistance study provides an example of a scenario of the future in which roles and responsibilities were assigned to several actors: “ The results show the Ministry of Health that what is happening in Accra is going on all over the country. From now on, the Regions must apply the law. The Ministry must take the results and use them to educate the pharmacists. They need to better explain how to take the medication. They must also educate the general population and thirdly, the herbalist who mix antimicrobial agents with their herbs. They have to stop that. ”

The stories about what results meant for action were not automatically accepted. Some people became inspired and put forward similar or somewhat modified actor-scenarios. Others started to resist the envisioned futures and roles they were assigned, and put forward alternative views in which the results had different implications for what should be done and who should do what. This could lead to further actions and interactions, after which a relatively stable set of ideas emerged about what results meant for action.

We analyzed who, according to the investigators, should play a role in the scenario’s which they described as leading to change (Table  5 ). In 14 cases, the investigators said that they should themselves play a key role in achieving change. In the other 12 cases, the investigators envisaged others playing the main role.Investigators gave different reasons why others had to play a key role in applying their results. Eight of them said they were constrained because they did not work in the health sector. For some, this was reason enough not to foresee a role for themselves in acting on the study results. “ I am an economist and I work here at the university. We presented to them [involved hospitals] and gave them the report […] How far they took it? It is up to them to use the result or not. ”

Not all investigators shared this idea. Three of the eight, also university-based researchers, saw a role for themselves even though they did not imagine that they would be the prime movers in achieving change. They described a strong motivation to encourage others to use their results for change. One of them explained: “ We did the study so it is logical for us to want to move the findings forward. I am not sure if anyone else would try to move the findings forward .”

Others said that they would not be able to take forward their results because they lacked the required seniority, influence or responsibility. One investigator said that, in Ghanaian culture, he would be considered too young to advise policymakers. Another investigator who was a district director and keen on the use of his findings, nonetheless felt that the results should be used in national policy processes, to which he had limited access and which were going in a different direction than the recommendation of his study.

Who mobilized results to achieve change?

Once the implications of results for action became accepted, people drew upon this accepted knowledge, and were influenced by it, to make real changes in policies and programs (Table  6 ). In 14 of the 20 used studies, one of the user-investigators played a key role in using this knowledge for achieving change. In the other six studies that were used, others, with whom the results had been personally discussed, played a key role in making change happen. In only one case we identified a translation process that happened without any interaction with the investigators, but this occurred more than 2 years after a study ended.

In all our cases, translations required efforts or support from people with a specific formal position, such as a regional health director, a program manager or a working group at the Ministry of Health. These formal positions were described as essential for acquiring support, mobilizing resources and making new knowledge part of concrete policies and practices.

Several investigators described how their position in policy processes became more influential because they were conducting a study. One of them said: “ When you do the study, then they know, it gives you a kind of authority in that area, they listen because you are involved, you have the data .”

In addition, some investigators said that they could scale up the use of their results when they themselves shifted positions, usually through promotions. An example is the district director in the immunization case, who was promoted to regional director during his study, and then transferred to lead a new region: “ When I came to this region, […] I found out that most of the things I saw over there, I am seeing here. I am carrying my luggage with me. Wherever I am going, the data goes with me .”

Interviewees pointed out that formal positions also had their limitations. Their influence was limited to specific subject areas, locations and directions. They also emphasized that trust, reputation, advocacy skills and sheer persistence could be just as important for gaining access to policy arenas, gathering support and mobilizing resources so that results could be turned into action.

The role of structures and dynamics in the context

Translations were not only shaped by actors and the coalitions they build, but also by the evolving world in which processes were embedded. Ideas, budgets, local practices, equipment and infrastructures that played a role in the envisioned change and in concrete actions could not be mobilized at will, but were entangled in a larger world full of existing structures and ongoing dynamics. Ideas were linked to value systems, budgets were part of financing schemes, practices were embedded in a social order, equipment depended on trained health workers and physical infrastructures were shaped by the local landscape. As a result of these entanglements, the structures and dynamics in the larger world enabled some translations and constrained others.

While we focused our analyses on the actions of individuals, in some cases, these structures and dynamics seemed just as important for how translations worked out. An example of a larger dynamic that influenced several translation processes was the design and implementation of the National Health Insurance Scheme. Cases 5 and 6 illustrate how this interacted with research and translation efforts. Case 5 examined the functioning of district level health insurance schemes, while the study in case 6 focused on community based health insurance. The study in case 5 was formulated and executed in relative isolation, with no involvement of potential key-users. Study 6 was led by a district health director, who interacted with potential key-users and was keen on making a contribution. When the national task force, which designed the blueprint for the National Health Insurance Scheme, was considering the role of the districts, some members became very interested in the recommendations from case 5. “ We had just finished the project when the government wanted to adapt the health insurance program. Most of our recommendations were incorporated in what was eventually adopted as national policy .” Despite the intentions and position of the district director, the results from case 6 were neglected by the national task force: “ It has not contributed to national policy because it didn’t fit the current agenda. It should have, and I think it is a prophecy document .”

The aim of our study was to map the contribution of health research to action and examine which features of research and translation processes were associated with the use of the results. All 30 cases in our sample were part of a program of health research which aimed particularly to foster locally led, demand-driven studies in Ghana.

Overall, we found that, in 20 of the 30 assessed research projects, a contribution to action for health could be identified between 6 and 12 months after studies were finalized. It is difficult to compare this use rate with other research programs, since data are sparse. The few studies that have been published tend to focus on a small number of cases, use self-reporting without triangulation, and/or interview a limited number of informants [ 13 , 36 , 37 ]. Perhaps the most similar study to ours was recently conducted in Australia, and used a questionnaire, one interview per case and a panel to assess the “ real world policy and practice impacts ” of 50 intervention studies within 5 years after finalization. In this study, 38% of the cases seemed to have ‘impact’, though this could not always be corroborated [ 15 ]. In our study of 30 cases, for which we interviewed several informants per case, the results of 67% of the studies were used to contribute to action for health within a year.

Our analysis suggests that this relatively high proportion is related to the strategy of the research program, which was designed specifically to enable studies that would be likely to contribute to action. Two aspects in particular seem to have made a difference. The first was the process of priority setting and study selection, which led to the funding of studies that were, from the onset, closely aligned with local health sector priorities, and that therefore posed questions that met the immediate information needs of those who shaped policy in health.

The second, and perhaps most important aspect of the program strategy in terms of the eventual use of research results, was that research had to be initiated and led by Ghanaians and that health sector professionals as well as academics were eligible to initiate studies.

Looking more closely at which features of research were most strongly associated with eventual use of study results, we found that one stood out above all. That was the presence of a single person who initiated the study, remained involved in the process, and was in a position to use the results in their own work. Critically, these user-investigators were likely to formulate ‘need-to-know’ research questions that filled urgently-felt information gaps and took initiatives to use their result. The results of 17 out of 18 of the studies involving user-investigators were translated into action.

The use of the results by the user-investigator was not the only reason why 17 of these 18 studies were used. Studies initiated by potential end-users were also more likely to involve other potential users in the formulation of research as well as in interpreting results and developing recommendations. In three cases, the user-investigators themselves did not have a major role in using study results. It was their ongoing interaction with other ‘external’ potential users that seemed to enable the use of the results.

If none of a study’s investigators was themselves a potential key user, interaction with external potential users seemed critical to the use of results. In one case, involving research in prisons, potential key-users contributed to both the study design and the interpretation of results. In three other cases, potential key-users were not involved until the field work was completed. It was their engagement in the interpretation of study results that appeared to contribute to the translation of research into action.

Our findings are in line with an analysis of research impact in the United Kingdom by Greenhalgh and Fahy [ 38 ] in which the use of research was characterized by an ethical commitment by researchers, strong institutional support and a proactive interdisciplinary approach to impact activities. Our findings contradict attempts to explain the use of research in terms of the characteristics of the results, such as their salience, applicability or validity [ 39 – 41 ]. While results could certainly play a role, we found that the use of research was strongly influenced by those who put forward what the results meant for action. The involvement of potential key users in this process seemed to contribute to developing recommendations and concrete plans that were broadly feasible, taking into account the validity of results, the specifics of the local situations and the aims of those who shaped policies for health.

Our analysis of how translation processes evolved suggests that there were two overall dynamics in the translation of knowledge into action: a first, in which investigators and others put forward stories about what results meant for action, which, after interaction and stabilization, could become part of the repertoire of locally accepted knowledge; and a second dynamic, in which actors drew upon this accepted knowledge, and were influenced by it, to actually effect change. These processes were not linear or isolated, but recursive, and embedded in, and interacting with, ongoing action and dynamics and structures in the context [ 42 ].

This perspective on knowledge translation may be useful to those who study how context influences the use of research, evidence briefs or other knowledge products [ 43 , 44 ]. The actor-scenario perspective suggests that context cannot easily be studied as a set of fixed factors that somehow have effect on the use of research, as different users may put forward very different actor-scenarios, in which the same results play a very different role and very different elements of ‘context’ are mobilized.

While several studies in other countries find that interaction with users enhances the likelihood that research is used, this is the first study to our knowledge in which the relation between what happens throughout research processes, and the use of the results, is systematically analyzed in a substantial number of demand-driven, locally led studies in a lower-income country.

Our findings support those of Walley et al. [ 45 ], who, based upon experiences in China and Pakistan, argued for an approach of “ getting practice into research: to get research into practice ”, especially for operational research in developing countries. An advantage of such an approach is that a problematic gap between researchers who ‘discover’ and policymakers and practitioners who ‘apply’ may not emerge [ 46 ]. Such an approach is perhaps not appropriate for research into new and untried treatments were the efficacy has not been established, but our study shows that it has great application for applied research.

A possible limitation of the focus on potential key users is that the use of results could be constrained by their authority or influence. In our study, we found this not generally to be the case. In part because they were promoted and transferred, and in part because the use of results created a concrete example to others, which helped to spread changes more widely. A potential risk of involving influential users in research processes is that their aims and interests may bias research. While this requires attention, it is important to recognize that researchers themselves have their own aims, interests and perspectives, which may also need to be reflected upon [ 23 ].

Researchers tend to focus on the written texts that they produce and disseminate and which they hope are picked up by others and then translated into action. We observed that results were mostly spread by people who were moving about, personal interaction and through the spread of successful innovations in which results were used. In none of our cases was the dissemination of written texts described as important for the use of results. This can be explained perhaps by the important role of user-investigators and personal interaction, which may have replaced the role of written texts, our selection of interviewees, and our 1-year follow-up, which seems short compared to other studies. Another explanation is that the use of written texts by unknown individuals, at unknown times and places is rather difficult to map, which may lead to an overestimation of the role of interaction [ 7 ].

This study shows both the potential and importance of locally led, demand-driven health research in lower-income countries. The approach of the research program was inspired by the critique in the early nineties that health research contributed little to health and development in poorer countries because it was dominated by foreign scholars instead of locally embedded researchers, and met international rather than local needs. The research program tried to turn this around by fostering research that was driven by local demands and led by local researchers. Our analysis shows the success of this approach in terms of contributing with research to action for health. This finding corresponds with analyses of research programs by others in several other countries [ 47 , 48 ]. Our analysis also shows the importance of local research for improving local action for health. While the studies did not produce major scientific breakthroughs, they often played a key role in improving local action for health, which is remarkable given their small budgets.

Considerations for research policy

The results allow us to formulate some suggestions for those who attempt to support research that more effectively contributes to health in low-income countries and elsewhere.

A first suggestion is to continue promoting national research priority setting, which is becoming increasingly common around the world [ 49 , 50 ]. While priority setting is only a first step, a demand-driven priority agenda can assist researchers in formulating proposals towards local needs and can help funders to select studies that are more likely to be used. A careful and inclusive priority setting process not only helps to orient research to needs, but also provides a platform for interaction, building trust and networking, which are important for the eventual use of research [ 50 – 52 ].

A second suggestion is to stimulate research that is initiated and conducted by those who can play a role in the use of the results. A challenge is that the number of professionals with an influential role in health policy, sufficient research skills and the necessary time for research is likely to be small [ 53 , 54 ]. It may be worth exploring how these user-investigators can best be incentivized and supported in their work, for example, by junior researchers [ 55 ].

The third suggestion is to engage potential key users in research processes from the start, especially in designing research proposals, interpreting results and formulating recommendations. To select potential key users, researchers can try to envision how results may be used and who will play a role in that process (an actor-scenario), and then try to involve those who seem most interested and influential.

While this study shows the advantages of demand-driven research, several cases show that more independent and critical research is also essential for improving global health [ 56 – 59 ]. A risk of a unilateral focus on demand-driven research is that it may take prevailing ideas, power relations and dominant elites as a starting point, and may lead to ignoring questions about what dominant views are based upon, the effects of power relations, and the needs of more marginalized groups [ 25 ].

Limitations

The detailed interviews showed that each case was unique, context-specific and far more nuanced that we have been able to describe in this paper. In order to assess what played a role in whether research was used or not, we have been obliged to reduce a precarious, ongoing and complex process into a snapshot of a limited period and number of actors and actions. The ‘use’ and ‘non-use’ of results, for example, actually covers a wide and dynamic spectrum which is not fully reflected in a binary categorization. Similarly, the roles of individuals in the inherently collaborative process of research did not always fit as neatly into binary categories of ‘user-initiated’ or ‘not user-initiated.’ Our iterative and inclusive research design aimed to minimize the subjectivity of these simplifications. The large number of interviews, openness of participants and the relatively small number of key actors involved in both the research and policy community helped us to examine how processes evolved and to triangulate claims. While some investigators had the initial tendency to under or overestimate the use of their research, the shared exploration of how processes evolved often helped to describe the role that research knowledge had played.

In examining the contribution of health research to action we identified a number of features which have implications for organizations that support research, especially but not exclusively, in low- and middle-income countries. Our study underlines the importance of supporting research that meets locally-expressed needs and that is led by people embedded in the contexts in which results can be used. Supporting the involvement of health sector professionals in the design, conduct and interpretation of research appears to be an especially worthwhile investment.

World Health Organization. The world health report 2013. Geneva: WHO; 2013.

Google Scholar  

Oliver K, Lorenc T, Innvar S. New directions in evidence-based policy research: a critical analysis of the literature. Health Res Policy Syst. 2014;12:34.

Article   PubMed   PubMed Central   Google Scholar  

Wehrens R. Beyond two communities–from research utilization and knowledge translation to co-production? Public Health. 2014;128:545–51.

Article   CAS   PubMed   Google Scholar  

Hanney SR, González-Block MA. Health research improves healthcare: now we have the evidence and the chance to help the WHO spread such benefits globally. Health Res Policy Syst. 2015;13:10–3.

Article   Google Scholar  

Wehrens R, Bekker M, Bal R. Coordination of research, policy and practice: a case study of collaboration in the field of public health. Sci Public Policy. 2011;38:755–66.

El-Jardali F, Lavis JN, Moat K, Pantoja T, Ataya N. Capturing lessons learned from evidence-to-policy initiatives through structured reflection. Health Res Policy Syst. 2014;12:2.

Lemay MA, Sa C. The use of academic research in public health policy and practice. Res Eval. 2013;23:79–88.

Uzochukwu B, Mbachu C, Onwujekwe O, Okwuosa C, Etiaba E, Nyström ME, et al. Health policy and systems research and analysis in Nigeria: examining health policymakers’ and researchers’ capacity assets, needs and perspectives in south-east Nigeria. Health Res Policy Syst. 2016;14:13.

Kok MO, Rodrigues A, Silva AP, de Haan S. The emergence and current performance of a health research system: lessons from Guinea Bissau. Health Res Policy Syst. 2012;10:5.

The Commission on Health Research for Development. Health Research, Essential Link to Equity in Development. Oxford: Oxford University Press; 1990.

Wolffers I, Adjei S, van der Drift R. Health research in the tropics. Lancet. 1998;351:1652–4.

Wooding S, Hanney SR, Buxton M, Grant J. Payback arising from research funding: evaluation of the Arthritis Research Campaign. Rheumatology (Oxford). 2005;44:1145–56.

Article   CAS   Google Scholar  

Oortwijn WJ, Hanney SR, Ligtvoet A, Hoorens S, Wooding S, Grant J, et al. Assessing the impact of health technology assessment in The Netherlands. Int J Technol Assess Health Care. 2008;24:259–69.

PubMed   Google Scholar  

Puri M, Regmi S, Tamang A, Shrestha P. Road map to scaling-up: translating operations research study’s results into actions for expanding medical abortion services in rural health facilities in Nepal. Health Res Policy Syst. 2014;12:24.

Cohen G, Schroeder J, Newson R, King L, Rychetnik L, Milat AJ, et al. Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool. Health Res Policy Syst. 2015;13:3.

Walugembe DR, Kiwanuka SN, Matovu JKB, Rutebemberwa E, Reichenbach L. Utilization of research findings for health policy making and practice: evidence from three case studies in Bangladesh. Health Res Policy Syst. 2015;13:26.

Burchett HED, Mayhew SH, Lavis JN, Dobrow MJ. When can research from one setting be useful in another? Understanding perceptions of the applicability and transferability of research. Health Promot Int. 2013;28:418–30.

Article   PubMed   Google Scholar  

Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS One. 2011;6:e21704.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Landry R, Lamari M, Amara N. The extent and determinants of the utilization of university research in government agencies. Public Adm Rev. 2003;63:192–205.

Oliver K, Innvaer S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2.

Lomas J. Using “linkage and exchange” to move research into policy at a Canadian foundation. Health Aff. 2000;19:236–40.

Redman S, Turner T, Davies H, Williamson A, Haynes A, Brennan S, et al. The SPIRIT Action Framework: A structured approach to selecting and testing strategies to increase the use of research in policy. Soc Sci Med. 2015;136–137:147–55.

Hegger I, Janssen SW, Keijsers JF, Schuit AJ, Van Oers HA. Analyzing the contributions of a government-commissioned research project: a case study. Health Res Policy Syst. 2014;12:8.

Cooke J, Ariss S, Smith C, Read J. On-going collaborative priority-setting for research activity: a method of capacity building to reduce the research-practice translational gap. Health Res Policy Syst. 2015;13:25.

Pratt B, Merritt M, Hyder AA. Towards deep inclusion for equity-oriented health research priority-setting: A working model. Soc Sci Med. 2016;151:215–24.

Viergever RF, Olifson S, Ghaffar A, Terry RF. A checklist for health research priority setting: nine common themes of good practice. Health Res policy Syst. 2010;8:36.

PubMed   PubMed Central   Google Scholar  

Hacking D, Cleary S. Setting priorities in health research using the model proposed by the World Health Organization: development of a quantitative methodology using tuberculosis in South Africa as a worked example. Health Res Policy Syst. 2016;14:10.

Hanney SR, González-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Heath Res Policy Syst. 2003;1:2.

Kok MO, Schuit AJ. Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Res Policy Syst. 2012;10:21.

Wolffers I, Adjei S. Research-agenda setting in developing countries. Lancet. 1999;353:2248–9.

Pope C, Ziebland S, Mays N. Qualitative research in health care. Analysing qualitative data. BMJ. 2000;320:114–6.

Dodor EA, Afenyadu GY. Factors associated with tuberculosis treatment default and completion at the Effia-Nkwanta Regional Hospital in Ghana. Trans R Soc Trop Med Hyg. 2005;99:827–32.

Adjei AA, Armah HB, Gbagbo F, Ampofo WK, Quaye IKE, Hesse IFA, et al. Prevalence of human immunodeficiency virus, hepatitis B virus, hepatitis C virus and syphilis among prison inmates and officers at Nsawam and Accra. Ghana J Med Microbiol. 2006;55(Pt 5):593–7.

Newman MJ, Frimpong E, Donkor ES, Opintan JA, Asamoah-Adu A. Resistance to antimicrobial drugs in Ghana. Infect Drug Resist. 2011;4:215–20.

Agyepong IA, Adjei S. Public social policy development and implementation: a case study of the Ghana National Health Insurance scheme. Health Policy Plan. 2008;23:150–60.

Hanney SR, Mugford M, Grant J, Buxton MJ. Assessing the benefits of health research: lessons from research into the use of antenatal corticosteroids for the prevention of neonatal respiratory distress syndrome. Soc Sci Med. 2005;60:937–47.

Kwan P, Johnston J, Fung AY, Chong DS, Collins RA, Lo SV. A systematic evaluation of payback of publicly funded health and health services research in Hong Kong. BMC Health Serv Res. 2007;7:121.

Greenhalgh T, Fahy N. Research impact in the community-based health sciences: an analysis of 162 case studies from the 2014 UK Research Excellence Framework. BMC Med. 2015;13:232.

Weiss CH, Bucuvalas MJ. Truth tests and utility tests: decision-makers’ frames of reference for social science research. Am Sociol Rev. 1980;45(2):302–13.

Dearing JW, Meyer G, Kazmiercak J. Portraying the new: communication between university innovators and potential users. Sci Commun. 1994;16(1):11–42.

Lomas J. Diffusion, dissemination, and implementation: who should do what? In: Doing More Good than Harm Eval Heal Care Interv. Edited by Warren KS, Mosteller F. New York: New York Academy of Science; 1993. p. 226–37.

Andermann A, Pang T, Newton JN, Davis A, Panisset U. Evidence for Health II: Overcoming barriers to using evidence in policy and practice. Health Res Policy Syst. 2016;14:17.

Squires JE, Graham ID, Hutchinson AM, Michie S, Francis JJ, Sales A, et al. Identifying the domains of context important to implementation science: a study protocol. Implement Sci. 2015;10:135.

Moat K, Lavis JN, Abelson J. How contexts and issues influence the use of policy-relevant research syntheses: a critical interpretive synthesis. Milbank Q. 2013;91:604–48.

Walley J, Khan AM, Shah SK, Witter S, Wei X. How to get research into practice: first get practice into research. Bull World Health Organ. 2007;85:424.

Lavis JN, Lomas J, Hamid M, Sewankambo NK. Assessing country-level efforts to link research to action. Bull World Health Organ. 2006;84:620–8.

Varkevisser CM, Mwaluko GM, Le Grand A. Research in action: the training approach of the Joint Health Systems Research Project for the Southern African Region. Health Policy Plan. 2001;16:281–91.

Brambila C, Ottolenghi E, Marin C, Bertrand JT. Getting results used: Evidence from reproductive health programmatic research in Guatemala. Health Policy Plan. 2007;22:234–45.

Becerra-Posada F, Minayo M, Quental C, de Haan S. National research for health systems in Latin America and the Caribbean: moving towards the right direction? Health Res Policy Syst. 2014;12:13.

Mugwagwa J, Edwards D, de Haan S. Assessing the implementation and influence of policies that support research and innovation systems for health: the cases of Mozambique, Senegal, and Tanzania. Health Res Policy Syst. 2015;13:21.

Mador RL, Kornas K, Simard A, Haroun V. Using the Nine Common Themes of Good Practice checklist as a tool for evaluating the research priority setting process of a provincial research and program evaluation program. Health Res Policy Syst. 2016;14:22.

Cole DC, Nyirenda LJ, Fazal N, Bates I. Implementing a national health research for development platform in a low-income country – a review of Malawi’s Health Research Capacity Strengthening Initiative. Health Res Policy Syst. 2016;14:24.

Erasmus E, Lehmann U, Agyepong IA, Alwar J, de Savigny D, Kamuzora P, et al. Strengthening post-graduate educational capacity for health policy and systems research and analysis: the strategy of the Consortium for Health Policy and Systems Analysis in Africa. Health Res Policy Syst. 2016;14:29.

Gonzalez Block MA, Mills A. Assessing capacity for health policy and systems research in low and middle income countries. Health Res Policy Syst. 2003;1:1.

Bates I, Boyd A, Smith H, Cole DC. A practical and systematic approach to organisational capacity strengthening for research in the health sector in Africa. Health Res Policy Syst. 2014;12:11.

Shin S, Furin J, Bayona J, Mate K, Kim JY, Farmer P. Community-based treatment of multidrug-resistant tuberculosis in Lima, Peru: 7 Years of experience. Soc Sci Med. 2004;59:1529–39.

Aaby P, Benn CS. Non-specific and sex-differential effects of routine vaccines: what evidence is needed to take these effects into consideration in low-income countries? Hum Vaccin. 2011;7:120–4.

Rodrigues A, Sandström A, Cá T, Steinsland H, Jensen H, Aaby P. Protection from cholera by adding lime juice to food - Results from community and laboratory studies in Guinea-Bissau, West Africa. Trop Med Int Heal. 2000;5:418–22.

Benn CS, Aaby P, Arts RJW, Jensen KJ, Netea MG, Fisker AB. An enigma: Why vitamin A supplementation does not always reduce mortality even though vitamin A deficiency is associated with increased mortality. Int J Epidemiol. 2015;44:906–18.

Download references

Acknowledgements

The team is grateful to Esther Vordzorgbe, Amanua Chinbuah and Mercy Abbey, the HRU team in Accra, and Rene van Veenhuizen. We would also like to thank Clement Amofah, Mandy Thijm, Jitske Both and Isabel Siemelink for their assistance in the data collection processes. We also gratefully acknowledge Jantine Schuit, Roland Bal and Arie Rip for their constructive suggestions to earlier versions of this manuscript, Elizabeth Pisani for her help with editing the final draft, and MvM for her invaluable support. We would like to reserve our final acknowledgement to all those professionals, from research, policy and practice for health, who somehow participated in this study. This work was supported by the SOR programme of the Netherlands National Institute for Public Health and the Environment and the Research Excellence and Innovation grant from Erasmus University.

Author information

Authors and affiliations.

Institute of Health Policy and Management, Erasmus University Rotterdam, Rotterdam, The Netherlands

Maarten Olivier Kok

VU University Amsterdam, Amsterdam, The Netherlands

Maarten Olivier Kok & Joost Ruitenberg

School of Public Health, University of Ghana, Accra, Ghana

John Owusu Gyapong

Department of Health Care and Culture, VU University Medical Centre Amsterdam, Amsterdam, The Netherlands

Ivan Wolffers

Noguchi Memorial Institute for Medical Research, University of Ghana, Accra, Ghana

David Ofori-Adjei

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Maarten Olivier Kok .

Additional information

Competing interests.

JOG was executive director of the HRDP and received a part time salary for this work. JR was co-chair of the Joint Programme Committee of the HRDP. DOA was co-chair of the Joint Programme Committee of the HRDP. IW was a member of the Joint Programme Committee of the HRDP. The remaining authors have no conflicts of interest to declare.

Authors’ contributions

The idea of conducting this study and writing this paper emerged collectively from the group of authors. MK developed the methods, conducted the empirical analyses together with three assistants and wrote the first draft of the manuscript, on which all authors commented. All authors read and approved the final manuscript.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Kok, M.O., Gyapong, J.O., Wolffers, I. et al. Which health research gets used and why? An empirical analysis of 30 cases. Health Res Policy Sys 14 , 36 (2016). https://doi.org/10.1186/s12961-016-0107-2

Download citation

Received : 26 October 2015

Accepted : 21 April 2016

Published : 17 May 2016

DOI : https://doi.org/10.1186/s12961-016-0107-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Research impact
• Contribution mapping
• Research policy
• Knowledge translation
• Research utilization
• Priority setting

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Open access
• Published: 27 March 2024

Exploring health and disease concepts in healthcare practice: an empirical philosophy of medicine study

• Rik R. van der Linden   ORCID: orcid.org/0000-0002-9642-3715 1 &
• Maartje H.N. Schermer   ORCID: orcid.org/0000-0003-4283-9659 1  

BMC Medical Ethics volume  25 , Article number:  38 ( 2024 ) Cite this article

223 Accesses

Metrics details

In line with recent proposals for experimental philosophy and philosophy of science in practice, we propose that the philosophy of medicine could benefit from incorporating empirical research, just as bioethics has. In this paper, we therefore take first steps towards the development of an empirical philosophy of medicine, that includes investigating practical and moral dimensions. This qualitative study gives insight into the views and experiences of a group of various medical professionals and patient representatives regarding the conceptualization of health and disease concepts in practice and the possible problems that surround them. This includes clinical, epistemological, and ethical issues. We have conducted qualitative interviews with a broad range of participants ( n  = 17), working in various health-related disciplines, fields and organizations. From the interviews, we highlight several different practical functions of definitions of health and disease. Furthermore, we discuss 5 types of problematic situations that emerged from the interviews and analyze the underlying conceptual issues. By providing theoretical frameworks and conceptual tools, and by suggesting conceptual changes or adaptations, philosophers might be able to help solve some of these problems. This empirical-philosophical study contributes to a more pragmatic way of understanding the relevance of conceptualizing health and disease by connecting the participants’ views and experiences to the theoretical debate. Going back and forth between theory and practice will likely result in a more complex but hopefully also better and more fruitful understanding of health and disease concepts.

Peer Review reports

In the philosophy of medicine, scholars have primarily addressed ‘health’ and ‘disease’ as theoretical concepts without exploring their actual use in practice all too much. Yet, it has been argued that the way we conceptualize health and disease also affects the practical and moral dimension of medicine [ 1 , 2 ]. While many philosophers recognize the practical consequences of defining health and disease in certain ways, most still tend to depart from theory to determine how health and disease should be defined. In the traditional analytical debate, only limited attention has been paid to the ways in which these concepts are embedded in the various practices they are deployed in. In the medical-philosophical literature, the conceptual, epistemic and bioethical issues associated with proposed disease-definitions, such as medicalization and overdiagnosis, have been primarily addressed as theoretical problems, often lacking contextualization and empirical foundation. Consequently, it is often not clear to what extent such conceptual issues are in fact experienced as problematic in practice and for whom exactly this is a problem. While it is increasingly recognized that the traditional method of conceptual analysis is ill-equipped to answer the various normative, ontological and epistemological questions surrounding the conceptualization of health and disease [ 2 , 3 , 4 ], new philosophical perspectives and research methods have to yet to be explored.

In recent contributions to the debate, several promising proposals have been made for a new direction, in which health and disease are viewed as plural concepts that need to be specified [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ]. Instead of formulating definitions on monistic grounds, it is proposed to continue the debate by philosophical explication [ 4 , 10 ], and by developing precising definitions [ 12 ]. This is important as concepts may serve various practical functions and are deployed in diverse contexts. As different practices may have different values, goals, and priorities, different types of definitions may be needed [ 7 ]. Moreover, we have recently suggested that we should assess the successfulness of concept definitions in relation to the function they serve in the context they are deployed in [ 5 ]. This shift towards a pragmatist stance requires scholars to look beyond theoretical arguments and to explore the various practical motivations of defining health and disease. Hence, when explicating concepts, it seems important to complement the theoretical debate by empirically studying the use of concepts in practice.

In contrast to the field of bioethics where empirical methods are commonly used to research attitudes, beliefs and perspectives of certain groups of people, empirical research is only seldomly conducted in philosophical studies on health, disease, and related concepts. Adding these methods to our philosophical toolbox enables us to investigate more closely how concepts of health and disease operate in medical practice and to explore what kind of problems occur in relation to them. We could use existing socio-empirical studies that, for example, investigate psychosocial and cultural aspects of certain diseases (e.g., see [ 13 ]), that review definitions and meanings of certain medical or bioethical concepts (e.g., see [ 14 , 15 ]), or that explore patients’ and professionals’ views towards certain research programs or medical developments (e.g., see [ 16 ]). Both quantative and qualitative methods can be useful, depending on the research question at stake. However, as we propose in this paper, besides making use of existing empirical literature, we can also conduct empirical philosophy of medicine studies that aim to explore philosophical questions head-on.

Referring to debates on empirical ethics, Seidlein & Salloch [ 17 ] recently argued that the reconciliation of perspectives in the philosophy of medicine and socio-empirical research will lead to a more nuanced discussion that includes experiences of patients. Drawing on Alexander Kon’s [ 18 ] pragmatic classification of empirical methods, they argue that this approach may be used to investigate current practices (‘Lay of the Land’), revealing differences between illness conceptions in different groups of people, or between notions of ‘disease’ and ‘illness’. Such studies may improve patient-centered and shared decision-making, as it becomes clearer ‘what’ should be treated (cf. [ 19 ]). In addition to this, we argue that studying the views, attitudes and beliefs of medical researchers, clinicians and other healthcare stakeholders, seems important for obtaining a better and wider understanding of how health and disease concepts are used in actual practice and why they are conceptualized in certain ways. This proposal for incorporating tools and methods of the social sciences in philosophical work on health and disease concepts resonates with calls for experimental philosophy of medicine Footnote 1 [ 20 , 21 ], and for more ‘philosophy of science in practice’ [ 22 , 23 ].

While there have not been many studies focused particularly on health and disease concepts in which empirical methods are used, some exceptions should be mentioned here. In Hofmann [ 24 ], physicians were presented a list of different conditions and were asked to classify them as disease or non-disease. Hofmann demonstrated that there are disparities between what physicians consider diseases. In Stronks et al. [ 25 ], lay people, randomly recruited on the streets, were asked to define what ‘health’ means to them. The study resulted in an extensive overview of different aspects of health and disease, categorized into multiple clusters, with interesting differences between socio-economic classes. In Kohne et al. [ 26 ], clinicians, patients, and clinicians who have been patients themselves, were interviewed to explore their ideas regarding the ontology of mental disorders. They observed that the ‘ontological palette’ is more diverse than is commonly perceived within the dominant scientific and educational discourse. In Van Heteren et al. [ 27 ], frontline professionals were interviewed to investigate their conceptions of health in clients with psychosocial problems. They observed that professionals define health in different ways but that they also accommodate for the views of their patients and to the broader context care is provided in.

As we understand health and disease concepts to be context-dependent, we believe it is important to investigate their function and problems arising in relation to them in various contexts. Regarding the methodology and the type of inquiry, our pragmatist approach encourages us to look for problematic situations . The term ‘problematic situations’ originates from the work of pragmatist John Dewey (see [ 28 ]), who argued that academic inquiry must always start with (solving) actual problems. Here, we will use the term problematic situation to describe as a situation in which current conceptions/definitions of health and disease are no longer sufficient for the continuation of a certain health care (related) practice, or the achievement of a goal of the specific practice that is at stake. Thus, besides mapping different health and disease conceptualizations, we primarily explore what kind of problematic situations are experienced in practice and investigate possible underlying conceptual issues. In doing so, we aim to further elucidate the philosophical debate on conceptualization of health and disease and give it more practical relevance. In this study we have therefore conducted qualitative interviews with a broad range of professionals and patient representatives, working in various health-related disciplines, fields and organizations. We chose qualitative methods because these are considered the most suitable for investigating new and underexplored areas.

Methodology

We have designed a qualitative interview study with professionals working in various fields and organizations. Interviews were conducted by RL. As the sample included a broad range of professionals and patient representatives, a one-size-fits-all approach was not considered to be useful. We used a semi-structured interview guide that could be adjusted and specified to each of the interviews. This structure allowed us to explore context specific problems in more detail and to respond more extensively to issues participants mentioned during the interviews. Examples of interview questions include (for the complete guide, see appendix): ‘How would you describe ‘health’ and ‘disease’ yourself?’; ‘Would colleagues in your field agree with your definitions?’; ‘Are there any specific problematic situations that you encounter in practice that are related to definitions of health and disease?’; ‘Do you see any solutions to such problematic situations or have there been solutions brought forward to solve these issues?’. From these broader, more abstract questions, the interview was subsequently narrowed down to more specific questions, in response to the answers given by the participants. The interviews were conducted digitally, via Microsoft Teams, and took 46 min on average (ranging from 37 to 57 min). Audio recordings of the interviews were transcribed verbatim.

Setting and recruitment

This study was conducted in The Netherlands. All participants were Dutch speaking and all were highly educated. All participants were selected following the principle of purposeful sampling. The reason for choosing for purposeful sampling was that we wanted to study definitions of disease and health in relation to actual problems arising in health-related practices. We recruited professionals who have spoken out in public or professionally about problems in relation to health and disease definitions and/or who work in fields/organizations that we considered to be interesting because we expected such issues to arise. Moreover, we aimed to cover a broad range of healthcare practices. The participants were recruited by e-mail.

Participants

The sample details a broad range of professionals ( n  = 17), including doctors, policy makers, representatives of patient organizations, humanities experts, and medical professionals working in various advisory boards and governmental organizations (see appendix for a specified overview of participants their expertise). All participants were Dutch speaking, highly educated and experienced professionals. The representatives of the patient organizations that we included were interviewed in their professional role and not as patients (if applicable). One of the interviews had to be excluded from analysis because the recording was unusable due to a technological error, bringing down the total number of transcripts from 17 to 16.

Data analysis

The data was analyzed using NVivo software (11th edition). The first interview reports and transcripts were discussed among RL and MS. Based on these discussions, RL made a first coding-scheme and discussed this with MS, which resulted in some adaptations. To reduce ‘tunnel vision’, transcripts were then analyzed and coded by RL and MS separately and compared afterwards. The interviews were analyzed in a way that may be best described as a method in between ‘grounded theory’ [ 29 ] and ‘directed content analysis’ [ 30 ]. That is, we did not build a conceptual scheme completely bottom-up as one would do with grounded theory. However, it was also not the case that we already had a solid theoretical framework at the start of the analysis which we would use to frame the issues discussed in the interviews, as is common in directed content analysis. We have taken the answers given by participants as a point of departure, exploring what their views are regarding the function of health and disease concepts, and exploring what kind of problematic situations they experience in practice. Sometimes, participants would already refer to specific theories, approaches or models themselves However, for other parts of the analysis, we have made use of distinctions and concepts from the academic literature to make sense of the many issues that were brought forward by participants. For instance, some issues mentioned by participants could be viewed as being practical examples of what is called a ‘line-drawing problem’ in the theoretical debate [ 10 , 31 ]. Such categories appeared useful for analyzing and interpreting the data but where not selected prior to the analysis.

Defining health and disease

In the interviews, respondents have pointed to various important practical functions of health and disease concepts. In some interviews the influence of certain definitions/approaches was explicitly articulated by participants. Participants talked about practical problems that they experienced and were often able to link these with how health and disease are conceptualized in their fields. For instance, some participants described specific models or definitions, such as the biopsychosocial model [ 32 , 33 , 34 ] and Positive Health [ 35 , 36 ] and talked about their significance for their professional fields. In other interviews, however, the link between conceptualizations of health and disease and practical issues was more implicit. Participants would, for example, speak more broadly about ‘biomedical’ and ‘holistic’ approaches, or discussed how thinking in terms of ‘evidence based medicine’ (EBM) could (negatively) affect clinical practice.

While some of the respondents mentioned that it would be convenient to have general, all-encompassing definitions, none of them thought it would be possible to formulate them in a way that they are exhaustive and practically useful at the same time. Instead, in some interviews, viewing health and disease as plural concepts was discussed as being a possible alternative. HD01, says in this regard:

I’m not saying that one type of concept is primary or more legitimate than the other. But if you are talking about a health concept for the use in scientific research, then I would argue for a concept that is more clearly defined. If you’re talking about how people experience things or use, for example laymen, you could be talking about a simpler concept. And I think those things can coexist just fine.

At the same time, other participants were more hesitant when discussing the possibility of having multiple definitions of health and disease. Concerns were raised that such a situation may lead to problems of communication between institutions, (medical) disciplines, but possibly also between doctor and patient. As defining health and disease was viewed by many to be important to facilitate communication, for some participants it also seemed to be problematic to have a plurality of definitions. Furthermore, some participants would also critically question the endeavor of defining health and disease, questioning the goal of defining concepts itself. In several interviews, defining health and disease is described as a continuous process of reflection and adjustment, rather than a pursuit of finding ultimate answers. One participant, HD02, describes that how we define our concepts always have an effect on practice:

I think that every description is functional, in the sense that it always has an effect. Words aren’t neutral so it’s not- I don’t believe in that correspondence theory of there being something in reality that you just have to put the right term on. A word always does something. And I think that’s what it’s more about, so when use a certain view of health, for example, the absence of diagnosis. Then it is important to see, what effect does that have? Who or what is excluded? Or who benefits from this? Who gets worse from this?

Health and disease concepts in practice

One of the key aims of this study was to explore how health and disease are conceptualized, defined or approached, in actual practice. In particular, we were interested what kind of practical functions health and disease concepts have in various contexts. In our analysis of the interviews, we observed that respondents discuss different types of health and disease concepts, working on different levels and as used for various kinds of purposes. If we look at the different type of functions and contexts the concepts are deployed in, and the levels on which they ‘operate’, an interesting picture emerges. We have categorized them broadly into three types of practical functions: (1) a ‘strategic, political and policy-making function’, (2) an ‘institutional and social function’, and (3), ‘guiding clinical practice and medical research’.

Strategic, political and policy-making function

In the context of strategic development, political debates and higher-order policy-making, definitions of health and disease can stay relatively broad and vague. Their function is not, for example, to give clinicians clear thresholds for line-drawing between the normal and pathological. Rather, their function is to steer public health policy, to change current practice within a healthcare organization, or to facilitate cooperation between organizations and institutions. Within this context, health and disease concepts do not need to have the analytical or explanatory power as may be needed in, for example, medical research or clinical practice. The definitions at stake may be demanding and idealistic, as they are used for questioning and/or changing the current state of affairs. Participant HD09 says in this regard:

If you want to explain to a politician why we are going to deploy all kinds of healthcare resources that are not directly focused physically, somatically, then you have to be able to explain it in clearly defined goals, objectives, and health definitions. And in that sense, it is of course also very important for the WHO to adjust such a definition. Because that changes your entire health policy worldwide. For example, it has an effect on what you use for prevention, but it also has an effect on what you use for treatment.

Embedded in these (inter)national discussions on definitions, goals and policies, we may find related discussions in the context of policy on local or organizational levels. Participant HD03 explains why defining health and disease concepts are considered to be important for organizational strategy and policy-making within healthcare organizations:

In the academic hospitals, we are primarily using a biomedical approach towards disease. At the same time, we have the ambition to expand to preventive medicine and to strive for positive health, public health, global health, that are all approaches of health. However, as an academic hospital you are only specialized in thinking about disease in biomedical terms.’’ … “So that’s the problem. If you make a strategy, what are you going to focus on? And so, what I say is, the wish is to focus on prevention, public health, global health and to look more broadly at health and disease.

Although broad and vague definitions may be used successfully for the purpose of guiding or changing policy, more concrete definitions may be needed in other contexts and for other purposes.

Institutional and social function

Another practical function that participants ascribed to the disease concept, and more concretely, to medical diagnosis, is a ‘gatekeeper function’ for issues regarding assessing eligibility for reimbursement of treatment and other healthcare arrangements. Examples mentioned by participants include debates on the legitimacy of viewing clinical conditions such as myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) and chronic pain disorders as ‘genuine diseases’. What we consider to be diseases may therefore also be viewed to be a social and political agreement, some argue. Participant HD05 explains why ‘disease’ could be viewed as an institutional concept:

Who will be reimbursed for their medical treatment? That is decided on a political level.’’ … ‘‘And you can say that, at some point you have to say that someone has a disease, within the framework of a certain law, because that is how it has been agreed upon. And that is an institutional fact, because that is what has been agreed upon by various authorities.

What our institutions acknowledge as ‘genuine diseases’ does not only have impact within the medical realm, but also plays an important role in societal and personal debates. What we define as disease has also a social function. It creates a situation in which others take care of you as a patient, but it can also excuse responsibility from social tasks and duties, for example. In this regard, HD09 says the following:

And no matter how you look at it, sickness creates privileges. Because if you’re sick, people will bring you breakfast in bed, or not if you’re vomiting. And then you get get-well cards, people send flowers and you get time off. Then you are very pathetic and that comes with all kinds of perks. And I’m not saying that people get sick on purpose because of the perks, but that is an automatic consequence. Because my demented patients don’t get get-well cards and flowers and breakfast in bed at all, they are looked at strangely in the supermarket. And patients with psychiatric disorders, well, let’s say… they are usually not the most popular. And that has to do with the fact that we, I think, as a society have determined that being sick has to do with physical ailments…. There’s a discrepancy there. Physically ill: pathetic, perks. Not visibly ill: poser, difficult, hassle, hassle, hassle. That stings.

Guiding clinical practice and medical research

In a clinical context, health and disease can be approached in different ways depending on the type and level of care that is provided. For example, in emergency situations a medical doctor needs to focus on the direct biological problem, but when the patient is in a recovery phase they may have to ‘switch’ and take psychological and social aspects more into account. When caring for a patient suffering from a chronic condition a medical doctor may want to focus on aspects such as resilience and adaptation, and supporting the patient in what they consider to be meaningful. By going through these levels of care, health and disease may be approached differently. Here, HD06 explains this process of ‘shifting’ between models:

Of course, healthcare is very broad. The trauma surgeon and the emergency room doctor who provide acute care for a trauma patient, they are mainly focused on the biomedical model, their A, B, C, D, E, breathing, blood pressure, circulation, you name it. But then you end up in a rehabilitation process in which the biopsychosocial model is used. And then you come to an occupational doctor and an insurance doctor where I think it is very important to also use that model of Positive Health. Because there- Well, we see that with trauma patients too. In our research, independent of the seriousness of the injury, impediments to the ability to function were actually caused by all sorts of personal factors. So, you have to support people in finding their own direction and adaptability.

While taking account for ‘personal’ factors such as adaptability (or resilience) and societal participation may be of relevance for the treatment and revalidation of patients, and thus could be considered as being part of ‘health’, in context of medical research such factors are usually separated from health and disease outcomes and viewed as determinants instead. This allows researchers to measure causal relations between factors such as societal participation and health in a better way. Taking all kinds of (intra)personal and societal factors as being part of the health concept may result in problems for causal explanations in scientific research. Participant HD01 says the following regarding this tension:

The moment you use a broad concept of health, in which all these things are lumped together, you risk that the causality is not actually clear. So, in that sense, I’d like to stick to defining health as biomedical and mental functioning. And I would like to keep those other factors in their own place. And then you can look much better at, what causes what? Or how are things connected?

Problematic situations in practice

A second key aim of this study was to ask participants if they did experience problematic situations in practice that are caused by or related to conceptual issues. In the interviews, a large variety of problematic situations were discussed, including various clinical, epistemological, and ethical issues. Some participants described more abstract problems such as ‘medicalization’ or ‘healthism’ in a broad sense, while others described more concrete issues, such as social or bureaucratic problems in case of patients with medically unexplained symptoms (MUS). Because of the diversity of participants included in our study (i.e., people working in different fields and organizations), the answers to our questions were also diverse and related to their particular context. We have clustered the problematic situations which were brought up in the interviews into 5 types:

1) Illness without identifiable pathology

2) Biomedical versus holistic approaches

3) Line-drawing and threshold problems

4) Problems with translational medicine: from research to the clinic

5) Communication problems

Illness without identifiable pathology

One issue that was discussed in several interviews is the problem of patients suffering from illness without identifiable pathology (or, ‘disease’). This includes patients suffering from ME/CFS, functional neurological disorders, chronic pain disorders, and other conditions that are often described under the umbrella term ‘medically unexplained symptoms’ (MUS). As illness is often viewed to be secondary to disease, and as it is commonplace to think that in order to overcome the illness, one has to cure the underlying disease, it seems only logical to search for the causing pathology. However, in many cases this search does not lead to a clearcut answer. As a result of this, unfortunately, the suffering of the patient is sometimes not taken seriously by medical professionals.

Besides being taken serious by medical professionals and getting the care they need, patients suffering from illness without known pathology may also encounter other type of problems. For example, for patients who cannot work due to illness a medical diagnosis is a necessary criterium to be met for being excused from work and to gain access to certain social and financial resources Footnote 2 . HD07 explains the institutional aspect of medical diagnosis:

Well, in this sense, we are dealing with legal frameworks. The law prescribes that to be able to claim a sickness benefit, one must be diagnosed with a disease. If it stops there, then we do not need to test those other two criteria. And sometimes you will find yourself in a gray area. Because yes, for example, I am also thinking about an example that I have. Social problems can also often lead to dysfunction. In the case of a social problem, there is not by definition disease, but can become one. And we often have to deal with those kinds of dilemmas, that if you see someone with informal care, with a financial problem, just to name a few- Those people who are walking on eggshells at a given moment when they come to us. We establish that, legally, there is no disease. But it might turn into disease.

In line with the situation sketched by HD07, HD15 argues that this problem of not getting recognized by our institutions as having a genuine disease, is a terrible experience for patients. HD15 explains that this in matter of fact urges their organization, a patient organization, to ‘medicalize’ the condition:

Then it will get very bad for them. Because people have a disease on the one hand, on the other hand, they always have to prove that they have it, and then there is also a financial need. So, that’s really the crux of the story. And, of course, we try with our work to make it clear as much as possible, that it is a progressive, biological condition, biomedical condition and that just needs research.

On the other hand, negative aspects of medicalization were also mentioned throughout the interviews. Participant HD14 mentions that including a condition in the ICD should be done with precaution:

The bottom line is that I’m a huge proponent of including pain in the ICD-11, the way as it is now. But I also see that there, I also see that in that balance of those arguments, there are, well, let’s just call it dangers. And that is that you do indeed have things that are normal part of life, which we are going to call disease. And that medical procedures are set up by people, who say, ‘hey, come to me, because I can solve it’. And that is, we have to be very careful about that, in communication, on the one hand to recognize that pain that is there, et cetera, and to take it seriously and with all the benefits that entails. But at the same time to ensure that we do not make it too medical where it is not desired.

In the interviews, many participants argue that, in clinical practice, the illness-experience of the patient is most important and deserves recognition. HD08 argues:

I think a disease is largely about the experience of the patient. And again, of course there is a biological construct underneath, but not always, eh. There are also people with a disease without a biological construct. And just to say, those people are not sick, I think that is far too short-sighted.’’ …. ‘‘We relatively often see people with a functional disorder, something that used to be called conversion or functional neurological symptoms. Those people can suffer a lot from this, but there is no biologically identifiable cause. And I think you shouldn’t dismiss those people as posers or say, you have nothing. No, they do have something and they do suffer from it and that leads to hindrance in daily life. So, I think you can speak of disease.

Biomedical versus holistic approaches

A broader issue that came up in many of the interviews is one that may be best described as problems that are due to biomedical versus holistic approaches towards health and disease. Participants discussed that focusing treatment primarily on a biomedical parameter while paying less attention to the experience of the patient as a whole can be problematic for providing good clinical care. That is, patients may be treated for their medical condition without taking sufficient account of their personal circumstances and/or life goals. Participant HD11 said in this regard:

Of course, you can approach disease in many different ways. If you approach it cell-chemically, so to speak, disease is what damages, or attacks, or if you will, the biochemical integrity of your cell. But if you look from a patient’s perspective, or from a doctor’s perspective, then a disease is something that hurts, bothers, hinders that patient. And the perspective of the patient, but also the approach of society, of course, plays a very important role in this.

In some cases, the emphasis on the biomedical paradigm may even lead to instances of ‘treating’ biomarkers that may not have a clear clinical significance. HD11, discussing the implications of the new drug (aducanumab) for Alzheimer’s Disease, explains that:

The bottom line is, there is a new drug that, if you look at the cellular level, biochemical level, it absolutely does something. It does something to the proteins in your brain, period. However, if you look at the clinical effect on the patient, and what it can do for the patient, it does nothing. Patients don’t improve, we have no improvement, cognition does not improve, general daily activities neither, nothing. The FDA has approved it on the grounds that, despite the fact that it doesn’t do anything clinically, biochemically the evidence is so clear that it does something, it’s bound to do something clinically. While it just doesn’t.

Yet, also in cases where a biomedical treatment has proven to be clinically effective, it could be nevertheless problematic to forget about the patient’s personal circumstances. Sometimes it may be more important to help people with psychosocial issues, for example, than to direct attention to the medical problem. Participant HD10 discusses person-centered care for diabetes patients and argues that taking care of the patient - improving their health - includes more than treating the disease biomedically:

That also touches on the need for person-centered care, - that the care providers really can actually see from the patient’s eyes which approach they should take. Do they really have to focus on that disorder or do they indeed have to focus on the social realm?

Another related problem that was mentioned in the interviews is that of prioritizing biomedical diagnosis over other holistic aspects when assessing the prognosis. Although the diagnosis may give important information regarding the development of a disease and about chances for successful treatment, other non-medical factors may have an underestimated influence on the prognosis as well. In some instances, psychosocial aspects may even show a stronger correlation with prognosis and treatment than the medical diagnosis does, participant HD04 says in this regard:

The classic assumption is very much like, if you know a diagnosis, then you know the prognosis and then you know whether or not you need to do something to influence that prognosis. Whether or not you can do something to influence that prognosis. And what we are gradually noticing is that that prognosis may well be determined by many other factors and that the diagnosis is only a small part of it and therefore only partly determines what the prognosis is. The prognosis is also determined by all kinds of other factors. other variables, to put it in scientific terms.

According to HD04, it is common for medical professionals to focus too much on biomedical diagnosis and to underestimate the influence of ‘non-medical’ variables on the prognosis and the well-being of patients – which, she beliefs, should be the ultimate aim. This does not only go for patients with medical unexplained symptoms, for which finding the right diagnosis is considered to be very difficult. Also for diseases that can be diagnosed straightforwardly there seems to exist a disparity between a biomedical view of disease and more holistic ones. HD04 gives the following example:

Examples abound. People with rheumatoid arthritis, we can diagnose rheumatoid arthritis fairly well with lab tests, with clinical tests, with imaging tests. We have criteria, you can always argue about that, but we generally agree on that. And then we also have a measure of the disease activity. So, if you have a very high sedimentation rate, then you have a high disease activity, for example. And if you then look at the severity of the complaints and the disability that people have and relate that to disease activity, then that is not a nice linear relationship. So, then there are people with, if you would look at it as a rheumatologist, as a doctor, if you look at it as a doctor, then well, that disease is just well under control, hardly swollen joints, no increased sedimentation rate, goes well, but in fact people suffer very much.

Line-drawing and treatment threshold problems

In the interviews, problems with drawing the lines between states of health, disease, or ‘being at-risk’, and problems with determining the right thresholds for starting medical interventions, were considered important reasons for having clear definitions. Having clear cut-offs for diagnosing disease and for starting treatment is seen as convenient for clinical practice. Participants expressed a desire to have objective measures to decide whether we are talking about disease, and when to start treatment. Yet, they were also highly doubtful if such clear lines could be drawn. On the one hand, they said diagnostic tests are used to examine if a patient deviates from the (objective, biomedical) norm. On the other hand, participants also argued that patients’ symptoms should be viewed as central to drawing the line. This also seems to be problematic, however, as patients may sometimes deviate from the norm but do not experience symptoms, or vice versa, patients may experience symptoms but test results do not show significant abnormalities. HD08 talks about the challenges of the line-drawing problem for clinical-decision making:

Of course, it is difficult, because doctors like to work neatly, like to work according to scientific evidence, like to work according to guidelines. And a guideline only works well if you can make hard statements, otherwise you have a guideline that says about everything: you ‘may consider this’. And yes, that is how guidelines end relatively often, but then it is not very useful in practice, because you want such a guideline to guide you. And the surgeon, just to name one, who wants to determine whether he should operate. And it’s easy if that just has a cut-off point that says, you have to operate above 23 and not below, just to name something. So, whenever there’s a big gray area, it’s complicated and leads to subjectivity and also different doctors making different decisions.

This was also discussed in relation to prevention, when patients are ‘treated’ with medication to prevent future disease(s) while they do not experience symptoms at that point of time. In particular, participants pointed to the lowering of diagnostic and treatment standards for risk-factors such as high blood pressure and high cholesterol as examples in which it is difficult to draw the line. Participant HD09, who reflect on this problem, says the following:

But you can get quit some muscle cramps from cholesterol lowering drugs. Yes, so it may be that he has one in twenty, one in thirty less chance of that stroke, but in the meantime, he is no longer able to walk down the stairs and do his own shopping because of those muscle complaints and perhaps even take a fall. Well, and it’s not the case that everyone has muscle problems, so for the people who don’t get this it might be the best treatment. That is the way you have to look at it. And also evaluate, eh, and that’s about when you start something, you have to follow up what it does to someone, even if someone has been using it for some time, because that can change.

When participants were asked if they could identify reasons for this trend of lowering diagnostic and treatment thresholds, some suggest that cultural values and norms play an important role. Not only there is an increasing societal pressure on living a healthy life, health is also increasingly viewed as a moral good. It is this normative shift, in combination with ever growing technological possibilities, that is suggested to lead medicine to focusing on early detection and treatment of health risks more and more – even if chances of developing actual diseases are expected to be low. Patients may desire more diagnostic testing or more frequent health check-ups and medical professionals may feel obliged to grant their requests, since the technology is available. This is not without consequences, however. HD11, for example, explains that excessive diagnostic testing may lead to overdiagnosis. In particular, ‘incidental findings’ Footnote 3 are considered to be a problematic situation:

And that is, I think, also an ethical dilemma that doctors have, because then you find something and what do I do now? They have no complaints at the moment, so I don’t really have to do something with it now. But imagine that it is cancer, and in four months they will come in with metastatic disease, and then I could have prevented that. That’s difficult. And then the technology renders it unlikely that such a patient says, never mind, we’ll see how things will go. Because everyone says oh, yes, if something can be done about it, then let’s do that scan, then do that biopsy, then do that incredibly complicated procedure.

Incidental findings may be clear instances of pathology, and in these cases, it may be regarded as fortunate that the patient can be treated for a disease that may otherwise have gone undetected until it was too late. However, in other cases incidental findings may be benign deviations or anomalies and it is questionable if the patient will benefit from further diagnostic testing and/or medical intervening, as it is not clear if the anomaly will ever lead to clinical symptoms. Again, this begs the question where to draw the line between normal and abnormal, between health and disease.

Problems with translational medicine: from research to the clinic, and beyond

In the interviews, some participants also discussed problems regarding translating medical scientific findings from a research context into clinical practice. One approach that was mentioned by participants as particularly problematic was ‘evidence-based medicine’ (EMB) Footnote 4 . While medical professionals may be aware of the different aims and goals of medical research versus clinical medicine, and of the problems surrounding EBM, they may feel bounded by institutional agreements and regulations. For example, insurers may only reimburse treatments that are proven to be effective according to standards of EBM and therefore may not sufficiently allow for tailoring treatment to the personal needs of the patient. HD09 explains how the broad implementation of the EBM style of reasoning, from research to the clinic and beyond, to institutional arrangements, is not without danger:

Evidence-based medicine, with its mono-focus thinking, traditionally, it’s fortunately changing, can also bring real dangers, because what you see is that politics and insurers are very much steering policy and reimbursing on the basis of guideline indicators.

HD13 goes even a step further by provocatively referring to EBM as ‘pharmaceutical-based medicine’. He argues that medical professionals are restricted by the rules and regulations of the healthcare institutions such as the National Healthcare Institute (‘Zorginstituut’), which require treatments to be ‘evidence-based’ before they can be considered eligible for reimbursement. As a result, HD13 claims, we end up with suboptimal medical treatments:

The entire ‘pharmaceutical-based medicine’ is currently ‘the’ steering element of the National Healthcare Institute and of affordable care in the Netherlands, of reimbursed care. And it’s not the best treatment that gets reimbursed, but the treatment that has been the most researched; not the one with the best outcomes.

Another problem that was particularly mentioned in the interviews was that of generalizing medical knowledge from the research context to the clinical context. As diseases and their treatments are commonly researched in study populations that do not represent patient populations in clinical practice - e.g., age range between 18 and 50, mostly Western, male subjects, having only one disease instead of several - a rather homogenous picture of specific disease entities with specific treatments is generated that often does not match the heterogeneous reality in clinical practice. Moreover, while medical research is often focused on curing a disease, or at least reducing its symptoms, patients may in fact have different goals and wishes that need a different approach. Participant HD09 argues that the goals of medical research do not always match the goals of clinical medicine:

So, the average patient in a trial is a middle-aged man. The average user, who is treated according to the guideline based on those trials, is an old woman or one who has more medical conditions and uses several medications. And then it is also the case that those trials are aimed at preventing a new event or surviving. And, for example, not having a second heart attack, not having a stroke. Well, those may be things that are important to someone, but I just said that is often not the most important thing. Those people are not all at about living longer, they care about function preservation. And then it can still be important to prevent that stroke, but then you really have to look at it in a different way.

Especially in case of (chronic) multimorbidity, in which patients suffer from multiple diseases at the same time and also use multiple medications, it can become questionable what is treated, exactly. A set of separate diseases, or the combined physiological effects and symptoms of a multitude of underlying pathologies, or even of the medications used? As a consequence, ‘evidence-based’ treatment protocols could potentially harm patient populations that do not fit the assumptions on which the treatment is found to be efficacious. Furthermore, diseases and also the medications that are used may interact, resulting in a clinical picture that is very different from what is expected. We might describe this situation as one that is epistemologically opaque : it seems to get very difficult, if not impossible, to distinguish cause and effect. HD09 explains:

And then the question is whether it will work the same way with that woman with all those old age conditions compared to what happened with that fifty-year-old man. So, it probably reacts differently as well. It reacts differently, because there are multiple diseases, interaction with disease. And it reacts differently because there are a whole lot more medications, interacting with medication. And it reacts differently because the body is different.

So, while medical research tries to reduce complexity and look into single homogenous diseases and patient groups, in clinical practice disease often manifests very differently.

Communication problems

While participants were generally doubtful about arriving at univocal and all-encompassing definitions of health and disease and favored the idea of conceptual pluralism, some participants also expressed concerns with regard to communication. If we all use different definitions or different health and disease concepts, how do we know we are still speaking of the same thing? As clear-cut definitions are often desired precisely for the purpose of solving ongoing problematic situations in medicine, it may seem paradoxical to accept conceptual pluralism. In practice, having multiple ways to understand a disease can lead to communication problems, participants fear. For example, when medical specialists’ views differ so significantly that they almost literally speak about different diseases, it is questionable if they are still able to sufficiently communicate with each other and their patients.

In an interview with HD08, opposing views on Alzheimer’s Disease among medical specialists were discussed. Alzheimer’s Disease was originally diagnosed on the basis of clinical signs and symptoms, but in recent years a part of the neurologist community has switched to prioritizing biomarker testing (i.e., primarily the presence of beta-amyloid) over clinical presentation. However, the problem is that the group of patients with positive biomarker tests do not completely match the group of patients who get symptoms. Therefore, changing the way of diagnosing Alzheimer’s disease in patients also seem to imply changing the definition of Alzheimer’s disease. Hence, it becomes unclear if medical specialists are still discussing the ‘same’ disease. HD08 says the following about the opposing views:

Well, I think there’s- You could almost say, it’s kind of a clash of civilizations. You have the people who just want a hardcore biological substrate and then have little regard for other aspects. And you have people who say yes, maybe it is not possible to classify it exactly into careful categories, let’s also take into account the less ‘hard’, less definable aspects that are important for the functioning of a patient.

While acknowledging the challenges and pitfalls that come with speaking different ‘medical languages’, at the same time, participants also see benefits of having different approaches towards health and disease. Some of them note that we already are using different languages, scientific explanations and medical classifications, and that this could be viewed as something valuable. In a combined interview with HD13 and HD14, HD14 discusses the different classification systems that are being used for chronic pain patients among different (para)medical professionals:

No, I think you should cherish that, because an anesthesiologist can do things that a rehabilitation doctor cannot do, and vice versa. So, you really have to use each other for that and that also applies to all those other medical specialists and paramedical specialists. So that in itself is not a big deal. What- Or rather, that’s very functional, that’s excellent. At the same time, we must speak each other’s language and that must be the same language with each other, but we must certainly not forget the patient. And, because the patient must also be at the center of our interprofessional communication. And, but also the wishes and needs of the patient. So, if HD13 says ‘I’m good at ICD’, and I’m good at ICF, to put it very bluntly, that’s not going to work. I need to know about ICD, enough to talk to HD13. And HD13 needs to know about ICF, enough to talk with me. But really, we should all be able to know enough to be able to talk to the patient properly.

Thus, interestingly, the suffering of one patient could be classified in several different ways, depending on the classification system that is used. While recognizing the challenges this brings for medical professionals, HD13 and HD14 also see the benefits of looking through different lenses – as long there is sufficient common ground to communicate with each other and the patient. So, concepts of health and disease seems to be approached differently at different levels of care (i.e., primary, secondary, and tertiary lines of healthcare) and between different types of (para)medical professionals. The situation as sketched by HD13 and HD14 seems evident for healthcare as arranged in The Netherlands, where various classification systems are indeed being used in different levels and types of healthcare practices Footnote 5 . Every classification system has its strengths and weaknesses. An ongoing challenge seems to lie in being able to sufficiently understand each other’s ‘medical language’.

Philosophers can contribute to medicine by exploring, analyzing and articulating conceptual issues. However, as we take health and disease concepts to be context-dependent, it is crucial to study their meaning in context. Building on recent proposals for a pragmatist understanding of health and disease that embraces conceptual pluralism, investigating different perspectives is very important. As Veit argues: “Questions such as how medical practitioners see, use, and evaluate concepts like health, pathology, and disease are important to the philosophy of medicine. Yet, these questions cannot be answered through introspection alone. They require investigative empirical methods” [ 21 ] (p.183). In similar vein, Seidlein & Salloch [ 17 ] argue that empirical methods can be used to gain better understanding of the complex relationship between illness and disease, by reflecting upon patient and professional perspectives. Including qualitative methods and other types of empirical research to our toolbox can bring theory and practice closer together and stimulate new medical-philosophical and bioethical explorations.

The current study differs from previous empirical studies [ 24 , 25 , 26 , 27 ], in that it was specifically focused on exploring how health and disease concepts have a function in practice and how they may lead to problematic situations . The existing studies have already shown a palette of different conceptualizations, but did not interpret these in terms of their practical function and role in problematic situations. In our interviews, various important practical functions of health and disease concepts were discussed and our participants suggest that different contexts and purposes may require different types of definitions. We agree with Veit that finding such a lack of consensus and a pluralism of concepts and functions, strengthens the case against conceptual monism, and favors positions that “relativise the concept to human interests and cultural dynamics’’ [ 21 ] (p.178). Indeed, our study reveals that “the notion [of disease] serves a variety of purposes that perhaps cannot be accomplished using a single concept” [ 21 ] (p.180).

However, the plurality of functions and the definitions that are used to serve them, may not always be compatible with each other. A broad concept definition of health may work, for example, to steer healthcare policy in a certain direction on a political or organizational level, but may cause problems when it must be implemented in a research setting. Of course, different functions and definitions do not exist in a vacuum but also interact. Moreover, as is evident from the interviews, although the plurality of definitions may sometimes be problematic for reasons of communication, it is also a reality. Therefore, it may be more fruitful to acknowledge this and to elucidate and explain the differences; this may actually enhance communication and understanding across domains.

In this article, we have highlighted 5 types of problematic situations that were discussed in the interviews and that can be related to the conceptualization of health and disease. Some problems are already recognized in the medical-philosophical literature, such as problem of line-drawing. Others may offer new starting points for medical-philosophical and bioethical inquiry. Philosophy of medicine might help to analyze and elucidate the conceptual components of these problems and come up with suggestions of how conceptual work might help to find solutions. For example, the work that has already been done by Rogers and Walker [ 12 , 31 ] regarding the line-drawing problem might be useful for medical practitioners and medical guideline developers. They propose using context specific précising definitions that serve to prevent overdiagnosis; such an approach may also be useful to help solve the line-drawing and treatment threshold problems, and the risks of over or undertreatment, that we encountered in this study.

Furthermore, tensions between biomedical and holistic approaches of health and disease – that have led to major debates in the philosophy of medicine and has important ethical implications – were also described by participants as problematic in practice. However, there was also a hint of a solution in the interviews. As one participant explained, different contexts may benefit from different approaches. Strictly biomedical definitions may be more useful for the emergency care doctor while during rehabilitation a holistic normative biopsychosocial model is considered more helpful. Footnote 6 This idea is in line with the proposal by Haverkamp et al. [ 7 ], to consider using concepts that fit best with the purposes and values of a specific healthcare practice. Some of the problematic situations described in the interviews may also give new input for investigating these purposes and values in different contexts. For example, the changing conceptualization of Alzheimer’s disease and the use of biomarker diagnostic testing, that was mentioned in the interviews, is a current topic of medical-philosophical and bioethical debate (e.g., see [ 43 , 44 , 45 ]).

Another role for philosophy can be to help healthcare professionals and policy makers to better understand how some of their problematic situations arise. For example, some of the issues we identified could be understood in terms of a disconnect between the three spheres of the conceptual triad of ‘disease, illness and sickness’, as originally presented by Twaddle [ 46 ] and as later updated by Hofmann [ 47 ]. As Hofmann already noted, cases of non-health are generally considered to be less controversial when two or three of the spheres align. However, when only one or two of the are deemed applicable to a certain condition, it becomes epistemically and normatively challenging [ 47 ]. This conceptual triad may help patients, healthcare professionals and policymakers to better understand issues around the problem of medically unexplained symptoms, also in relation to the institutional and social function of the disease concept. At this point, it may also be significant to note that in the Dutch language, in which the interviews were conducted, the distinction between disease, illness and sickness is not available. A single word, ‘ziek’ or ziekte’, is used to cover all three notions, making the conceptual confusion perhaps even more salient than in the English-speaking community.

Some of the problematic situations that we have described may, at first glance, be viewed as practical problems with only little conceptual basis. For example, when discussing disease as an institutional and social concept, and describing problems that patients who suffer from medically unexplained symptoms may face (e.g., problems with accessing certain healthcare resources, or social and financial arrangements), one might question to what extent this is a problem with the conceptualization of disease.

One might argue, as Hesslow [ 48 ] did, that we have been misled by the idea that we need a concept of disease to make normative decisions on clinical, moral or socially important issues. However, from a pragmatist perspective, the theoretical, practical and normative dimensions of concepts are inherently related. As De Vreese argues: ‘‘it seems undeniable that the health/disease distinctions made on the basis of tacit understandings of the disease notion do play an important role in the background of health care-related research and decision-making processes (clinical, moral, legal, social, or otherwise), which might have important consequences in practice’’ [ 6 ] (p.429). Starting from this observation, we might consider adapting our concepts to better fit the social and institutional arrangements (cf. [ 49 , 50 ]). or we might propose better concepts or criteria to base these decisions on (e.g., see [ 51 ]). Both seem to be pre-eminently tasks for philosophers and ethicists to pursue. Additionally, empirical studies may help to further explore these ‘tacit understandings of the disease notion’ and investigate what these ‘important consequences in practice’ entail, as starting points for further philosophical and ethical reflection.

Limitations

As is common for qualitative research, results cannot be generalized and results may not represent the views, attitudes and beliefs of the whole community of medical professionals or patient organizations. As the sample of this study is relatively small and consisted of a broad range of professionals, the findings should be viewed as starting points for further investigation, not definitive answers. Moreover, as indicated in the methods section, the sample consisted of a group of highly educated and experienced professionals. Although there were good reasons to select them, it is important to remark that as a consequence, we did not study the views and experiences of other, more ‘ordinary’ healthcare workers and patients. Also, we did not include the views of different nationalities, cultures, and/or for example less educated or marginalized people. Indeed, we should ask: ‘who are the rightful owners of the concepts disease, illness and sickness’ [ 9 ]? If we view health and disease as plural concepts then an empirical philosophy of medicine should do justice to this plurality by including the views and experiences of these groups as well. Future studies may focus on investigating more specific groups (e.g., a specific medical specialist field or certain group of patients) and/or institutional contexts.

Furthermore, as we have learned from discussions on the empirical turn in medical ethics [ 52 ], one should be careful and considerate when making normative claims on basis of empirical data. However, given the explorative character of this study, this is not deemed a significant problem. Our aim was to explore the range of views regarding health and disease concepts, and the existence of problematic situations related to health and disease concepts, not to give an exhaustive or quantitative overview of such concepts and situations. Furthermore, in qualitative research, it is generally acknowledged that the researcher is not merely a ‘neutral observer’ but also an actor who actively engages with participants in the research process, and thus, is part of the data that is generated [ 53 ]. In this study in particular, with its aim of exploring how health and disease concepts function in practice and examining whether they could lead to problems, the interview guide was drafted from a specific theoretical angle. Moreover, the interviews were analyzed with existing theoretical discussions and frameworks in the back of our minds. By being open and reflexive about this process, and by making our interpretations as transparent as possible, we hope to have gained sufficient rigor.

The traditional debate on health and disease concepts commonly departs from theory rather than from practice. In line with recent calls for experimental philosophy of medicine and empirical philosophy of science, we suggest that theoretical work could benefit from incorporating empirical research. In this qualitative interview study, we have examined the relevance and significance of health and disease concepts, as experienced by participants in various healthcare practices. We found that there are three types of functions that health and disease concepts serve in practice: (1) ‘Strategic development, politics and policy-making’, (2) ‘Institutional and social function’, and (3), ‘Guiding clinical practice and medical research’. Being aware of these different purposes may prevent bluntly using concepts beyond their functional scope. We also explored what kind of difficulties participants experienced in relation to the conceptualization of health and disease in practice, and found five main types of problematic situations: (1) Illness without identifiable pathology, (2) Biomedical versus holistic approaches, (3) Line-drawing and treatment threshold problems, (4) Problems with translational medicine: from research to the clinic, and beyond, and (5), Communication problems.

This study demonstrates how concepts of health and disease can influence different aspects of healthcare and healthcare-related practices and may sometimes contribute to complex problematic situations. By analyzing these influences, by making underlying implicit assumptions explicit, giving further interpretation to the problems observed in practice, providing theoretical frameworks and conceptual tools, and by suggesting conceptual changes or adaptations, we might be able to help solve some of these problems. To do this in a proper way, we need both theoretical and empirical work. If we want our philosophical definitions to be a part of the solution for real-world problems, it is important to consider the intuitions and ideas of people working in different types of medical fields, patients, researchers, and all other stakeholders [ 20 ]. Paraphrasing Immanuel Kant, we may conclude that philosophy of medicine without empirical research risks being empty, while empirical research without philosophical theorizing will still leave us blind. Going back and forth between theory and practice will probably result in a more complex but hopefully also in a better and more fruitful understanding of concepts of health and disease.

Data availability

The data that support the findings of this study are available from the Erasmus Medical Center but GDPR restrictions apply to the availability of these data and are therefore not publicly available.

The notion of experimental philosophy is relatively new and its definition is therefore not yet solidified. Sometimes it is used broadly, including various kinds of empirical research methods. In other instances, it refers specifically to philosophical studies with an experimental design, in which one variable is changed in isolation to measure changes in a philosophically relevant outcome (e.g., moral judgement). We believe that the latter, more narrow definition is useful to distinguish between experimental and other empirical studies. Therefore, in the title of our study, we explicitly use the term empirical philosophy instead of experimental philosophy.

This is, at least, how things are arranged in the Netherlands. Similar arrangements are in place in many other countries worldwide.

Incidental findings are anomalies that are detected in clinical tests that were in fact aimed on testing something else. As the clinical significance of these findings is often not clear, clinicians and/or clinical researchers are confronted with ethical dilemmas [ 37 , 38 , 39 ].

EBM can be described as an approach towards medicine that takes scientific evidence as a central point for guiding clinical decision-making. Typically, in EBM meta-analyses and randomized clinical trials (RCTs) are considered to be the highest forms of scientific evidence. While these methods can indeed have strong benefits over other types of medical research, there is ample discussion about its down sides as well [ 40 , 41 , 42 ].

For instance, general physicians, who provide primary care, use a different classification system (International Classification of Primary Care; ICPC) than a medical specialist in a hospital (International Statistical Classification of Diseases and Related Health Problems; ICD), who provides secondary and tertiary care, uses. Physiotherapists (International Classification of Functioning, Disability and Health; ICF), and psychologists (Diagnostic and Statistical Manual of Mental Disorders; DSM), in turn, also use different types of classification systems.

Another way to frame this would be to say that in emergency care, only ‘disease’ may be relevant to provide proper medical care, whereas in a rehabilitation setting the whole triad of disease, illness and sickness is being addressed.

Hofmann B. Managing the moral expansion of medicine. BMC Med Ethics. 2022;23(97). https://doi.org/10.1186/s12910-022-00836-2 .

Hofmann B. Does disease incite a stronger moral appeal than health? BMC Med. 2023;21(419). https://doi.org/10.1186/s12916-023-03110-3 .

Lemoine M. Defining disease beyond conceptual analysis: an analysis of conceptual analysis in philosophy of medicine. Theor Med Bioeth. 2013;34:309–25. https://doi.org/10.1007/s11017-013-9261-5 .

Article   Google Scholar  

Schwartz PH. Decision and discovery in defining ‘disease’. In: Kincaid H, McKitrick J, editors. Establishing medical reality. Dordrecht: Springer; 2007. pp. 47–63.

Chapter   Google Scholar  

Van der Linden R, Schermer M. Health and disease as practical concepts: exploring function in context-specific definitions. Med Health Care Philos. 2022;25:131–40. https://doi.org/10.1007/s11019-021-10058-9 .

De Vreese L. How to proceed in the disease concept debate? A pragmatic approach. J Med Philos. 2017;42(4):424–46. https://doi.org/10.1093/jmp/jhx011 .

Haverkamp B, Bovenkerk B, Verweij MA, Practice-Oriented. Review of Health concepts. J Med Philos. 2018;43(4):381–401. https://doi.org/10.1093/jmp/jhy011 .

Nordby H. The analytic-synthetic distinction and conceptual analysis of basic health concepts. Med Health Care Philos. 2006;9(2):169–80. https://doi.org/10.1007/s11019-006-0002-7 .

Nordby H. Who are the rightful owners of the concepts disease, illness and sickness? A pluralistic analysis of basic health concepts. Open J Philos. 2019;9:470–92. https://doi.org/10.4236/ojpp.2019.94029 .

Schwartz PH. Progress in defining disease: improved approaches and increased impact. J Med Philos. 2017;42:485–502. https://doi.org/10.1093/jmp/jhx012 .

Walker MJ, Rogers WA. A new approach to defining disease. J Med Philos. 2018;43(4):402–20. https://doi.org/10.1093/jmp/jhy014 .

Rogers WA, Walker MJ. Précising definitions as a way to combat overdiagnosis. J Eval Clin Pract. 2018;24:1019–25. https://doi.org/10.1111/jep.12909 .

Schulz S, Harzheim L, Hübner C, Lorke M, Jünger S, Woopen C. Patient-centered empirical research on ethically relevant psychosocial and cultural aspects of cochlear, glaucoma and cardiovascular implants – a scoping review. BMC Med Ethics. 2023;24(68). https://doi.org/10.1186/s12910-023-00945-6 .

Sanchini V, Sala R, Gastmans C. The concept of vulnerability in aged care: a systematic review of argument-based ethics literature. BMC Med Ethics. 2022;23(84). https://doi.org/10.1186/s12910-022-00819-3 .

Schleidgen S, Klingler C, Bertram T, Rogowski WH, Marckmann G. What is personalized medicine: sharpening a vague term based on a systematic literature review. BMC Med Ethics. 2013;14(55). https://doi.org/10.1186/1472-6939-14-55 .

Erdmann A, Rehmann-Sutter C, Bozzaro C. Patients’ and professionals’ views related to ethical issues in precision medicine: a mixed research synthesis. BMC Med Ethics. 2021;22:116. https://doi.org/10.1186/s12910-021-00682-8 .

Seidlein AH, Salloch S. Illness and disease: an empirical-ethical viewpoint. BMC Med Ethics. 2019;20(5). https://doi.org/10.1186/s12910-018-0341-y .

Kon AA. The role of empirical research in bioethics. Am J Bioeth. 2009;9:59–65. https://doi.org/10.1080/15265160902874320 .

Verberne WR, Stiggelbout AM, Bos WJW, van Delden JJM. Asking the right questions: towards a person-centered conception of shared decision-making regarding treatment of advanced chronic kidney disease in older patients. BMC Med Ethics. 2022;23(47). https://doi.org/10.1186/s12910-022-00784-x .

De Block A, Hens K. A plea for an experimental philosophy of medicine. Theor Med Bioeth. 2021;42:81–9. https://doi.org/10.1007/s11017-021-09551-2 .

Veit W. Experimental philosophy of medicine and the concepts of health and disease. Theor Med Bioeth. 2021;42:169–86. https://doi.org/10.1007/s11017-021-09550-3 .

Ankeny R, Chang H, Boumans M, Boon M. Introduction: philosophy of science in practice. Eur J Philos Sci. 2011;1:303. https://doi.org/10.1007/s13194-011-0036-4 .

Kampourakis K. Empirical philosophy of Science. Sci Educ. 2019;28:1–3. https://doi.org/10.1007/s11191-019-00035-3 .

Hofmann B. Do health professionals have a prototype concept of disease? The answer is no. Philos Ethics Humanit Med. 2017;12(6):1–7. https://doi.org/10.1186/s13010-017-0047-7 .

Stronks K, Hoeymans N, Haverkamp B, den Hertog FRJ, van Bon-Martens MJH, Galenkamp H, Verweij M, van Oers HAM. Do conceptualisations of health differ across social strata? A concept mapping study among lay people. BMJ Open. 2018;8:e020210. https://doi.org/10.1136/bmjopen-2017-020210 .

Kohne ACJ, De Graauw LP, Leenhouts-van der Maas R, van Os J. Clinician and patient perspectives on the ontology of mental disorder: a qualitative study. Front Psychiatry. 2023;14:1081925. https://doi.org/10.3389/fpsyt.2023.1081925 .

Van Heteren F, Raaphorst N, Groeneveld S, Bussemaker J. Professionals’ health conceptions of clients with psychosocial problems: an analysis based on an empirical exploration of semi-structured interviews. Int J Nurs Stud Adv. 2023;5:100120. https://doi.org/10.1016/j.ijnsa.2023.100120 .

Wright HG. John Dewey’s view of situations, problems, means and ends. In: Wright HG, editor. Means, ends and Medical Care. Dordrecht: Springer; 2007. p. 92. https://doi.org/10.1007/978-1-4020-5292-7_4 .

Corbin J, Strauss A. Basics of qualitative research: techniques and procedures for developing grounded theory. California: Sage; 2008.

Book   Google Scholar  

Hsieh H, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15:1277–88. https://doi.org/10.1177/1049732305276687 .

Rogers WA, Walker MJ. The line-drawing problem in disease definition. J Med Philos. 2017;42(4):405–23. https://doi.org/10.1093/jmp/jhx010 .

Engel GL. The need for a new medical model: a challenge for biomedicine. Science. 1977;196(4286):129–36. https://doi.org/10.1126/science.847460 .

Bolton D. Looking forward to a decade of the biopsychosocial model. BJPsych Bull. 2022;46(4):228–32. https://doi.org/10.1192/bjb.2022.34 .

Bolton D, Gillett G. The Biopsychosocial Model of Health and Disease: New Philosophical and Scientific developments. London: Palgrave Pivot Cham; 2019. https://doi.org/10.1007/978-3-030-11899-0_1 .

Huber M, Knottnerus JA, Green L, van der Horst H, Jadad AR, Kromhout D, Leonard B, et al. How should we Define Health? BMJ. 2011;343(7817):235–37. https://doi.org/10.1136/bmj.d4163 .

Huber M, van Vliet M, Giezenberg M, Winkens B, Heerkens Y, Dagnelie PC, Knottnerus JA. Towards a ‘patient-centred’ operationalisation of the new dynamic concept of health: a mixed methods study. BMJ Open. 2016;6(1):1–12. https://doi.org/10.1136/bmjopen-2015-010091 .

Bomhof C, van Bodegom L, Vernooij M, Pinxten W, de Beaufort I, Bunnik E. The impact of incidental findings detected during brain imaging on Research participants of the Rotterdam Study: an interview study. Camb Q Healthc Ethics. 2020;29(4):542–56. https://doi.org/10.1017/S0963180120000304 .

Wagner J, Aron DC. Incidentalomas – A ‘‘disease’’ of modern imaging technology. Best Pract Res Clin Endocrinol Metab. 2012;26(1):3–8. https://doi.org/10.1016/j.beem.2011.08.006 .

Wolf SM, Introduction. The challenge of incidental findings. J Law Med Ethics. 2008;26(2):216–8. https://doi.org/10.1111/j.1748-720X.2008.00265.x .

Bolt TC, Huisman F. Disease specificity and evidence-based medicine: a historical perspective. Eur J Person Cent Healthc. 2020;8:308–14. https://doi.org/10.5750/ejpch.v8i3.1795 .

Borck C. Negotiating Epistemic hierarchies in Biomedicine: the rise of evidence-based medicine. In: Epple M, Imhausen A, Müller F, editors. Weak knowledge: forms, functions, and Dynamics. Frankfurt: Campus; 2020. pp. 449–82.

Google Scholar  

Parker M. Whither our art? Clinical wisdom and evidence-based medicine. Med Health Care Philos. 2002;5:273–80. https://doi.org/10.1023/A:1021116516342 .

Alpinar-Sencan Z, Schicktanz S. Addressing ethical challenges of disclosure in dementia prediction: limitations of current guidelines and suggestions to proceed. BMC Med Ethics. 2020;21(33). https://doi.org/10.1186/s12910-020-00476-4 .

Schermer MHN. Preclinical disease or risk factor? Alzheimer’s Disease as a case study of changing conceptualizations of Disease. J Med Philos. 2023;48(4):322–34. https://doi.org/10.1093/jmp/jhad009 .

Tromp K, Smedinga M, Richard E, Perry M, Schermer M. Views on early diagnosis of Alzheimer’s Disease among Dutch Physicians: a qualitative interview study. J Alzheimers Dis. 2021;79(2):917–27.

Twaddle A. ‘Disease, illness and sickness revisited’. In: Twaddle A, Nordenfelt L, editors. Disease, illness and sickness: three Central concepts in the theory of Health. Linköping: Linköping University; 1994. pp. 1–18.

Hofmann B. On the Triad Disease, illness and sickness. J Med Philos. 2002;27(6):651–73. https://doi.org/10.1076/jmep.27.6.651.13793 .

Hesslow G. Do we need a concept of disease? Theor Med. 1993;14:1–14. https://doi.org/10.1007/BF00993984 .

Kukla QR. What counts as disease, and why does it matter? J Philos Disabil. 2022;2:130–56. https://doi.org/10.5840/jpd20226613 .

Kukla R. Infertility, epistemic risk, and disease definitions. Synthese. 2019;196:4409–28. https://doi.org/10.1007/s112 .

Htut Maung H. Is infertility a disease and does it matter? Bioethics. 2019;33:43–53. https://doi.org/10.1111/bioe.12495 .

Davies R, Ives J, Dunn M. A systematic review of empirical bioethics methodologies. BMC Med Ethics. 2015;16(15). https://doi.org/10.1186/s12910-015-0010-3 .

Ives J, Dunn M. Who’s arguing? A call for reflexivity in bioethics. Bioethics. 2010;24(5):256–65. https://doi.org/10.1111/j.1467-8519.2010.01809.x .

Download references

Acknowledgements

The authors would like to thank all the participants of the interview study for their input.

This research is funded by the Dutch Scientific Organization (NWO), Project Number 406.18.FT.002. The funding body played no role in the design of the study and collection, analysis, interpretation of data, and in writing the manuscript.

Author information

Authors and affiliations.

department of Medical Ethics, Philosophy & History of Medicine, Erasmus MC University Medical Center, Rotterdam, The Netherlands

Rik R. van der Linden & Maartje H.N. Schermer

You can also search for this author in PubMed   Google Scholar

Contributions

RL and MS were both involved in conceptualizing and designing the qualitative study, and both wrote the main manuscript. RL conducted the interviews with the participants. Both authors analyzed and interpreted the qualitative data. MS has secured the funding. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Rik R. van der Linden .

Ethics declarations

Ethics approval and consent to participate.

The research proposal was submitted for review by the research ethics review committee of the Erasmus Medical Center. A waiver was granted (MEC-2021-0584), as the study did not fall within the scope of the WMO (Dutch Medical Research Involving Human Subject Act). This study was carried out in accordance with General Data Protection Regulation (GDPR), and the Privacy Regulation of Erasmus Medical Center. Written and oral informed consent was obtained before the start of the interviews.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

van der Linden, R.R., Schermer, M.H. Exploring health and disease concepts in healthcare practice: an empirical philosophy of medicine study. BMC Med Ethics 25 , 38 (2024). https://doi.org/10.1186/s12910-024-01037-9

Download citation

Received : 12 December 2023

Accepted : 13 March 2024

Published : 27 March 2024

DOI : https://doi.org/10.1186/s12910-024-01037-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health and disease concepts
• Empirical philosophy
• Philosophy of medicine

BMC Medical Ethics

ISSN: 1472-6939

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.14(8); 2017 Aug

Assessing the impact of healthcare research: A systematic review of methodological frameworks

Samantha cruz rivera.

Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom

Derek G. Kyte

Olalekan lee aiyegbusi, thomas j. keeley, melanie j. calvert, associated data.

All relevant data are within the paper and supporting files.

Increasingly, researchers need to demonstrate the impact of their research to their sponsors, funders, and fellow academics. However, the most appropriate way of measuring the impact of healthcare research is subject to debate. We aimed to identify the existing methodological frameworks used to measure healthcare research impact and to summarise the common themes and metrics in an impact matrix.

Methods and findings

Two independent investigators systematically searched the Medical Literature Analysis and Retrieval System Online (MEDLINE), the Excerpta Medica Database (EMBASE), the Cumulative Index to Nursing and Allied Health Literature (CINAHL+), the Health Management Information Consortium, and the Journal of Research Evaluation from inception until May 2017 for publications that presented a methodological framework for research impact. We then summarised the common concepts and themes across methodological frameworks and identified the metrics used to evaluate differing forms of impact. Twenty-four unique methodological frameworks were identified, addressing 5 broad categories of impact: (1) ‘primary research-related impact’, (2) ‘influence on policy making’, (3) ‘health and health systems impact’, (4) ‘health-related and societal impact’, and (5) ‘broader economic impact’. These categories were subdivided into 16 common impact subgroups. Authors of the included publications proposed 80 different metrics aimed at measuring impact in these areas. The main limitation of the study was the potential exclusion of relevant articles, as a consequence of the poor indexing of the databases searched.

Conclusions

The measurement of research impact is an essential exercise to help direct the allocation of limited research resources, to maximise research benefit, and to help minimise research waste. This review provides a collective summary of existing methodological frameworks for research impact, which funders may use to inform the measurement of research impact and researchers may use to inform study design decisions aimed at maximising the short-, medium-, and long-term impact of their research.

Derek Kyte and colleagues systematically review approaches to the evaluation of health research.

Author summary

Why was this study done.

• There is a growing interest in demonstrating the impact of research in order to minimise research waste, allocate resources efficiently, and maximise the benefit of research. However, there is no consensus on which is the most appropriate tool to measure the impact of research.
• To our knowledge, this review is the first to synthesise existing methodological frameworks for healthcare research impact, and the associated impact metrics by which various authors have proposed impact should be measured, into a unified matrix.

What did the researchers do and find?

• We conducted a systematic review identifying 24 existing methodological research impact frameworks.
• We scrutinised the sample, identifying and summarising 5 proposed impact categories, 16 impact subcategories, and over 80 metrics into an impact matrix and methodological framework.

What do these findings mean?

• This simplified consolidated methodological framework will help researchers to understand how a research study may give rise to differing forms of impact, as well as in what ways and at which time points these potential impacts might be measured.
• Incorporating these insights into the design of a study could enhance impact, optimizing the use of research resources.

Introduction

In 2010, approximately US$240 billion was invested in healthcare research worldwide [ 1 ]. Such research is utilised by policy makers, healthcare providers, and clinicians to make important evidence-based decisions aimed at maximising patient benefit, whilst ensuring that limited healthcare resources are used as efficiently as possible to facilitate effective and sustainable service delivery. It is therefore essential that this research is of high quality and that it is impactful—i.e., it delivers demonstrable benefits to society and the wider economy whilst minimising research waste [ 1 , 2 ]. Research impact can be defined as ‘any identifiable ‘benefit to, or positive influence on the economy, society, public policy or services, health, the environment, quality of life or academia’ (p. 26) [ 3 ].

There are many purported benefits associated with the measurement of research impact, including the ability to (1) assess the quality of the research and its subsequent benefits to society; (2) inform and influence optimal policy and funding allocation; (3) demonstrate accountability, the value of research in terms of efficiency and effectiveness to the government, stakeholders, and society; and (4) maximise impact through better understanding the concept and pathways to impact [ 4 – 7 ].

Measuring and monitoring the impact of healthcare research has become increasingly common in the United Kingdom [ 5 ], Australia [ 5 ], and Canada [ 8 ], as governments, organisations, and higher education institutions seek a framework to allocate funds to projects that are more likely to bring the most benefit to society and the economy [ 5 ]. For example, in the UK, the 2014 Research Excellence Framework (REF) has recently been used to assess the quality and impact of research in higher education institutions, through the assessment of impact cases studies and selected qualitative impact metrics [ 9 ]. This is the first initiative to allocate research funding based on the economic, societal, and cultural impact of research, although it should be noted that research impact only drives a proportion of this allocation (approximately 20%) [ 9 ].

In the UK REF, the measurement of research impact is seen as increasingly important. However, the impact element of the REF has been criticised in some quarters [ 10 , 11 ]. Critics deride the fact that REF impact is determined in a relatively simplistic way, utilising researcher-generated case studies, which commonly attempt to link a particular research outcome to an associated policy or health improvement despite the fact that the wider literature highlights great diversity in the way research impact may be demonstrated [ 12 , 13 ]. This led to the current debate about the optimal method of measuring impact in the future REF [ 10 , 14 ]. The Stern review suggested that research impact should not only focus on socioeconomic impact but should also include impact on government policy, public engagement, academic impacts outside the field, and teaching to showcase interdisciplinary collaborative impact [ 10 , 11 ]. The Higher Education Funding Council for England (HEFCE) has recently set out the proposals for the REF 2021 exercise, confirming that the measurement of such impact will continue to form an important part of the process [ 15 ].

With increasing pressure for healthcare research to lead to demonstrable health, economic, and societal impact, there is a need for researchers to understand existing methodological impact frameworks and the means by which impact may be quantified (i.e., impact metrics; see Box 1 , 'Definitions’) to better inform research activities and funding decisions. From a researcher’s perspective, understanding the optimal pathways to impact can help inform study design aimed at maximising the impact of the project. At the same time, funders need to understand which aspects of impact they should focus on when allocating awards so they can make the most of their investment and bring the greatest benefit to patients and society [ 2 , 4 , 5 , 16 , 17 ].

Box 1. Definitions

• Research impact: ‘any identifiable benefit to, or positive influence on, the economy, society, public policy or services, health, the environment, quality of life, or academia’ (p. 26) [ 3 ].
• Methodological framework: ‘a body of methods, rules and postulates employed by a particular procedure or set of procedures (i.e., framework characteristics and development)’ [ 18 ].
• Pathway: ‘a way of achieving a specified result; a course of action’ [ 19 ].
• Quantitative metrics: ‘a system or standard of [quantitative] measurement’ [ 20 ].
• Narrative metrics: ‘a spoken or written account of connected events; a story’ [ 21 ].

Whilst previous researchers have summarised existing methodological frameworks and impact case studies [ 4 , 22 – 27 ], they have not summarised the metrics for use by researchers, funders, and policy makers. The aim of this review was therefore to (1) identify the methodological frameworks used to measure healthcare research impact using systematic methods, (2) summarise common impact themes and metrics in an impact matrix, and (3) provide a simplified consolidated resource for use by funders, researchers, and policy makers.

Search strategy and selection criteria

Initially, a search strategy was developed to identify the available literature regarding the different methods to measure research impact. The following keywords: ‘Impact’, ‘Framework’, and ‘Research’, and their synonyms, were used during the search of the Medical Literature Analysis and Retrieval System Online (MEDLINE; Ovid) database, the Excerpta Medica Database (EMBASE), the Health Management Information Consortium (HMIC) database, and the Cumulative Index to Nursing and Allied Health Literature (CINAHL+) database (inception to May 2017; see S1 Appendix for the full search strategy). Additionally, the nonindexed Journal of Research Evaluation was hand searched during the same timeframe using the keyword ‘Impact’. Other relevant articles were identified through 3 Internet search engines (Google, Google Scholar, and Google Images) using the keywords ‘Impact’, ‘Framework’, and ‘Research’, with the first 50 results screened. Google Images was searched because different methodological frameworks are summarised in a single image and can easily be identified through this search engine. Finally, additional publications were sought through communication with experts.

Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (see S1 PRISMA Checklist ), 2 independent investigators systematically screened for publications describing, evaluating, or utilising a methodological research impact framework within the context of healthcare research [ 28 ]. Papers were eligible if they included full or partial methodological frameworks or pathways to research impact; both primary research and systematic reviews fitting these criteria were included. We included any methodological framework identified (original or modified versions) at the point of first occurrence. In addition, methodological frameworks were included if they were applicable to the healthcare discipline with no need of modification within their structure. We defined ‘methodological framework’ as ‘a body of methods, rules and postulates employed by a particular procedure or set of procedures (i.e., framework characteristics and development)’ [ 18 ], whereas we defined ‘pathway’ as ‘a way of achieving a specified result; a course of action’ [ 19 ]. Studies were excluded if they presented an existing (unmodified) methodological framework previously available elsewhere, did not explicitly describe a methodological framework but rather focused on a single metric (e.g., bibliometric analysis), focused on the impact or effectiveness of interventions rather than that of the research, or presented case study data only. There were no language restrictions.

Data screening

Records were downloaded into Endnote (version X7.3.1), and duplicates were removed. Two independent investigators (SCR and OLA) conducted all screening following a pilot aimed at refining the process. The records were screened by title and abstract before full-text articles of potentially eligible publications were retrieved for evaluation. A full-text screening identified the publications included for data extraction. Discrepancies were resolved through discussion, with the involvement of a third reviewer (MJC, DGK, and TJK) when necessary.

Data extraction and analysis

Data extraction occurred after the final selection of included articles. SCR and OLA independently extracted details of impact methodological frameworks, the country of origin, and the year of publication, as well as the source, the framework description, and the methodology used to develop the framework. Information regarding the methodology used to develop each methodological framework was also extracted from framework webpages where available. Investigators also extracted details regarding each framework’s impact categories and subgroups, along with their proposed time to impact (‘short-term’, ‘mid-term’, or ‘long-term’) and the details of any metrics that had been proposed to measure impact, which are depicted in an impact matrix. The structure of the matrix was informed by the work of M. Buxton and S. Hanney [ 2 ], P. Buykx et al. [ 5 ], S. Kuruvila et al. [ 29 ], and A. Weiss [ 30 ], with the intention of mapping metrics presented in previous methodological frameworks in a concise way. A consensus meeting with MJC, DGK, and TJK was held to solve disagreements and finalise the data extraction process.

Included studies

Our original search strategy identified 359 citations from MEDLINE (Ovid), EMBASE, CINAHL+, HMIC, and the Journal of Research Evaluation, and 101 citations were returned using other sources (Google, Google Images, Google Scholar, and expert communication) (see Fig 1 ) [ 28 ]. In total, we retrieved 54 full-text articles for review. At this stage, 39 articles were excluded, as they did not propose new or modified methodological frameworks. An additional 15 articles were included following the backward and forward citation method. A total of 31 relevant articles were included in the final analysis, of which 24 were articles presenting unique frameworks and the remaining 7 were systematic reviews [ 4 , 22 – 27 ]. The search strategy was rerun on 15 May 2017. A further 19 publications were screened, and 2 were taken forward to full-text screening but were ineligible for inclusion.

Methodological framework characteristics

The characteristics of the 24 included methodological frameworks are summarised in Table 1 , 'Methodological framework characteristics’. Fourteen publications proposed academic-orientated frameworks, which focused on measuring academic, societal, economic, and cultural impact using narrative and quantitative metrics [ 2 , 3 , 5 , 8 , 29 , 31 – 39 ]. Five publications focused on assessing the impact of research by focusing on the interaction process between stakeholders and researchers (‘productive interactions’), which is a requirement to achieve research impact. This approach tries to address the issue of attributing research impact to metrics [ 7 , 40 – 43 ]. Two frameworks focused on the importance of partnerships between researchers and policy makers, as a core element to accomplish research impact [ 44 , 45 ]. An additional 2 frameworks focused on evaluating the pathways to impact, i.e., linking processes between research and impact [ 30 , 46 ]. One framework assessed the ability of health technology to influence efficiency of healthcare systems [ 47 ]. Eight frameworks were developed in the UK [ 2 , 3 , 29 , 37 , 39 , 42 , 43 , 45 ], 6 in Canada [ 8 , 33 , 34 , 44 , 46 , 47 ], 4 in Australia [ 5 , 31 , 35 , 38 ], 3 in the Netherlands [ 7 , 40 , 41 ], and 2 in the United States [ 30 , 36 ], with 1 model developed with input from various countries [ 32 ].

Methodological framework development

The included methodological frameworks varied in their development process, but there were some common approaches employed. Most included a literature review [ 2 , 5 , 7 , 8 , 31 , 33 , 36 , 37 , 40 – 46 ], although none of them used a recognised systematic method. Most also consulted with various stakeholders [ 3 , 8 , 29 , 31 , 33 , 35 – 38 , 43 , 44 , 46 , 47 ] but used differing methods to incorporate their views, including quantitative surveys [ 32 , 35 , 43 , 46 ], face-to-face interviews [ 7 , 29 , 33 , 35 , 37 , 42 , 43 ], telephone interviews [ 31 , 46 ], consultation [ 3 , 7 , 36 ], and focus groups [ 39 , 43 ]. A range of stakeholder groups were approached across the sample, including principal investigators [ 7 , 29 , 43 ], research end users [ 7 , 42 , 43 ], academics [ 3 , 8 , 39 , 40 , 43 , 46 ], award holders [ 43 ], experts [ 33 , 38 , 39 ], sponsors [ 33 , 39 ], project coordinators [ 32 , 42 ], and chief investigators [ 31 , 35 ]. However, some authors failed to identify the stakeholders involved in the development of their frameworks [ 2 , 5 , 34 , 41 , 45 ], making it difficult to assess their appropriateness. In addition, only 4 of the included papers reported using formal analytic methods to interpret stakeholder responses. These included the Canadian Academy of Health Sciences framework, which used conceptual cluster analysis [ 33 ]. The Research Contribution [ 42 ], Research Impact [ 29 ], and Primary Health Care & Information Service [ 31 ] used a thematic analysis approach. Finally, some authors went on to pilot their framework, which shaped refinements on the methodological frameworks until approval. Methods used to pilot the frameworks included a case study approach [ 2 , 3 , 30 , 32 , 33 , 36 , 40 , 42 , 44 , 45 ], contrasting results against available literature [ 29 ], the use of stakeholders’ feedback [ 7 ], and assessment tools [ 35 , 46 ].

Major impact categories

1. primary research-related impact.

A number of methodological frameworks advocated the evaluation of ‘research-related impact’. This encompassed content related to the generation of new knowledge, knowledge dissemination, capacity building, training, leadership, and the development of research networks. These outcomes were considered the direct or primary impacts of a research project, as these are often the first evidenced returns [ 30 , 62 ].

A number of subgroups were identified within this category, with frameworks supporting the collection of impact data across the following constructs: ‘research and innovation outcomes’; ‘dissemination and knowledge transfer’; ‘capacity building, training, and leadership’; and ‘academic collaborations, research networks, and data sharing’.

1 . 1 . Research and innovation outcomes . Twenty of the 24 frameworks advocated the evaluation of ‘research and innovation outcomes’ [ 2 , 3 , 5 , 7 , 8 , 29 – 39 , 41 , 43 , 44 , 46 ]. This subgroup included the following metrics: number of publications; number of peer-reviewed articles (including journal impact factor); citation rates; requests for reprints, number of reviews, and meta-analysis; and new or changes in existing products (interventions or technology), patents, and research. Additionally, some frameworks also sought to gather information regarding ‘methods/methodological contributions’. These advocated the collection of systematic reviews and appraisals in order to identify gaps in knowledge and determine whether the knowledge generated had been assessed before being put into practice [ 29 ].

1 . 2 . Dissemination and knowledge transfer . Nineteen of the 24 frameworks advocated the assessment of ‘dissemination and knowledge transfer’ [ 2 , 3 , 5 , 7 , 29 – 32 , 34 – 43 , 46 ]. This comprised collection of the following information: number of conferences, seminars, workshops, and presentations; teaching output (i.e., number of lectures given to disseminate the research findings); number of reads for published articles; article download rate and number of journal webpage visits; and citations rates in nonjournal media such as newspapers and mass and social media (i.e., Twitter and blogs). Furthermore, this impact subgroup considered the measurement of research uptake and translatability and the adoption of research findings in technological and clinical applications and by different fields. These can be measured through patents, clinical trials, and partnerships between industry and business, government and nongovernmental organisations, and university research units and researchers [ 29 ].

1 . 3 . Capacity building , training , and leadership . Fourteen of 24 frameworks suggested the evaluation of ‘capacity building, training, and leadership’ [ 2 , 3 , 5 , 8 , 29 , 31 – 35 , 39 – 41 , 43 ]. This involved collecting information regarding the number of doctoral and postdoctoral studentships (including those generated as a result of the research findings and those appointed to conduct the research), as well as the number of researchers and research-related staff involved in the research projects. In addition, authors advocated the collection of ‘leadership’ metrics, including the number of research projects managed and coordinated and the membership of boards and funding bodies, journal editorial boards, and advisory committees [ 29 ]. Additional metrics in this category included public recognition (number of fellowships and awards for significant research achievements), academic career advancement, and subsequent grants received. Lastly, the impact metric ‘research system management’ comprised the collection of information that can lead to preserving the health of the population, such as modifying research priorities, resource allocation strategies, and linking health research to other disciplines to maximise benefits [ 29 ].

1 . 4 . Academic collaborations , research networks , and data sharing . Lastly, 10 of the 24 frameworks advocated the collection of impact data regarding ‘academic collaborations (internal and external collaborations to complete a research project), research networks, and data sharing’ [ 2 , 3 , 5 , 7 , 29 , 34 , 37 , 39 , 41 , 43 ].

2. Influence on policy making

Methodological frameworks addressing this major impact category focused on measurable improvements within a given knowledge base and on interactions between academics and policy makers, which may influence policy-making development and implementation. The returns generated in this impact category are generally considered as intermediate or midterm (1 to 3 years). These represent an important interim stage in the process towards the final expected impacts, such as quantifiable health improvements and economic benefits, without which policy change may not occur [ 30 , 62 ]. The following impact subgroups were identified within this category: ‘type and nature of policy impact’, ‘level of policy making’, and ‘policy networks’.

2 . 1 . Type and nature of policy impact . The most common impact subgroup, mentioned in 18 of the 24 frameworks, was ‘type and nature of policy impact’ [ 2 , 7 , 29 – 38 , 41 – 43 , 45 – 47 ]. Methodological frameworks addressing this subgroup stressed the importance of collecting information regarding the influence of research on policy (i.e., changes in practice or terminology). For instance, a project looking at trafficked adolescents and women (2003) influenced the WHO guidelines (2003) on ethics regarding this particular group [ 17 , 21 , 63 ].

2 . 2 . Level of policy impact . Thirteen of 24 frameworks addressed aspects surrounding the need to record the ‘level of policy impact’ (international, national, or local) and the organisations within a level that were influenced (local policy makers, clinical commissioning groups, and health and wellbeing trusts) [ 2 , 5 , 8 , 29 , 31 , 34 , 38 , 41 , 43 – 47 ]. Authors considered it important to measure the ‘level of policy impact’ to provide evidence of collaboration, coordination, and efficiency within health organisations and between researchers and health organisations [ 29 , 31 ].

2 . 3 . Policy networks . Five methodological frameworks highlighted the need to collect information regarding collaborative research with industry and staff movement between academia and industry [ 5 , 7 , 29 , 41 , 43 ]. A policy network emphasises the relationship between policy communities, researchers, and policy makers. This relationship can influence and lead to incremental changes in policy processes [ 62 ].

3. Health and health systems impact

A number of methodological frameworks advocated the measurement of impacts on health and healthcare systems across the following impact subgroups: ‘quality of care and service delivering’, ‘evidence-based practice’, ‘improved information and health information management’, ‘cost containment and effectiveness’, ‘resource allocation’, and ‘health workforce’.

3 . 1 . Quality of care and service delivery . Twelve of the 24 frameworks highlighted the importance of evaluating ‘quality of care and service delivery’ [ 2 , 5 , 8 , 29 – 31 , 33 – 36 , 41 , 47 ]. There were a number of suggested metrics that could be potentially used for this purpose, including health outcomes such as quality-adjusted life years (QALYs), patient-reported outcome measures (PROMs), patient satisfaction and experience surveys, and qualitative data on waiting times and service accessibility.

3 . 2 . Evidence-based practice . ‘Evidence-based practice’, mentioned in 5 of the 24 frameworks, refers to making changes in clinical diagnosis, clinical practice, treatment decisions, or decision making based on research evidence [ 5 , 8 , 29 , 31 , 33 ]. The suggested metrics to demonstrate evidence-based practice were adoption of health technologies and research outcomes to improve the healthcare systems and inform policies and guidelines [ 29 ].

3 . 3 . Improved information and health information management . This impact subcategory, mentioned in 5 of the 24 frameworks, refers to the influence of research on the provision of health services and management of the health system to prevent additional costs [ 5 , 29 , 33 , 34 , 38 ]. Methodological frameworks advocated the collection of health system financial, nonfinancial (i.e., transport and sociopolitical implications), and insurance information in order to determine constraints within a health system.

3 . 4 . Cost containment and cost-effectiveness . Six of the 24 frameworks advocated the subcategory ‘cost containment and cost-effectiveness’ [ 2 , 5 , 8 , 17 , 33 , 36 ]. ‘Cost containment’ comprised the collection of information regarding how research has influenced the provision and management of health services and its implication in healthcare resource allocation and use [ 29 ]. ‘Cost-effectiveness’ refers to information concerning economic evaluations to assess improvements in effectiveness and health outcomes—for instance, the cost-effectiveness (cost and health outcome benefits) assessment of introducing a new health technology to replace an older one [ 29 , 31 , 64 ].

3 . 5 . Resource allocation . ‘Resource allocation’, mentioned in 6frameworks, can be measured through 2 impact metrics: new funding attributed to the intervention in question and equity while allocating resources, such as improved allocation of resources at an area level; better targeting, accessibility, and utilisation; and coverage of health services [ 2 , 5 , 29 , 31 , 45 , 47 ]. The allocation of resources and targeting can be measured through health services research reports, with the utilisation of health services measured by the probability of providing an intervention when needed, the probability of requiring it again in the future, and the probability of receiving an intervention based on previous experience [ 29 , 31 ].

3 . 6 . Health workforce . Lastly, ‘health workforce’, present in 3 methodological frameworks, refers to the reduction in the days of work lost because of a particular illness [ 2 , 5 , 31 ].

4. Health-related and societal impact

Three subgroups were included in this category: ‘health literacy’; ‘health knowledge, attitudes, and behaviours’; and ‘improved social equity, inclusion, or cohesion’.

4 . 1 . Health knowledge , attitudes , and behaviours . Eight of the 24 frameworks suggested the assessment of ‘health knowledge, attitudes, behaviours, and outcomes’, which could be measured through the evaluation of levels of public engagement with science and research (e.g., National Health Service (NHS) Choices end-user visit rate) or by using focus groups to analyse changes in knowledge, attitudes, and behaviour among society [ 2 , 5 , 29 , 33 – 35 , 38 , 43 ].

4 . 2 . Improved equity , inclusion , or cohesion and human rights . Other methodological frameworks, 4 of the 24, suggested capturing improvements in equity, inclusion, or cohesion and human rights. Authors suggested these could be using a resource like the United Nations Millennium Development Goals (MDGs) (superseded by Sustainable Development Goals [SDGs] in 2015) and human rights [ 29 , 33 , 34 , 38 ]. For instance, a cluster-randomised controlled trial in Nepal, which had female participants, has demonstrated the reduction of neonatal mortality through the introduction of maternity health care, distribution of delivery kits, and home visits. This illustrates how research can target vulnerable and disadvantaged groups. Additionally, this research has been introduced by the World Health Organisation to achieve the MDG ‘improve maternal health’ [ 16 , 29 , 65 ].

4 . 3 . Health literacy . Some methodological frameworks, 3 of the 24, focused on tracking changes in the ability of patients to make informed healthcare decisions, reduce health risks, and improve quality of life, which were demonstrably linked to a particular programme of research [ 5 , 29 , 43 ]. For example, a systematic review showed that when HIV health literacy/knowledge is spread among people living with the condition, antiretroviral adherence and quality of life improve [ 66 ].

5. Broader economic impacts

Some methodological frameworks, 9 of 24, included aspects related to the broader economic impacts of health research—for example, the economic benefits emerging from the commercialisation of research outputs [ 2 , 5 , 29 , 31 , 33 , 35 , 36 , 38 , 67 ]. Suggested metrics included the amount of funding for research and development (R&D) that was competitively awarded by the NHS, medical charities, and overseas companies. Additional metrics were income from intellectual property, spillover effects (any secondary benefit gained as a repercussion of investing directly in a primary activity, i.e., the social and economic returns of investing on R&D) [ 33 ], patents granted, licences awarded and brought to the market, the development and sales of spinout companies, research contracts, and income from industry.

The benefits contained within the categories ‘health and health systems impact’, ‘health-related and societal impact’, and ‘broader economic impacts’ are considered the expected and final returns of the resources allocated in healthcare research [ 30 , 62 ]. These benefits commonly arise in the long term, beyond 5 years according to some authors, but there was a recognition that this could differ depending on the project and its associated research area [ 4 ].

Data synthesis

Five major impact categories were identified across the 24 included methodological frameworks: (1) ‘primary research-related impact’, (2) ‘influence on policy making’, (3) ‘health and health systems impact’, (4) ‘health-related and societal impact’, and (5) ‘broader economic impact’. These major impact categories were further subdivided into 16 impact subgroups. The included publications proposed 80 different metrics to measure research impact. This impact typology synthesis is depicted in ‘the impact matrix’ ( Fig 2 and Fig 3 ).

CIHR, Canadian Institutes of Health Research; HTA, Health Technology Assessment; PHC RIS, Primary Health Care Research & Information Service; RAE, Research Assessment Exercise; RQF, Research Quality Framework.

AIHS, Alberta Innovates—Health Solutions; CAHS, Canadian Institutes of Health Research; IOM, Impact Oriented Monitoring; REF, Research Excellence Framework; SIAMPI, Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions between science and society.

Commonality and differences across frameworks

The ‘Research Impact Framework’ and the ‘Health Services Research Impact Framework’ were the models that encompassed the largest number of the metrics extracted. The most dominant methodological framework was the Payback Framework; 7 other methodological framework models used the Payback Framework as a starting point for development [ 8 , 29 , 31 – 35 ]. Additional methodological frameworks that were commonly incorporated into other tools included the CIHR framework, the CAHS model, the AIHS framework, and the Exchange model [ 8 , 33 , 34 , 44 ]. The capture of ‘research-related impact’ was the most widely advocated concept across methodological frameworks, illustrating the importance with which primary short-term impact outcomes were viewed by the included papers. Thus, measurement of impact via number of publications, citations, and peer-reviewed articles was the most common. ‘Influence on policy making’ was the predominant midterm impact category, specifically the subgroup ‘type and nature of policy impact’, in which frameworks advocated the measurement of (i) changes to legislation, regulations, and government policy; (ii) influence and involvement in decision-making processes; and (iii) changes to clinical or healthcare training, practice, or guidelines. Within more long-term impact measurement, the evaluations of changes in the ‘quality of care and service delivery’ were commonly advocated.

In light of the commonalities and differences among the methodological frameworks, the ‘pathways to research impact’ diagram ( Fig 4 ) was developed to provide researchers, funders, and policy makers a more comprehensive and exhaustive way to measure healthcare research impact. The diagram has the advantage of assorting all the impact metrics proposed by previous frameworks and grouping them into different impact subgroups and categories. Prospectively, this global picture will help researchers, funders, and policy makers plan strategies to achieve multiple pathways to impact before carrying the research out. The analysis of the data extraction and construction of the impact matrix led to the development of the ‘pathways to research impact’ diagram ( Fig 4 ). The diagram aims to provide an exhaustive and comprehensive way of tracing research impact by combining all the impact metrics presented by the different 24 frameworks, grouping those metrics into different impact subgroups, and grouping these into broader impact categories.

NHS, National Health Service; PROM, patient-reported outcome measure; QALY, quality-adjusted life year; R&D, research and development.

This review has summarised existing methodological impact frameworks together for the first time using systematic methods ( Fig 4 ). It allows researchers and funders to consider pathways to impact at the design stage of a study and to understand the elements and metrics that need to be considered to facilitate prospective assessment of impact. Users do not necessarily need to cover all the aspects of the methodological framework, as every research project can impact on different categories and subgroups. This review provides information that can assist researchers to better demonstrate impact, potentially increasing the likelihood of conducting impactful research and reducing research waste. Existing reviews have not presented a methodological framework that includes different pathways to impact, health impact categories, subgroups, and metrics in a single methodological framework.

Academic-orientated frameworks included in this review advocated the measurement of impact predominantly using so-called ‘quantitative’ metrics—for example, the number of peer-reviewed articles, journal impact factor, and citation rates. This may be because they are well-established measures, relatively easy to capture and objective, and are supported by research funding systems. However, these metrics primarily measure the dissemination of research finding rather than its impact [ 30 , 68 ]. Whilst it is true that wider dissemination, especially when delivered via world-leading international journals, may well lead eventually to changes in healthcare, this is by no means certain. For instance, case studies evaluated by Flinders University of Australia demonstrated that some research projects with non-peer-reviewed publications led to significant changes in health policy, whilst the studies with peer-reviewed publications did not result in any type of impact [ 68 ]. As a result, contemporary literature has tended to advocate the collection of information regarding a variety of different potential forms of impact alongside publication/citations metrics [ 2 , 3 , 5 , 7 , 8 , 29 – 47 ], as outlined in this review.

The 2014 REF exercise adjusted UK university research funding allocation based on evidence of the wider impact of research (through case narrative studies and quantitative metrics), rather than simply according to the quality of research [ 12 ]. The intention was to ensure funds were directed to high-quality research that could demonstrate actual realised benefit. The inclusion of a mixed-method approach to the measurement of impact in the REF (narrative and quantitative metrics) reflects a widespread belief—expressed by the majority of authors of the included methodological frameworks in the review—that individual quantitative impact metrics (e.g., number of citations and publications) do not necessary capture the complexity of the relationships involved in a research project and may exclude measurement of specific aspects of the research pathway [ 10 , 12 ].

Many of the frameworks included in this review advocated the collection of a range of academic, societal, economic, and cultural impact metrics; this is consistent with recent recommendations from the Stern review [ 10 ]. However, a number of these metrics encounter research ‘lag’: i.e., the time between the point at which the research is conducted and when the actual benefits arise [ 69 ]. For instance, some cardiovascular research has taken up to 25 years to generate impact [ 70 ]. Likewise, the impact may not arise exclusively from a single piece of research. Different processes (such as networking interactions and knowledge and research translation) and multiple individuals and organisations are often involved [ 4 , 71 ]. Therefore, attributing the contribution made by each of the different actors involved in the process can be a challenge [ 4 ]. An additional problem associated to attribution is the lack of evidence to link research and impact. The outcomes of research may emerge slowly and be absorbed gradually. Consequently, it is difficult to determine the influence of research in the development of a new policy, practice, or guidelines [ 4 , 23 ].

A further problem is that impact evaluation is conducted ‘ex post’, after the research has concluded. Collecting information retrospectively can be an issue, as the data required might not be available. ‘ex ante’ assessment is vital for funding allocation, as it is necessary to determine the potential forthcoming impact before research is carried out [ 69 ]. Additionally, ex ante evaluation of potential benefit can overcome the issues regarding identifying and capturing evidence, which can be used in the future [ 4 ]. In order to conduct ex ante evaluation of potential benefit, some authors suggest the early involvement of policy makers in a research project coupled with a well-designed strategy of dissemination [ 40 , 69 ].

Providing an alternate view, the authors of methodological frameworks such as the SIAMPI, Contribution Mapping, Research Contribution, and the Exchange model suggest that the problems of attribution are a consequence of assigning the impact of research to a particular impact metric [ 7 , 40 , 42 , 44 ]. To address these issues, these authors propose focusing on the contribution of research through assessing the processes and interactions between stakeholders and researchers, which arguably take into consideration all the processes and actors involved in a research project [ 7 , 40 , 42 , 43 ]. Additionally, contributions highlight the importance of the interactions between stakeholders and researchers from an early stage in the research process, leading to a successful ex ante and ex post evaluation by setting expected impacts and determining how the research outcomes have been utilised, respectively [ 7 , 40 , 42 , 43 ]. However, contribution metrics are generally harder to measure in comparison to academic-orientated indicators [ 72 ].

Currently, there is a debate surrounding the optimal methodological impact framework, and no tool has proven superior to another. The most appropriate methodological framework for a given study will likely depend on stakeholder needs, as each employs different methodologies to assess research impact [ 4 , 37 , 41 ]. This review allows researchers to select individual existing methodological framework components to create a bespoke tool with which to facilitate optimal study design and maximise the potential for impact depending on the characteristic of their study ( Fig 2 and Fig 3 ). For instance, if researchers are interested in assessing how influential their research is on policy making, perhaps considering a suite of the appropriate metrics drawn from multiple methodological frameworks may provide a more comprehensive method than adopting a single methodological framework. In addition, research teams may wish to use a multidimensional approach to methodological framework development, adopting existing narratives and quantitative metrics, as well as elements from contribution frameworks. This approach would arguably present a more comprehensive method of impact assessment; however, further research is warranted to determine its effectiveness [ 4 , 69 , 72 , 73 ].

Finally, it became clear during this review that the included methodological frameworks had been constructed using varied methodological processes. At present, there are no guidelines or consensus around the optimal pathway that should be followed to develop a robust methodological framework. The authors believe this is an area that should be addressed by the research community, to ensure future frameworks are developed using best-practice methodology.

For instance, the Payback Framework drew upon a literature review and was refined through a case study approach. Arguably, this approach could be considered inferior to other methods that involved extensive stakeholder involvement, such as the CIHR framework [ 8 ]. Nonetheless, 7 methodological frameworks were developed based upon the Payback Framework [ 8 , 29 , 31 – 35 ].

Limitations

The present review is the first to summarise systematically existing impact methodological frameworks and metrics. The main limitation is that 50% of the included publications were found through methods other than bibliographic databases searching, indicating poor indexing. Therefore, some relevant articles may not have been included in this review if they failed to indicate the inclusion of a methodological impact framework in their title/abstract. We did, however, make every effort to try to find these potentially hard-to-reach publications, e.g., through forwards/backwards citation searching, hand searching reference lists, and expert communication. Additionally, this review only extracted information regarding the methodology followed to develop each framework from the main publication source or framework webpage. Therefore, further evaluations may not have been included, as they are beyond the scope of the current paper. A further limitation was that although our search strategy did not include language restrictions, we did not specifically search non-English language databases. Thus, we may have failed to identify potentially relevant methodological frameworks that were developed in a non-English language setting.

In conclusion, the measurement of research impact is an essential exercise to help direct the allocation of limited research resources, to maximise benefit, and to help minimise research waste. This review provides a collective summary of existing methodological impact frameworks and metrics, which funders may use to inform the measurement of research impact and researchers may use to inform study design decisions aimed at maximising the short-, medium-, and long-term impact of their research.

Supporting information

S1 appendix, s1 prisma checklist, acknowledgments.

We would also like to thank Mrs Susan Bayliss, Information Specialist, University of Birmingham, and Mrs Karen Biddle, Research Secretary, University of Birmingham.

Abbreviations

Funding statement.

Funding was received from Consejo Nacional de Ciencia y Tecnología (CONACYT). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript ( http://www.conacyt.mx/ ).

Data Availability