rigour in qualitative research meaning

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Chapter 26: Rigour

Darshini Ayton

Learning outcomes

Upon completion of this chapter, you should be able to:

• Understand the concepts of rigour and trustworthiness in qualitative research.
• Describe strategies for dependability, credibility, confirmability and transferability in qualitative research.
• Define reflexivity and describe types of reflexivity

What is rigour?

In qualitative research, rigour, or trustworthiness, refers to how researchers demonstrate the quality of their research. 1, 2 Rigour is an umbrella term for several strategies and approaches that recognise the influence on qualitative research by multiple realities; for example, of the researcher during data collection and analysis, and of the participant. The research process is shaped by multiple elements, including research skills, the social and research environment and the community setting. 2

Research is considered rigorous or trustworthy when members of the research community are confident in the study’s methods, the data and its interpretation. 3 As mentioned in Chapters 1 and 2, quantitative and qualitative research are founded on different research paradigms and, hence, quality in research cannot be addressed in the same way for both types of research studies. Table 26.1 provides a comparison overview of the approaches of quantitative and qualitative research in ensuring quality in research.

Table 26.1: Comparison of quantitative and qualitative approaches to ensuring quality in research

Below is an overview of the main approaches to rigour in qualitative research. For each of the approaches, examples of how rigour was demonstrated are provided from the author’s PhD thesis.

Approaches to dependability

Dependability requires the researcher to provide an account of changes to the research process and setting. 3 The main approach to dependability is an audit trail.

• Audit trail – the researcher records or takes notes on the conduct of the research and the process of reaching conclusions from the data. The audit trail includes information on the data collection and data analysis, including decision-making and interpretations of the data that influence the study’s results. 8 , 9

The interview questions for this study evolved as the study progressed, and accordingly, the process was iterative. I spent 12 months collecting data, and as my understanding and responsiveness to my participants and to the culture and ethos of the various churches developed, so did my line of questioning. For example, in the early interviews for phase 2, I included questions regarding the qualifications a church leader might look for in hiring someone to undertake health promotion activities. This question was dropped after the first couple of interviews, as it was clear that church leaders did not necessarily view their activities as health promoting and therefore did not perceive the relevance of this question. By ‘being church’, they were health promoting, and therefore activities that were health promoting were not easily separated from other activities that were part of the core mission of the church 10 ( pp93–4)

Approaches to credibility

Credibility requires the researcher to demonstrate the truth or confidence in the findings. The main approaches to credibility include triangulation, prolonged engagement, persistent observation, negative case analysis and member checking. 3

• Triangulation – the assembly of data and interpretations from multiple methods (methods triangulation), researchers (research triangulation), theory (theory triangulation) and data sources (different participant groups). 9 Refer to Chapter 28 for a detailed discussion of this process.
• Prolonged engagement – the requirement for researchers to spend sufficient time with participants and/or within the research context to familiarise them with the research setting, to build trust and rapport with participants and to recognise and correct any misinformation. 9

Prolonged engagement with churches was also achieved through the case study phase as the ten case study churches were involved in more than one phase of data collection. These ten churches were the case studies in which significant time was spent conducting interviews and focus groups, and attending activities and programs. Subsequently, there were many instances where I interacted with the same people on more than one occasion, thereby facilitating the development of interactive and deeper relationships with participants 10 (pp.94–5)

• Persistent observation – the identification of characteristics and elements that are most relevant to the problem or issue under study, and upon which the research will focus in detail. 9

In the following chapters, I present my analysis of the world of churches in which I was immersed as I conducted fieldwork. I describe the processes of church practice and action, and explore how this can be conceptualised into health promotion action 10 (p97)

• Negative case analysis – the process of finding and discussing data that contradicts the study’s main findings. Negative case analysis demonstrates that nuance and granularity in perspectives of both shared and divergent opinions have been examined, enhancing the quality of the interpretation of the data.

Although I did not use negative case selection, the Catholic churches in this study acted as examples of the ‘low engagement’ 10 (p97 )

• Member checking – the presentation of data analysis, interpretations and conclusions of the research to members of the participant groups. This enables participants or people with shared identity with the participants to provide their perspectives on the research. 9

Throughout my candidature – during data collection and analysis, and in the construction of my results chapters – I engaged with a number of Christians, both paid church staff members and volunteers, to test my thoughts and concepts. These people were not participants in the study, but they were embedded in the cultural and social context of churches in Victoria. They were able to challenge and also affirm my thinking and so contributed to a process of member checking 10 (p96)

Approaches to confirmability

Confirmability is demonstrated by grounding the results in the data from participants. 3 This can be achieved through the use of quotes, specifying the number of participants and data sources and providing details of the data collection.

• Quotes from participants are used to demonstrate that the themes are generated from the data. The results section of the thesis chapters commences with a story based on the field notes or recordings, with extensive quotes from participants presented throughout. 10
• The number of participants in the study provides the context for where the data is ‘sourced’ from for the results and interpretation. Table 26.2 is reproduced with permission from the Author’s thesis and details the data sources for the project. This also contributes to establishing how triangulation across data sources and methods was achieved.
• Details of data collection – Table 26.2 provides detailed information about the processes of data collection, including dates and locations but the duration of each research encounter was not specified.

Table 26.2 Data sources for the PhD research project of the Author.

Approaches to transferability.

To enable the transferability of qualitative research, researchers need to provide information about the context and the setting. A key approach for transferability is thick description. 6

• Thick description – detailed explanations and descriptions of the research questions are provided, including about the research setting, contextual factors and changes to the research setting. 9

I chose to include the Catholic Church because it is the largest Christian group in Australia and is an example of a traditional church. The Protestant group were represented through the Uniting, Anglican Baptist and Church of Christ denominations. The Uniting Church denomination is unique to Australia and was formed in 1977 through the merging of the Methodist, Presbyterian and Congregationalist denominations. The Church of Christ denomination was chosen to represent a contemporary less hierarchical denomination in comparison to the other protestant denominations. The last group, the Salvation Army, was chosen because of its high profile in social justice and social welfare, therefore offering different perspectives on the role and activities of the church in health promotion 10 (pp82–3)

What is reflexivity?

Reflexivity is the process in which researchers engage to explore and explain how their subjectivity (or bias) has influenced the research. 12 Researchers engage in reflexive practices to ensure and demonstrate rigour, quality and, ultimately, trustworthiness in their research. 13 The researcher is the instrument of data collection and data analysis, and hence awareness of what has influenced their approach and conduct of the research – and being able to articulate them – is vital in the creation of knowledge. One important element is researcher positionality (see Chapter 27), which acknowledges the characteristics, interests, beliefs and personal experiences of the researcher and how this influences the research process. Table 26.3 outlines different types of reflexivity, with examples from the author’s thesis.

Table 26.3: Types of reflexivity

The quality of qualitative research is measured through the rigour or trustworthiness of the research, demonstrated through a range of strategies in the processes of data collection, analysis, reporting and reflexivity.

• Chowdhury IA. Issue of quality in qualitative research: an overview. Innovative Issues and Approaches in Social Sciences . 2015;8(1):142-162. doi:10.12959/issn.1855-0541.IIASS-2015-no1-art09
• Cypress BS. Rigor or reliability and validity in qualitative research: perspectives, strategies, reconceptualization, and recommendations. Dimens Crit Care Nurs . 2017;36(4):253-263. doi:10.1097/DCC.0000000000000253
• Connelly LM. Trustworthiness in qualitative research. Medsurg Nurs . 2016;25(6):435-6.
• Golafshani N. Understanding reliability and validity in qualitative research. Qual Rep . 2003;8(4):597-607. Accessed September 18, 2023. https://nsuworks.nova.edu/tqr/vol8/iss4/6/
• Yilmaz K. Comparison of quantitative and qualitative research traditions: epistemological, theoretical, and methodological differences. Eur J  Educ . 2013;48(2):311-325. doi:10.1111/ejed.12014
• Shenton AK. Strategies for ensuring trustworthiness in qualitative research projects. Education for Information 2004;22:63-75. Accessed September 18, 2023. https://content.iospress.com/articles/education-for-information/efi00778
• Varpio L, O’Brien B, Rees CE, Monrouxe L, Ajjawi R, Paradis E. The applicability of generalisability and bias to health professions education’s research. Med Educ . Feb 2021;55(2):167-173. doi:10.1111/medu.14348
• Carcary M. The Research Audit Trail: Methodological guidance for application in practice. Electronic Journal of Business Research Methods . 2020;18(2):166-177. doi:10.34190/JBRM.18.2.008
• Korstjens I, Moser A. Series: Practical guidance to qualitative research. Part 4: Trustworthiness and publishing. Eur J Gen Pract . Dec 2018;24(1):120-124. doi:10.1080/13814788.2017.1375092
• Ayton D. ‘From places of despair to spaces of hope’ – the local church and health promotion in Victoria . PhD. Monash University; 2013. https://figshare.com/articles/thesis/_From_places_of_despair_to_spaces_of_hope_-_the_local_church_and_health_promotion_in_Victoria/4628308/1
• Hanson A. Negative case analysis. In: Matthes J, ed. The International Encyclopedia of Communication Research Methods . John Wiley & Sons, Inc.; 2017. doi: 10.1002/9781118901731.iecrm0165
• Olmos-Vega FM. A practical guide to reflexivity in qualitative research: AMEE Guide No. 149. Med Teach . 2023;45(3):241-251. doi: 10.1080/0142159X.2022.2057287
• Dodgson JE. Reflexivity in qualitative research. J Hum Lact . 2019;35(2):220-222. doi:10.1177/08903344198309

Qualitative Research – a practical guide for health and social care researchers and practitioners Copyright © 2023 by Darshini Ayton is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License , except where otherwise noted.

Share This Book

MS in Nursing (MSN)

A Guide To Qualitative Rigor In Research

Advances in technology have made quantitative data more accessible than ever before; but in the human-centric discipline of nursing, qualitative research still brings vital learnings to the health care industry. It is sometimes difficult to derive viable insights from qualitative research; but in the article below, the authors identify three criteria for developing acceptable qualitative studies.

Qualitative rigor in research explained

Qualitative rigor. It’s one of those terms you either understand or you don’t. And it seems that many of us fall into the latter of those two categories. From novices to experienced qualitative researchers, qualitative rigor is a concept that can be challenging. However, it also happens to be one of the most critical aspects of qualitative research, so it’s important that we all start getting to grips with what it means.

Rigor, in qualitative terms, is a way to establish trust or confidence in the findings of a research study. It allows the researcher to establish consistency in the methods used over time. It also provides an accurate representation of the population studied. As a nurse, you want to build your practice on the best evidence you can and to do so you need to have confidence in those research findings.

This article will look in more detail at the unique components of qualitative research in relation to qualitative rigor. These are: truth-value (credibility); applicability (transferability); consistency (dependability); and neutrality (confirmability).

Credibility

Credibility allows others to recognize the experiences contained within the study through the interpretation of participants’ experiences. In order to establish credibility, a researcher must review the individual transcripts, looking for similarities within and across all participants.

A study is considered credible when it presents an interpretation of an experience in such a way that people sharing that experience immediately recognize it. Examples of strategies used to establish credibility include:

• Reflexivity
• Member checking (aka informant feedback)
• Peer examination
• Peer debriefing
• Prolonged time spent with participants
• Using the participants’ words in the final report

Transferability

The ability to transfer research findings or methods from one group to another is called transferability in qualitative language, equivalent to external validity. One way of establishing transferability is to provide a dense description of the population studied by describing the demographics and geographic boundaries of the study.

Ways in which transferability can be applied by researchers include:

• Using the same data collection methods with different demographic groups or geographical locations
• Giving a range of experiences on which the reader can build interventions and understanding to decide whether the research is applicable to practice

Dependability

Related to reliability in quantitative terms, dependability occurs when another researcher can follow the decision trail used by the researcher. This trail is achieved by:

• Describing the specific purpose of the study
• Discussing how and why the participants were selected for the study
• Describing how the data was collected and how long collection lasted
• Explaining how the data was reduced or transformed for analysis
• Discussing the interpretation and presentation of the findings
• Explaining the techniques used to determine the credibility of the data

Strategies used to establish dependability include:

• Having peers participate in the analysis process
• Providing a detailed description of the research methods
• Conducting a step-by-step repetition of the study to identify similarities in results or to enhance findings

Confirmability

Confirmability occurs once credibility, transferability and dependability have been established. Qualitative research must be reflective, maintaining a sense of awareness and openness to the study and results. The researcher needs a self-critical attitude, taking into account how his or her preconceptions affect the research.

Techniques researchers use to achieve confirmability include:

• Taking notes regarding personal feelings, biases and insights immediately after an interview
• Following, rather than leading, the direction of interviews by asking for clarifications when needed

Reflective research produces new insights, which lead the reader to trust the credibility of the findings and applicability of the study

Become a Champion of Qualitative Rigor

Clinical Nurse Leaders, or CNLs, work with interdisciplinary teams to improve care for populations of patients. CNLs can impact quality and safety by assessing risks and utilizing research findings to develop quality improvement strategies and evidence-based solutions.

As a student in Queens University of Charlotte’s online Master of Science in Nursing program , you will solidify your skills in research and analysis allowing you to make informed, strategic decisions to drive measurable results for your patients.

Request more information to learn more about how this degree can improve your nursing practice, or call 866-313-2356.

Adapted from: Thomas, E. and Magilvy, J. K. (2011), Qualitative Rigor or Research Validity in Qualitative Research. Journal for Specialists in Pediatric Nursing, 16: 151–155. [WWW document]. URL  http://onlinelibrary.wiley.com/doi/10.1111/j.1744-6155.2011.00283.x  [accessed 2 July 2014]

Recommended Articles

Tips for nurse leaders to maintain moral courage amid ethical dilemmas, relationship between nursing leadership & patient outcomes, day in the life of a clinical nurse leader®, 8 leadership skills nurses need to be successful, the rise of medical errors in hospitals, get started.

The Ultimate Guide to Qualitative Research - Part 3: Presenting Qualitative Data

• Presenting qualitative data
• Data visualization
• Research paper writing
• Introduction

What is rigor in qualitative research?

Why is transparent research important, how do you achieve transparency and rigor in research.

• How to publish a research paper

Transparency and rigor in research

Qualitative researchers face particular challenges in convincing their target audience of the value and credibility of their subsequent analysis . Numbers and quantifiable concepts in quantitative studies are relatively easier to understand than their counterparts associated with qualitative methods . Think about how easy it is to make conclusions about the value of items at a store based on their prices, then imagine trying to compare those items based on their design, function, and effectiveness.

The goal of qualitative data analysis is to allow a qualitative researcher and their audience to make determinations about the value and impact of the research. Still, before the audience can reach these determinations, the process of conducting research that produces the qualitative analysis must first be perceived as credible. It is the responsibility of the researcher to persuade their audience that their data collection process and subsequent analysis are rigorous.

Qualitative rigor refers to the meticulousness, consistency, and transparency of the research. It is the application of systematic, disciplined, and stringent methods to ensure the credibility, dependability, confirmability, and transferability of research findings. In qualitative inquiry, these attributes ensure the research accurately reflects the phenomenon it is intended to represent, that its findings can be used by others, and that its processes and results are open to scrutiny and validation.

Credibility

Credibility refers to the extent to which the results accurately represent the participants' experiences. To achieve credibility, qualitative researchers, especially those conducting research on human research participants, employ a number of strategies to bolster the credibility of the data and the subsequent analysis. Prolonged engagement and persistent observation , for example, involve spending significant time in the field to gain a deep understanding of the research context and to continuously observe the phenomenon under study. Peer debriefing involves discussing the research and findings with knowledgeable peers to assess their validity . Member checking involves sharing the findings with the research participants to confirm that they accurately reflect their experiences. These and other methods ensure an abundantly rich data set from which the researcher describes in vivid detail the phenomenon under study, and which other scholars can audit to challenge the strength of the findings if necessary.

Dependability

Dependability refers to the consistency of the research process such that it is logical and clearly documented. It addresses the potential for others to build on the research through subsequent studies. To achieve dependability, researchers should provide a 'decision trail' detailing all decisions made during the course of the study. This allows others to understand how conclusions were reached and to replicate the study if necessary. Ultimately, the documentation of a researcher's process while collecting and analyzing data provides a clear record not only for other scholars to consider but also for those conducting the study and refining their methods for future research.

Confirmability

Confirmability requires the research findings to be directly linked to the data. While it is important to acknowledge researcher positionality (e.g., through reflexive memos) in social science research, researchers still have a responsibility to make assertions and identify insights rooted in the data for the resulting knowledge to be considered confirmable. By transparently communicating how the data was analyzed and conclusions were reached, researchers can allow their audience to perform a sort of audit of the study. This practice helps remind researchers about the importance of ensuring that there are sufficient connections to the raw data collected from the field and the findings that are presented as consequential developments of theory.

Transferability

Transferability refers to the applicability of the research findings in other contexts or with other participants. While dependability is more relevant to the application of research within its own situated context, transferability is determined by how findings generated in one set of circumstances (e.g., a geographic location or a culture) apply to another set of circumstances. This is essentially a significant challenge since, given the unique focus on context in qualitative research , researchers can't usually claim that their findings are universally applicable. Instead, they provide a rich, detailed description of the context and participants, enabling others to determine if the findings may apply to their own context. As a result, such detail necessitates discussion of transparency in research, which will be discussed later in this section.

Reflexivity

The concept of reflexivity also contributes to rigor in qualitative research. Reflexivity involves the researcher critically reflecting on the research and their own role in it, including how their biases , values, experiences, and presence may influence the research. Any discussion of reflexivity necessitates a recognition that knowledge about the social world is never objective, but always from a particular perspective. Reflexivity begins with an acknowledgment that those who conduct qualitative research do so while perceiving the social world through an analytical lens that is unique and distinct from that of others. As subjectivity is an inevitable circumstance in any research involving humans as sources or instruments of data collection , the researcher is responsible for providing a thick description of the environment in which they are collecting data as well as a detailed description of their own place in the research. Subjectivity can be considered as an asset, whereby researchers acknowledge and indicate how their positionality informed the analysis in ways that were insightful and productive.

Triangulation

Triangulation is another key aspect of rigor, referring to the use of multiple data sources, researchers, or methods to cross-check and validate findings. This can increase the depth and breadth of the research, improve its quality, and decrease the likelihood of researcher bias influencing the findings. Particularly given the complexity and dynamic nature of the social world, one method or one analytical approach will seldom be sufficient in holistically understanding the phenomenon or concept under study. Instead, a researcher benefits from examining the world through multiple methods and multiple analytical approaches, not to garner perfectly consistent results but to gather as much rich detail as possible to strengthen the analysis and subsequent findings.

In qualitative research , rigor is not about seeking a single truth or reality, but rather about being thorough, transparent, and critical in the research to ensure the integrity and value of the study. Rigor can be seen as a commitment to best practices in research, with researchers consistently questioning their methods and findings, checking for alternative interpretations , and remaining open to critique and revision. This commitment to rigor helps ensure that qualitative research provides valid, reliable , and meaningful contributions to our understanding of the complex social world.

Transparently document your research with ATLAS.ti

Choose ATLAS.ti for getting the most out of your data. Download a free trial of our software today.

When you read a story in a newspaper or watch a news report on television, do you ever get the feeling that you may not be receiving all the information or context necessary to understand the overarching messages being conveyed? Perhaps a salesperson is trying to convince you to buy something from them by explaining all the benefits of a product but doesn't tell you how they know these benefits are real. When you're choosing a movie to watch, you might look at a critic's review or a score in an online movie database without actually knowing how that review or score was actually determined.

In all of these situations, it is easier to trust the information presented to you if there is a rigorous analysis process behind that information and if that process is explicitly detailed. The same is true for qualitative research results, making transparency a key element in qualitative research methodologies . Transparency is a fundamental aspect of rigor in qualitative research. It involves the clear, detailed, and explicit documentation of all stages of the research process. This allows other researchers to understand, evaluate, transfer, and build upon the study. The key aspects of transparency in qualitative research include methodological transparency, analytical transparency, and reflexive transparency.

Methodological transparency involves providing a comprehensive description of the research methods and procedures used in the study. This includes detailing the research design, sampling strategy, data collection methods , and ethical considerations . For example, researchers should thoroughly describe how participants were selected, how and where data were collected (e.g., interviews , focus groups , observations ), and how ethical issues such as consent, confidentiality , and potential harm were addressed. They should also clearly articulate the theoretical and conceptual frameworks that guided the study. Methodological transparency allows other researchers to understand how the study was conducted and assess its trustworthiness.

Analytical transparency refers to the clear and detailed documentation of the data analysis process. This involves explaining how the raw data were transformed into findings, including the coding process , theme/category development, and interpretation of results . Researchers should describe the specific analytical strategies they used, such as thematic analysis , grounded theory , or discourse analysis . They should provide evidence to support their findings, such as direct quotes from participants. They may also describe any software they used to assist with analyzing data . Analytical transparency allows other researchers to understand how the findings were derived and assess their credibility and confirmability.

Reflexive transparency involves the researcher reflecting on and disclosing their own role in the research, including their potential biases , assumptions, and influences. This includes recognizing and discussing how the researcher's background, beliefs, and interactions with participants may have shaped the data collection and analysis . Reflexive transparency may be achieved through the use of a reflexivity journal, where the researcher regularly records their thoughts, feelings, and reactions during research. This aspect of transparency ensures that the researcher is open about their subjectivity and allows others to assess the potential impact of the researcher's positionality on the findings.

Transparency in qualitative research is essential for maintaining rigor, trustworthiness, and ethical integrity . By being transparent, researchers allow their work to be scrutinized, critiqued, and improved upon, contributing to the ongoing development and refinement of knowledge in their field.

Rigorous, trustworthy research is research that applies the appropriate research tools to meet the stated objectives of the investigation. For example, to determine if an exploratory investigation was rigorous, the investigator would need to answer a series of methodological questions: Do the data collection tools produce appropriate information for the level of precision required in the analysis ? Do the tools maximize the chance of identifying the full range of what there is to know about the phenomenon? To what degree are the collection techniques likely to generate the appropriate level of detail needed for addressing the research question(s) ? To what degree do the tools maximize the chance of producing data with discernible patterns?

Once the data are collected, to what degree are the analytic techniques likely to ensure the discovery of the full range of relevant and salient themes and topics? To what degree do the analytic strategies maximize the potential for finding relationships among themes and topics? What checks are in place to ensure that the discovery of patterns and models are relevant to the research question? Finally, what standards of evidence are required to ensure readers that results are supported by the data?

The clear challenge is to identify what questions are most important for establishing research rigor (trustworthiness) and to provide examples of how such questions could be answered for those using qualitative data . Clearly, rigorous research must be both transparent and explicit; in other words, researchers need to be able to describe to their colleagues and their audiences what they did (or plan to do) in clear, simple language. Much of the confusion that surrounds qualitative data collection and analysis techniques comes from practitioners who shroud their behaviors in mystery and jargon. For example, clearly describing how themes are identified, how codebooks are built and applied, and how models were induced helps to bring more rigor to qualitative research .

Researchers also must become more familiar with the broad range of methodological techniques available, such as content analysis , grounded theory , and discourse analysis . Cross-fertilization across methodological traditions can also be extremely valuable to generate meaningful understanding rather than attacking all problems with the same type of methodological tool.

The introduction of methodologically neutral and highly flexible qualitative analysis software like ATLAS.ti can be considered as extremely helpful indeed. It is highly apt to both support interdisciplinary cross-pollination and to bring about a great deal of trust in the presented results. By allowing the researcher to combine both the source material and his/her findings in a structured, interactive platform while producing both quantifiable reports and intuitive visual renderings of their results, ATLAS.ti adds new levels of trustworthiness to qualitative research . Moreover, it permits the researcher to apply multiple approaches to their research, to collaborate across philosophical boundaries, and thus significantly enhance the level of rigor in qualitative research. Dedicated research software like ATLAS.ti helps the researcher to catalog, explore and competently analyze the data generated in a given research project.

Ultimately, transparency and rigor are indispensable elements of any robust research study. Achieving transparency requires a systematic, deliberate, and thoughtful approach. It revolves around clarity in the formulation of research objectives, comprehensiveness in methods, and conscientious reporting of the results. Here are several key strategies for achieving transparency and rigor in research:

Clear research objectives and methods

Transparency begins with the clear and explicit statement of research objectives and questions. Researchers should explain why they are conducting the study, what they hope to learn, and how they plan to achieve their objectives. This involves identifying and articulating the study's theoretical or conceptual framework and delineating the key research questions . Ensuring clarity at this stage sets the groundwork for transparency throughout the rest of the study.

Transparent research includes a comprehensive and detailed account of the research design and methodology. Researchers should describe all stages of their research process, including the selection and recruitment of participants, the data collection methods , the setting of the research, and the timeline. Each step should be explained in enough detail that another researcher could replicate the study. Furthermore, any modifications to the research design or methodology over the course of the study should be clearly documented and justified.

Thorough data documentation and analysis

In the data collection phase, researchers should provide thorough documentation, including original data records such as transcripts , field notes , or images . The specifics of how data was gathered, who was involved, and when and where it took place should be meticulously recorded.

During the data analysis phase , researchers should clearly describe the steps taken to analyze the data, including coding processes , theme identification , and how conclusions were drawn. Researchers should provide evidence to support their findings and interpretations , such as verbatim quotes or detailed examples from the data. They should also describe any analytic software or tools used, including how they were used and why they were chosen.

Reflexivity and acknowledgment of bias

Transparent research involves a process of reflexivity , where researchers critically reflect on their own role in the research process. This includes considering how their own beliefs, values, experiences, and relationships with participants may have influenced the data collection and analysis . Researchers should maintain reflexivity journals to document these reflections, which can then be incorporated into the final research report. Researchers should also explicitly acknowledge potential biases and conflicts of interest that could influence the research. This includes personal, financial, or institutional interests that could affect the conduct or reporting of the research.

Transparent reporting and publishing

Transparency also involves the open sharing of research materials and data, where ethical and legal guidelines permit. This may include providing access to interview guides , survey instruments , data analysis scripts, raw data , and other research materials. Open sharing allows others to scrutinize, transfer, or extend the research, thereby enhancing its transparency and trustworthiness.

Finally, the reporting and publishing phase should adhere to the principles of transparency. Researchers should follow the relevant reporting guidelines for their field. Such guidelines provide a framework for reporting research in a comprehensive, systematic, and transparent manner.

Furthermore, researchers should choose to publish in open-access journals or other accessible formats whenever possible, to ensure the research is publicly accessible. They should also be open to critique and engage in post-publication discussion and debate about their findings.

By adhering to these strategies, researchers can ensure the transparency of their research, enhancing its credibility, trustworthiness, and contribution to their field. Transparency is more than just a good research practice—it's a fundamental ethical obligation to the research community, participants, and wider society.

Rigorous research starts with ATLAS.ti

Click here for a free trial of our powerful and intuitive data analysis software.

Criteria for Good Qualitative Research: A Comprehensive Review

• Regular Article
• Open access
• Published: 18 September 2021
• Volume 31 , pages 679–689, ( 2022 )

Cite this article

You have full access to this open access article

• Drishti Yadav   ORCID: orcid.org/0000-0002-2974-0323 1  

78k Accesses

28 Citations

72 Altmetric

Explore all metrics

This review aims to synthesize a published set of evaluative criteria for good qualitative research. The aim is to shed light on existing standards for assessing the rigor of qualitative research encompassing a range of epistemological and ontological standpoints. Using a systematic search strategy, published journal articles that deliberate criteria for rigorous research were identified. Then, references of relevant articles were surveyed to find noteworthy, distinct, and well-defined pointers to good qualitative research. This review presents an investigative assessment of the pivotal features in qualitative research that can permit the readers to pass judgment on its quality and to condemn it as good research when objectively and adequately utilized. Overall, this review underlines the crux of qualitative research and accentuates the necessity to evaluate such research by the very tenets of its being. It also offers some prospects and recommendations to improve the quality of qualitative research. Based on the findings of this review, it is concluded that quality criteria are the aftereffect of socio-institutional procedures and existing paradigmatic conducts. Owing to the paradigmatic diversity of qualitative research, a single and specific set of quality criteria is neither feasible nor anticipated. Since qualitative research is not a cohesive discipline, researchers need to educate and familiarize themselves with applicable norms and decisive factors to evaluate qualitative research from within its theoretical and methodological framework of origin.

Similar content being viewed by others

Good Qualitative Research: Opening up the Debate

Beyond qualitative/quantitative structuralism: the positivist qualitative research and the paradigmatic disclaimer.

What is Qualitative in Research

Avoid common mistakes on your manuscript.

Introduction

“… It is important to regularly dialogue about what makes for good qualitative research” (Tracy, 2010 , p. 837)

To decide what represents good qualitative research is highly debatable. There are numerous methods that are contained within qualitative research and that are established on diverse philosophical perspectives. Bryman et al., ( 2008 , p. 262) suggest that “It is widely assumed that whereas quality criteria for quantitative research are well‐known and widely agreed, this is not the case for qualitative research.” Hence, the question “how to evaluate the quality of qualitative research” has been continuously debated. There are many areas of science and technology wherein these debates on the assessment of qualitative research have taken place. Examples include various areas of psychology: general psychology (Madill et al., 2000 ); counseling psychology (Morrow, 2005 ); and clinical psychology (Barker & Pistrang, 2005 ), and other disciplines of social sciences: social policy (Bryman et al., 2008 ); health research (Sparkes, 2001 ); business and management research (Johnson et al., 2006 ); information systems (Klein & Myers, 1999 ); and environmental studies (Reid & Gough, 2000 ). In the literature, these debates are enthused by the impression that the blanket application of criteria for good qualitative research developed around the positivist paradigm is improper. Such debates are based on the wide range of philosophical backgrounds within which qualitative research is conducted (e.g., Sandberg, 2000 ; Schwandt, 1996 ). The existence of methodological diversity led to the formulation of different sets of criteria applicable to qualitative research.

Among qualitative researchers, the dilemma of governing the measures to assess the quality of research is not a new phenomenon, especially when the virtuous triad of objectivity, reliability, and validity (Spencer et al., 2004 ) are not adequate. Occasionally, the criteria of quantitative research are used to evaluate qualitative research (Cohen & Crabtree, 2008 ; Lather, 2004 ). Indeed, Howe ( 2004 ) claims that the prevailing paradigm in educational research is scientifically based experimental research. Hypotheses and conjectures about the preeminence of quantitative research can weaken the worth and usefulness of qualitative research by neglecting the prominence of harmonizing match for purpose on research paradigm, the epistemological stance of the researcher, and the choice of methodology. Researchers have been reprimanded concerning this in “paradigmatic controversies, contradictions, and emerging confluences” (Lincoln & Guba, 2000 ).

In general, qualitative research tends to come from a very different paradigmatic stance and intrinsically demands distinctive and out-of-the-ordinary criteria for evaluating good research and varieties of research contributions that can be made. This review attempts to present a series of evaluative criteria for qualitative researchers, arguing that their choice of criteria needs to be compatible with the unique nature of the research in question (its methodology, aims, and assumptions). This review aims to assist researchers in identifying some of the indispensable features or markers of high-quality qualitative research. In a nutshell, the purpose of this systematic literature review is to analyze the existing knowledge on high-quality qualitative research and to verify the existence of research studies dealing with the critical assessment of qualitative research based on the concept of diverse paradigmatic stances. Contrary to the existing reviews, this review also suggests some critical directions to follow to improve the quality of qualitative research in different epistemological and ontological perspectives. This review is also intended to provide guidelines for the acceleration of future developments and dialogues among qualitative researchers in the context of assessing the qualitative research.

The rest of this review article is structured in the following fashion: Sect.  Methods describes the method followed for performing this review. Section Criteria for Evaluating Qualitative Studies provides a comprehensive description of the criteria for evaluating qualitative studies. This section is followed by a summary of the strategies to improve the quality of qualitative research in Sect.  Improving Quality: Strategies . Section  How to Assess the Quality of the Research Findings? provides details on how to assess the quality of the research findings. After that, some of the quality checklists (as tools to evaluate quality) are discussed in Sect.  Quality Checklists: Tools for Assessing the Quality . At last, the review ends with the concluding remarks presented in Sect.  Conclusions, Future Directions and Outlook . Some prospects in qualitative research for enhancing its quality and usefulness in the social and techno-scientific research community are also presented in Sect.  Conclusions, Future Directions and Outlook .

For this review, a comprehensive literature search was performed from many databases using generic search terms such as Qualitative Research , Criteria , etc . The following databases were chosen for the literature search based on the high number of results: IEEE Explore, ScienceDirect, PubMed, Google Scholar, and Web of Science. The following keywords (and their combinations using Boolean connectives OR/AND) were adopted for the literature search: qualitative research, criteria, quality, assessment, and validity. The synonyms for these keywords were collected and arranged in a logical structure (see Table 1 ). All publications in journals and conference proceedings later than 1950 till 2021 were considered for the search. Other articles extracted from the references of the papers identified in the electronic search were also included. A large number of publications on qualitative research were retrieved during the initial screening. Hence, to include the searches with the main focus on criteria for good qualitative research, an inclusion criterion was utilized in the search string.

From the selected databases, the search retrieved a total of 765 publications. Then, the duplicate records were removed. After that, based on the title and abstract, the remaining 426 publications were screened for their relevance by using the following inclusion and exclusion criteria (see Table 2 ). Publications focusing on evaluation criteria for good qualitative research were included, whereas those works which delivered theoretical concepts on qualitative research were excluded. Based on the screening and eligibility, 45 research articles were identified that offered explicit criteria for evaluating the quality of qualitative research and were found to be relevant to this review.

Figure  1 illustrates the complete review process in the form of PRISMA flow diagram. PRISMA, i.e., “preferred reporting items for systematic reviews and meta-analyses” is employed in systematic reviews to refine the quality of reporting.

PRISMA flow diagram illustrating the search and inclusion process. N represents the number of records

Criteria for Evaluating Qualitative Studies

Fundamental criteria: general research quality.

Various researchers have put forward criteria for evaluating qualitative research, which have been summarized in Table 3 . Also, the criteria outlined in Table 4 effectively deliver the various approaches to evaluate and assess the quality of qualitative work. The entries in Table 4 are based on Tracy’s “Eight big‐tent criteria for excellent qualitative research” (Tracy, 2010 ). Tracy argues that high-quality qualitative work should formulate criteria focusing on the worthiness, relevance, timeliness, significance, morality, and practicality of the research topic, and the ethical stance of the research itself. Researchers have also suggested a series of questions as guiding principles to assess the quality of a qualitative study (Mays & Pope, 2020 ). Nassaji ( 2020 ) argues that good qualitative research should be robust, well informed, and thoroughly documented.

Qualitative Research: Interpretive Paradigms

All qualitative researchers follow highly abstract principles which bring together beliefs about ontology, epistemology, and methodology. These beliefs govern how the researcher perceives and acts. The net, which encompasses the researcher’s epistemological, ontological, and methodological premises, is referred to as a paradigm, or an interpretive structure, a “Basic set of beliefs that guides action” (Guba, 1990 ). Four major interpretive paradigms structure the qualitative research: positivist and postpositivist, constructivist interpretive, critical (Marxist, emancipatory), and feminist poststructural. The complexity of these four abstract paradigms increases at the level of concrete, specific interpretive communities. Table 5 presents these paradigms and their assumptions, including their criteria for evaluating research, and the typical form that an interpretive or theoretical statement assumes in each paradigm. Moreover, for evaluating qualitative research, quantitative conceptualizations of reliability and validity are proven to be incompatible (Horsburgh, 2003 ). In addition, a series of questions have been put forward in the literature to assist a reviewer (who is proficient in qualitative methods) for meticulous assessment and endorsement of qualitative research (Morse, 2003 ). Hammersley ( 2007 ) also suggests that guiding principles for qualitative research are advantageous, but methodological pluralism should not be simply acknowledged for all qualitative approaches. Seale ( 1999 ) also points out the significance of methodological cognizance in research studies.

Table 5 reflects that criteria for assessing the quality of qualitative research are the aftermath of socio-institutional practices and existing paradigmatic standpoints. Owing to the paradigmatic diversity of qualitative research, a single set of quality criteria is neither possible nor desirable. Hence, the researchers must be reflexive about the criteria they use in the various roles they play within their research community.

Improving Quality: Strategies

Another critical question is “How can the qualitative researchers ensure that the abovementioned quality criteria can be met?” Lincoln and Guba ( 1986 ) delineated several strategies to intensify each criteria of trustworthiness. Other researchers (Merriam & Tisdell, 2016 ; Shenton, 2004 ) also presented such strategies. A brief description of these strategies is shown in Table 6 .

It is worth mentioning that generalizability is also an integral part of qualitative research (Hays & McKibben, 2021 ). In general, the guiding principle pertaining to generalizability speaks about inducing and comprehending knowledge to synthesize interpretive components of an underlying context. Table 7 summarizes the main metasynthesis steps required to ascertain generalizability in qualitative research.

Figure  2 reflects the crucial components of a conceptual framework and their contribution to decisions regarding research design, implementation, and applications of results to future thinking, study, and practice (Johnson et al., 2020 ). The synergy and interrelationship of these components signifies their role to different stances of a qualitative research study.

Essential elements of a conceptual framework

In a nutshell, to assess the rationale of a study, its conceptual framework and research question(s), quality criteria must take account of the following: lucid context for the problem statement in the introduction; well-articulated research problems and questions; precise conceptual framework; distinct research purpose; and clear presentation and investigation of the paradigms. These criteria would expedite the quality of qualitative research.

How to Assess the Quality of the Research Findings?

The inclusion of quotes or similar research data enhances the confirmability in the write-up of the findings. The use of expressions (for instance, “80% of all respondents agreed that” or “only one of the interviewees mentioned that”) may also quantify qualitative findings (Stenfors et al., 2020 ). On the other hand, the persuasive reason for “why this may not help in intensifying the research” has also been provided (Monrouxe & Rees, 2020 ). Further, the Discussion and Conclusion sections of an article also prove robust markers of high-quality qualitative research, as elucidated in Table 8 .

Quality Checklists: Tools for Assessing the Quality

Numerous checklists are available to speed up the assessment of the quality of qualitative research. However, if used uncritically and recklessly concerning the research context, these checklists may be counterproductive. I recommend that such lists and guiding principles may assist in pinpointing the markers of high-quality qualitative research. However, considering enormous variations in the authors’ theoretical and philosophical contexts, I would emphasize that high dependability on such checklists may say little about whether the findings can be applied in your setting. A combination of such checklists might be appropriate for novice researchers. Some of these checklists are listed below:

The most commonly used framework is Consolidated Criteria for Reporting Qualitative Research (COREQ) (Tong et al., 2007 ). This framework is recommended by some journals to be followed by the authors during article submission.

Standards for Reporting Qualitative Research (SRQR) is another checklist that has been created particularly for medical education (O’Brien et al., 2014 ).

Also, Tracy ( 2010 ) and Critical Appraisal Skills Programme (CASP, 2021 ) offer criteria for qualitative research relevant across methods and approaches.

Further, researchers have also outlined different criteria as hallmarks of high-quality qualitative research. For instance, the “Road Trip Checklist” (Epp & Otnes, 2021 ) provides a quick reference to specific questions to address different elements of high-quality qualitative research.

Conclusions, Future Directions, and Outlook

This work presents a broad review of the criteria for good qualitative research. In addition, this article presents an exploratory analysis of the essential elements in qualitative research that can enable the readers of qualitative work to judge it as good research when objectively and adequately utilized. In this review, some of the essential markers that indicate high-quality qualitative research have been highlighted. I scope them narrowly to achieve rigor in qualitative research and note that they do not completely cover the broader considerations necessary for high-quality research. This review points out that a universal and versatile one-size-fits-all guideline for evaluating the quality of qualitative research does not exist. In other words, this review also emphasizes the non-existence of a set of common guidelines among qualitative researchers. In unison, this review reinforces that each qualitative approach should be treated uniquely on account of its own distinctive features for different epistemological and disciplinary positions. Owing to the sensitivity of the worth of qualitative research towards the specific context and the type of paradigmatic stance, researchers should themselves analyze what approaches can be and must be tailored to ensemble the distinct characteristics of the phenomenon under investigation. Although this article does not assert to put forward a magic bullet and to provide a one-stop solution for dealing with dilemmas about how, why, or whether to evaluate the “goodness” of qualitative research, it offers a platform to assist the researchers in improving their qualitative studies. This work provides an assembly of concerns to reflect on, a series of questions to ask, and multiple sets of criteria to look at, when attempting to determine the quality of qualitative research. Overall, this review underlines the crux of qualitative research and accentuates the need to evaluate such research by the very tenets of its being. Bringing together the vital arguments and delineating the requirements that good qualitative research should satisfy, this review strives to equip the researchers as well as reviewers to make well-versed judgment about the worth and significance of the qualitative research under scrutiny. In a nutshell, a comprehensive portrayal of the research process (from the context of research to the research objectives, research questions and design, speculative foundations, and from approaches of collecting data to analyzing the results, to deriving inferences) frequently proliferates the quality of a qualitative research.

Prospects : A Road Ahead for Qualitative Research

Irrefutably, qualitative research is a vivacious and evolving discipline wherein different epistemological and disciplinary positions have their own characteristics and importance. In addition, not surprisingly, owing to the sprouting and varied features of qualitative research, no consensus has been pulled off till date. Researchers have reflected various concerns and proposed several recommendations for editors and reviewers on conducting reviews of critical qualitative research (Levitt et al., 2021 ; McGinley et al., 2021 ). Following are some prospects and a few recommendations put forward towards the maturation of qualitative research and its quality evaluation:

In general, most of the manuscript and grant reviewers are not qualitative experts. Hence, it is more likely that they would prefer to adopt a broad set of criteria. However, researchers and reviewers need to keep in mind that it is inappropriate to utilize the same approaches and conducts among all qualitative research. Therefore, future work needs to focus on educating researchers and reviewers about the criteria to evaluate qualitative research from within the suitable theoretical and methodological context.

There is an urgent need to refurbish and augment critical assessment of some well-known and widely accepted tools (including checklists such as COREQ, SRQR) to interrogate their applicability on different aspects (along with their epistemological ramifications).

Efforts should be made towards creating more space for creativity, experimentation, and a dialogue between the diverse traditions of qualitative research. This would potentially help to avoid the enforcement of one's own set of quality criteria on the work carried out by others.

Moreover, journal reviewers need to be aware of various methodological practices and philosophical debates.

It is pivotal to highlight the expressions and considerations of qualitative researchers and bring them into a more open and transparent dialogue about assessing qualitative research in techno-scientific, academic, sociocultural, and political rooms.

Frequent debates on the use of evaluative criteria are required to solve some potentially resolved issues (including the applicability of a single set of criteria in multi-disciplinary aspects). Such debates would not only benefit the group of qualitative researchers themselves, but primarily assist in augmenting the well-being and vivacity of the entire discipline.

To conclude, I speculate that the criteria, and my perspective, may transfer to other methods, approaches, and contexts. I hope that they spark dialog and debate – about criteria for excellent qualitative research and the underpinnings of the discipline more broadly – and, therefore, help improve the quality of a qualitative study. Further, I anticipate that this review will assist the researchers to contemplate on the quality of their own research, to substantiate research design and help the reviewers to review qualitative research for journals. On a final note, I pinpoint the need to formulate a framework (encompassing the prerequisites of a qualitative study) by the cohesive efforts of qualitative researchers of different disciplines with different theoretic-paradigmatic origins. I believe that tailoring such a framework (of guiding principles) paves the way for qualitative researchers to consolidate the status of qualitative research in the wide-ranging open science debate. Dialogue on this issue across different approaches is crucial for the impending prospects of socio-techno-educational research.

Amin, M. E. K., Nørgaard, L. S., Cavaco, A. M., Witry, M. J., Hillman, L., Cernasev, A., & Desselle, S. P. (2020). Establishing trustworthiness and authenticity in qualitative pharmacy research. Research in Social and Administrative Pharmacy, 16 (10), 1472–1482.

Article   Google Scholar  

Barker, C., & Pistrang, N. (2005). Quality criteria under methodological pluralism: Implications for conducting and evaluating research. American Journal of Community Psychology, 35 (3–4), 201–212.

Bryman, A., Becker, S., & Sempik, J. (2008). Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy. International Journal of Social Research Methodology, 11 (4), 261–276.

Caelli, K., Ray, L., & Mill, J. (2003). ‘Clear as mud’: Toward greater clarity in generic qualitative research. International Journal of Qualitative Methods, 2 (2), 1–13.

CASP (2021). CASP checklists. Retrieved May 2021 from https://casp-uk.net/casp-tools-checklists/

Cohen, D. J., & Crabtree, B. F. (2008). Evaluative criteria for qualitative research in health care: Controversies and recommendations. The Annals of Family Medicine, 6 (4), 331–339.

Denzin, N. K., & Lincoln, Y. S. (2005). Introduction: The discipline and practice of qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), The sage handbook of qualitative research (pp. 1–32). Sage Publications Ltd.

Google Scholar  

Elliott, R., Fischer, C. T., & Rennie, D. L. (1999). Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology, 38 (3), 215–229.

Epp, A. M., & Otnes, C. C. (2021). High-quality qualitative research: Getting into gear. Journal of Service Research . https://doi.org/10.1177/1094670520961445

Guba, E. G. (1990). The paradigm dialog. In Alternative paradigms conference, mar, 1989, Indiana u, school of education, San Francisco, ca, us . Sage Publications, Inc.

Hammersley, M. (2007). The issue of quality in qualitative research. International Journal of Research and Method in Education, 30 (3), 287–305.

Haven, T. L., Errington, T. M., Gleditsch, K. S., van Grootel, L., Jacobs, A. M., Kern, F. G., & Mokkink, L. B. (2020). Preregistering qualitative research: A Delphi study. International Journal of Qualitative Methods, 19 , 1609406920976417.

Hays, D. G., & McKibben, W. B. (2021). Promoting rigorous research: Generalizability and qualitative research. Journal of Counseling and Development, 99 (2), 178–188.

Horsburgh, D. (2003). Evaluation of qualitative research. Journal of Clinical Nursing, 12 (2), 307–312.

Howe, K. R. (2004). A critique of experimentalism. Qualitative Inquiry, 10 (1), 42–46.

Johnson, J. L., Adkins, D., & Chauvin, S. (2020). A review of the quality indicators of rigor in qualitative research. American Journal of Pharmaceutical Education, 84 (1), 7120.

Johnson, P., Buehring, A., Cassell, C., & Symon, G. (2006). Evaluating qualitative management research: Towards a contingent criteriology. International Journal of Management Reviews, 8 (3), 131–156.

Klein, H. K., & Myers, M. D. (1999). A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly, 23 (1), 67–93.

Lather, P. (2004). This is your father’s paradigm: Government intrusion and the case of qualitative research in education. Qualitative Inquiry, 10 (1), 15–34.

Levitt, H. M., Morrill, Z., Collins, K. M., & Rizo, J. L. (2021). The methodological integrity of critical qualitative research: Principles to support design and research review. Journal of Counseling Psychology, 68 (3), 357.

Lincoln, Y. S., & Guba, E. G. (1986). But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. New Directions for Program Evaluation, 1986 (30), 73–84.

Lincoln, Y. S., & Guba, E. G. (2000). Paradigmatic controversies, contradictions and emerging confluences. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (2nd ed., pp. 163–188). Sage Publications.

Madill, A., Jordan, A., & Shirley, C. (2000). Objectivity and reliability in qualitative analysis: Realist, contextualist and radical constructionist epistemologies. British Journal of Psychology, 91 (1), 1–20.

Mays, N., & Pope, C. (2020). Quality in qualitative research. Qualitative Research in Health Care . https://doi.org/10.1002/9781119410867.ch15

McGinley, S., Wei, W., Zhang, L., & Zheng, Y. (2021). The state of qualitative research in hospitality: A 5-year review 2014 to 2019. Cornell Hospitality Quarterly, 62 (1), 8–20.

Merriam, S., & Tisdell, E. (2016). Qualitative research: A guide to design and implementation. San Francisco, US.

Meyer, M., & Dykes, J. (2019). Criteria for rigor in visualization design study. IEEE Transactions on Visualization and Computer Graphics, 26 (1), 87–97.

Monrouxe, L. V., & Rees, C. E. (2020). When I say… quantification in qualitative research. Medical Education, 54 (3), 186–187.

Morrow, S. L. (2005). Quality and trustworthiness in qualitative research in counseling psychology. Journal of Counseling Psychology, 52 (2), 250.

Morse, J. M. (2003). A review committee’s guide for evaluating qualitative proposals. Qualitative Health Research, 13 (6), 833–851.

Nassaji, H. (2020). Good qualitative research. Language Teaching Research, 24 (4), 427–431.

O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine, 89 (9), 1245–1251.

O’Connor, C., & Joffe, H. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19 , 1609406919899220.

Reid, A., & Gough, S. (2000). Guidelines for reporting and evaluating qualitative research: What are the alternatives? Environmental Education Research, 6 (1), 59–91.

Rocco, T. S. (2010). Criteria for evaluating qualitative studies. Human Resource Development International . https://doi.org/10.1080/13678868.2010.501959

Sandberg, J. (2000). Understanding human competence at work: An interpretative approach. Academy of Management Journal, 43 (1), 9–25.

Schwandt, T. A. (1996). Farewell to criteriology. Qualitative Inquiry, 2 (1), 58–72.

Seale, C. (1999). Quality in qualitative research. Qualitative Inquiry, 5 (4), 465–478.

Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22 (2), 63–75.

Sparkes, A. C. (2001). Myth 94: Qualitative health researchers will agree about validity. Qualitative Health Research, 11 (4), 538–552.

Spencer, L., Ritchie, J., Lewis, J., & Dillon, L. (2004). Quality in qualitative evaluation: A framework for assessing research evidence.

Stenfors, T., Kajamaa, A., & Bennett, D. (2020). How to assess the quality of qualitative research. The Clinical Teacher, 17 (6), 596–599.

Taylor, E. W., Beck, J., & Ainsworth, E. (2001). Publishing qualitative adult education research: A peer review perspective. Studies in the Education of Adults, 33 (2), 163–179.

Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19 (6), 349–357.

Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative Inquiry, 16 (10), 837–851.

Download references

Open access funding provided by TU Wien (TUW).

Author information

Authors and affiliations.

Faculty of Informatics, Technische Universität Wien, 1040, Vienna, Austria

Drishti Yadav

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Drishti Yadav .

Ethics declarations

Conflict of interest.

The author declares no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Yadav, D. Criteria for Good Qualitative Research: A Comprehensive Review. Asia-Pacific Edu Res 31 , 679–689 (2022). https://doi.org/10.1007/s40299-021-00619-0

Download citation

Accepted : 28 August 2021

Published : 18 September 2021

Issue Date : December 2022

DOI : https://doi.org/10.1007/s40299-021-00619-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative research
• Evaluative criteria
• Find a journal
• Publish with us
• Track your research

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

20.1 Introduction to qualitative rigor

We hear a lot about fake news these days. Fake news has to do with the quality of journalism that we are consuming. It begs questions like: does it contain misinformation, is it skewed or biased in its portrayal of stories, does it leave out certain facts while inflating others. If we take this news at face value, our opinions and actions may be intentionally manipulated by poor quality information. So, how do we avoid or challenge this? The oversimplified answer is, we find ways to check for quality. While this isn’t a chapter dedicated to fake news, it does offer an important comparison for the focus of this chapter, rigor in qualitative research. Rigor is concerned with the quality of research that we are designing and consuming. While I devote a considerable amount of time in my clinical class talking about the importance of adopting a non-judgmental stance in practice, that is not the case here; I want you to be judgmental, critical thinkers about research! As a social worker who will hopefully be producing research (we need you!) and definitely consuming research, you need to be able to differentiate good science from rubbish science. Rigor will help you to do this.

This chapter will introduce you to the concept of rigor and specifically, what it looks like in qualitative research. We will begin by considering how rigor relates to issues of ethics and how thoughtfully involving community partners in our research can add additional dimensions in planning for rigor. Next, we will look at rigor in how we capture and manage qualitative data, essentially helping to ensure that we have quality raw data to work with for our study. Finally, we will devote time to discussing how researchers, as human instruments, need to maintain accountability throughout the research process. Finally, we will examine tools that encourage this accountability and how they can be integrated into your research design. Our hope is that by the end of this chapter, you will begin to be able to identify some of the hallmarks of quality in qualitative research, and if you are designing a qualitative research proposal, that you consider how to build these into your design.

19.1 Introduction to qualitative rigor

Learning objectives.

Learners will be able to…

• Identify the role of rigor in qualitative research and important concepts related to qualitative rigor
• Discuss why rigor is an important consideration when conducting, critiquing and consuming qualitative research
• Differentiate between quality in quantitative and qualitative research studies

In Chapter 11 we talked about quality in quantitative studies, but we built our discussion around concepts like reliability and validity . With qualitative studies, we generally think about quality in terms of the concept of rigor . The difference between quality in quantitative research and qualitative research extends beyond the type of data (numbers vs. words/sounds/images). If you sneak a peek all the way back to Chapter 7 , we discussed the idea of different paradigms or fundamental frameworks for how we can think about the world. These frameworks value different kinds of knowledge, arrive at knowledge in different ways, and evaluate the quality of knowledge with different criteria. These differences are essential in differentiating qualitative and quantitative work.

Quantitative research generally falls under a positivist paradigm, seeking to uncover knowledge that holds true across larger groups of people. To accomplish this, we need to have tools like reliability and validity to help produce internally consistent and externally generalizable findings (i.e. was our study design dependable and do our findings hold true across our population).

In contrast, qualitative research is generally considered to fall into an alternative paradigm (other than positivist), such as the interpretive paradigm which is focused on the subjective experiences of individuals and their unique perspectives. To accomplish this, we are often asking participants to expand on their ideas and interpretations. A positivist tradition requires the information collected to be very focused and discretely defined (i.e. closed questions with prescribed categories). With qualitative studies, we need to look across unique experiences reflected in the data and determine how these experiences develop a richer understanding of the phenomenon we are studying, often across numerous perspectives.

Rigor is a concept that reflects the quality of the process used in capturing, managing, and analyzing our data as we develop this rich understanding. Rigor helps to establish standards through which qualitative research is critiqued and judged, both by the scientific community and by the practitioner community.

For the scientific community, people who review qualitative research studies submitted for publication in scientific journals or for presentations at conferences will specifically look for indications of rigor, such as the tools we will discuss in this chapter. This confirms for them that the researcher(s) put safeguards in place to ensure that the research took place systematically and that consumers can be relatively confident that the findings are not fabricated and can be directly connected back to the primary sources of data that was gathered or the secondary data that was analyzed.

As a note here, as we are critiquing the research of others or developing our own studies, we also need to recognize the limitations of rigor. No research design is flawless and every researcher faces limitations and constraints. We aren’t looking for a researcher to adopt every tool we discuss below in their design. In fact, one of my mentors, speaks explicitly about “misplaced rigor”, that is, using techniques to support rigor that don’t really fit what you are trying to accomplish with your research design. Suffice it to say that we can go overboard in the area of rigor and it might not serve our study’s best interest. As a consumer or evaluator of research, you want to look for steps being taken to reflect quality and transparency throughout the research process, but they should fit within the overall framework of the study and what it is trying to accomplish.

From the perspective of a practitioner, we also need to be acutely concerned with the quality of research. Social work has made a commitment, outlined in our Code of Ethics (NASW,2017) , to competent practice in service to our clients based on “empirically based knowledge” (subsection 4.01). When I think about my own care providers, I want them to be using “good” research—research that we can be confident was conducted in a credible way and whose findings are honestly and clearly represented. Don’t our clients deserve the same from us?

As providers, we will be looking to qualitative research studies to provide us with information that helps us better understand our clients, their experiences, and the problems they encounter. As such, we need to look for research that accurately represents:

• Who is participating in the study
• What circumstances is the study being conducted under
• What is the research attempting to determine

Further, we want to ensure that:

• Findings are presented accurately and reflect what was shared by participants ( raw data )
•  A reasonably good explanation of how the researcher got from the raw data to their findings is presented
• The researcher adequately considered and accounted for their potential influence on the research process

As we talk about different tools we can use to help establish qualitative rigor, I will try to point out tips for what to look for as you are reading qualitative studies that can reflect these. While rigor can’t “prove” quality, it can demonstrate steps that are taken that reflect thoughtfulness and attention on the part of the researcher(s). This is a link from the American Psychological Association on the topic of reviewing qualitative research manuscripts. It’s a bit beyond the level of critiquing that I would expect from a beginning qualitative research student, however, it does provide a really nice overview of this process. Even if you aren’t familiar with all the terms, I think it can be helpful in giving an overview of the general thought process that should be taking place.

To begin breaking down how to think about rigor, I find it helpful to have a framework to help understand different concepts that support or are associated with rigor. Lincoln and Guba (1985) have suggested such a framework for thinking about qualitative rigor that has widely contributed to standards that are often employed for qualitative projects. The overarching concept around which this framework is centered is trustworthiness . Trustworthiness is reflective of how much stock we should put in a given qualitative study—is it really worth our time, headspace, and intellectual curiosity? A study that isn’t trustworthy suggests poor quality resulting from inadequate forethought, planning, and attention to detail in how the study was carried out. This suggests that we should have little confidence in the findings of a study that is not trustworthy.

According to Lincoln and Guba (1985) [1] trustworthiness is grounded in responding to four key ideas and related questions to help you conceptualize how they relate to your study. Each of these concepts is discussed below with some considerations to help you to compare and contrast these ideas with more positivist or quantitative constructs of research quality.

Truth value

You have already been introduced to the concept of internal validity . As a reminder, establishing internal validity is a way to ensure that the change we observe in the dependent variable is the result of the variation in our independent variable—did we actually design a study that is truly testing our hypothesis. In much/most qualitative studies we don’t have hypotheses, independent or dependent variables, but we do still want to design a study where our audience (and ourselves) can be relatively sure that we as the researcher arrived at our findings through a systematic and scientific process, and that those findings can be clearly linked back to the data we used and not some fabrication or falsification of that data; in other words, the truth value of the research process and its findings. We want to give our readers confidence that we didn’t just make up our findings or “see what we wanted to see”.

Applicability

• who we were studying
• how we went about studying them
• what we found

Consistency

These concepts reflect a set of standards that help to determine the integrity of qualitative studies. At the end of this chapter you will be introduced to a range of tools to help support or reflect these various standards in qualitative research. Because different qualitative designs (e.g. phenomenology, narrative, ethnographic), that you will learn more about in Chapter 22 emphasize or prioritize different aspects of quality, certain tools will be more appropriate for these designs. Since this chapter is intended to give you a general overview of rigor in qualitative studies, exploring additional resources will be necessary to best understand which of these concepts are prioritized in each type of design and which tools best support them.

Key Takeaways

• Qualitative research is generally conducted within an interpretativist paradigm. This differs from the post-positivist paradigm in which most quantitative research originates. This fundamental difference means that the overarching aim of these different approaches to knowledge building differ, and consequently, our standards for judging the quality of research within these paradigms differ.
• Assessing the quality of qualitative research is important, both from a researcher and a practitioner perspective. On behalf of our clients and our profession, we are called to be critical consumers of research. To accomplish this, we need strategies for assessing the scientific rigor with which research is conducted.
• Trustworthiness and associated concepts, including credibility, transferablity, dependability and confirmability, provide a framework for assessing rigor or quality in qualitative research.
• Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry . Newberry Park, CA: Sage ↵

Rigor is the process through which we demonstrate, to the best of our ability, that our research is empirically sound and reflects a scientific approach to knowledge building.

The degree to which an instrument reflects the true score rather than error.  In statistical terms, reliability is the portion of observed variability in the sample that is accounted for by the true variability, not by error. Note : Reliability is necessary, but not sufficient, for measurement validity.

The extent to which the scores from a measure represent the variable they are intended to.

a paradigm guided by the principles of objectivity, knowability, and deductive logic

Findings form a research study that apply to larger group of people (beyond the sample). Producing generalizable findings requires starting with a representative sample.

a paradigm based on the idea that social context and interaction frame our realities

in a literature review, a source that describes primary data collected and analyzed by the author, rather than only reviewing what other researchers have found

Data someone else has collected that you have permission to use in your research.

unprocessed data that researchers can analyze using quantitative and qualitative methods (e.g., responses to a survey or interview transcripts)

Trustworthiness is a quality reflected by qualitative research that is conducted in a credible way; a way that should produce confidence in its findings.

Ability to say that one variable "causes" something to happen to another variable. Very important to assess when thinking about studies that examine causation such as experimental or quasi-experimental designs.

The level of confidence that research is obtained through a systematic and scientific process and that findings can be clearly connected to the data they are based on (and not some fabrication or falsification of that data).

The ability to apply research findings beyond the study sample to some broader population,

This is a synonymous term for generalizability - the ability to apply the findings of a study beyond the sample to a broader population.

The potential for qualitative research findings to be applicable to other situations or with other people outside of the research study itself.

Consistency is the idea that we use a systematic (and potentially repeatable) process when conducting our research.

a single truth, observed without bias, that is universally applicable

one truth among many, bound within a social and cultural context

The idea that qualitative researchers attempt to limit or at the very least account for their own biases, motivations, interests and opinions during the research process.

Doctoral Research Methods in Social Work Copyright © by Mavs Open Press. All Rights Reserved.

Share This Book

• Search Menu
• Advance Articles
• Editor's Choice
• Author Guidelines
• Submission Site
• Open Access
• About Journal of Public Administration Research and Theory
• About the Public Management Research Association
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, where qualitative methods shine, forget qualitative versus quantitative, rigor: the point of departure, deductive, mixed, and hybrid qualitative methods.

• < Previous

A Reviewer’s Guide to Qualitative Rigor

• Article contents
• Figures & tables
• Supplementary Data

Branda Nowell, Kate Albrecht, A Reviewer’s Guide to Qualitative Rigor, Journal of Public Administration Research and Theory , Volume 29, Issue 2, April 2019, Pages 348–363, https://doi.org/10.1093/jopart/muy052

• Permissions Icon Permissions

Institutions are useful for advancing methodologies within disciplines. Through required coursework, doctoral students are indoctrinated into basic guidelines and frameworks that provide a common foundation for scholars to interact with one another. Lacking such forums in many of our doctoral granting institutions ( Stout 2013 ), the field of public management continues to struggle with an ambivalence toward qualitative approaches. Lack of shared understanding concerning basic tenets of qualitative methodology abounds. This article is intended for qualitative consumers, those not formally trained in qualitative methods but who serve as peer reviewers, content experts, and advisors in arenas where qualitative methods are encountered. Adopting a postpositivistic stance dominant in the field, we seek to offer a pragmatic perspective on qualitative methods with regards to some basic tenets of rigor appropriate (and inappropriate) for assessing the contribution of qualitative research. We argue that the first step in this effort is to stop conflating data type (qualitative versus quantitative) with inductive versus deductive modes of inquiry. Using deductive modes as the basis for comparison, we discuss both common, as well as, diverging criteria of quality and rigor for inductive modes of inquiry. We conclude with a discussion of rigor in emerging methods which utilize qualitative data but from within a deductive, mixed, or hybrid mode of inquiry.

The field of public management continues to have a rocky relationship with qualitative methods. Like most methods, qualitative research has both its champions and its critics in the field. However, it is our sense that the majority of the field sits somewhere a bit right of center, open to a discussion but still suspect of what to do with findings from any study consisting of a small unrepresentative sample and no standard error. Much of this stems from fundamental misunderstandings about what qualitative inquiry is, and is not, designed to do. The cost of this to our discipline is significant. In a recent review, Ospina and colleagues (2017) reported only 7.5% of the articles published in top PA journals over the past 5 years relied solely on qualitative methods. This is not particularly surprising as our doctoral training institutions allow graduates to remain largely uninformed about qualitative approaches ( Stout 2013 ). However, there are many questions germane to our discipline that are best suited to qualitative inquiry (for discussion, see Brower, Abolafia, and Carr 2000 ; Milward forthcoming , Ospina et al. 2017 ). In order to advance the contribution qualitative methods can make to the field, some foundational understanding about qualitative rigor is needed.

In embarking on this effort, we join an esteemed cadre of scholars who have grappled with the issue of qualitative rigor in public management (e.g., Brower et al. 2000 ; Dodge et al. 2005 ; Lowery and Evans 2004 ; Ospina et al. 2017 ). However, we seek a very specific audience. This is not an article written for the initiated qualitative scholar; we are not seeking to offer advancements in qualitative techniques or further the discourse on the precepts of qualitative inquiry. Nor is this an article particularly aimed at the edification of the novice qualitative scholar looking to embark upon qualitative inquiry for the first time; there are many excellent texts out there that deal with the issues contained in this article in a much more thorough manner. Rather, this article was conceptualized and written primarily for the qualitative consumer who, at present, represents the over-whelming majority in the field of public management.

As we are envisioning our intended audience, three general categories of consumers come to mind. First, this article is for the quantitatively trained peer reviewer who finds themselves asked to assess the quality and contribution of a qualitative study brought to them for review. These folks serve as the gatekeepers and a quality assurance mechanism critical to the advancement of the discipline. Second, this article is for the scholar reviewing the literature within a content domain populated by both qualitative and quantitative studies. If we want qualitative research to have a greater substantive impact on the discipline, we need to give non-qualitatively trained scholars the tools to assess the contribution of qualitative research within their own research paradigm. Otherwise, citations will inevitably trend into methodological silos. Finally, this article is written for the quantitatively trained professor who finds themselves on a committee trying to support a student pursuing a qualitative or mixed method dissertation. We have a beloved colleague who routinely asks students whether their dissertations are going to be empirical or qualitative. Her intent is not to be pejorative; she simply has no frame of reference for how to think about quotations as data.

A Brief Note on Epistemology

We recognize that the writing of this article requires the adoption of some normative stances linked to the philosophy of science; namely, an epistemological stance that is primarily postpositivist in nature. We have intentionally deviated from normative practice in qualitative scholarship in minimizing our discussion of epistemology (for further discussions, see Creswell 2018 ; Creswell and Miller 2000 ; Raadschelders 2011 ; Riccucci 2010 ). This is not because we do not appreciate the value and relevance of alternative epistemological stances for the field of public management. However, many methods associated with qualitative rigor can be applied across different epistemological stances, varying in intention and orientation rather than practical execution. 1 For example, Lincoln and Guba’s (1986) criteria of trustworthiness are useful. This is true regardless of whether you are utilizing those practices because you believe in the postpositivistic limitations of humans to fully comprehend social processes present in natural settings, or because you believe these social processes are co-constructed in an inseparable relationship between the researcher and the participant. In a similar way, reflexivity 2 is relevant to both the postpositivist as well as the interpretivists regardless of whether you embrace the inseparability between the knower and knowledge (constructivism) or just view humans as fallible in part because they cannot fully and objectively separate who they are from the questions they ask and the answers they find (postpositivism; Guba 1990 ).

In this paper, we seek to offer a pragmatic perspective on qualitative inquiry with a focus on how to conceptualize and assess quality and rigor within a postpositivistic framework; the dominant philosophical stance of most qualitative consumers within public management. We do this with the aim of widening the pathways through which qualitative studies might influence scholarship in public management. We recognize that such an endeavor may be highly controversial within some branches of the qualitative community which maintain a strong allegiance to advancing a constructivist philosophy of science (e.g., Carter and Little 2007 ; Rolfe 2006 ). However, we argue it is neither reasonable nor necessary for qualitative consumers to suspend their fundamental view of reality in order to appreciate and assess the contribution of qualitative work to the broader field. There is a rich history of the integration of qualitative research within postpositivism (e.g., Clark 1998 ; Glaser and Strauss 2017 ; Prasad 2015 ; Yin 2017 ), particularly in the organizational sciences ( Eisenhardt and Graebner 2007 ).

We do not foresee a reconciliation between constructivism and postpositivist philosophies occurring any time soon. However, we do see sizable opportunity for naturalistic, inductive qualitative inquiry to have a broader impact in the field of public management if we start from the perspective that both qualitative and quantitative methods are compatible and complementary buckets of tools within social science. Different tools are best suited for different jobs and there is almost as much variation within each bucket as there is between them. Regardless, the world is an increasingly complex place. As a discipline that routinely trudges off into some really messy domains of inquiry and holds itself accountable to informing practice as well as advancing theory ( Brooks 2002 ; Denhardt 2001 ; Gill and Meier 2000 ; Head 2010 ; Weber and Khademian 2008 ), we need every tool we can get.

To address building this toolbox for qualitative consumers, we present first an overview of critical domains of inquiry in the field of public management where we see qualitative methods as being particularly well suited to advancing scholarship. This review highlights some of the most cited and celebrated theories of our field that have been initially shaped or meaningfully re-imagined from qualitative approaches. Next, we argue for a reframing of the question of qualitative rigor, asserting the more productive distinction lies in differentiating inductive versus deductive modes of inquiry. Leveraging this perspective, we discuss both commonalities and points of departure in appropriate criteria of quality and rigor between deductive versus inductive models. Finally, we discuss issues of rigor in three emerging methods in public management that use qualitative data in deductive, mixed and hybrid models of inquiry.

If qualitative methods are viewed as a category of tools, it is relevant to next consider some of the functionality one maximizes through the use of such tools. Although this list is not exhaustive, it is intended to provide a general grounding into the types of situations where qualitative approaches are particularly well equipped to make a contribution to the field of public management.

Advancing New Theory and Discovering Nuance in Existing Theory

Quantitative hypothesis testing requires a priori theory. Arbitrarily searching for significant correlations between variables in a dataset without a theoretically grounded hypothesis to direct the analysis is infamously problematic for well-documented reasons ( Kuhn 1996 ; Steiner 1988 ). Theory is a combination of a premise as well as a well-explicated mechanism that explains the why behind the premise or proposition.

Cross-sectional quantitative designs can test the strength and nature of association between two or more constructs. Longitudinal quantitative designs can examine the patterning of association over time, and experimental designs can even narrow in on causality. These are powerful tools, but none are well equipped to discover the mechanisms by which these observed patterns are operating or identifying intervening factors that explain inconsistencies across cases. We use existing theory to infer the mechanism associated with an observed pattern but this is generally not an empirical exercise, it is a conceptual one. Further, it often requires the extrapolation of theoretical mechanisms conceptualized in one organizational context (e.g., private firms) to be applied in a completely different organizational context (e.g., public organizations). When the hypothesized association holds, we generally conclude that the same mechanisms are in operation in the same manner. How critically do we look at this assumption? What else might be going on? Qualitative methods offer tools specifically designed to empirically shed light on these questions.

Qualitative methods are particularly useful in the theory development process because they are able to provide detailed description of a phenomenon as it occurs in context. These methods do not require the scholar to guess in advance the most important factors and their relationship to each other. Mechanisms associated with the co-occurrence of two phenomena can be observed in real time or described by first hand informants who experienced it. For example, Feldman’s (e.g. Feldman 2000 ; Feldman and Pentland 2003 ) seminal work on the role of routines as sources of change and innovation in organizations was based on organizational ethnography. Some other classic examples of theory development in public management that began as qualitative research can be found in organizational culture and sense making case studies ( Schein 2003 ; Weick 1993 ). Toepler’s (2005) case study of a CEO in crisis, the phenomena of iron triangles ( Freeman 1965 ), and the social construction of target populations ( Schneider and Ingram 1993 ) are also illustrations of theoretical advances through qualitative inquiry. Additionally, a major contribution to theory of both formal and informal accountability in the public sector and multi-sector collaboration was a direct result of a grounded theory qualitative approach ( Romzek and Dubnick 1987 ; Romzek, LeRoux, and Blackmar 2012 ; Romzek et al. 2014 ). All of these examples leverage a qualitative researcher’s ability to harness an inductive approach that allows for the emergence of our understanding of the nature of phenomena from those organizations and people who experienced it.

Beyond advancing new theories, qualitative methods have a strong tradition of clarifying and expanding upon existing theory. Underpinning many public management research areas is the ever-present politics-administration dichotomy. Maynard-Moody and Kelly’s (1993) foundational piece used a phenomenological approach to present the views of public workers who must navigate their administrative and political responsibilities every day. Agency and stewardship theories have also been examined and further delineated using qualitative methods ( Schillemans 2013 ). Theories of goal-directed networks and managerial tensions around unity and diversity have been expanded through qualitative studies ( Saz-Carranza and Ospina 2010 ). Finally, the nature of public participation has been theorized and statistically tested, but along the way the notion of authentic engagement—described as “deep and continuous involvement…with the potential for all involved to have an effect on the situation” (p. 320) was introduced to clarify theories, in part as a result of King et al.’s (1998) qualitative study.

Developing New Constructs, Frameworks, and Typologies

Quantitative hypothesis testing and construct validation requires the conceptualization and suggested operationalization of a construct. The development or usage of a new measure is aptly treated with skepticism if it is not empirically and theoretically grounded. In this way, many variables that we quantitatively leverage could not exist without prior development through qualitative research. For example, a foundational idea, and the basis for subsequent quantitative considerations of the differences between managers and front-line workers, is rooted in Lipsky’s (1971) examination and discussion of the street-level bureaucrat. Drawing from case studies and interviews, Lipsky highlights the nature of front-line worker discretion and challenges public management scholars to include this important context in future research.

Public Service Motivation (PSM), public management’s very own and home-grown construct, was born from Perry and Wise’s (1990) discussion citing both cases and quotes from public servants. Their argument for PSM to be more fully operationalized and then measured is rooted in their content analysis. Although they do not explicitly state the qualitative nature of their article, their argument for, and legacy of PSM scale measures, is drawn directly from the words and actions of public servants themselves.

Defining Mechanisms Underlying Statistical Associations

Although some quantitative articles do include mechanisms in their discussion sections, many simply rehash results and what hypotheses were or were not supported. Indeed, quantitative research in public management gives considerable weight to well-documented statistical association, even when the causal mechanism is ambiguous. In this world, how then do mechanisms get clarified when an association is found? This is an area where qualitative researchers have been working with less recognition of the importance of their research striving to answer “how” and “why” questions. The literature mentioned here again is not an exhaustive list, but emblematic of some prime examples of how our field’s understanding of a statistical result has been given more texture and a much richer application to both theory and practice through qualitative methods.

In the area of government contracting, Dias and Maynard-Moody (2007) further examine past quantitative findings that turn on Transaction Cost Economics (TCE) ( Williamson 1981 ) by explicating how and why implementing competing contracting philosophies of agencies and service providers underpins the nature of the transaction itself. Another qualitative piece examining the deeper mechanisms behind TCE is Van Slyke’s (2003) discussion of the “mythology” of contracting. In his research, data from semi-structured interviews suggests competition is not a simple construct in testing TCE interactions between governments and service providers because of the nature of environmental constraints, actions by nonprofit organizations, networked relationships, and government-enacted barriers have important dynamics. Honig (2018) offers another apt example in a mixed method study in which he demonstrates how comparative case study designs can reveal insights about the role of the environment in moderating the relationship between managerial control and success that were not possible to capture through quantitative modeling.

We have observed many scholars get conceptually hung up on the numbers versus text dichotomy associated with qualitative versus quantitative traditions. Although it is true that qualitative methods generally involve the analysis of some form of text and quantitative methods always involve the analysis of numbers, this focus on data type is largely a distraction from the more important distinction of inductive versus deductive forms of inquiry ( Eisenhardt and Graebner 2007 ). Deductive approaches to inquiry start with a general premise or proposition and then investigate whether this premise holds within a specific sample intended to represent a broader population. Inductive approaches start with a specific case or set of cases of theoretical importance and seek to describe a phenomenon of interest within that case in such a manner as to draw rich insight into that phenomenon (for discussion, see Eisenhardt and Graebner 2007 ; McNabb 2014 ). Although there are a handful of qualitative and/or hybrid qualitative/quantitative methods intended for deductive inquiry (more on this below), the bulk of tools in the qualitative bucket are intended for inductive inquiry.

Overarching Principles of Quality Between Inductive and Deductive Inquiry

Before we get into differences, it is important to first consider similarities. Although inductive and deductive traditions of scholarship differ in many important respects, they also share some commonalities that form the mutual basis of considerations of quality in terms of assessing their contribution to the literature. In our exuberance to elaborate their differences, we can forget what these forms of inquiry can hold in common. We argue that inductive and deductive approaches share in common three core values that are foundational to the notion of quality scholarship in public management: 1) the importance of scholarship that advances theory, 2) the principle of inquiry-driven design, and 3) the criticality of gap-driven inquiry.

Relevance of Scholarship for Advancing Theory

In public management, our focus is to inform practice as well as advance theory ( Kettl 2000 ). As a result, we give the greatest currency to knowledge that has relevance beyond the boundaries of the specific case, individual, or instance. Thus, within our field, the degree to which findings can have relevance beyond the study case or sample is foundational to conceptualizations of quality regardless of inductive or deductive approach ( Dubnick 1999 ). Inductive scholarship, different from most deductive studies, allows for a plurality of truths and an equifinality of pathways to the same outcome ( Eisenhardt, Graebner, and Sonenshein 2016 ), but the same standards of quality still apply. In other words, in inductive approaches, one need not argue an observed finding is the only explanation for a given outcome observed in another space or time, but it must be a plausible explanation for a similar outcome given a similar set of circumstances ( Holland 1986 ; Lewis 1973 ).

As such, both inductive and deductive studies are in the same boat of trying to figure out the extent to which and ways in which their limited study has broader implications for the field. The criteria and processes used to establish this element of quality certainly differs, but the precept that findings must have relevance beyond the scope of the data analyzed is common to both qualitative and quantitative scholarship in the field of public management ( McNabb 2015 ).

Inquiry-Driven Design

Both inductive and deductive traditions are inquiry driven. This means that evaluating the quality of any design—qualitative or quantitative—is inseparable from understanding the research question the study is designed to address. It is possible to hammer a nail with a screwdriver, but it is not considered good practice as you are likely to just make a mess of it. In the same way, different research questions are more or less appropriate to different designs. Thus, while it is possible to attempt to describe the different ways in which people experience transformational leadership with an exploratory survey or use a series of focus groups to examine the relative prevalence of public service motivation among different groups, it is not a good practice as you are likely to just make a mess of it.

A common misconception is that inductive qualitative methods seek to ask and answer the same questions as quantitative methods, just using different types of data and standards of rigor. This is not the case. Inductive approaches are designed to pose and address fundamentally different kinds of questions that necessitate different types of data and criteria of rigor. However, methodological appropriateness ( Haverland and Yanow 2012 ), or using the right tool for the job, is a value common to both inductive and deductive traditions and a key element of quality for all public management scholarship.

Gap-Driven Inquiry

Both inductive and deductive traditions recognize that knowledge does not advance in isolation—it takes a community of scholars to build a body of knowledge ( Kuhn 1996 ; Gill and Meier 2000 ). The practice of positioning a research question in terms of its relevance within broader conversations that are taking place within the literature is mainstream to both traditions ( McNabb 2015 ). In the field of public management—as elsewhere—the greatest currency is given to studies that clearly identify and address a significant gap within the literature; we seek to investigate something overlooked, under-appreciated, or potentially misunderstood in our current understanding of a given phenomenon. The extent to which a study accomplishes such a contribution is a shared element of quality for both deductive and inductive traditions.

In the previous section, we have argued that inductive and deductive approaches in public management share a common foundation in conceptualizing the quality of inquiry. Specifically, we suggest quality can be conceptualized as inquiry that addresses a significant gap in the literature in a manner that advances our general understanding of a broader phenomenon through the use of a method appropriate to the nature of the research question. Rigor, then, can be conceptualized as the appropriate execution of that method. Put simply, if quality is the what, rigor for our purposes becomes the how. It is here that inductive and deductive traditions diverge in a significant way.

It is useful to start with the negative case. Two criteria appropriate for deductive research but NOT appropriate for inductive inquiry include:

1) Is there evidence that the causal factors, processes, nature, meaning, and/or significance of the phenomenon generalize to the broader population?

2) Are the findings able to be replicated in the sense that two researchers asking the same question would come to the same interpretation of the data?

These two criteria, held sacred as cornerstones of rigor in deductive inquiry, seem to cause the greatest amount of heartburn within the field of public management and its relationship to inductive qualitative inquiry. If it is not generalizable and it does not replicate, how is that possibly science? This results in on-going frustration among qualitative scholars as they attempt to respond to criticisms of their design by reviewers, colleagues, and advisors in terms of the lack of representative sampling and/or inter-rater reliability measures. This is rooted in some fundamental misunderstandings about what inductive inquiry is and what it seeks to accomplish.

Generalizability

In deductive methods, when there are more cases that conform to an a priori hypothesis than do not, relative to the standard error and controlling for all other factors in the model, we reject the null hypothesis that this pattern could have been observed merely by random chance. However, in every deductive sample, there can be numerous observations which do not conform to our models. These we vaguely disregard as “error.” When cases deviate substantially, we call them “outliers” and may remove them from consideration entirely. This is reasonable because the aim of deductive inquiry is to test the presence of an a priori relationship in the population based on a limited, representative sample ( Neuman and Robson 2014 ). Associations deal with probabilities and likelihoods; not all cases must conform to a pattern to conclude that an association exists as long as the sample is reasonably representative and sufficient to detect differences ( Wasserman 2013 ).

Inductive research is attempting to do something quite different. The sample of an inductive study is never purely random nor convenient. Instead, each case or participant should be purposively selected because they represent a theoretically interesting exemplar of, or key informant about, a phenomenon of interest ( Patton 2014 ). In other words, by nature of being selected for inclusion in an inductive study, the scholar is making the argument that we should care about understanding the experience of this person(s) or the events of this case. Whether a pattern discerned in an inductive study is common in the general population is not the question an inductive scholar is seeking to answer. In fact, the case may have been selected specifically because it represents something rare or unusual. Rather, they are seeking to use a systematic method to interpret and represent, in rich detail, what is true for a particular set of individual(s) and/or cases, identifying themes and patterns across cases that add insight into the phenomenon of interest. Cases with divergent patterns or informants with contradictory experiences are not ignored or discounted as measurement error or outliers. Rather, the inductive scholar seeks to understand the factors and mechanisms that explain these points of divergence ( Eisenhardt et al. 2016 ).

Although the inductive scholar does not empirically test the extent to which an association or experience is common in the general population, this does not mean that inductive findings are not intended to have relevance for advancing general theory and practice. If done well, an inductive study should provide a detailed, contextualized, and empirically grounded interpretation of what was true in one or more cases of interest. Just as one experience in one setting should never be assumed to dictate what one might experience in another setting, it would likewise be absurd to assume prior experience is totally irrelevant if a similar set of conditions are present. In this way, qualitative inductive scholarship seeks to systematically describe and interpret what is occurring in a finite set of cases in sufficient detail as to lend insight into what might be going on in cases like these . Discerning the quantitative prevalence of these key patterns or experiences within populations is where deductive methods can pick up where inductive leave off. However, it is only through also gaining a grounded and detailed understanding of phenomenon of theoretical interest do we gain new insights and have hope of developing understanding and theory that has relevance to field of practice.

Replication

As mentioned previously, inductive methods are seeking to develop a deep understanding of causal factors, processes, nature, meaning, and/or significance of a particular phenomenon ( Creswell and Poth 2018 ; Denzin and Lincoln 2012 ; Patton 2014 ). This understanding generally comes from asking a lot of questions, observing settings and behavior, and collecting stories, images, and other artifacts that aid the scholar in also gaining insight into their phenomenon of interest. Different approaches have been created to narrow in on specific types of phenomenon. For example, phenomenology looks at how individuals experience and ascribe meaning to a given phenomenon ( Giorgi 1997 ; Moran 2002 ; Waugh and Waugh 2003 ). Grounded theory seeks to identify the causal relationships that give rise to, and result from, a given phenomenon ( Glaser and Strauss 2017 ; Morse et al. 2016 ). Ethnography seeks to uncover the cultural elements within human systems ( Agar 1996 ; Hammersley 1983 ; Preissle and Le Compte 1984 ).

Each tradition has its own systematic process of data collection and analysis. However, regardless of the tradition, it is always the analyst who must draw inference and interpretation from the vast array of qualitative information in front of them . Just as there are some doctors who can observe the same patient information to diagnose root causes while others focus on first order symptomology, multiple analysts working independently on the same data sources may also come to different interpretations of what is going on ( Langley 1999 ). One doctor is not necessarily right and the others wrong; rather the same thing can be many things at once (e.g., structural, psychological, cultural). Therefore, the appropriate criteria of rigor is not whether the same interpretation would be independently arrived upon by different analysts. Rather, in inductive analysis, the criteria is: based on the evidence provided, is a given interpretation credible ( Patton 1999 )? In other words, if an independent analyst were informed of another analyst’s interpretation and then given all the same source information, would the interpretation stand up to scrutiny as being a justified, empirically grounded, exposition of the phenomenon?

Elements of Rigor

If we cannot assess inductive studies in terms of generalizability and replication, what are valid criteria upon which they might be evaluated? In very global terms, rigorous inductive research in public management can be judged on two core criteria:

1) Does the research design and its execution generate new insight into the causal factors, processes, nature, meaning, and/or significance of a phenomenon of interest to the field? (reviewed in Table 1 ) and

2) Is the account of these causal factors, processes, nature, meaning, and/or significance within these cases trustworthy? (reviewed in Table 3 )

Relevant and Inappropriate Criteria of Rigor for Inductive Research

The trustworthiness and depth of insight of an inductive study is manifest in its research design, execution, reporting.

Research Design

Because the contribution of inductive qualitative research fundamentally hinges on the theoretical relevance of the units (e.g., individuals, cases, texts) selected for study, sampling is of paramount importance. Different approaches of qualitative analysis have specific guidance on sampling consistent with that approach. For example, grounded theory uses a protocol of proposition-driven sampling in which the investigator strategically chooses cases iteratively in conjunction with data analysis in an effort to examine variation in patterns observed in the previous cases (for discussion, see Corbin and Strauss 1990 ; Glaser 2002 ). However, regardless of which analysis tradition an inductive scholar is using, the inductive qualitative sample must always be justified in terms of why the informants, texts, and/or cases selected should be considered of theoretical interest to the field. This description should be situated in terms of who these informants are in the broader population of possible informants relevant to the research question. Inductive scholarship should include a clear explication of why these individuals were chosen specifically and what they represent. What qualifies them as key informants of this phenomenon? Why would we expect them to have insight into this question that is particularly information rich and/or relevant to the field? How might their position likely influence the perspective they offer about the phenomenon (for discussion, see Marshall 1996 ; for exemplar, see Saz-Carranza and Ospina 2010 and Romzek et al. 2012 justification of both case and informant selection)?

As outlined in most introductory texts in qualitative analysis (e.g., Denzin and Lincoln 2012 ; Miles, Huberman, and Saldana 2013 ; Patton 2014 ), there are numerous sampling strategies that may guide participant or case selection in an inductive study. Common approaches include efforts to capture the “typical case,” the “extreme case,” the disconfirming case, or the “unusual case.” Sampling is also often purposefully stratified to represent informants from theoretically important sub-populations. In studies of individual level phenomenon, this may include stratifying samples to include men and women, young/middle age/old, more or less experience, or different ethnicities/racial groups. In studies of higher order phenomenon such as at the organizational, coalition, group, or network level, the scholar may choose to stratify cases across geographic region or based on some developmental phase (e.g., new versus old organizations). Although there are numerous potential sampling strategies for an inductive study, they all share in common the criteria that whatever or whomever is chosen for inclusion or exclusion of an inductive study, sampling decisions must be inquiry driven, theoretically justified, and information rich.

How Many is Enough?

The question of sample size in inductive qualitative research is less straight forward than it is in deductive research. In the deductive world, the sample size criteria turns primarily on the power to detect differences given the model applied to the data ( Wasserman 2013 ). In inductive research, the sample size question focuses on the sources of variability of the phenomenon of interest that are of theoretical importance to the field given the research question. However, inductive studies complicate the sample size question because numerous and varied sources of data can be, and often are, integrated. For example, in several qualitative approaches, triangulation of findings among multiple data sources is one of elements of the rigor (e.g., for review see Jonsen and Jehn 2009 ).

Just as with deductive research, no one inductive study can address every dimension or perspective that might be relevant to understanding a phenomenon of interest. Therefore, in addition to clearly articulating the criteria upon which individuals or other data sources were sampled for inclusion into the study, there is need to explicate the boundary criteria that sets the limits for who or what is not considered within the scope of the inquiry. Following this, the authors must clearly articulate the unit or units of analysis that define the phenomenon of inquiry. Is it case-based such as an inquiry into the factors that hindered international NGO community from being effective contributors to the response phase of Hurricane Katrina (e.g., Eikenberry, Arroyave, and Cooper 2007 )? Is it organizational such as a study of service providers’ usage of monitoring tools based on agency theory (e.g., Lambright 2008 ). Is it focused on the individual, such as examining public service motivation and transformation leadership (e.g., Andersen et al. 2016 )? Or is it episodic such as a study of the process through which routines can lead to a source of innovation within an organization (e.g., Feldman 2003 )? Higher order phenomenon (i.e., case-level, coalition-level, organizational-level, etc.) often require multiple data sources or informants associated with that case, group, or organization to gain sufficient depth of understanding of the dynamics present. This will necessarily place limits on the number of cases that can be studied comparatively. Alternatively, a single informant may be able to reflect on multiple episodic units based on varied experiences over time.

Qualitative Saturation

Qualitative saturation is a technique commonly referenced in inductive research to demonstrate that the dataset is robust in terms of capturing the important variability that exists around the phenomenon of interest ( O’Reilly and Parker 2013 ). However, we advise caution in the use of saturation in defending the sample characteristics of a qualitative sample. Qualitative saturation refers to a point at which the analyst has obtained a sort of information redundancy such that continued analysis has revealed no new insight not already captured by previous cases ( Morse 1995 ). Generally, during analysis, scholars do reach a point at which no new themes or propositions emerge and analysis of new transcripts leads only to additional instances of existing themes or relationships. However, this standard is problematic as a criterion for rigor in public management for two reasons.

First, in order to be used as a condition of sampling rigor, it requires that the scholar analyze their data as it is being collected so as to recognize the point at which no additional data collection is needed. Although this design feature is integral to grounded theory, it is uncommon in other qualitative traditions which often mimic deductive models having a distinct data collection phase preceding a data analysis phase ( Miles, Huberman, and Saldana 2013 ). Second, the methods by which a scholar determines saturation are generally methodologically difficult to standardize or demonstrate as a criteria of rigor ( Morse 1995 ; O’Reilly and Parker 2013 ). Therefore, while saturation is an important heuristic in guiding data analysis—for example, for informing the analyst when they should transition from open coding to axial coding, we do not find it is a particularly useful concept for qualitative consumers to evaluate the suitability of a dataset in terms of whether it should be considered theoretically robust.

Consequently, qualitative consumers generally must rely on qualitative as opposed to quantitative benchmarks for determining the suitability of a given dataset for addressing an inductive research question. The questions qualitative consumers need to answer are these: 1) is the dataset analyzed information rich and 2) does it have a reasonable chance of representing variability of the phenomena of interest that are of theoretical importance given the research question ( Brower, Abolafia, and Carr 2000 )? In efforts to orient new inductive scholars into the general ballpark of sample expectations, some scholars have cautiously made heavily caveated recommendations (for review, see Onwuegbuzie and Leech 2007 ). Qualitative studies of 100 or more units are unusual and generally unnecessary for most inductive analytic traditions unless some type of quantification is desired (see below discussion on hybrid designs; Gentles et al. 2015 ). Studies of ten or less units would require a unique justification in terms of how such a data set provides a theoretically robust perspective on the phenomenon of interest. Within that sizeable range, qualitative consumers will have to make a subjective call about the theoretical robustness of a given dataset in relation to the research question asked, the phenomenon of interest, the analytic tradition used, and the interpretive claims made. Benchmarking sampling approaches against existing literature utilizing the same analytic approach is helpful for creating consistency within the field. Additionally, qualitative consumers may find the following questions a useful rubric in determining how theoretically robust a given dataset might be considered to be:

1) Is the phenomenon rare or infrequently encountered?

2) Are the data rare or particularly difficult to obtain?

3) Is the phenomenon simple or complex?

4) Is the phenomenon new or well investigated in the literature?

5) How information rich is each unit in relation to the phenomenon of interest?

6) Is the same unit being engaged at multiple points in time?

Data Collection Protocols and Procedures

In deductive research, constructs and relationships are articulated prior to analysis, and what one can discover is therefore necessarily constrained to what one looks to find. In inductive research, constructs and relationships are articulated through analysis, and the scholar seeks to minimize constraint on what can be discovered ( Lincoln and Guba 1986 ). However, because in most inductive studies, the data must still be collected from individuals, the actions of the investigator will inevitably constrain and shape what the data looks like. This is done a priori through the creation of protocols which guide the types of questions that the investigator asks informants or the elements the investigator observes and records their observations. While these protocols can, and often should evolve over the course of the study, it is the execution of these protocols that create the data used in analysis. Consequently, the quality of these protocols and their execution is an important consideration in determining the rigor of an inductive study ( Miles, Huberman, and Saldana 2013 ).

In demonstrating the rigor of an inductive research design, the investigator should be able to clearly describe what data was considered relevant for a given research question and how this data was obtained. Data collection protocol design should be consistent with the specific methodological tradition embraced by the study (see Table 2 ). Vague descriptors such as “data were obtained through open ended interviews” is not sufficient description to determine rigor. Just as in deductive research the same construct can be operationalized in multiple ways, two inductive investigators may be interested in the same research question but ask very different types of interview questions of their informants. Researchers should be able to describe the types of questions the investigator asked informants related to the phenomenon of interest. These questions should have a clear conceptual linkage to the research question of concern, the analytic tradition embraced, and be a key consideration in the analysis and interpretation of the findings (for exemplar, see Rerup and Feldman’s (2011) , description of the interview protocol used to illicit espoused schemas of staff in a tech start up). It is also important for the qualitative consumer to recognize that data looks different depending on the different analytic tradition one uses. Table 2 outlines some of the more prevalent qualitative traditions.

Qualitative Data Collection and Analysis Traditions

Data Analysis and Interpretation

Like deductive approaches, inductive qualitative data analysis come in many forms linked to different analytic traditions and are more or less appropriate to different types of research questions. These traditions carry with them specific guidance on design, sampling, and analysis. Methodological deviations or qualitative “mixology” ( Kahlke 2014 ) in which design element from multiple traditions are combined or certain design elements omitted should be well-justified and evaluated carefully by the qualitative consumer to ensure the resulting design remains robust. Just as with deductive designs, robust inductive designs should have a clear logical flow from the research question, to the data collection protocol, to the description of the analysis procedure, to the explication of the findings. There should be no black curtain behind which hundreds of pages of transcripts are magically transformed into seven key findings. Rather, the scholar should be able to provide a clear and concise description of their analysis process and its relationship to the reported findings (for exemplar, see Rivera’s (2017) case analysis description in her study of gender discrimination in academic hiring committees).

As discussed, the overarching criteria of rigor associated with an inductive study is not reliability or replication. Rather, rigorous analysis is based on 1) whether the interpretation is credible in light of the data, 2) whether it was the result of a robust and systematic analytical process designed to move beyond superficial findings and minimize and/or account for investigator bias, and 3) whether it is reported with sufficient attention to context so as to facilitate the potential relevance of insights to similar contexts. These features were first described by Lincoln and Guba (1986) as the criteria of qualitative trustworthiness. They developed an initial set of practices designed to achieve these elements of rigor that have since been expanded upon by various qualitative scholars. Although these elements remain under development and debate, especially in public management (for discussion see, Lowery and Evans 2004 ), Table 3 offers a broad overview of some of the more commonly advocated strategies and associated aims that qualitative consumers might consider when evaluating the rigor of an inductive study. However, it is important to note that these elements represent strategies. They are not a checklist and certain strategies may be more or less appropriate in certain study designs. As such, we argue rigor is best conceptualized in terms of its functionality. Was the design logically coherent in relation to the research question? Was the analysis systematically designed to move beyond superficial findings and minimize and/or account for investigator bias? Did the design result in both credible and insightful findings? Were the findings reported with sufficient attention to context so as to facilitate empirically grounded theory building?

Elements of Qualitative Rigor (Adapted From Creswell and Poth 2018 ; Denzin and Lincoln 2003 ; Lincoln and Guba 1986 ; Morse 2015 )

We have argued that the distinction between inductive versus deductive approaches is a most relevant delineation for identifying appropriate criteria of rigor. Up to this point, we have focused primarily on inductive applications of qualitative data. However, as noted previously, not all qualitative data analysis is inductive. In this final section, we give special consideration to qualitative approaches in the field of public management that that are either deductive, mixed, and hybrid methods.

Narrative Policy Framework

In policy process research, the Narrative Policy Framework (NPF) has more recently emerged as an approach for quantifying qualitative data that has been coded from policy documents and various mediums of public comment ( Shanahan, Jones, and McBeth 2013 ). The NPF was designed to address postpositivist challenges to policy process theories by taking into account the critical role that narratives play in generating and facilitating meaning for people and how those narratives then relate to the politics of constructing reality ( Shanahan, Jones, and McBeth 2013 ). Within the NPF, narratives are considered to be constructed of important elements that include many traditional parts of stories like a hero, a villain, a plot, and a moral. These narrative elements are applied as codes in a more directly deductive approach and then often used for hypothesis testing at micro , meso , and macro levels ( McBeth, Jones, and Shanahan 2014 ).

Despite being derived from qualitative data, much of the work on NPF embraces a deductive model of hypothesis testing ( Shanahan, Jones, and McBeth 2017 ). In deductive applications, the standards of rigor as it relates to representative sampling, construct validity, reliability, statistical power, and generalizability apply. These methods require the development of a stable coding framework that can be applied by multiple coders with a high degree of reliability. As such, metrics such as inter-rater reliability are appropriate tools for demonstrating that the coding framework is being applied in a consistent manner. Another design challenge with NPF is the fact that its core propositions are associated with discrete “narratives” as the unit of analysis, which can be difficult to isolate in a standardized way across different types of policy documents which may contain multiple narratives (for discussion, see Shanahan, Jones, and McBeth 2018 ). Further, the representative sampling of policy documents relative to a defined population can be difficult to conceptualize ( Shanahan, Jones, and McBeth 2018 ). Despite these challenges, NPF is valuable in its ability to examine whether specific narrative patterns have a stable and generalizable influence on different outcomes of the policy process ( McBeth, Jones, and Shanahan 2014 ); a question ill-suited to an inductive narrative analysis approach.

Mixed Methods

Another development that has gained popularity in public management and applied social sciences more generally is the mixed methods study (see Honig, this issue). A mixed methods study is often characterized as one that uses a combination of both qualitative and quantitative data ( Creswell and Clark 2018 ; for alternative definitions see Johnson et al. 2007 ). It is generally assumed that mixed methods studies will also utilize a combination of inductive and deductive approaches. The ordering of the inductive/deductive mixture can vary. For example, the scholar may use an inductive qualitative phase aimed at gaining a greater insight about a poorly understood phenomenon. Constructs, dimensions, and propositions resulting in the findings from this first inductive phase of analysis can then be translated into a second confirmatory phase in the form of survey measure development, psychometrics, and hypothesis testing. In a second variation, a scholar may use existing literature and theory to deductively create measures and propose and test hypotheses. The scholar may then design an inductive phase in which the mechanisms and contextual factor underlying these hypotheses are explored in great depth through qualitative methods (for discussion of various design options, see Mele & Belardinelli, this issue; Creswell, Clark, Gutmann, and Hanson 2003 ).

Considerations of rigor in a mixed methods study are two pronged. First, mixed methods studies have the dual burden of adhering to all the requirements of rigorous design associated with both inductive and deductive models. For example, the sample for the inductive phase must meet the criteria of offering an information rich, inquiry-driven sample while the sample for the deductive phase must have still sufficient power to detect differences and be a reasonably representative sample of the population. This generally makes such studies relatively large and ambitious. Second, a rigorous mixed methods study should ideally reflect some degree of complementarity between the approaches, maximizing the different advantages in inductive versus deductive designs. Each design element should reflect thoughtful attention to the points at which the findings from the different phases of analysis co-inform one another ( Johnson, Burke, and Onwuegbuzie 2004 ).

Qualitative Comparative Analysis

Qualitative Comparative Analysis (QCA; Ragin 1998 ; Ragin and Rihoux 2004 ) represents a hybrid approach, being neither fully inductive or deductive. QCA has an established presence in public management ( Cristofoli and Markovic 2016 ; Hudson and Kuhner 2013 ; Malatesta and Carboni 2015 ; Pattyn, Molevald, and Befani 2017 ; Raab, Mannak, and Cabre 2015 ; Sanger 2013 ; Thomann 2015 ). Like NPF, QCA involves the quantification of qualitative data and the application of mathematical models. However, different from NPF, which is principally deductive in its approach, QCA can use inductive qualitative methods to identify outcomes of interest and factors of relevance to explaining that outcome. These interpretations of the data are then quantified and entered into mathematical models designed to examine pathways of necessary and sufficient conditions that are derived from a researcher creating a numeric data table, often using binary codes.

QCA, first introduced by Ragin (1987) , is intended to unify aspects of qualitative, case-based research, and quantitative, variable-based, approaches ( Fischer 2011 ). QCA is rooted in the assumption of equifinality; that different causal conditions can lead to the same outcome, and that the effect of each condition is dependent on how it is combined with other conditions ( Fischer 2011 ; Ragin 1987 ). Accordingly, QCA is not hindered by the assumptions of homogeneous effects that encumber many quantitative approaches. Rather, it enables the researcher to consider multiple pathways and combinations that may lead to the same outcome. Also unique to QCA is the focus on combinatorial logic that assumes that cases should be viewed holistically within the context of all conditions combined. As such, QCA can reveal patterns across cases that might be difficult to discern through purely qualitative approaches (for discussion see Rihoux and Ragin 2008 ).

One of the challenges to assessing QCA from a rigor perspective stems from its inherently hybrid nature. The samples in QCA are generally small and presumably inductively selected ( Hug 2013 ). As such, an inductive criteria of rigor could apply. However, the results of a QCA have a distinctive deductive flavor in both the style of analysis and interpretation. For example, the process by which the specific constructs are identified for inclusion is often not well explicated and may contain a mixture of a priori theory and inductively derived theory. Some authors embrace a fully deductive hypothesis driven approaches based on theory and using predetermined codebooks (e.g., Raab, Mannak, and Cambre 2015 ; Thomman 2015 ). Cases, which do not fit into one of the identified pathways are excluded from the output due to criteria like relevancy and consistency that enable the Boolean algebra of QCA to more readily converge on causal pathways. 3 Publications of QCA findings generally focus primarily on the pathways identified with little or no attention to the cases that deviated from these patterns.

It is our belief that as QCA applications evolve, scholars will need to, metaphorically, pick a horse to ride in their utilization of this technique in order for a study to be associated with the appropriate standards of rigor. In other words, QCA is a descriptive tool that can be used either inductively or deductively. Is a study a deductive effort to examine possible configurations of pathways toward a predefined outcome using a priori factors examined within a representative sample of a population? If so, deductive criteria of rigor would apply to a QCA as it relates to construct validity, inter-rater reliability, and representative sampling. On the other hand, QCA could also be a powerful tool used within an inductive model of research with associated inductive criteria of rigor. In this model, cases would be purposively justified as theoretically important to understanding a given phenomenon. The QCA would represent a tool within a broader process of inquiry for examining complex patterning across cases that may be difficult to otherwise discern. The inductive process by which coding categories were generated and qualitative variability that exists within coding delineations would be central concerns of the analysis. The analysis would include an empirically grounded contextualization and interpretation of the cases that conform to, as well as deviate from, the identified patterns so as to inform the mechanisms by which one pattern versus another may emerge. Either application of QCA, whether deductive or inductive, holds promise as a technique but murky applications which do not fully commit to either standard of rigor seem problematic (for additional discussion, see Hug 2013 ).

We began this article with the assertion that qualitative methods are poised to make a greater contribution in shaping our understanding of public management. We view this as a good thing; having the potential to inject new insight and depth of understanding into the questions that define the field. We perceive a general openness in the discipline to advancing bodies of literature through the integration of contributions from both inductive and deductive styles of inquiry. However, much of the discipline lacks even basic training in inductive approaches to research (see Stout 2013 ) which serves as a barrier. Deductive models—by virtue of the more structured task they are designed to accomplish coupled with the greater duration of time this approach has had to institutionalize—are simply more straightforward in their precepts of rigor. However, advancing the contribution of qualitative methods in public management will not happen without some shared construction of rigor that is compatible with a postpositivistic stance on science. We argue that the first step in advancing this agenda is to stop conflating data type (qualitative versus quantitative) with methodological approach (inductive versus deductive).

Beyond this, this article is positioned as a conversation-starter and as a resource for breaking down barriers for meaningful interactions that have put qualitative and quantitative methods at odds. We argue here that these past misunderstandings have less to do with the analysis of text versus number-based data, and more to do with murky or altogether misunderstood differences between the requirements of quality and rigor for inductive versus deductive methods. In clearing some of the air on quality and rigor of both kinds of methods in this space, we put forth a postpositivist stance with the understanding that not all scholars will agree, but that this perspective offers a productive pathway for broadly engaging the most common public management researcher today.

Agar , Michael H . 1996 . The professional stranger: An informal introduction to ethnography . 2nd ed. UK : Emerald Publishing Group .

Google Scholar

Google Preview

Andersen , Lotte Bøgh , Bente Bjørnholt , Louise Ladegaard Bro , and Christina Holm-Petersen . 2016 . Leadership and motivation: A qualitative study of transformational leadership and public service motivation . International Review of Administrative Sciences : 1 – 17 .

Brooks , Arthur C . 2002 . Can nonprofit management help answer public management’s “big questions” ? Public Administration Review , 62 ( 3 ): 259 – 66 .

Brower , Ralph S. , Mitchel Y. Abolafia , and Jered B. Carr . 2000 . On improving qualitative methods in public administration research . Administration & Society 32 ( 4 ): 363 – 97 . doi: 10.1177/00953990022019470 .

Carter , Stacy M. , and Miles Little . 2007 . Justifying knowledge, justifying method, taking action: Epistemologies, methodologies, and methods in qualitative research . Qualitative Health Research 17 ( 10 ): 1316 – 28 .

Caelli , Kate , Lynne Ray , and Judy Mill . 2003 . ‘Clear as mud’: Toward greater clarity in generic qualitative research . International Journal of Qualitative Methods 2 ( 2 ): 1 – 13 .

Clark , Alexander M . 1998 . The qualitative-quantitative debate: Moving from positivism and confrontation to post-positivism and reconciliation . Journal of Advanced Nursing 27 ( 6 ): 1242 – 9 .

Corbin , Juliet M. , and Anselm Strauss . 1990 . Grounded theory research: Procedures, canons, and evaluative criteria . Qualitative sociology 13 ( 1 ): 3 – 21 .

Corbin , Juliet , and Anselm L. Strauss . 2014 . Basics of qualitative research . Thousand Oaks, CA : Sage .

Creswell , John W. , and Dana L. Miller . 2000 . “ Determining validity in qualitative inquiry .” Theory into Practice . 39 ( 3 ): 124 – 130 .

Creswell , John W. , and Vicki L. Plano Clark . 2018 . Designing and conducting mixed methods research , 3rd ed. Thousand Oaks, CA : Sage .

Creswell , John W. and Cheryl Poth . 2018 . Qualitative inquiry and research design: Choosing among five approaches . 4th ed. Thousand Oaks, CA : Sage .

Creswell , John W. , Vicki L. Plano Clark , Michelle L. Gutmann , and William E. Hanson . 2003 . “ Advanced mixed methods research designs .” Handbook of Mixed Methods in Social and Behavioral Research 209 : 240 .

Cristofoli , Daniela and Josip Markovic . 2016 . How to make public networks really work: A qualitative comparative analysis . Public Administration 94 : 89 – 110 . doi: 10.1111/padm.12192

Dias , Janice Johnson , and Steven Maynard-Moody . 2007 . For-Profit Welfare: Contracts, Conflicts, and the Performance Paradox . Journal of Public Administration Research and Theory 17 : 189 – 211 .

Denhardt , Robert B . 2001 . The big questions of public administration education . Public Administration Review . 61 ( 5 ): 526 – 34 .

Denzin , Norman K. , and Yvonna S. Lincoln . 2003 . The landscape of qualitative research: Theories and issues . 2nd ed. Thousand Oaks, CA : Sage .

Denzin , Norman K. , and Yvonna S. Lincoln . 2012 . Strategies of qualitative inquiry . Vol. 4 . Thousand Oaks, CA : Sage .

Dodge , J. , S. M. Ospina , and E. G. Foldy . 2005 . Integrating rigor and relevance in public administration scholarship: The contribution of narrative inquiry . Public Administration Review 65 ( 3 ): 286 – 300 .

Dubnick , M. J . 1999 . Demons, spirits, and elephants: Reflections on the failure of public administration theory . Paper presented at the annual meeting of the American Political Science Association , Atlanta, GA .

Eikenberry , Angela M. , Verónica Arroyave , and Tracy Cooper . 2007 . Administrative failure and the international NGO response to Hurricane Katrina . Public Administration Review 67 ( 1 ): 160 – 70 .

Eisenhardt , Kathleen M. , and Melissa E. Graebner . 2007 . Theory building from cases: Opportunities and challenges . Academy of Management Journal . 50 ( 1 ): 25 – 32 .

Eisenhardt , Kathleen M. , Melissa E. Graebner , and Scott Sonenshein . 2016 . “ Grand challenges and inductive methods: Rigor without rigor mortis .” Academy Of Management Journal 59 ( 4 ): 1113 – 1123 .

Feldman , Martha S . 2000 . “ Organizational routines as a source of continuous change .” Organization Science 11 ( 6 ): 611 – 629 .

Feldman , Martha S. , and B. T. Pentland . 2003 . Reconceptualizing organizational routines as a source of flexibility and change . Administrative Science Quarterly 48 ( 1 ): 94 – 118 .

Feldman , Martha S. , Kaj Sköldberg , Ruth Nicole Brown , and Debra Horner . 2004 . Making sense of stories: A rhetorical approach to narrative analysis . Journal of Public Administration Research and Theory 14 ( 2 ): 147 – 70 .

Fischer , Manuel . 2011 . Social Network Analysis and Qualitative Comparative Analysis: Their mutual benefit for the explanation of policy network structures . Methodological Innovations Online 6 ( 2 ): 27 – 51 .

Freeman , John Leiper . 1965 . The Political Process: Executive Bureau-Legislative, Committee Relations . Vol. 13 . New York : Random House .

Glaser , Barney G . 2002 . Conceptualization: On theory and theorizing using grounded theory . International Journal of Qualitative Methods 1 ( 2 ): 23 – 38 .

Glaser , Barney G. , and Anselm L. Strauss . 2017 . Discovery of grounded theory: Strategies for qualitative research . London : Routledge .

Gentles , Stephen J. , Cathy Charles , Jenny Ploeg , and K. Ann McKibbon . 2015 . Sampling in qualitative research: Insights from an overview of the methods literature . The Qualitative Report 20 ( 11 ): 1772 .

Gill , J. , and K. J. Meier . 2000 . Public administration research and practice: A methodological manifesto . Journal of Public Administration Research and Theory 10 ( 1 ): 157 – 99 .

Giorgi , Amedeo . 1997 . The theory, practice, and evaluation of the phenomenological method as a qualitative research procedure . Journal of Phenomenological Psychology 28 ( 2 ): 235 – 60 .

Guba Egon G ., ed. 1990 . The paradigm dialog . Newbury Park, CA : Sage .

Hammersley , Martyn . 1983 . Ethnography . San Francisco, CA : John Wiley & Sons .

Haverland , Markus , and Dvora Yanow . 2012 . A hitchhiker ‘ s guide to the public administration research universe: Surviving conversations on methodologies and methods . Public Administration Review 72 ( 3 ): 401 – 8 .

Head , Brian William . 2010 . Public management research: Towards relevance . Public Management Review 12 ( 5 ): 571 – 85 .

Holland , P. W . 1986 . Statistics and causal inference . Journal of the American Statistical Association . 81 ( 396 ): 945 – 960 .

Honig , Dan . 2018 . Case study design and analysis as a complementary empirical strategy to econometric analysis in the study of public agencies: deploying mutually supportive mixed methods . Current issue.

Hudson , John , and Stefan Kühner . 2013 . Qualitative comparative analysis and applied public policy analysis: New applications of innovative methods . Policy and Society 32 ( 4 ): 279 – 87 .

Hug , Simon . 2013 . Qualitative comparative analysis: How inductive use and measurement error lead to problematic inference . Political Analysis 21 ( 2 ): 252 – 65 .

Johnson , R. Burke , and Anthony J. Onwuegbuzie . 2004 . Mixed methods research: A research paradigm whose time has come . Educational Researcher 33 ( 7 ): 14 – 26 .

Johnson , R. Burke , Anthony J. Onwuegbuzie , and Lisa A. Turner . 2007 . Toward a definition of mixed methods research . Journal of Mixed Methods Research 1 ( 2 ): 112 – 33 .

Jonsen , K. , and K. A. Jehn . 2009 . Using triangulation to validate themes in qualitative studies . Qualitative Research in Organizations and Management: An International Journal 4 ( 2 ): 123 – 50 .

Kahlke , Renate M . 2014 . Generic qualitative approaches: Pitfalls and benefits of methodological mixology . International Journal of Qualitative Methods 13 ( 1 ): 37 – 52 .

Kettl , Donald F . 2000 . Public administration at the millennium: The state of the field . Journal of Public Administration Research and Theory 10 ( 1 ): 7 – 34 .

King , Cheryl Simrell , Kathryn M. Feltey , and Bridget O’Neill Susel . 1998 . The Question of Participation: Toward Authentic Public Participation in Public Administration . Public Administration Review 58 ( 4 ): 317 . doi: 10.2307/977561 .

Kuhn , Thomas S . 1996 . The nature and necessity of scientific revolutions . In The structure of scientific revolutions , Kihn T. S . ed. 3rd ed. Chicago : University of Chicago Press .

Lambright , Kristina T . 2008 . Agency theory and beyond: Contracted providers’ motivations to properly use service monitoring tools . Journal of Public Administration Research and Theory 19 ( 2 ): 207 – 27 .

Langley , Ann . 1999 . Strategies for theorizing from process data . Academy of Management review 24 ( 4 ): 691 – 710 .

Lewis , D . 1973 . Causation . The Journal of Philosophy 70 ( 17 ): 556 – 67 .

Lincoln , Yvonna S. , and Egon G. Guba . 1986 . But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation . New Directions for Evaluation 30 : 73 – 84 .

Lipsky , Michael . 1971 . Street-level bureaucracy and the analysis of urban reform . Urban Affairs Quarterly 6 ( 4 ): 391 – 409 . doi: 10.1177/107808747100600401

Lowery , Daniel , and Karen G. Evans . 2004 . The iron cage of methodology: The vicious circle of means limiting ends limiting means ... Administration & Society 36 ( 3 ): 306 – 27 .

Malatesta , Deanna , and Julia L. Carboni . 2015 . The public–private distinction: Insights for public administration from the state action doctrine . Public Administration Review 75 ( 1 ): 63 – 74 .

Marshall , M. N . 1996 . Sampling for qualitative research . Family Practice 13 ( 6 ): 522 – 6 .

Maynard-Moody , Steven , and Marisa Kelly . 1993 . Stories managers tell about elected officials: Making sense of the politics-administration dichotomy . In Public management: The state of the art , ed. B. Bozeman, 71 – 90 . San Francisco : Jossey-Bass .

McNabb , David E . 2014 . Case research in public management . London : Routledge .

McNabb , David E . 2015 . Research methods in public administration and nonprofit management . London : Routledge .

McBeth , Mark K. , Michael D. Jones , and Elizabeth A. Shanahan . 2014 . “ The narrative policy framework .” In Theories of the policy process , edited by Sabatier , Paul A. , and Weible Christopher M , 225 – 266 . Boulder : Westview Press .

Mertens , Donna M. , and Amy T. Wilson . 2012 . Program evaluation theory and practice: A comprehensive guide . New York : Guilford Press .

Milward , H. Brinton. Forthcoming. Toward a theory of organizational networks: Structure, process, and people . Perspectives on Public Management and Governance .

Miles , Matthew B. , A. Michael Huberman , and Johnny Saldana . 2013 . Qualitative data analysis . Thousand Oaks, CA : Sage .

Moran , Dermot . 2002 . Introduction to phenomenology . London : Routledge .

Morse , Janice M . 1995 . The significance of saturation . Qualitative Health Research 5 ( 2 ): 147 – 9 .

Morse , Janice M . 2015 . Critical analysis of strategies for determining rigor in qualitative inquiry . Qualitative Health Research 25 ( 9 ): 1212 – 22 .

Morse , Janice M. , Phyllis Noerager Stern , Juliet Corbin , Barbara Bowers , Kathy Charmaz , and Adele E. Clarke . 2016 . Developing grounded theory: The second generation . London : Routledge .

Neuman , William Lawrence , and Karen Robson . 2014 . Basics of social research . Canada : Pearson .

Onwuegbuzie , A. J. , and N. L. Leech . 2007 . A call for qualitative power analyses . Quality & Quantity 41 ( 1 ): 105 – 21 .

O’Reilly , Michelle , and Nicola Parker . 2013 . ‘Unsatisfactory Saturation’: A critical exploration of the notion of saturated sample sizes in qualitative research . Qualitative Research 13 ( 2 ): 190 – 7 .

Ospina , Sonia M. , and Jennifer Dodge . 2005 . It’s about time: Catching method up to meaning—the usefulness of narrative inquiry in public administration research . Public Administration Review 65 ( 2 ): 143 – 57 .

Ospina , Sonia M. , Marc Esteve , and Seulki Lee . 2017 . Assessing Qualitative Studies in Public Administration Research . Public Administration Review 78 ( 4 ): 593 – 605 . doi: 10.1111/puar.12837

Patton , Michael Quinn . 1999 . Enhancing the quality and credibility of qualitative analysis . Health Services Research 34 ( 5 ): 1189 – 208 .

Patton , Michael Quinn . 2014 . Qualitative Research & Evaluation Methods: Integrating Theory and Practice . Thousand Oaks, CA : Sage .

Pattyn , Valérie , Astrid Molenveld , and Barbara Befani . 2017 . Qualitative comparative analysis as an evaluation tool: Lessons from an application in development cooperation . American Journal of Evaluation .

Perry , James L. , and Lois Recascino Wise . 1990 . The motivational bases of public service . Public Administration Review 50 ( 3 ): 367 . doi: 10.2307/976618

Pillow , W . 2003 . Confession, catharsis, or cure? Rethinking the uses of reflexivity as methodological power in qualitative research . International Journal of Qualitative Studies in Education 16 ( 2 ): 175 – 96 .

Prasad , Pushkala . 2015 . Crafting qualitative research: Working in the postpositivist traditions . London : Routledge .

Preissle , Judith , and Margaret D. Le Compte . 1984 . Ethnography and qualitative design in educational research . New York : Academic Press .

Raab , Jörg. , R. S. Mannak , and B. Cambre . 2015 . Combining structure, governance, and context: A configurational approach to network effectiveness . Journal of Public Administration Research and Theory 25 ( 2 ): 479 – 511 . doi: 10.1093/jopart/mut039

Raadschelders , J. C . 2011 . The future of the study of public administration: Embedding research object and methodology in epistemology and ontology . Public Administration Review 71 ( 6 ): 916 – 24 .

Ragin , Charles . 1987 . The comparative method: Moving beyond qualitative and quantitative methods . Berkeley : University of California Press .

Ragin , Charles C . 1998 . The logic of qualitative comparative analysis . International Review of Social History 43 ( S6 ): 105 – 24 . doi: 10.1017/S0020859000115111

Ragin , Charles C. , and Benoit Rihoux . 2004 . Qualitative Comparative Analysis (QCA): State of the Art and Prospects . Qualitative Methods 2 ( 2 ): 3 – 13 .

Rerup , C. , and M. S. Feldman . 2011 . Routines as a source of change in organizational schemata: The role of trial-and-error learning . Academy of Management Journal 54 ( 3 ): 577 – 610 .

Riccucci , Norma M . 2010 . Public administration: Traditions of inquiry and philosophies of knowledge . Washington, DC : Georgetown University Press .

Riessman , Catherine Kohler . 1993 . Narrative analysis . Vol. 30 . Thousand Oaks, CA : Sage .

Rihoux , Benoît , and Charles C. Ragin . 2008 . Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques . Vol. 51 . Thousand Oaks, CA : Sage .

Rivera , Lauren A . 2017 . “ When two bodies are (not) a problem: Gender and relationship status discrimination in academic hiring .” American Sociological Review 82 ( 6 ): 1111 – 1138 .

Rolfe , Gary . 2006 . Validity, trustworthiness and rigour: Quality and the idea of qualitative research . Journal of Advanced Nursing 53 ( 3 ): 304 – 10 .

Romzek , B. , and Melvin J. Dubnick . 1987 . Accountability in the Public Sector: Lessons from the Challenger Disaster . Public Administration Review 47 ( 3 ): 227 – 38 .

Romzek , B. , Kelly LeRoux , and Jeannette M. Blackmar . 2012 . A preliminary theory of informal accountability among network organizational actors . Public Administration Review 72 ( 3 ): 442 – 53 .

Romzek , B. , K. LeRoux , J. Johnston , R. J. Kempf , and J. S. Piatak . 2014 . Informal accountability in multisector service delivery collaborations . Journal of Public Administration Research and Theory 24 ( 4 ): 813 – 42 . doi: 10.1093/jopart/mut027

Sanger , Mary Bryna . 2013 . Does measuring performance lead to better performance ?. Journal of Policy Analysis and Management 32 ( 1 ): 185 – 203 .

Saz-Carranza , Angel , and Sonia M. Ospina . 2010 . The behavioral dimension of governing interorganizational goal-directed networks—Managing the unity-diversity tension . Journal of Public Administration Research and Theory . 21 ( 2 ): 327 – 65 .

Schein , Edgar H . 2003 . On dialogue, culture, and organizational learning . Reflections: The SoL Journal 4 ( 4 ): 27 – 38 . doi: 10.1162/152417303322004184

Schillemans , Thomas . 2013 . Moving beyond the clash of interests: On stewardship theory and the relationships between central government departments and public agencies . Public Management Review 15 ( 4 ): 541 – 62 .

Shanahan , Elizabeth A. , Michael D. Jones , and Mark K. McBeth . 2018 . How to conduct a Narrative Policy Framework study . The Social Science Journal . 55 ( 3 ): 332 – 345 .

Shanahan , Elizabeth A. , Michael D. Jones , Mark K. McBeth , and Ross R. Lane . 2013 . An angel on the wind: How heroic policy narratives shape policy realities: Narrative policy framework . Policy Studies Journal 41 ( 3 ): 453 – 83 . doi: 10.1111/psj.12025

Schneider , Anne , and Helen Ingram . 1993 . Social construction of target populations: Implications for politics and policy . American Political Science Review 87 ( 2 ): 334 – 47 .

Steiner , Elizabeth . 1988 . Methodology of theory building . Sydney : Educology Research Associates .

Stout , Margaret . 2013 . Preparing public administration scholars for qualitative inquiry: A status report . Public Administration Research 2 ( 1 ): 11 – 28 .

Thomann , Eva . 2015 . Is output performance all about the resources? A fuzzy ‐ set qualitative comparative analysis of street ‐ level bureaucrats in Switzerland . Public Administration 93 ( 1 ): 177 – 94 .

Toepler , Stefan . 2005 . Called to order: A board president in trouble . Nonprofit Management and Leadership 15 ( 4 ): 469 – 76 .

Van Slyke , David M . 2003 . The mythology of privatization in contracting for social services . Public Administration Review 63 ( 3 ): 296 – 315 . doi:10.1111/1540–6210.00291

Wasserman , Larry . 2013 . All of statistics: A concise course in statistical inference . New York : Springer Science & Business Media .

Waugh , William L. Jr , and Wesley W. Waugh . 2003 . Phenomenology and public administration . International Journal of Organization Theory & Behavior . 7 ( 3 ): 405 – 31 .

Weber , E. P. , and A. M. Khademian . 2008 . Wicked problems, knowledge challenges, and collaborative capacity builders in network settings . Public administration review 68 ( 2 ): 334 – 49 .

Weick , Karl E . 1993 . The collapse of sensemaking in organizations: The mann gulch disaster . Administrative Science Quarterly 38 ( 4 ): 628 . doi: 10.2307/2393339

Williamson , O. E . 1981 . The economics of organization: The transaction cost approach . American Journal of Sociology 87 ( 3 ): 548 – 77 .

Yin , Robert K . 2017 . Case study research and applications: Design and methods . Thousand Oaks, CA : Sage .

The method, methodology, epistemology coupling is a topic of considerable debate and concern in the field of qualitative methods ( Corbin and Strauss 2014 ; Haverland and Yanow 2012 ; Ospina et al 2017 ). Whether certain methods can or should be implemented by scholars embracing diverging epistemological stances is a topic warranting further discourse in the field of public management.

Reflexivity refers to the practice of being intentionally reflective about who you are both as a person situated within society and as a scholar professionally socialized within a cultural and institutional milieu. Specifically, reflexive practice calls upon scholars to consider how the totality of who they are as individuals influences the manner in which they approach scholarship, the questions they ask, the way the subjects of one’s inquiry may react/respond, and how one interprets what they observe. This is done with an eye toward critically examining how these factors may shape and constrain what one “finds” (for discussion, see Pillow 2003 ).

Within the QCA lexicon, results are referred to as causal pathways, although researchers are cautioned against the use of terms like causation as QCA uses a combinatorial logic/conjunctural causation instead of main effect/parameter estimate logic.

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1477-9803
• Print ISSN 1053-1858
• Copyright © 2024 Public Management Research Association
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Rigor or Reliability and Validity in Qualitative Research: Perspectives, Strategies, Reconceptualization, and Recommendations

Affiliation.

• 1 Brigitte S. Cypress, EdD, RN, CCRN, is an assistant professor of nursing, Lehman College and The Graduate Center, City University of New York.
• PMID: 28570380
• DOI: 10.1097/DCC.0000000000000253

Issues are still raised even now in the 21st century by the persistent concern with achieving rigor in qualitative research. There is also a continuing debate about the analogous terms reliability and validity in naturalistic inquiries as opposed to quantitative investigations. This article presents the concept of rigor in qualitative research using a phenomenological study as an exemplar to further illustrate the process. Elaborating on epistemological and theoretical conceptualizations by Lincoln and Guba, strategies congruent with qualitative perspective for ensuring validity to establish the credibility of the study are described. A synthesis of the historical development of validity criteria evident in the literature during the years is explored. Recommendations are made for use of the term rigor instead of trustworthiness and the reconceptualization and renewed use of the concept of reliability and validity in qualitative research, that strategies for ensuring rigor must be built into the qualitative research process rather than evaluated only after the inquiry, and that qualitative researchers and students alike must be proactive and take responsibility in ensuring the rigor of a research study. The insights garnered here will move novice researchers and doctoral students to a better conceptual grasp of the complexity of reliability and validity and its ramifications for qualitative inquiry.

• Qualitative Research*
• Reproducibility of Results
• Research Design*

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Qualitative Research:...

Qualitative Research: Rigour and qualitative research

• Related content
• Peer review
• Nicholas Mays , director of health services research a ,
• Catherine Pope , lecturer in social and behavioural medicine b
• a King's Fund Institute, London W2 4HT
• b Department of Epidemiology and Public Health, University of Leicester, Leicester LE1 6TP
• Correspondence to: Mr Mays.

Various strategies are available within qualitative research to protect against bias and enhance the reliability of findings. This paper gives examples of the principal approaches and summarises them into a methodological checklist to help readers of reports of qualitative projects to assess the quality of the research.

Criticisms of qualitative research

In the health field--with its strong tradition of biomedical research using conventional, quantitative, and often experimental methods--qualitative research is often criticised for lacking scientific rigour. To label an approach “unscientific” is peculiarly damning in an era when scientific knowledge is generally regarded as the highest form of knowing. The most commonly heard criticisms are, firstly, that qualitative research is merely an assembly of anecdote and personal impressions, strongly subject to researcher bias; secondly, it is argued that qualiative research lacks reproducibility--the research is so personal to the researcher that there is no guarantee that a different researcher would not come to radically different conclusions; and, finally, qualitative research is criticised for lacking generalisability. It is said that qualitative methods tend to generate large amounts of detailed information about a small number of settings.

Is qualitative research different?

The pervasive assumption underlying all these criticisms is that quantitative and qualitative approaches are fundamentally different in their ability to ensure the validity and reliability of their findings. This distinction, however, is more one of degree than of type. The problem of the relation of a piece of research to some presumed underlying “truth” applies to the conduct of any form of social research. “One of the greatest methodological fallacies of the last half century in social research is the belief that science is a particular set of techniques; it is, rather, a state of mind, or attitude, and the organisational conditions which allow that attitude to be expressed.” 1 In quantitative data analysis it is possible to generate statistical representations of phenomena which may or may not be fully justified since, just as in qualitative work, they will depend on the judgment and skill of the researcher and the appropriateness to the question answered of the data collected. All research is selective--there is no way that the researcher can in any sense capture the literal truth of events. All research depends on collecting particular sorts of evidence through the prism of particular methods, each of which has its strengths and weaknesses. For example, in a sample survey it is difficult for the researcher to ensure that the questions, categories, and language used in the questionnaire are shared uniformly by respondents and that the replies returned have the same meanings for all respondents. Similarly, research that relies exclusively on observation by a single researcher is limited by definition to the perceptions and introspection of the investigator and by the possibility that the presence of the observer may, in some way that is hard to characterise, have influenced the behaviour and speech that was witnessed. Britten and Fisher summarise the position neatly by pointing out that “there is some truth in the quip that quantitative methods are reliable but not valid and that qualitative methods are valid but not reliable.” 2

Strategies to ensure rigour in qualitative research

As in quantitative research, the basic strategy to ensure rigour in qualitative research is systematic and self conscious research design, data collection, interpretation, and communication. Beyond this, there are two goals that qualitative researchers should seek to achieve: to create an account of method and data which can stand independently so that another trained researcher could analyse the same data in the same way and come to essentially the same conclusions; and to produce a plausible and coherent explanation of the phenomenon under scrutiny. Unfortunately, many qualitative researchers have neglected to give adequate descriptions in their research reports of their assumptions and methods, particularly with regard to data analysis. This has contributed to some of the criticisms of bias from quantitative researchers.

Yet the integrity of qualitative projects can be protected throughout the research process. The remainder of this paper discusses how qualitative researchers attend to issues of validity, reliability, and generalisability.

Much social science is concerned with classifying different “types” of behaviour and distinguishing the “typical” from the “atypical.” In quantitative research this concern with similarity and difference leads to the use of statistical sampling so as to maximise external validity or generalisability. Although statistical sampling methods such as random sampling are relatively uncommon in qualitative investigations, there is no reason in principle why they cannot be used to provide the raw material for a comparative analysis, particularly when the researcher has no compelling a priori reason for a purposive approach. For example, a random sample of practices could be studied in an investigation of how and why teamwork in primary health care is more and less successful in different practices. However, since qualitative data collection is generally more time consuming and expensive than, for example, a quantitative survey, it is not usually practicable to use a probability sample. Furthermore, statistical representativeness is not a prime requirement when the objective is to understand social processes.

An alternative approach, often found in qualitative research and often misunderstood in medical circles, is to use systematic, non-probabilistic sampling. The purpose is not to establish a random or representative sample drawn from a population but rather to identify specific groups of people who either possess characteristics or live in circumstances relevant to the social phenomenon being studied. Informants are identified because they will enable exploration of a particular aspect of behaviour relevant to the research. This approach to sampling allows the researcher deliberately to include a wide range of types of informants and also to select key informants with access to important sources of knowledge.

“Theoretical” sampling is a specific type of non-probability sampling in which the objective of developing theory or explanation guides the process of sampling and data collection. 3 Thus, the analyst makes an initial selection of informants; collects, codes, and analyses the data; and produces a preliminary theoretical explanation before deciding which further data to collect and from whom. Once these data are analysed, refinements are made to the theory, which may in turn guide further sampling and data collection. The relation between sampling and explanation is iterative and theoretically led.

To return to the example of the study of primary care team working, some of the theoretically relevant characteristics of general practices affecting variations in team working might be the range of professions represented in the team, the frequency of opportunities for communication among team members, the local organisation of services, and whether the practice is in an urban, city, or rural area. These factors could be identified from other similar research and within existing social science theories of effective and ineffective team working and would then be used explicitly as sampling categories. Though not statistically representative of general practices, such a sample is theoretically informed and relevant to the research questions. It also minimises the possible bias arising from selecting a sample on the basis of convenience.

ENSURING THE RELIABILITY OF AN ANALYSIS

In many forms of qualitative research the raw data are collected in a relatively unstructured form such as tape recordings or transcripts of conversations. The main ways in which qualitative researchers ensure the retest reliability of their analyses is in maintaining meticulous records of interviews and observations and by documenting the process of analysis in detail. While it is possible to analyse such data singlehandedly and use ways of classifying and categorising the data which emerge from the analysis and remain implicit, more explicit group approaches, which perhaps have more in common with the quantitative social sciences, are increasingly used. The interpretative procedures are often decided on before the analysis. Thus, for example, computer software is available to facilitate the analysis of the content of interview transcripts. 4 A coding frame can be developed to characterise each utterance (for example, in relation to the age, sex, and role of the speaker; the topic; and so on), and transcripts can then be coded by more than one researcher. 5 One of the advantages of audiotaping or videotaping is the opportunity the tapes offer for subsequent analysis by independent observers.

The reliability of the analysis of qualitative data can be enhanced by organising an independent assessment of transcripts by additional skilled qualitative researchers and comparing agreement between the raters. For example, in a study of clinical encounters between cardiologists and their patients which looked at the differential value each derived from the information provided by echocardiography, transcripts of the clinic interviews were analysed for content and structure by the principal researcher and by an independent panel, and the level of agreement was assessed. 6

SAFEGUARDING VALIDITY

Alongside issues of reliability, qualitative researchers give attention to the validity of their findings. “Triangulation” refers to an approach to data collection in which evidence is deliberately sought from a wide range of different, independent sources and often by different means (for instance, comparing oral testimony with written records). This approach was used to good effect in a qualitative study of the effects of the introduction of general management into the NHS. The accounts of doctors, managers, and patient advocates were explored in order to identify patterns of convergence between data sources to see whether power relations had shifted appreciably in favour of professional managers and against the medical profession. 7

The differences in GPs' interviews with parents of handicapped and non-handicapped children have been shown by qualitative methods

• Download figure
• Open in new tab
• Download powerpoint

Validation strategies sometimes used in qualitative research are to feed the findings back to the participants to see if they regard the findings as a reasonable account of their experience 8 and to use interviews or focus groups with the same people so that their reactions to the evolving analysis become part of the emerging research data. 9 If used in isolation these techniques assume that fidelity to the participants' commonsense perceptions is the touchstone of validity. In practice, this sort of validation has to be set alongside other evidence of the plausibility of the research account since different groups are likely to have different perspectives on what is happening. 10

A related analytical and presentational issue is concerned with the thoroughness with which the researcher examines “negative” or “deviant” cases--those in which the researcher's explanatory scheme appears weak or is contradicted by the evidence. The researcher should give a fair account of these occasions and try to explain why the data vary. 11 In the same way, if the findings of a single case study diverge from those predicted by a previously stated theory, they can be useful in revising the existing theory in order to increase its reliability and validity.

VALIDITY AND EXPLANATION

It is apparent in qualitative research, particularly in observational studies (see the next paper in this series for more on observational methods), that the researcher can be regarded as a research instrument. 12 Allowing for the inescapable fact that purely objective observation is not possible in social science, how can the reader judge the credibility of the observer's account? One solution is to ask a set of questions: how well does this analysis explain why people behave in the way they do; how comprehensible would this explanation be to a thoughtful participant in the setting; and how well does the explanation it advances cohere with what we already know?

This is a challenging enough test, but the ideal test of a qualitative analysis, particularly one based on observation, is that the account it generates should allow another person to learn the “rules” and language sufficiently well to be able to function in the research setting. In other words, the report should carry sufficient conviction to enable someone else to have the same experience as the original observer and appreciate the truth of the account. 13 Few readers have the time or inclination to go to such lengths, but this provides an ideal against which the quality of a piece of qualitative work can be judged.

The development of “grounded theory” 3 offers another response to this problem of objectivity. Under the strictures of grounded theory, the findings must be rendered through a systematic account of a setting that would be clearly recognisable to the people in the setting (by, for example, recording their words, ideas, and actions) while at the same time being more structured and self consciously explanatory than anything that the participants themselves would produce.

Attending to the context

Some pieces of qualitative research consist of a case study carried out in considerable detail in order to produce a naturalistic account of everyday life. For example, a researcher wishing to observe care in an acute hospital around the clock may not be able to study more than one hospital. Again the issue of generalisability, or what can be learnt from a single case, arises. Here, it is essential to take care to describe the context and particulars of the case study and to flag up for the reader the similarities and differences between the case study and other settings of the same type. A related way of making the best use of case studies is to show how the case study contributes to and fits with a body of social theory and other empirical work. 12 The final paper in this series discusses qualitative case studies in more detail.

COLLECTING DATA DIRECTLY

Another defence against the charge that qualitative research is merely impressionistic is that of separating the evidence from secondhand sources and hearsay from the evidence derived from direct observation of behaviour in situ. It is important to ensure that the observer has had adequate time to become thoroughly familiar with the milieu under scrutiny and that the participants have had the time to become accustomed to having the researcher around. It is also worth asking whether the observer has witnessed a wide enough range of activities in the study site to be able to draw conclusions about typical and atypical forms of behaviour--for example, were observations undertaken at different times? The extent to which the observer has succeeded in establishing an intimate understanding of the research setting is often shown in the way in which the subsequent account shows sensitivity to the specifics of language and its meanings in the setting.

MINIMISING RESEARCHER BIAS IN THE PRESENTATION OF RESULTS

Although it is not normally appropriate to write up qualitative research in the conventional format of the scientific paper, with a rigid distinction between the results and discussion sections of the account, it is important that the presentation of the research allows the reader as far as possible to distinguish the data, the analytic framework used, and the interpretation. 1 In quantitative research these distinctions are conventionally and neatly presented in the methods section, numerical tables, and the accompanying commentary. Qualitative research depends in much larger part on producing a convincing account. 14 In trying to do this it is all too easy to construct a narrative that relies on the reader's trust in the integrity and fairness of the researcher. The equivalent in quantitative research is to present tables of data setting out the statistical relations between operational definitions of variables without giving any idea of how the phenomena they represent present themselves in naturally occurring settings. 1 The need to quantify can lead to imposing arbitrary categories on complex phenomena, just as data extraction in qualitative research can be used selectively to tell a story that is rhetorically convincing but scientifically incomplete.

The problem with presenting qualitative analyses objectively is the sheer volume of data customarily available and the relatively greater difficulty faced by the researcher in summarising qualitative data. It has been suggested that a full transcript of the raw data should be made available to the reader on microfilm or computer disk, 11 although this would be cumbersome. Another partial solution is to present extensive sequences from the original data (say, of conversations), followed by a detailed commentary.

Another option is to combine a qualitative analysis with some quantitative summary of the results. The quantification is used merely to condense the results to make them easily intelligible; the approach to the analysis remains qualitative since naturally occurring events identified on theoretical grounds are being counted. The table shows how Silverman compared the format of the doctor's initial questions to parents in a paediatric cardiology clinic when the child was not handicapped with a smaller number of cases when the child had Down's syndrome. A minimum of interpretation was needed to contrast the two sorts of interview. 15 16

Assessing a piece of qualitative research

This short paper has shown some of the ways in which researchers working in the qualitative tradition have endeavoured to ensure the rigour of their work. It is hoped that this summary will help the prospective reader of reports of qualitative research to identify some of the key questions to ask when trying to assess its quality. A range of helpful checklists has been published to assist readers of quantitative research assess the design 17 and statistical 18 and economic 19 aspects of individual published papers and review articles. 20 Likewise, the contents of this paper have been condensed into a checklist for readers of qualitative studies, covering design, data collection, analysis, and reporting (box). We hope that the checklist will give readers of studies in health and health care research that use qualitative methods the confidence to subject them to critical scrutiny.

Questions to ask of a qualitative study

Overall, did the researcher make explicit in the account the theoretical framework and methods used at every stage of the research?

Was the context clearly described?

Was the sampling strategy clearly described and justified?

Was the sampling strategy theoretically com-prehensive to ensure the generalisability of the conceptual analyses (diverse range of individuals and settings, for example)?

How was the fieldwork undertaken? Was it described in detail?

Could the evidence (fieldwork notes, inter-view transcripts, recordings, documentary analysis, etc) be inspected independently by others; if relevant, could the process of transcription be independently inspected?

Were the procedures for data analysis clearly described and theoretically justified? Did they relate to the original research questions? How were themes and concepts identified from the data?

Was the analysis repeated by more than one researcher to ensure reliability?

Did the investigator make use of quantitative evidence to test qualitative conclusions where appropriate?

Did the investigator give evidence of seeking out observations that might have contradicted or modified the analysis?

Was sufficient of the original evidence pre-sented systematically in the written account to satisfy the sceptical reader of the relation between the interpretation and the evidence (for example, were quotations numbered and sources given)?

Form of doctor's questions to parents at a paediatric cardiology clinic 15

• View inline

Further reading

Hammersley M. Reading ethnographic research. London: Longman, 1990.

• MacDonald I ,
• Britten N ,
• Glaser BG ,
• Krippendorff K
• Pollitt C ,
• Harrison S ,
• Hunter DJ ,
• McKeganey NP ,
• Glassner B ,
• Silverman D
• Fowkes FGR ,
• Gardner MJ ,
• Campbell MJ
• Department of Clinical Epidemiology and Biostatistics

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

17 20. Quality in qualitative studies: Rigor in research design

Chapter outline.

• Introduction to qualitative rigor (13 minute read)
• Ethical responsibility and cultural respectfulness (4 minute read)
• Critical considerations (6 minute read)
• Data capture: Striving for accuracy in our raw data (6 minute read)
• Data management: Keeping track of our data and our analysis (8 minute read)
• Tools to account for our influence (22 minute read)

Content warning: Examples in this chapter contain references to fake news, mental health treatment, peer-support, misrepresentation, equity and (dis)honesty in research.

We hear a lot about fake news these days. Fake news has to do with the quality of journalism that we are consuming. It begs questions like: does it contain misinformation, is it skewed or biased in its portrayal of stories, does it leave out certain facts while inflating others. If we take this news at face value, our opinions and actions may be intentionally manipulated by poor quality information. So, how do we avoid or challenge this? The oversimplified answer is, we find ways to check for quality. While this isn’t a chapter dedicated to fake news, it does offer an important comparison for the focus of this chapter, rigor in qualitative research.  Rigor is concerned with the quality of research that we are designing and consuming. While I devote a considerable amount of time in my clinical class talking about the importance of adopting a non-judgmental stance in practice, that is not the case here; I want you to be judgmental, critical thinkers about research! As a social worker who will hopefully be producing research (we need you!) and definitely consuming research, you need to be able to differentiate good science from rubbish science. Rigor will help you to do this.

This chapter will introduce you to the concept of rigor and specifically, what it looks like in qualitative research.  We will begin by considering how rigor relates to issues of ethics and how thoughtfully involving community partners in our research can add additional dimensions in planning for rigor. Next, we will look at rigor in how we capture and manage qualitative data, essentially helping to ensure that we have quality raw data to work with for our study.  Finally, we will devote time to discussing how researchers, as human instruments, need to maintain accountability throughout the research process. Finally, we will examine tools that encourage this accountability and how they can be integrated into your research design. Our hope is that by the end of this chapter, you will begin to be able to identify some of the hallmarks of quality in qualitative research, and if you are designing a qualitative research proposal, that you consider how to build these into your design.

20.1 Introduction to qualitative rigor

Learning Objectives

Learners will be able to…

• Identify the role of rigor in qualitative research and important concepts related to qualitative rigor
• Discuss why rigor is an important consideration when conducting, critiquing and consuming qualitative research
• Differentiate between quality in quantitative and qualitative research studies

In Chapter 10 we talked about quality in quantitative studies, but we built our discussion around concepts like reliability and validity .  With qualitative studies, we generally think about quality in terms of the concept of rigor . The difference between quality in quantitative research and qualitative research extends beyond the type of data (numbers vs. words/sounds/images). If you sneak a peek all the way back to Chapter 5 , we discussed the idea of different paradigms or fundamental frameworks for how we can think about the world. These frameworks value different kinds of knowledge, arrive at knowledge in different ways, and evaluate the quality of knowledge with different criteria. These differences are essential in differentiating qualitative and quantitative work.

Quantitative research generally falls under a positivist paradigm, seeking to uncover knowledge that holds true across larger groups of people.  To accomplish this, we need to have tools like reliability and validity to help produce internally consistent and externally generalizable findings (i.e. was our study design dependable and do our findings hold true across our population).

In contrast, qualitative research is generally considered to fall into an alternative paradigm (other than positivist), such as the interpretive paradigm which is focused on the subjective experiences of individuals and their unique perspectives. To accomplish this, we are often asking participants to expand on their ideas and interpretations. A positivist tradition requires the information collected to be very focused and discretely defined (i.e. closed questions with prescribed categories). With qualitative studies, we need to look across unique experiences reflected in the data and determine how these experiences develop a richer understanding of the phenomenon we are studying, often across numerous perspectives.

Rigor is a concept that reflects the quality of the process used in capturing, managing, and analyzing our data as we develop this rich understanding. Rigor helps to establish standards through which qualitative research is critiqued and judged, both by the scientific community and by the practitioner community.

For the scientific community, people who review qualitative research studies submitted for publication in scientific journals or for presentations at conferences will specifically look for indications of rigor, such as the tools we will discuss in this chapter.  This confirms for them that the researcher(s) put safeguards in place to ensure that the research took place systematically and that consumers can be relatively confident that the findings are not fabricated and can be directly connected back to the primary sources of data that was gathered or the secondary data that was analyzed.

As a note here, as we are critiquing the research of others or developing our own studies, we also need to recognize the limitations of rigor.  No research design is flawless and every researcher faces limitations and constraints. We aren’t looking for a researcher to adopt every tool we discuss below in their design.  In fact, one of my mentors, speaks explicitly about “misplaced rigor”, that is, using techniques to support rigor that don’t really fit what you are trying to accomplish with your research design. Suffice it to say that we can go overboard in the area of rigor and it might not serve our study’s best interest.  As a consumer or evaluator of research, you want to look for steps being taken to reflect quality and transparency throughout the research process, but they should fit within the overall framework of the study and what it is trying to accomplish. 

From the perspective of a practitioner, we also need to be acutely concerned with the quality of research. Social work has made a commitment, outlined in our Code of Ethics (NASW,2017) , to competent practice in service to our clients based on “empirically based knowledge” (subsection 4.01). When I think about my own care providers, I want them to be using “good” research – research that we can be confident was conducted in a credible way and whose findings are honestly and clearly represented. Don’t our clients deserve the same from us?

As providers, we will be looking to qualitative research studies to provide us with information that helps us better understand our clients, their experiences, and the problems they encounter.  As such, we need to look for research that accurately represents:

• Who is participating in the study
• What circumstances is the study being conducted under
• What is the research attempting to determine

Further, we want to ensure that:

• Findings are presented accurately and reflect what was shared by participants ( raw data )
•  A reasonably good explanation of how the researcher got from the raw data to their findings is presented
• The researcher adequately considered and accounted for their potential influence on the research process

As we talk about different tools we can use to help establish qualitative rigor, I will try to point out tips for what to look for as you are reading qualitative studies that can reflect these. While rigor can’t “prove” quality, it can demonstrate steps that are taken that reflect thoughtfulness and attention on the part of the researcher(s). This is a link from the American Psychological Association on the topic of reviewing qualitative research manuscripts. It’s a bit beyond the level of critiquing that I would expect from a beginning qualitative research student, however, it does provide a really nice overview of this process.  Even if you aren’t familiar with all the terms, I think it can be helpful in giving an overview of the general thought process that should be taking place.

To begin breaking down how to think about rigor, I find it helpful to have a framework to help understand different concepts that support or are associated with rigor. Lincoln and Guba (1985) have suggested such a framework for thinking about qualitative rigor that has widely contributed to standards that are often employed for qualitative projects. The overarching concept around which this framework is centered is trustworthiness .  Trustworthiness is reflective of how much stock we should put in a given qualitative study – is it really worth our time, headspace, and intellectual curiosity? A study that isn’t trustworthy suggests poor quality resulting from inadequate forethought, planning, and attention to detail in how the study was carried out.  This suggests that we should have little confidence in the findings of a study that is not trustworthy.

According to Lincoln and Guba (1985) [1] trustworthiness is grounded in responding to four key ideas and related questions to help you conceptualize how they relate to your study. Each of these concepts is discussed below with some considerations to help you to compare and contrast these ideas with more positivist or quantitative constructs of research quality.

Truth value

You have already been introduced to the concept of internal validity . As a reminder, establishing internal validity is a way to ensure that the change we observe in the dependent variable is the result of the variation in our independent variable – did we actually design a study that is truly testing our hypothesis.  In much/most qualitative studies we don’t have hypotheses, independent or dependent variables, but we do still want to design a study where our audience (and ourselves) can be relatively sure that we as the researcher arrived at our findings through a systematic and scientific process, and that those findings can be clearly linked back to the data we used and not some fabrication or falsification of that data; in other words, the truth value of the research process and its findings. We want to give our readers confidence that we didn’t just make up our findings or “see what we wanted to see”.

Applicability

• who we were studying
• how we went about studying them
• what we found

Consistency

These concepts reflect a set of standards that help to determine the integrity of qualitative studies. At the end of this chapter you will be introduced to a range of tools to help support or reflect these various standards in qualitative research. Because different qualitative designs (e.g. phenomenology, narrative, ethnographic), that you will learn more about in chapter 22 emphasize or prioritize different aspects of quality, certain tools will be more appropriate for these designs. Since this chapter is intended to give you a general overview of rigor in qualitative studies, exploring additional resources will be necessary to best understand which of these concepts are prioritized in each type of design and which tools best support them.

Key Takeaways

• Qualitative research is generally conducted within an interpretativist paradigm.  This differs from the post-positivist paradigm in which most quantitative research originates. This fundamental difference means that the overarching aim of these different approaches to knowledge building differ, and consequently, our standards for judging the quality of research within these paradigms differ.
• Assessing the quality of qualitative research is important, both from a researcher and a practitioner perspective.  On behalf of our clients and our profession, we are called to be critical consumers of research. To accomplish this, we need strategies for assessing the scientific rigor with which research is conducted.
• Trustworthiness and associated concepts, including credibility, transferablity, dependability and confirmability, provide a framework for assessing rigor or quality in qualitative research.

20.2 Ethical responsibility and cultural respectfulness

• Discuss the connection between rigor and ethics as they relate to the practice of qualitative research
• Explain how the concepts of accountability and transparency lay an ethical foundation for rigorous qualitative research

The two concepts of rigor and ethics in qualitative research are closely intertwined.  It is a commitment to ethical research that leads us to conduct research in rigorous ways, so as not to put forth research that is of poor quality, misleading, or altogether false.  Furthermore, the tools that demonstrate rigor in our research are reinforced by solid ethical practices.  For instance, as we build a rigorous protocol for collecting interview data, part of this protocol must include a well-executed, ethical informed consent process; otherwise, we hold little hope that our efforts will lead to trustworthy data . Both ethics and rigor shine a light on our behaviors as researchers.  These concepts offer standards by which others can critique our commitment to quality in the research we produce. They are both tools for accountability in the practice of research.

Related to this idea of accountability, rigor requires that we promote a sense of t ransparenc y in the qualitative research process.  We will talk extensively in this chapter about tools to help support this sense of transparency, but first, I want to explore why transparency is so important for ethical qualitative research. As social workers, our own knowledge, skills, and abilities to help serve our clients are our tools.  Similarly, qualitative research demands the social work researcher be an actively involved human instrument in the research process.

While quantitative researchers also makes a commitment to transparency, they may have an easier job of demonstrating it.  Let’s just think about the data analysis stage of research. The quantitative researcher has a data set, and based on that data set there are certain tests that they can run. Those tests are mathematically defined and computed by statistical software packages and we have established guidelines for interpreting the results and reporting the findings. There is most certainly tremendous skill and knowledge exhibited in the many decisions that go into this analysis process; however, the rules and requirements that lay the foundation for these mathematical tests mean that much of this process is prescribed for us. The prescribed procedures offer quantitative researchers a shorthand for talking about their transparency.

In comparison, the qualitative researcher, sitting down with their data for analysis will engage in a process that will require them to make hundreds or thousands of decisions about what pieces of data mean, what label they should have, how they relate to other ideas, what the larger significance is as it relates to their final results. That isn’t to say that we don’t have procedures and processes as qualitative researchers, we just can’t rely on mathematics to make these decisions precise and clear.  We have to rely on ourselves as human instruments. Adopting a commitment to transparency in our research as qualitative researchers means that we are actively describing for our audience the role we have as human instruments and we consider how this is shaping the research process. This allows us to avoid unethically representing what we did in our research process and what we found.

I think that as researchers we can sometimes think of data as an object that is not inherently valuable, but rather a means to an end.  But if we see qualitative data as part of sacred stories that are being shared with us, doesn’t it feel like a more precious resource? Something worthy of thoughtfully and even gently gathering, something that needs protecting and safe-keeping. Adhering to a rigorous research process can help to honor these commitments and avoid the misuse of data as a precious resource. Thinking like this will hopefully help us to demonstrate greater cultural humility as social work researchers.

• Ethics and rigor both are interdependent and call attention to our behaviors as researchers and the quality and care with which our research is conducted.
• Accountability and transparency in qualitative research helps to demonstrate that as researchers we are acting with integrity. This means that we are clear about how we are conducting our research, what decisions we are making during the research process, and how we have arrived at these decisions.

While this activity is early in the chapter, I want you to consider for a few moments about how accountability relates to your research proposal.

• Who are you accountable to as you carry plan and carry out your research
• In what ways are you accountable to each of the people you listed in the previous question?

20.3 Critical considerations

• Identify some key questions for a critical critique of research planning and design
• Differentiate some alternative standards for rigor according to more participatory research approaches

As I discussed above, rigor shines a spotlight on our actions as researchers.  A critical perspective is one that challenges traditional arrangements of power, control and the role of structural forces in maintaining oppression and inequality in society. From this perspective, rigor takes on additional meaning beyond the internal integrity of the qualitative processes used by you or I as researchers, and suggest that standards of quality need to address accountability to our participants and the communities that they represent, NOT just the scientific community. There are many evolving dialogues about what criteria constitutes “good” research from critical traditions, including participatory and empowerment approaches that have their roots in critical perspective. These discussions could easily stand as their own chapter, however, for our purposes, we will borrow some questions from these critical debates to consider how they might inform the work we do as qualitative researchers.

Who gets to ask the questions?

In the case of your research proposal, chances are you are outlining your research question. Because our research question truly drives our research process, it carries a lot of weight in the planning and decision-making process of research. In many instances, we bring our f ully-formed research projects to participants, and they are only involved in the collection of data.  But critical approaches would challenge us to involve people who are impacted by issues we are studying from the onset. How can they be involved in the early stages of study development, even in defining our question? If we treat their lived experience as expertise on the topic, why not start early using this channel to guide how we think about the issue? This challenges us to give up some of our control and to listen for the “right” question before we ask it.

Who owns the data and the findings?

Answering this question from a traditional research approach is relatively clear – the researcher or rather, the university or research institution they represent. However, critical approaches question this.  Think about this specifically in terms of qualitative research. Should we be “owning” pieces of other people’s stories, since that is often the data we are working with? What say do people get in what is done with their stories and the findings that are derived from them? Unfortunately, there aren’t clear answers. These are some critical questions that we need to struggle with as qualitative researchers.

• How can we disrupt or challenge current systems of data ownership, empowering participants to maintain greater rights?
• What could more reciprocal research ownership arrangments look like?
• What are the benefits and consequences of disrupting this system? 
• What are the benefits and consequences of perpetuating our current system?

What is the sustained impact of what I’m doing?

As qualitative researchers, our aim is often exploring meaning and developing understanding of social phenomena. However, criteria from more critical traditions challenge us to think more tangibly and with more immediacy. They require us to answer questions about how our involvement with this specific group of people within the context of this project may directly benefit or harm the people involved. This not only applies in the present but also in the future.

We need to consider questions like:

• How has our interaction shaped participants’ perceptions of research?
• What are the ripple effects left behind from the questions we raised by our study?
• What thoughts or feelings have been reinforced or challenged, both within the community but also for outsiders?
• Have we built/strengthened/damaged relationships?
• Have we expanded/depleted resources for participants?

We need to reflect on these topics in advance and carefully considering the potential ramifications of our research before we begin. This helps to demonstrate critical rigor in our approach to research planning.  Furthermore, research that is being conducted in participatory traditions should actively involve participants and other community members to define what the immediate impacts of the research should be.  We need to ask early and often, what do they need as a community and how can research be a tool for accomplishing this? Their answers to these questions then become the criteria on which our research is judged.  In designing research for direct and immediate change and benefit to the community, we also need to think about how well we are designing for sustainable change.  Have we crafted a research project that creates lasting transformation, or something that will only be short-lived?

As students and as scholars we are often challenged by constraints as we address issues of rigor, especially some of the issues raised here. One of the biggest constraints is time.  As a student, you are likely building a research proposal while balancing many demands on your time. To actively engage community members and to create sustainable research projects takes considerable time and commitment. Furthermore, we often work in highly structured systems that have many rules and regulations that can make doing things differently or challenging convention quite hard.  However, we can begin to make a more equity-informed research agenda by:

• Reflecting on issues of power and control in our own projects
• Learning from research that models more reciprocal relationships between researcher and researched
• Finding new and creative ways to actively involve participants in the process of research and in sharing the benefits of research

In the resource box below, you will find links for a number of sources to learn more about participatory research methods that embody the critical perspective in research that we have been discussing.

As we turn our attention to rigor in the various aspects of the qualitative research process, continue to think about what critical criteria might also apply to each of these areas.

• Traditional research methods, including many qualitative approaches, may fail to challenge the ways that the practice of research can disenfranchise and disempower individauls and communities.
• Researchers from critical perspectives often question the power arrangments, roles, and objectives of more traditional research methods, and have been developing alternatives such as participatory research approaches. These participatory approaches engage participants in much more active ways and furthermore, they evaluate the quality of research by the direct and sustained benefit that it brings to participants and their communities.

Bergold, J., & Thomas, S. (2012). Participatory research methods: A methodological approach in motion.  Historical Social Research/Historische Sozialforschun g, 13 (1), 191-222. https://www.jstor.org/stable/41756482?casa_token=RSMr_e37Hp0AAAAA%3ApH–Cs2L–zUUf7uUi_arUBZtHurNvVFE9P5anZVLtV0VgPSATK54BuTux3ZzE9BgqHoSU006D6-04OD_9LW50dpLb0t8CEocWGmX-G8LebgoC3Bvbv6&seq=1#metadata_info_tab_contents

Center for Community Health and Development, University of Kansas. (n.d.). Community toolbox: Section.2 Community-based participatory research [webpage]. https://ctb.ku.edu/en/table-of-contents/evaluate/evaluation/intervention-research/main

New Tactics in Human Rights. (n.d.). Participatory research for action [webpage]. https://www.newtactics.org/conversation/participatory-research-action

Pain, R., Whitman, G., Milledge, D., & Lune Rivers Trust. (2010). Participatory action research toolkit: An introduction to using PAR as an approach to learning, research and action. http://www.communitylearningpartnership.org/wp-content/uploads/2017/01/PARtoolkit.pdf

Participate. (n.d.). Participatory research methods [webpage]. https://participatesdgs.org/methods/

20.4 Data capture: Striving for accuracy in our raw data

• Explain the importance of paying attention to the data collection process for ensuring rigor in qualitative research
• Identify key points that they will need to consider and address in developing a plan for gathering data to support rigor in their study

It is very hard to make a claim that research was conducted in a rigorous way if we don’t start with quality raw data. That is to say, if we botch our data collection, we really can’t produce trustworthy findings, no matter how good our analysis is. So what is quality raw data? From a qualitative research perspective, quality raw data means that the data we capture provides an accurate representation of what was shared with us by participants or through other data sources, such as documents. This section is meant to help you consider rigor as it pertains to how you capture your data. This might mean how you document the information from your interviews or focus groups, how you record your field notes as you are conducting observations, what you track down and access for other artifacts, or how you produce your entries in your reflexive journal (as this can become part of your data, as well).

This doesn’t mean that all your data will look the same. However, you will want to anticipate the type(s) of data you will be collecting and what format they will be in.  In addition, whenever possible and appropriate, you will want the data you collect to be in a consistent format.  So, if you are conducting interviews and you decide that you will be capturing this data by taking field notes, you will use a similar strategy for gathering information at each interview.  You would avoid using field notes for some, recording and transcribing others, and then having emailed responses from the remaining participants.  You might be wondering why this matters, after all, you are asking them the same questions. However, using these different formats to capture your data can make your data less comparable. This may have led to different information being shared by the participant and different information being captured by the researcher.  For instance, if you rely on email responses, you lose the ability to follow up with probing questions you may have introduced in an in-person interview. Those participants who were recorded may not have felt as free to share information when compared to those interviews where you took field notes. It becomes harder to know if variation in your data is due to diversity in peoples’ experiences or just differences in how you went about capturing your data. Now we will turn our attention to quality in different types of data.

As qualitative researchers, we often are dealing with written data. At times, it may be participants who are doing the writing.  We may ask participants to provide written responses to questions or we may use writing samples as artifacts that are produced for some other purpose that we have permission to include in our study. In either case, ideally we are including this written data with as little manipulation as possible. If we do things like take passages or ideas out of context or interpret segments in our own words, we run a much greater risk of misrepresenting the data that is being shared with us. This is a direct threat to rigor, compromising the quality of the raw data we are collecting.   If we need to clarify what a participant means by one of their responses and we have the opportunity to follow up with them, we want to capture their own words as closely as we can when they provide their explanation.  This is also true if we ask participants to provide us with drawings.  For instance, we may ask youth to provide a drawn response to a question as an age-appropriate way to respond to a question, but we might follow-up by asking them to explain their drawing to us.  We would want to capture their description as close to their own words as possible, including both the drawing and the description in our data.

Researchers may also be responsible for producing written data. Rigorous field notes strive to capture participants’ words as accurately as possible, which usually means quoting more and paraphrasing less.  Of course we can’t avoid paraphrasing altogether (unless you have incredible shorthand skills, which I definitely do not), but the more interpreting or filtering we do as we capture our data, the less trustworthy it becomes. You also want to stick to a consistent method of recording your field notes.  It becomes much harder to analyze your data if you have one system one day and another system another day.  The quality of the notes may differ greatly and differences in organization may make it challenging to compare across your data. Finally, rigorous field notes usually capture context, as well.  If you are gathering field notes for an interview or during a focus group, this may mean that you take note of non-verbal information during the exchange. If you are conducting an observation, your field notes might contain detailed information about the setting and circumstances of the observation.

As qualitative researchers, we may also be working with audio, video, or other forms of media data.  Much of what we have already discussed in respect to written data also applies to these data formats, as well. The less we manipulate or change the original data source, the better. For example, if you have an audio recording of your focus group, you want your transcript to be as close to verbatim as possible. Also, if we are working with a visual or aural medium, like a performance, capturing context and description – including audience reactions – with as much detail as possible is vital if we are looking to analyze the meaning of such an event or experience.

This topic shouldn’t require more than a couple sentences as you write up your research proposal. However, these sentences should reflect some careful forethought and planning. Remember, this is the hand-off! If you are a relay runner, this is the point where the baton gets passed as the participant or source transfers information to the study. Also, you want to ensure that you select a strategy that can be consistent and part of systematic process. Now we need to come up with a plan for managing our data.

Data will be collected using semi-structured interviews. Interviews will be digitally recorded and transcribed verbatim. In addition, the researcher will take field notes during each interview (see field note template, appendix A).  

As they are gathered, documents will be assigned a study identification number. Along with their study ID, a brief description of the document, its source, and any other historical information will be kept in the data tracking log (see data tracking log, appendix B).   

• Anticipating and planning for how you will systematically and consistently gather your data is crucial for a rigorous qualitative research project.
• When conducting qualitative research, we not only need to consider the data that we collect from other sources, but the data that we produce ourselves.  As human instruments in the research process, our reaction to the data also becomes a form of data that can shape our findings.  As such, we need to think about how we can capture this as well.

How will you ensure that you use a consistent and systematic approach for qualitative data collection in your proposal?

20.5 Data management: Keeping track of our data and our analysis

• Explain how data management and data analysis in qualitative projects can present unique challenges or opportunities for demonstrating quality in the research process
• Plan for key elements to address or include in a data management plan that supports qualitative rigor in the study

Elements to think about

Once data collection begins, we need a plan for what we are going to do with it. As we talked about in our chapter devoted to qualitative data collection, this is often an important point of departure between quantitative and qualitative methods.  Quantitative research tends to be much more sequential, meaning that first we collect all the data, then we analyze the data. If we didn’t do it this way, we wouldn’t know what numbers we are dealing with.  However, with qualitative data, we are usually collecting and beginning to analyze our data simultaneously. This offers us a great opportunity to learn from our data as we are gathering it. However, it also means that if you don’t have a plan for how you are going to manage these dual processes of data collection and data analysis, you are going to get overwhelmed twice as fast!  A rigorous process will have a clearly defined process for labeling and tracking your data artifacts, whether they are text documents (e.g. transcripts, newspaper clippings, advertisements), photos, videos, or audio recordings. These may be physical documents, but more often than not, they are electronic. In either case, a clear, documented labeling system is required. This becomes very important because you are going to need to come back to this artifact at some point during your analysis and you need to have a way of tracking it down. Let’s talk a bit more about this.

You were introduced to the term iterative process in our previous discussions about qualitative data analysis. As a reminder, an iterative process is one that involves repetition, so in the case of working with qualitative data, it means that we will be engaging in a repeating and evolving cycle of reviewing our data, noting our initial thoughts and reactions about what the data means, collecting more data, and going back to review the data again.  Figure 20.1 depicts this iterative process. To adopt a rigorous approach to qualitative analysis, we need to think about how we will capture and document each point of this iterative process. This ishow we demonstrate transparency in our data analysis process, how we detail the work that we are doing as human instruments.

During this process, we need to consider:

• How will we capture our thoughts about the data, including what we are specifically responding to in the data?
• How do we introduce new data into this process? 
• How do we record our evolving understanding of the data and what those changes are prompted by?

So we have already talked about the importance of labeling our artifacts, but each artifact is likely to contain many ideas.  For instance, think about the many ideas that are shared in a single interview.  Because of this, we need to also have a clear and standardized way of labeling smaller segments of data within each artifact that represent discrete or separate ideas. If you recall back to our analysis chapter, these labels are called units . You are likely to have many, many units in each artifact. Additionally, as suggested above, you need a way to capture your thought process as you respond to the data.  This documentation is called memoing , a term you were introduced to in our analysis chapter.  These various components, labeling your artifacts, labeling your units, and memoing, come together as you produce a rigorous plan for how you document your data analysis.  Again, rigor here is closely associated with transparency.  This means that you are using these tools to document a clear road map for how you got from your raw data to your findings. The term for this road map is an audit trail , and we will speak more about it in the next section. The test of this aspect of rigor becomes your ability to work backwards, or better yet, for someone else to work backwards.  Could someone not connected with your project look at your findings, and using your audit trail, trace these ideas all the way back to specific points in your raw data? The term for this is having an external audit and will also be further explained below.  If you can do this, we sometimes say that your findings are clearly “grounded in your data”.

What our plan for data management might look like.

If you are working with physical data, you will need a system of logging and storing your artifacts.  In addition, as you break your artifacts down into units you may well be copying pieces of these artifacts onto small note cards or post-its that serve as your data units.  These smaller units become easier to manipulate and move around as you think about what ideas go together and what they mean collectively.  However, each of these smaller units need a label that links them back to their artifact. But why do I have to go through all this? Well, it isn’t just for the sake of transparency and being able to link your findings back to the original raw data, although that is certainly important. You also will likely reach a point in your analysis where themes are coming together and you are starting to make sense of things. When this occurs, you will have a pile of units from various artifacts under each of these themes.  At this point you will want to know where the information in the units came from. If it was verbal data, you will want to know who said it or what source it came from.  This offers us important information about the context of our findings and who/what they are connected to. We can’t determine this unless we have a good labeling system.

You will need to come up with a system that makes sense to you and fits for your data.  As an example, I’m often working with transcripts from interviews or focus groups.  As I am collecting my data, each transcript is numbered as I obtain it.  Also, the transcripts themselves have continuous line numbers on them.  When I start to break-up or deconstruct my data, each unit gets a label that consists of two numbers separated by a period. The number before the period is the transcript that the unit came from and the number after the period is the line number within that transcript so that I can find exactly where the information is.  So, if I have a unit labeled 3.658, it means that this data can be found in my transcript labeled 3 and on line 658.

Now, I often use electronic versions of my transcripts when I break them up. As I showed in our data analysis chapter, I create an excel file where I can cut and paste the data units, their label, and the preliminary code I am assigning to this idea.  I find excel useful because I can easily sort my data by codes and start to look for emerging themes.  Furthermore, above I mentioned memoing, or recording my thoughts and responses to the data.  I can easily do this in excel, by adding an additional column for memoing where I can put my thoughts/responses by a particular unit and date it, so I know when I was having that thought.  Generally speaking, I find that excel makes it pretty easy for me to manipulate or move my data around while I’m making sense of it, while also documenting this.  Of course, the qualitative data analysis software packages that I mentioned in our analysis chapter all have their own systems for activities such as assigning labels, coding , and memoing .  And if you choose to use one of these, you will want to be well acquainted with how to do this before you start collecting data. That being said, you don’t need software or even excel to do this work.  I know many qualitative researchers who prefer having physical data in front of them, allowing them to shift note cards around and more clearly visualize their emerging themes. If you elect for this, you just need to make sure you track the moves you are making and your thought process during the analysis. And be careful if you have a cat, mine would have a field day with piles of note cards left on my desk!

• Due to the dynamic and often iterative nature of qualitative research, we need to proactively consider how we will store and analyze our qualitative data, often at the same time we are collecting it.
• Whatever data management system we plan for, it needs to have consistent ways of documenting our evolving understanding of what our data mean. This documentation acts as an important bridge between our raw qualitative data and our qualitative research findings, helping to support rigor in our design.

20.6 Tools to account for our influence

• Identify key tools for enhancing qualitative rigor at various stages of the research process
• Begin to critique the quality of existing qualitative studies based on the use of these tools
• Determine which tools may strengthen the quality of our own qualitative research designs

So I’ve saved the best for last. This is a concrete discussion about tools that you can utilize to demonstrate qualitative rigor in your study.  The previous sections in this chapter suggest topics you need to think about related to rigor, but this suggests strategies to actually accomplish it. Remember, these are tools you should also be looking for as you examine other qualitative research studies.  As I previously mentioned, you won’t be looking to use all of these in any one study, but rather determining which tools make the most sense based on your study design.

Some of these tools apply throughout the research process, while others are more specifically applied at one stage of research. For instance, an audit trail is created during your analysis phase, while peer debriefing can take place throughout all stages of your research process. These come to us from the work of Lincoln and Guba (1985) [2] . Along with the argument that we need separate criteria for judging the quality of from the Interpretivist paradigm (as opposed to Positivist criteria of reliability and validity ), they also proposed a compendium of tools to help meet these criteria. We will review each of these tools and an example will be provided after the description.

Observer triangulation

Observer triangulation involves including more than one member of your research team to aid in analyzing the data. Essentially, you will have at least two sets of eyes looking at the data, drawing it out, and then comparing findings, converging on agreement about what the final results should be. This helps us to ensure that we aren’t just seeing what we want to see.

Data triangulation

Data triangulation is a strategy that you build into your research design where you include data from multiple sources to help enhance your understanding of a topic.  This might mean that you include a variety of groups of people to represent different perspectives on the issue. This can also mean that you collect different types of data. The main idea here is that by incorporating different sources of data (people or types), you are seeking to get a more well-rounded or comprehensive understanding of the focus of your study.

People: Instead of just interviewing mental health consumers about their treatment, you also include family members and providers.

Types: I have conducted a case study where we included interviews and the analysis of multiple documents, such as emails, agendas, and meeting minutes.

Peer debriefing

Peer debriefing means that you intentionally plan for and meet with a qualitative researcher outside of your team to discuss your process and findings and to help examine the decisions you are making, the logic behind them, and your potential influence and accountability in the research process. You will often meet with a peer debriefer multiple times during your research process and may do things like: review your reflexive journal ; review certain aspects of your project, such as preliminary findings; discuss current decisions you are considering; and review the current status of your project. The main focus here is building in some objectivity to what can become a very subjective process. We can easily become very involved in this research and it can be hard for us to step back and thoughtfully examine the decisions we are making.

Member-checking

Member-checking has to do with incorporating research participants into the data analysis process. This may mean actively including them throughout the analysis, either as a co-researcher or as a consultant. This can also mean that once you have the findings from your analysis, you take these to your participants (or a subset of your participants) and ask them to review these findings and provide you feedback about their accuracy.  I will often ask participants when I member-check, can you hear your voice in these findings? Do you recognize what you shared with me in these results? We often need to preface member-checking by saying that we are bringing together many people’s ideas, so we are often trying to represent multiple perspectives, but we want to make sure that their perspective is included in there.  This can be a very important step in ensuring that we did a reasonable job getting from our raw data to our findings…did we get it right. It also gives some power back to participants, as we are giving them some say in what our findings look like.

Thick description

Providing a thick description means that you are giving your audience a rich, detailed description of your findings and the context in which they exist.  As you read a thick description, you walk away feeling like you have a very vivid picture of what the research participants felt, thought, or experienced, and that you now have a more complete understanding of the topic being studied. Of course, a thick description can’t just be made up at the end. You can’t hope to produce a thick description if you haven’t done work early on to collect detailed data and performed a thorough analysis.  Our main objective with a thick description is being accountable to our audience in helping them to understand what we learned in the most comprehensive way possible.

Reflexivity

Reflexivity pertains to how we understand and account for our influence, as researchers, on the research process. In social work practice, we talk extensively about our “use of self” as social workers, meaning that we work to understanding how our unique personhood (who we are) impacts or influences how we work with our clients.  Reflexivity is about applying this to the process of research, rather than practice. It assumes that our values, beliefs, understanding, and experiences all may influence the decisions that we make as we engage in research.  By engaging in qualitative research with reflexivity, we are attempting to be transparent about how we are shaping and being shaped by the research we are conducting.

Prolonged engagement

Prolonged engagement means that we are extensively spending time with participants or are in the community we are studying. We are visiting on multiple occasions during the study in an attempt to get the most complete picture or understanding possible. This can be very important for us as we attempt to analyze and interpret our data. If we haven’t spent enough time getting to know our participants and their community, we may miss the meaning of data that is shared with us because we don’t understand the cultural subtext in which this data exists.  The main idea here is that we don’t know what we don’t know; furthermore, we can’t know it unless we invest time getting to know it! There’s no short-cut here, you have to put in the time.

Audit trail

Creating an audit trail is something we do during our data analysis process as qualitative researchers. An audit trail is essentially creating a map of how you got from your raw data to your research findings. This means that we should be able to work backwards, starting with your research findings and trace them back to your raw data.  It starts with labeling our data as we begin to break it apart (deconstruction) and then reassemble it (reconstruction). It allows us to determine where ideas came from and how/why we put ideas together to form broader themes.  An audit trail offers transparency in our data analysis process. It is the opposite of the “black box” we spoke about in our qualitative analysis chapter, making it clear how we got from point A to point B.

External audit

An external audit is when we actually bring in a qualitative researcher not connected to our project once the study has been completed to examine the research project and the findings to “evaluate the accuracy and evaluate whether or not the findings, interpretations and conclusions are supported by the data” (Robert Wood Johnson Foundation, External Audits). An external auditor will likely look at all of our research materials, but will likely make extensive use of our audit trail to ensure that a clear link can be established between our findings and the raw data we collected by an external observer.  Much like a peer debriefer, an external auditor can offer an outside critique of the study, thereby helping us to reflect on the work we are doing and how we are going about it.

Negative case analysis

Negative case analysis involves including data that contrasts, contradicts, or challenges the majority of evidence that we have found or expect to find. This may come into play in our sampling, meaning that we may seek to recruit or include a specific participant or group of participants because they represent a divergent opinion. Or, as we begin our analysis, we may identify a unique or contrasting idea or opinion that seems to contradict the majority of what our other data seem to be point to.  In this case, we choose to intentionally analyze and work to understand this unique perspective in our data. As with a thick description, a negative case analysis is attempting to offer the most comprehensive and complete understanding of the phenomenon we are studying, including divergent or contradictory ideas that may be held about it.

Now let’s take some time to think through each of the stages of the design process and consider how we might apply some of these strategies.  Again, these tools are to help us, as human instruments, better account for our role in the qualitative research process and also to enhance the trustworthiness of our research when we share it with others. It is unrealistic that you would apply all of these, but attention to some will indicate that you have been thoughtful in your design and concerned about the quality of your work and the confidence in your findings.

First let’s discuss sampling. We have already discussed that qualitative research generally relies on non-probability sampling and have reviewed some specific non-probability strategies you might use.  However, along with selecting a strategy, you might also include a couple of the rigor-related tools discussed above.  First, you might choose to employ data triangulation.  For instance, maybe you are conducting an ethnography studying the culture of a peer-support clubhouse.  As you are designing your study, along with extensive observations you plan to make in the clubhouse, you are also going to conduct interviews with staff, board members, and focus groups with members.  In this way you are combining different types of data (i.e. observations, focus groups, interviews) and perspectives (i.e. yourself as the researcher, members, staff, board). In addition, you might also consider using negative case analysis. At the planning stage, this could involve you intentionally sampling a case or set of cases that are likely to provide an alternative view or perspective compared to what you might expect to find. Finally, specifically articulating your sampling rationale can also enhance the rigor of your research (Barusch, Gringeri, & George, 2011) [3] . While this isn’t listed in our tools table, it is generally a good practice when reporting your research (qualitative or quantitative) to outline your sampling strategy with a brief rationale for the choices you made. This helps to improve the transparency of your study.

Next, we can progress to data gathering. The main rigor-related tool that directly applies to this stage of your design is likely prolonged engagement.  Here we build in or plan to spend extensive time with participants gathering data.  This might mean that we return for repeated interviews with the same participants or that we go back numerous times to make observations and take field notes. While this can take many forms, the overarching idea here is that you build in time to immerse yourself in the context and culture that you are studying.  Again, there is no short-cut here, it demands time in the field getting to know people, places, significance, history, etc. You need to appreciate the context and the culture of the situation you are studying. Something special to consider here is insider/outsider status.  If you would consider yourself an “outsider”, that is to say someone who does not belong to the same group or community of people you are studying, it may be quite obvious that you will need to spend time getting to know this group and take considerable time observing and reflecting on the significance of what you see. However, if you are a researcher who is a member of the particular community you are studying, or an “insider”, I would suggest that you still need to work to objectively to take a step back, make observations, and try to reflect on what you see, what you thought you knew, and what you come to know about the community you belong to.  In both cases, prolonged engagement requires good self-reflection and observation skills.

A number of these tools may be applied during the data analysis process. First, if you have a research team, you might use observer triangulation, although this might not be an option as a student unless you are building a proposal as a group. As explained above, observer triangulation means that more than one of you will be examining the data that has been collected and drawing results from it. You will then compare these results and ultimately converge on your findings.

Example.  I’m currently using the following strategy on a project where we are analyzing focus group data that was collected over a number of focus groups. We have a team of four researchers and our process involves:

• reviewing our initial focus group transcripts
• individually identifying important categories that were present
• collectively processing these together and identifying specific labels we would use for a second round of coding
• individually returning to the transcripts with our codes and coding all the transcripts
• collectively meeting again to discuss what subthemes fell under each of the codes and if the codes fit or needed to be changed/merged/expanded

While the process was complex, I do believe this triangulation of observers enriched our analysis process. It helped us to gain a clearer understanding of our results as we collectively discussed and debated what each theme meant based on our individual understandings of the data.

While we did discuss negative case analysis above in the sampling phase, it is also worth mentioning here. Contradictory findings may creep up during our analysis. One of our participants may share something or we may find something in a document that seemingly is at odds with the majority of the rest of our data. Rather than ignoring this, negative case analysis would seek to understand this perspective and what might be behind this contradiction. In addition, we may choose to construct an audit trail as we move from raw data to our research findings during our data analysis. This means that we will institute a strategy for tracking our analysis process. I imagine that most researchers develop their own variation on this tracking process, but at its core, you need to find a way to label your segments of data so that you know where they came from once you start to break them up. Furthermore, you will be making decisions about what groups of data belong together and what they mean. Your tracking process for your audit trail will also have to provide a way to document how you arrived at these decisions. Often towards the end of an analysis process, researchers may choose to employ member checking (although you may also implement this throughout your analysis). In the example above where I was discussing our focus group project, we plan to take our findings back to some of our focus group participants to see if they feel that we captured the important information based on what they shared with us. As discussed in sampling, it is also a good practice to make sure to articulate your qualitative analysis process clearly. Unfortunately, I’ve read a number of qualitative studies where the researchers provide little detail regarding what their analysis looked like and how they arrived at their results. This often leaves me with questions about the quality of what was done.

Now we need to share our research with others. The most relevant tool specific to this phase is providing a thick description of our results.  As indicated in the table, a thick description means that we offer our audience a very detailed, rich narrative in helping them to interpret and make sense of our results.  Remember, the main aim of qualitative research is not necessarily to produce results that generalize to a large group of people.  Rather, we are seeking to enhance understanding about a particular experience, issue, or phenomenon by studying it very extensively for a relatively small sample. This produces a deep, as opposed to, a broad understanding. A thick description can be very helpful by offering detailed information about the sample, the context in which the study takes place, and a thorough explanation of findings and often how they relate to each other.  As a consumer of research, a thick description can help us to make our own judgments about the implications of these results and what other situations or populations these findings might apply to.

You may have noticed that a few of the tools in our table haven’t yet been discussed in the qualitative process yet. This is because some of these rigor-related tools are meant to span the researcher process. To begin with, reflexivity is a tool that best applied through qualitative research. I encourage students in my social work practice classes to find ways to build reflexivity into their professional lives as a way of improving their professional skills. This is no less true of qualitative research students. Throughout our research process, we need to consider how our use-of-self is shaping the decisions we are making and how the research may be transforming us during the process.  What led you to choose your research question? Why did you group those ideas together? What caused you to label your theme that? What words do you use to talk about your study at a conference? The qualitative researcher has much influence throughout this process, and self-examination of that influence can be an important piece of rigor.  As an example, one step that I sometimes build into qualitative projects is reflexively journaling before and after interviews.  I’m often driving to these interviews, so I’ll turn my bluetooth on in the car and capture my thoughts before and after, transcribing them later.  This helps me to check-in with myself during data collection and can help me illuminate insights I might otherwise miss.  I have also found this to be helpful to use in my peer debriefing. Peer debriefing can be used throughout the research process. Meeting with a peer debriefer throughout the research process can be a good way to consistently reflect on your progress and the decisions you are making throughout a project. A peer debriefer can make connections that we may otherwise miss and question aspects of our project that may be important for us to explore.  As I mentioned, combining reflexivity with peer debriefing can be a powerful tool for processing our self-reflection in connection with the progress of our project.

Finally, the use of an external audit really doesn’t come into play until the end of the research process, but an external auditor will look extensively at the whole research process. Again, this is a researcher who is unattached to the project and seeking to follow the path of the project in hopes of providing an external perspective on the trustworthiness of the research process and its findings. Often, these auditors will begin at the end, starting with the findings, and attempt to trace backwards to the beginning of the project. This is often quite a laborious task and some qualitative scholars debate whether the attention to objectivity in this strategy may be at odds with the aims of qualitative research in illuminating the uniquely subjective experiences of participants by inherently subjective researchers. However, it can be a powerful tool for demonstrating that a systematic approach was used.

As you are thinking about designing your qualitative research proposal, consider how you might use some of these tools to strengthen the quality of your proposed research.  Again, you might be using these throughout the entire research process, or applying them more specifically to one stage of the process (e.g. data collection, data analysis).  In addition, as you are reviewing qualitative studies to include in your literature review or just in developing your understanding of the topic, make sure to look out for some of these tools being used.  They are general indicators that we can use to assess the attention and care that was given to using a scientific approach to producing the knowledge that is being shared.

• As qualitative researchers there are a number of tools at your disposal to help support quality and rigor. These tools can aid you in assessing the quality of others’ work and in supporting the quality of your own design.
• Qualitative rigor is not a box we can tick complete somewhere along our research project’s timeline.  It is something that needs to be attended to thoughtfully throughout the research process; it is a commitment we make to our participants and to our potential audience.

List out 2-3 tools that seem like they would be a good fit for supporting the rigor of your qualitative proposal. Also, provide a justification as to why they seem relevant to the design of your research and what you are trying to accomplish.

• Justification:
• Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry . Newberry Park, CA: Sage ↵
• Lincoln, YS. & Guba, EG. (1985). Naturalistic inquiry. Newbury Park, CA: Sage Publications. ↵
• Barusch, A., Gringeri, C., & George, M. (2011). Rigor in qualitative social work research: A review of strategies used in published articles. Social Work Research, 35 (1), 11-19. ↵

Rigor is the process through which we demonstrate, to the best of our ability, that our research is empirically sound and reflects a scientific approach to knowledge building.

The ability of a measurement tool to measure a phenomenon the same way, time after time. Note: Reliability does not imply validity.

The extent to which the scores from a measure represent the variable they are intended to.

Findings form a research study that apply to larger group of people (beyond the sample). Producing generalizable findings requires starting with a representative sample.

in a literature review, a source that describes primary data collected and analyzed by the author, rather than only reviewing what other researchers have found

Data someone else has collected that you have permission to use in your research.

Ability to say that one variable "causes" something to happen to another variable. Very important to assess when thinking about studies that examine causation such as experimental or quasi-experimental designs.

a single truth, observed without bias, that is universally applicable

one truth among many, bound within a social and cultural context

The idea that qualitative researchers attempt to limit or at the very least account for their own biases, motivations, interests and opinions during the research process.

The process of research is record and described in such a way that the steps the researcher took throughout the research process are clear.

A research journal that helps the researcher to reflect on and consider their thoughts and reactions to the research process and how it may be shaping the study

Notes that are taken by the researcher while we are in the field, gathering data.

An iterative approach means that after planning and once we begin collecting data, we begin analyzing as data as it is coming in.  This early analysis of our (incomplete) data, then impacts our planning, ongoing data gathering and future analysis as it progresses.

Memoing is the act of recording your thoughts, reactions, quandaries as you are reviewing the data you are gathering.

An audit trail is a system of documenting in qualitative research analysis that allows you to link your final results with your original raw data. Using an audit trail, an independent researcher should be able to start with your results and trace the research process backwards to the raw data. This helps to strengthen the trustworthiness of the research.

Context is the circumstances surrounding an artifact, event, or experience.

A code is a label that we place on segment of data that seems to represent the main idea of that segment.

Part of the qualitative data analysis process where we begin to interpret and assign meaning to the data.

including more than one member of your research team to aid in analyzing the data

Including data from multiple sources to help enhance your understanding of a topic

Member checking involves taking your results back to participants to see if we "got it right" in our analysis. While our findings bring together many different peoples' data into one set of findings, participants should still be able to recognize their input and feel like their ideas and experiences have been captured adequately.

A thick description is a very complete, detailed, and illustrative of the subject that is being described.

How we understand and account for our influence, as researchers, on the research process.

As researchers, this means we are extensively spending time with participants or are in the community we are studying.

Including data that contrasts, contradicts, or challenges the majority of evidence that we have found or expect to find

sampling approaches for which a person’s likelihood of being selected for membership in the sample is unknown

Ethnography is a qualitative research design that is used when we are attempting to learn about a culture by observing people in their natural environment.

Graduate research methods in social work Copyright © 2020 by Matthew DeCarlo, Cory Cummings, Kate Agnelli is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Login to your account

If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password

If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password

• AACP Member Login      Submit

Download started.

• PDF [516 KB] PDF [516 KB]
• Figure Viewer
• Download Figures (PPT)
• Add To Online Library Powered By Mendeley
• Add To My Reading List
• Export Citation
• Create Citation Alert

A Review of the Quality Indicators of Rigor in Qualitative Research

• Jessica L. Johnson, PharmD Jessica L. Johnson Correspondence Corresponding Author: Jessica L. Johnson, William Carey University School of Pharmacy, 19640 Hwy 67, Biloxi, MS 39574. Tel: 228-702-1897. Contact Affiliations William Carey University School of Pharmacy, Biloxi, Mississippi Search for articles by this author
• Donna Adkins, PharmD Donna Adkins Affiliations William Carey University School of Pharmacy, Biloxi, Mississippi Search for articles by this author
• Sheila Chauvin, PhD Sheila Chauvin Affiliations Louisiana State University, School of Medicine, New Orleans, Louisiana Search for articles by this author
• qualitative research design
• standards of rigor
• best practices

INTRODUCTION

• Denzin Norman
• Lincoln Y.S.
• Google Scholar
• Anderson C.
• Full Text PDF
• Scopus (561)
• Santiago-Delefosse M.
• Stephen S.L.
• Scopus (83)
• Scopus (32)
• Levinson W.
• Scopus (504)
• Dixon-Woods M.
• Scopus (439)
• Malterud K.
• Midtgarden T.
• Scopus (203)
• Wasserman J.A.
• Wilson K.L.
• Scopus (68)
• Barbour R.S.
• Sale J.E.M.
• Scopus (12)
• Fraser M.W.
• Scopus (35)
• Sandelowski M.
• Scopus (1554)

BEST PRACTICES: STEP-WISE APPROACH

Step 1: identifying a research topic.

• Scopus (279)
• Creswell J.
• Maxwell J.A.
• Glassick C.E.
• Maeroff G.I.
• Scopus (264)

• Scopus (271)
• Ringsted C.
• Scherpbier A.
• Scopus (132)
• Ravitch S.M.

• View Large Image
• Download Hi-res image
• Download (PPT)
• Huberman M.

Step 2: Qualitative Study Design

• Whittemore R.
• Mandle C.L.
• Scopus (969)
• Marshall M.N.
• Scopus (2208)
• Horsfall J.
• Scopus (182)
• O’Reilly M.
• Scopus (1036)
• Burkard A.W.
• Scopus (164)
• Patton M.Q.
• Scopus (353)
• Scopus (4166)
• Johnson R.B.

Step 3: Data Analysis

Step 4: drawing valid conclusions.

• Swanwick T.
• Swanwick T.O.
• O’Brien B.C.
• Harris I.B.
• Beckman T.J.
• Scopus (4705)

Step 5: Reporting Research Results

• Shenton A.K.
• Scopus (4115)

Article info

Publication history, identification.

DOI: https://doi.org/10.5688/ajpe7120

ScienceDirect

• Download .PPT

Related Articles

• Access for Developing Countries
• Articles & Issues
• Articles In Press
• Current Issue
• Past Issues
• Journal Information
• About Open Access
• Aims & Scope
• Editorial Board
• Editorial Team
• History of AJPE
• Contact Information
• For Authors
• Guide for Authors
• Researcher Academy
• Rights & Permissions
• Submission Process
• Submit Article
• For Reviewers
• Reviewer Instructions
• Reviewer Frequently Asked Questions

The content on this site is intended for healthcare professionals.

• Privacy Policy   
• Terms and Conditions   
• Accessibility   
• Help & Contact

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• BMC Health Serv Res

Application of four-dimension criteria to assess rigour of qualitative research in emergency medicine

Roberto forero.

1 The Simpson Centre for Health Services Research, South Western Sydney Clinical School and the Ingham Institute for Applied Research, Liverpool Hospital, UNSW, Liverpool, NSW 1871 Australia

Shizar Nahidi

Josephine de costa, mohammed mohsin.

2 Psychiatry Research and Teaching Unit, Liverpool Hospital, NSW Health, Sydney, Australia

3 School of Psychiatry, Faculty of Medicine, University of New South Wales, Sydney, Australia

Gerry Fitzgerald

4 School - Public Health and Social Work, Queensland University of Technology (QUT), Brisbane, Qld Australia

5 Australasian College for Emergency Medicine (ACEM), Melbourne, VIC Australia

Nick Gibson

6 School of Nursing and Midwifery, Edith Cowan University (ECU), Perth, WA Australia

Sally McCarthy

7 Emergency Care Institute (ECI), NSW Agency for Clinical Innovation (ACI), Sydney, Australia

Patrick Aboagye-Sarfo

8 Clinical Support Directorate, System Policy & Planning Division, Department of Health WA, Perth, WA Australia

Associated Data

All data generated or analysed during this study are included in this published article and its supplementary information files have been included in the appendix. No individual data will be available.

The main objective of this methodological manuscript was to illustrate the role of using qualitative research in emergency settings. We outline rigorous criteria applied to a qualitative study assessing perceptions and experiences of staff working in Australian emergency departments.

We used an integrated mixed-methodology framework to identify different perspectives and experiences of emergency department staff during the implementation of a time target government policy. The qualitative study comprised interviews from 119 participants across 16 hospitals. The interviews were conducted in 2015–2016 and the data were managed using NVivo version 11. We conducted the analysis in three stages, namely: conceptual framework, comparison and contrast and hypothesis development. We concluded with the implementation of the four-dimension criteria (credibility, dependability, confirmability and transferability) to assess the robustness of the study,

We adapted four-dimension criteria to assess the rigour of a large-scale qualitative research in the emergency department context. The criteria comprised strategies such as building the research team; preparing data collection guidelines; defining and obtaining adequate participation; reaching data saturation and ensuring high levels of consistency and inter-coder agreement.

Based on the findings, the proposed framework satisfied the four-dimension criteria and generated potential qualitative research applications to emergency medicine research. We have added a methodological contribution to the ongoing debate about rigour in qualitative research which we hope will guide future studies in this topic in emergency care research. It also provided recommendations for conducting future mixed-methods studies. Future papers on this series will use the results from qualitative data and the empirical findings from longitudinal data linkage to further identify factors associated with ED performance; they will be reported separately.

Electronic supplementary material

The online version of this article (10.1186/s12913-018-2915-2) contains supplementary material, which is available to authorized users.

Qualitative research methods have been used in emergency settings in a variety of ways to address important problems that cannot be explored in another way, such as attitudes, preferences and reasons for presenting to the emergency department (ED) versus other type of clinical services (i.e., general practice) [ 1 – 4 ].

The methodological contribution of this research is part of the ongoing debate of scientific rigour in emergency care, such as the importance of qualitative research in evidence-based medicine, its contribution to tool development and policy evaluation [ 2 , 3 , 5 – 7 ]. For instance, the Four-Hour Rule and the National Emergency Access Target (4HR/NEAT) was an important policy implemented in Australia to reduce EDs crowding and boarding (access block) [ 8 – 13 ]. This policy generated the right conditions for using mixed methods to investigate the impact of 4HR/NEAT policy implementation on people attending, working or managing this type of problems in emergency departments [ 2 , 3 , 5 – 7 , 14 – 17 ].

The rationale of our study was to address the perennial question of how to assess and establish methodological robustness in these types of studies. For that reason, we conducted this mixed method study to explore the impact of the 4HR/NEAT in 16 metropolitan hospitals in four Australian states and territories, namely: Western Australia (WA), Queensland (QLD), New South Wales (NSW), and the Australian Capital Territory (ACT) [ 18 , 19 ].

The main objectives of the qualitative component was to understand the personal, professional and organisational perspectives reported by ED staff during the implementation of 4HR/NEAT, and to explore their perceptions and experiences associated with the implementation of the policy in their local environment.

This is part of an Australian National Health and Medical Research Council (NH&MRC) Partnership project to assess the impact of the 4HR/NEAT on Australian EDs. It is intended to complement the quantitative streams of a large data-linkage/dynamic modelling study using a mixed-methods approach to understand the impact of the implementation of the four-hour rule policy.

Methodological rigour

This section describes the qualitative methods to assess the rigour of the qualitative study. Researchers conducting quantitative studies use conventional terms such as internal validity, reliability, objectivity and external validity [ 17 ]. In establishing trustworthiness, Lincoln and Guba created stringent criteria in qualitative research, known as credibility, dependability, confirmability and transferability [ 17 – 20 ]. This is referred in this article as “the Four-Dimensions Criteria” (FDC). Other studies have used different variations of these categories to stablish rigour [ 18 , 19 ]. In our case, we adapted the criteria point by point by selecting those strategies that applied to our study systematically. Table  1 illustrates which strategies were adapted in our study.

Key FDC strategies adapted from Lincoln and Guba [ 23 ]

Study procedure

We carefully planned and conducted a series of semi-structured interviews based on the four-dimension criteria (credibility, dependability, confirmability and transferability) to assess and ensure the robustness of the study. These criteria have been used in other contexts of qualitative health research; but this is the first time it has been used in the emergency setting [ 20 – 26 ].

Sampling and recruitment

We employed a combination of stratified purposive sampling (quota sampling), criterion-based and maximum variation sampling strategies to recruit potential participants [ 27 , 28 ]. The hospitals selected for the main longitudinal quantitative data linkage study, were also purposively selected in this qualitative component.

We targeted potential individuals from four groups, namely: ED Directors, ED physicians, ED nurses, and data/admin staff. The investigators identified local site coordinators who arranged the recruitment in each of the participating 16 hospitals (6 in NSW, 4 in QLD, 4 in WA and 2 in the ACT) and facilitated on-site access to the research team. These coordinators provided a list of potential participants for each professional group. By using this list, participants within each group were selected through purposive sampling technique. We initially planned to recruit at least one ED director, two ED physicians, two ED nurses and one data/admin staff per hospital. Invitation emails were circulated by the site coordinators to all potential participants who were asked to contact the main investigators if they required more information.

We also employed criterion-based purposive sampling to ensure that those with experience relating to 4HR/NEAT were eligible. For ethical on-site restrictions, the primary condition of the inclusion criteria was that eligible participants needed to be working in the ED during the period that the 4HR/NEAT policy was implemented. Those who were not working in that ED during the implementation period were not eligible to participate, even if they had previous working experience in other EDs.

We used maximum variation sampling to ensure that the sample reflects a diverse group in terms of skill level, professional experience and policy implementation [ 28 ]. We included study participants irrespective of whether their role/position was changed (for example, if they received a promotion during their term of service in ED).

In summary, over a period of 7 months (August 2015 to March 2016), we identified all the potential participants (124) and conducted 119 interviews (5 were unable to participate due to workload availability). The overall sample comprised a cohort of people working in different roles across 16 hospitals. Table  2 presents the demographic and professional characteristics of the participants.

Demographic and professional characteristics of the staff participated in the study

Dir represents ‘Director’, NUM Nursing unit manager, CNC Clinical nurse consultant

Data collection

We employed a semi-structured interview technique. Six experienced investigators (3 in NSW, 1 in ACT, 1 in QLD and 1 in WA) conducted the interviews (117 face-to-face on site and 2 by telephone). We used an integrated interview protocol which consisted of a demographically-oriented question and six open-ended questions about different aspects of the 4HR/NEAT policy (see Additional file  1 : Appendix 1).

With the participant’s permission, interviews were audio-recorded. All the hospitals provided a quiet interview room that ensured privacy and confidentiality for participants and investigators.

All the interviews were transcribed verbatim by a professional transcriber with reference to a standardised transcription protocol [ 29 ]. The data analysis team followed a stepwise process for data cleaning, and de-identification. Transcripts were imported to qualitative data analysis software NVivo version 11 for management and coding [ 30 ].

Data analysis

The analyses were carried out in three stages. In the first stage, we identified key concepts using content analysis and a mind-mapping process from the research protocol and developed a conceptual framework to organise the data [ 31 ]. The analysis team reviewed and coded a selected number of transcripts, then juxtaposed the codes against the domains incorporated in the interview protocol as indicated in the three stages of analysis with the conceptual framework (Fig.  1 ).

Conceptual framework with the three stages of analysis used for the analysis of the qualitative data

In this stage, two cycles of coding were conducted: in the first one, all the transcripts were revised and initially coded, key concepts were identified throughout the full data set. The second cycle comprised an in-depth exploration and creation of additional categories to generate the codebook (see Additional file  2 : Appendix 2). This codebook was a summary document encompassing all the concepts identified as primary and subsequent levels. It presented hierarchical categorisation of key concepts developed from the domains indicated in Fig. ​ Fig.1 1 .

A summarised list of key concepts and their definitions are presented in Table  3 . We show the total number of interviews for each of the key concepts, and the number of times (i.e., total citations) a concept appeared in the whole dataset.

Summary of key concepts, their definition, total number of citations and total number of interviews

Citations refer to the number of times a coded term was counted in NVivo

The second stage of analysis compared and contrasted the experiences, perspectives and actions of participants by role and location. The third and final stage of analysis aimed to generate theory-driven hypotheses and provided an in-depth understanding of the impact of the policy. At this stage, the research team explored different theoretical perspectives such as the carousel model and models of care approach [ 16 , 32 – 34 ]. We also used iterative sampling to reach saturation and interpret the findings.

Ethics approval and consent to participate

Ethics approval was obtained for all participating hospitals and the qualitative methods are based on the original research protocol approved by the funding organisations [ 18 ].

This section described the FDC and provided a detailed description of the strategies used in the analysis. It was adapted from the FDC methodology described by Lincoln and Guba [ 23 – 26 ] as the framework to ensure a high level of rigour in qualitative research. In Table ​ Table1, 1 , we have provided examples of how the process was implemented for each criterion and techniques to ensure compliance with the purpose of FDC.

Credibility

Prolonged and varied engagement with each setting.

All the investigators had the opportunity to have a continued engagement with each ED during the data collection process. They received a supporting material package, comprising background information about the project; consent forms and the interview protocol (see Additional file 1 : Appendix 1). They were introduced to each setting by the local coordinator and had the chance to meet the ED directors and potential participants, They also identified local issues and salient characteristics of each site, and had time to get acquainted with the study’s participants. This process allowed the investigators to check their personal perspectives and predispositions, and enhance their familiarity with the study setting. This strategy also allowed participants to become familiar with the project and the research team.

Interviewing process and techniques

In order to increase credibility of the data collected and of the subsequent results, we took a further step of calibrating the level of awareness and knowledge of the research protocol. The research team conducted training sessions, teleconferences, induction meetings and pilot interviews with the local coordinators. Each of the interviewers conducted one or two pilot interviews to refine the overall process using the interview protocol, time-management and the overall running of the interviews.

The semi-structured interview procedure also allowed focus and flexibility during the interviews. The interview protocol (Additional file 1 : Appendix 1) included several prompts that allowed the expansion of answers and the opportunity for requesting more information, if required.

Establishing investigators’ authority

In relation to credibility, Miles and Huberman [ 35 ] expanded the concept to the trustworthiness of investigators’ authority as ‘human instruments’ and recommended the research team should present the following characteristics:

• Familiarity with phenomenon and research context : In our study, the research team had several years’ experience in the development and implementation of 4HR/NEAT in Australian EDs and extensive ED-based research experience and track records conducting this type of work.
• Investigative skills: Investigators who were involved in data collections had three or more years’ experience in conducting qualitative data collection, specifically individual interview techniques.
• Theoretical knowledge and skills in conceptualising large datasets: Investigators had post-graduate experience in qualitative data analysis and using NVivo software to manage and qualitative research skills to code and interpret large amounts of qualitative data.
• Ability to take a multidisciplinary approach: The multidisciplinary background of the team in public health, nursing, emergency medicine, health promotion, social sciences, epidemiology and health services research, enabled us to explore different theoretical perspectives and using an eclectic approach to interpret the findings.

These characteristics ensured that the data collection and content were consistent across states and participating hospitals.

Collection of referential adequacy materials

In accordance with Guba’s recommendation to collect any additional relevant resources, investigators maintained a separate set of materials from on-site data collection which included documents and field notes that provided additional information in relation to the context of the study, its findings and interpretation of results. These materials were collected and used during the different levels of data analysis and kept for future reference and secure storage of confidential material [ 26 ].

Peer debriefing

We conducted several sessions of peer debriefing with some of the Project Management Committee (PMC) members. They were asked at different stages throughout the analysis to reflect and cast their views on the conceptual analysis framework, the key concepts identified during the first level of analysis and eventually the whole set of findings (see Fig. ​ Fig.1). 1 ). We also have reported and discussed preliminary methods and general findings at several scientific meetings of the Australasian College for Emergency Medicine.

Dependability

Rich description of the study protocol.

This study was developed from the early stages through a systematic search of the existing literature about the four-hour rule and time-target care delivery in ED. Detailed draft of the study protocol was delivered in consultation with the PMC. After incorporating all the comments, a final draft was generated for the purpose of obtaining the required ethics approvals for each ED setting in different states and territories.

To maintain consistency, we documented all the changes and revisions to the research protocol, and kept a trackable record of when and how changes were implemented.

Establishing an audit trail

Steps were taken to keep a track record of the data collection process [ 24 ]: we have had sustained communication within the research team to ensure the interviewers were abiding by an agreed-upon protocol to recruit participants. As indicated before, we provided the investigators with a supporting material package. We also instructed the interviewers on how to securely transfer the data to the transcriber. The data-analysis team systematically reviewed the transcripts against the audio files for accuracy and clarifications provided by the transcriber.

All the steps in coding the data and identification of key concepts were agreed upon by the research team. The progress of the data analysis was monitored on a weekly basis. Any modifications of the coding system were discussed and verified by the team to ensure correct and consistent interpretation throughout the analysis.

The codebook (see Additional file 2 : Appendix 2) was revised and updated during the cycles of coding. Utilisation of the mind-mapping process described above helped to verify consistency and allowed to determine how precise the participants’ original information was preserved in the coding [ 31 ].

As required by relevant Australian legislation [ 36 ], we maintained complete records of the correspondence and minutes of meetings, as well as all qualitative data files in NVivo and Excel on the administrative organisation’s secure drive. Back-up files were kept in a secure external storage device, for future access if required.

Stepwise replication—measuring the inter-coders’ agreement

To assess the interpretative rigour of the analysis, we applied inter-coder agreement to control the coding accuracy and monitor inter-coder reliability among the research team throughout the analysis stage [ 37 ]. This step was crucially important in the study given the changes of staff that our team experienced during the analysis stage. At the initial stages of coding, we tested the inter-coder agreement using the following protocol:

• Step 1 – Two data analysts and principal investigator coded six interviews, separately.
• Step 2 – The team discussed the interpretation of the emerging key concepts, and resolved any coding discrepancies.
• Step 3 – The initial codebook was composed and used for developing the respective conceptual framework.
• Step 4 – The inter-coder agreement was calculated and found a weighted Kappa coefficient of 0.765 which indicates a very good agreement (76.5%) of the data.

With the addition of a new analyst to the team, we applied another round of inter-coder agreement assessment. We followed the same steps to ensure the inter-coder reliability along the trajectory of data analysis, except for step 3—a priori codebook was used as a benchmark to compare and contrast the codes developed by the new analyst. The calculated Kappa coefficient 0.822 indicates a very good agreement of the data (See Table  4 ).

Inter-coder analysis using Cohen’s Kappa coefficients

Confirmability

Reflexivity.

The analysis was conducted by the research team who brought different perspectives to the data interpretation. To appreciate the collective interpretation of the findings, each investigator used a separate reflexive journal to record the issues about sensitive topics or any potential ethical issues that might have affected the data analysis. These were discussed in the weekly  meetings.

After completion of the data collection, reflection and feedback from all the investigators conducting the interviews were sought in both written and verbal format.

Triangulation

To assess the confirmability and credibility of the findings, the following four triangulation processes were considered: methodological, data source, investigators and theoretical triangulation.

Methodological triangulation is in the process of being implemented using the mixed methods approach with linked data from our 16 hospitals.

Data source triangulation was achieved by using several groups of ED staff working in different states/territories and performing different roles. This triangulation offered a broad source of data that contributed to gain a holistic understanding of the impact of 4HR/NEAT on EDs across Australia. We expect to use data triangulation with linked-data in future secondary analysis.

Investigators triangulation was obtained by consensus decision making though collaboration, discussion and participation of the team holding different perspectives. We also used the investigators’ field notes, memos and reflexive journals as a form of triangulation to validate the data collected. This approach enabled us to balance out the potential bias of individual investigators and enabling the research team to reach a satisfactory consensus level.

Theoretical triangulation was achieved by using and exploring different theoretical perspectives such as the carousel model and models of care approach [ 16 , 32 – 34 ]. that could be applied in the context of the study to generate hypotheses and theory driven codes [ 16 , 32 , 38 ].

Transferability

Purposive sampling to form a nominated sample.

As outlined in the methods section, we used a combination of three purposive sampling techniques to make sure that the selected participants were representative of the variety of views of ED staff across settings. This representativeness was critical for conducting comparative analysis across different groups.

Data saturation

We employed two methods to ensure data saturation was reached, namely: operational and theoretical. The operational method was used to quantify the number of new codes per interview over time. It indicates that the majority of codes were identified in the first interviews, followed by a decreasing frequency of codes identified from other interviews.

Theoretical saturation and iterative sampling were achieved through regular meetings where progress of coding and identification of variations in each of the key concepts were reported and discussed. We also used iterative sampling to reach saturation and interpret the findings. We continued this iterative process until no new codes emerged from the dataset and all the variations of an observed phenomenon were identified [ 39 ] (Fig.  2 ).

Data saturation gain per interview added based on the chronological order of data collection in the hospitals. Y axis = number of new codes, X axis = number of interviews over time

Scientific rigour in qualitative research assessing trustworthiness is not new. Qualitative researchers have used rigour criteria widely [ 40 – 42 ]. The novelty of the method described in this article rests on the systematic application of these criteria in a large-scale qualitative study in the context of emergency medicine.

According to the FDC, similar findings should be obtained if the process is repeated with the same cohort of participants in the same settings and organisational context. By employing the FDC and the proposed strategies, we could enhance the dependability of the findings. As indicated in the literature, qualitative research has many times been questioned in history for its validity and credibility [ 3 , 20 , 43 , 44 ].

Nevertheless, if the work is done properly, based on the suggested tools and techniques, any qualitative work can become a solid piece of evidence. This study suggests that emergency medicine researchers can improve their qualitative research if conducted according to the suggested criteria. The triangulation and reflexivity strategies helped us to minimise the investigators’ bias, and affirm that the findings were objective and accurately reflect the participants’ perspectives and experiences. Abiding by a consistent method of data collection (e.g., interview protocol) and conducting the analysis with a team of investigators, helped us minimise the risk of interpretation bias.

Employing several purposive sampling techniques enabled us to have a diverse range of opinions and experiences which at the same time enhanced the credibility of the findings. We expect that the outcomes of this study will show a high degree of applicability, because any resultant hypotheses may be transferable across similar settings in emergency care. The systematic quantification of data saturation at this scale of qualitative data has not been demonstrated in the emergency medicine literature before.

As indicated, the objective of this study was to contribute to the ongoing debate about rigour in qualitative research by using our mixed methods study as an example. In relation to innovative application of mixed-methods, the findings from this qualitative component can be used to explain specific findings from the quantitative component of the study. For example, different trends of 4HR/NEAT performance can be explained by variations in staff relationships across states (see key concept 1, Table ​ Table3). 3 ). In addition, some experiences from doctors and nurses may explain variability of performance indicators across participating hospitals. The robustness of the qualitative data will allow us to generate hypotheses that in turn can be tested in future research.

Careful planning is essential in any type of research project which includes the importance of allocating sufficient resources both human and financial. It is also required to organise precise arrangements for building the research team; preparing data collection guidelines; defining and obtaining adequate participation. This may allow other researchers in emergency care to replicate the use of the FDC in the future.

This study has several limitations. Some limitations of the qualitative component include recall bias or lack of reliable information collected about interventions conducted in the past (before the implementation of the policy). As Weber and colleagues [ 45 ] point out, conducting interviews with clinicians at a single point in time may be affected by recall bias. Moreover, ED staff may have left the organisation or have progressed in their careers (from junior to senior clinical roles, i.e. junior nursing staff or junior medical officers, registrars, etc.), so obtaining information about pre/during/post-4HR/NEAT was a difficult undertaking. Although the use of criterion-based and maximum-variation sampling techniques minimised this effect, we could not guarantee that the sampling techniques could have reached out all those who might be eligible to participate.

In terms of recruitment, we could not select potential participants who were not working in that particular ED during the implementation, even if they had previous working experience in other hospital EDs. This is a limitation because people who participated in previous hospitals during the intervention could not provide valuable input to the overall project.

In addition, one would claim that the findings could have been ‘ED-biased’ due to the fact that we did not interview the staff or administrators outside the ED. Unfortunately, interviews outside the ED were beyond the resources and scope of the project.

With respect to the rigour criteria, we could not carry out a systematic member checking as we did not have the required resources for such an expensive follow-up. Nevertheless, we have taken extensive measures to ensure confirmation of the integrity of the data.

Conclusions

The FDC presented in this manuscript provides an important and systematic approach to achieve trustworthy qualitative findings. As indicated before, qualitative research credentials have been questioned. However, if the work is done properly based on the suggested tools and techniques described in this manuscript, any work can become a very notable piece of evidence. This study concludes that the FDC is effective; any investigator in emergency medicine research can improve their qualitative research if conducted accordingly.

Important indicators such as saturation levels and inter-coder reliability should be considered in all types of qualitative projects. One important aspect is that by using FDC we can demonstrate that qualitative research is not less rigorous than quantitative methods.

We also conclude that the FDC is a valid framework to be used in qualitative research in the emergency medicine context. We recommend that future research in emergency care should consider the FDC to achieve trustworthy qualitative findings. We can conclude that our method confirms the credibility (validity) and dependability (reliability) of the analysis which are a true reflection of the perspectives reported by the group of participants across different states/territories.

We can also conclude that our method confirms the objectivity of the analyses and reduces the risk for interpretation bias. We encourage adherence to practical frameworks and strategies like those presented in this manuscript.

Finally, we have highlighted the importance of allocating sufficient resources. This is essential if other researchers in emergency care would like to replicate the use of the FDC in the future.

Following papers in this series will use the empirical findings from longitudinal data linkage analyses and the results from the qualitative study to further identify factors associated with ED performance before and after the implementation of the 4HR/NEAT.

Additional files

Appendix 1. Interview form. Text. (PDF 445 kb)

Appendix 2. Codebook NVIVO. Text code. (PDF 335 kb)

Appendix 3. Acknowledgements. Text. (PDF 104 kb)

Acknowledgements

We acknowledge Brydan Lenne who was employed in the preliminary stages of the project, for ethics application preparation and ethics submissions, and her contribution in the planning stages, data collection of the qualitative analysis and preliminary coding of the conceptual framework is appreciated. Fenglian Xu, who was also employed in the initial stages of the project in the data linkage component. Jenine Beekhuyzen, CEO Adroit Research, for consultancy and advice on qualitative aspects of the manuscript and Liz Brownlee, owner/manager Bostock Transcripts services for the transcription of the interviews. Brydan Lenne, Karlene Dickens; Cecily Scutt and Tracey Hawkins who conducted the interviews across states. We also thank Anna Holdgate, Michael Golding, Michael Hession, Amith Shetty, Drew Richardson, Daniel Fatovich, David Mountain, Nick Gibson, Sam Toloo, Conrad Loten, John Burke and Vijai Joseph who acted as site contacts on each State/Territory. We also thank all the participants for their contribution in time and information provided. A full acknowledgment of all investigators and partner organisations is enclosed as an attachment (see Additional file  3 : Appendix 3).

This project was funded by the Australian National Health and Medical Research Council (NH&MRC) Partnership Grant No APP1029492 with cash contributions from the following organisations: Department of Health of Western Australia, Australasian College for Emergency Medicine, Ministry of Health of NSW and the Emergency Care Institute, NSW Agency for Clinical Innovation, and Emergency Medicine Foundation, Queensland.

Availability of data and materials

Abbreviations, authors’ contributions.

RF, GF, SMC made substantial contributions to conception, design and funding of the study. RF, SN, NG, SMC with acquisition of data. RF, SN, JDC for the analysis and interpretation of data. RF, SN, JDC, MM, GF, NG, SMC and PA were involved in drafting the manuscript and revising it critically for important intellectual content and gave final approval of the version to be published. All authors have participated sufficiently in the work to take public responsibility for appropriate portions of the content; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

As indicated in the background, our study received ethics approval from the respective Human Research Ethics Committees of Western Australian Department of Health (DBL.201403.07), Cancer Institute NSW (HREC/14/CIPHS/30), ACT Department of Health (ETH.3.14.054) and Queensland Health (HREC/14/QGC/30) as well as governance approval from the 16 participating hospitals. All participants received information about the project; received an invitation to participate and signed a consent form and agreed to allow an audio recording to be conducted.

Consent for publication

All the data used from the interviews were de-identified for the analysis. No individual details, images or recordings, were used apart from the de-identified transcription.

Competing interests

RF is an Associate Editor of the Journal. No other authors have declared any competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

• Technical advance
• Open access
• Published: 17 February 2018

Application of four-dimension criteria to assess rigour of qualitative research in emergency medicine

• Roberto Forero   ORCID: orcid.org/0000-0001-6031-6590 1 ,
• Shizar Nahidi 1 ,
• Josephine De Costa 1 ,
• Mohammed Mohsin 2 , 3 ,
• Gerry Fitzgerald 4 , 5 ,
• Nick Gibson 6 ,
• Sally McCarthy 5 , 7 &
• Patrick Aboagye-Sarfo 8  

BMC Health Services Research volume  18 , Article number:  120 ( 2018 ) Cite this article

88k Accesses

243 Citations

16 Altmetric

Metrics details

The main objective of this methodological manuscript was to illustrate the role of using qualitative research in emergency settings. We outline rigorous criteria applied to a qualitative study assessing perceptions and experiences of staff working in Australian emergency departments.

We used an integrated mixed-methodology framework to identify different perspectives and experiences of emergency department staff during the implementation of a time target government policy. The qualitative study comprised interviews from 119 participants across 16 hospitals. The interviews were conducted in 2015–2016 and the data were managed using NVivo version 11. We conducted the analysis in three stages, namely: conceptual framework, comparison and contrast and hypothesis development. We concluded with the implementation of the four-dimension criteria (credibility, dependability, confirmability and transferability) to assess the robustness of the study,

We adapted four-dimension criteria to assess the rigour of a large-scale qualitative research in the emergency department context. The criteria comprised strategies such as building the research team; preparing data collection guidelines; defining and obtaining adequate participation; reaching data saturation and ensuring high levels of consistency and inter-coder agreement.

Based on the findings, the proposed framework satisfied the four-dimension criteria and generated potential qualitative research applications to emergency medicine research. We have added a methodological contribution to the ongoing debate about rigour in qualitative research which we hope will guide future studies in this topic in emergency care research. It also provided recommendations for conducting future mixed-methods studies. Future papers on this series will use the results from qualitative data and the empirical findings from longitudinal data linkage to further identify factors associated with ED performance; they will be reported separately.

Peer Review reports

Qualitative research methods have been used in emergency settings in a variety of ways to address important problems that cannot be explored in another way, such as attitudes, preferences and reasons for presenting to the emergency department (ED) versus other type of clinical services (i.e., general practice) [ 1 , 2 , 3 , 4 ].

The methodological contribution of this research is part of the ongoing debate of scientific rigour in emergency care, such as the importance of qualitative research in evidence-based medicine, its contribution to tool development and policy evaluation [ 2 , 3 , 5 , 6 , 7 ]. For instance, the Four-Hour Rule and the National Emergency Access Target (4HR/NEAT) was an important policy implemented in Australia to reduce EDs crowding and boarding (access block) [ 8 , 9 , 10 , 11 , 12 , 13 ]. This policy generated the right conditions for using mixed methods to investigate the impact of 4HR/NEAT policy implementation on people attending, working or managing this type of problems in emergency departments [ 2 , 3 , 5 , 6 , 7 , 14 , 15 , 16 , 17 ].

The rationale of our study was to address the perennial question of how to assess and establish methodological robustness in these types of studies. For that reason, we conducted this mixed method study to explore the impact of the 4HR/NEAT in 16 metropolitan hospitals in four Australian states and territories, namely: Western Australia (WA), Queensland (QLD), New South Wales (NSW), and the Australian Capital Territory (ACT) [ 18 , 19 ].

The main objectives of the qualitative component was to understand the personal, professional and organisational perspectives reported by ED staff during the implementation of 4HR/NEAT, and to explore their perceptions and experiences associated with the implementation of the policy in their local environment.

This is part of an Australian National Health and Medical Research Council (NH&MRC) Partnership project to assess the impact of the 4HR/NEAT on Australian EDs. It is intended to complement the quantitative streams of a large data-linkage/dynamic modelling study using a mixed-methods approach to understand the impact of the implementation of the four-hour rule policy.

Methodological rigour

This section describes the qualitative methods to assess the rigour of the qualitative study. Researchers conducting quantitative studies use conventional terms such as internal validity, reliability, objectivity and external validity [ 17 ]. In establishing trustworthiness, Lincoln and Guba created stringent criteria in qualitative research, known as credibility, dependability, confirmability and transferability [ 17 , 18 , 19 , 20 ]. This is referred in this article as “the Four-Dimensions Criteria” (FDC). Other studies have used different variations of these categories to stablish rigour [ 18 , 19 ]. In our case, we adapted the criteria point by point by selecting those strategies that applied to our study systematically. Table  1 illustrates which strategies were adapted in our study.

Study procedure

We carefully planned and conducted a series of semi-structured interviews based on the four-dimension criteria (credibility, dependability, confirmability and transferability) to assess and ensure the robustness of the study. These criteria have been used in other contexts of qualitative health research; but this is the first time it has been used in the emergency setting [ 20 , 21 , 22 , 23 , 24 , 25 , 26 ].

Sampling and recruitment

We employed a combination of stratified purposive sampling (quota sampling), criterion-based and maximum variation sampling strategies to recruit potential participants [ 27 , 28 ]. The hospitals selected for the main longitudinal quantitative data linkage study, were also purposively selected in this qualitative component.

We targeted potential individuals from four groups, namely: ED Directors, ED physicians, ED nurses, and data/admin staff. The investigators identified local site coordinators who arranged the recruitment in each of the participating 16 hospitals (6 in NSW, 4 in QLD, 4 in WA and 2 in the ACT) and facilitated on-site access to the research team. These coordinators provided a list of potential participants for each professional group. By using this list, participants within each group were selected through purposive sampling technique. We initially planned to recruit at least one ED director, two ED physicians, two ED nurses and one data/admin staff per hospital. Invitation emails were circulated by the site coordinators to all potential participants who were asked to contact the main investigators if they required more information.

We also employed criterion-based purposive sampling to ensure that those with experience relating to 4HR/NEAT were eligible. For ethical on-site restrictions, the primary condition of the inclusion criteria was that eligible participants needed to be working in the ED during the period that the 4HR/NEAT policy was implemented. Those who were not working in that ED during the implementation period were not eligible to participate, even if they had previous working experience in other EDs.

We used maximum variation sampling to ensure that the sample reflects a diverse group in terms of skill level, professional experience and policy implementation [ 28 ]. We included study participants irrespective of whether their role/position was changed (for example, if they received a promotion during their term of service in ED).

In summary, over a period of 7 months (August 2015 to March 2016), we identified all the potential participants (124) and conducted 119 interviews (5 were unable to participate due to workload availability). The overall sample comprised a cohort of people working in different roles across 16 hospitals. Table  2 presents the demographic and professional characteristics of the participants.

Data collection

We employed a semi-structured interview technique. Six experienced investigators (3 in NSW, 1 in ACT, 1 in QLD and 1 in WA) conducted the interviews (117 face-to-face on site and 2 by telephone). We used an integrated interview protocol which consisted of a demographically-oriented question and six open-ended questions about different aspects of the 4HR/NEAT policy (see Additional file  1 : Appendix 1).

With the participant’s permission, interviews were audio-recorded. All the hospitals provided a quiet interview room that ensured privacy and confidentiality for participants and investigators.

All the interviews were transcribed verbatim by a professional transcriber with reference to a standardised transcription protocol [ 29 ]. The data analysis team followed a stepwise process for data cleaning, and de-identification. Transcripts were imported to qualitative data analysis software NVivo version 11 for management and coding [ 30 ].

Data analysis

The analyses were carried out in three stages. In the first stage, we identified key concepts using content analysis and a mind-mapping process from the research protocol and developed a conceptual framework to organise the data [ 31 ]. The analysis team reviewed and coded a selected number of transcripts, then juxtaposed the codes against the domains incorporated in the interview protocol as indicated in the three stages of analysis with the conceptual framework (Fig.  1 ).

Conceptual framework with the three stages of analysis used for the analysis of the qualitative data

In this stage, two cycles of coding were conducted: in the first one, all the transcripts were revised and initially coded, key concepts were identified throughout the full data set. The second cycle comprised an in-depth exploration and creation of additional categories to generate the codebook (see Additional file  2 : Appendix 2). This codebook was a summary document encompassing all the concepts identified as primary and subsequent levels. It presented hierarchical categorisation of key concepts developed from the domains indicated in Fig. 1 .

A summarised list of key concepts and their definitions are presented in Table  3 . We show the total number of interviews for each of the key concepts, and the number of times (i.e., total citations) a concept appeared in the whole dataset.

The second stage of analysis compared and contrasted the experiences, perspectives and actions of participants by role and location. The third and final stage of analysis aimed to generate theory-driven hypotheses and provided an in-depth understanding of the impact of the policy. At this stage, the research team explored different theoretical perspectives such as the carousel model and models of care approach [ 16 , 32 , 33 , 34 ]. We also used iterative sampling to reach saturation and interpret the findings.

Ethics approval and consent to participate

Ethics approval was obtained for all participating hospitals and the qualitative methods are based on the original research protocol approved by the funding organisations [ 18 ].

This section described the FDC and provided a detailed description of the strategies used in the analysis. It was adapted from the FDC methodology described by Lincoln and Guba [ 23 , 24 , 25 , 26 ] as the framework to ensure a high level of rigour in qualitative research. In Table 1 , we have provided examples of how the process was implemented for each criterion and techniques to ensure compliance with the purpose of FDC.

Credibility

Prolonged and varied engagement with each setting.

All the investigators had the opportunity to have a continued engagement with each ED during the data collection process. They received a supporting material package, comprising background information about the project; consent forms and the interview protocol (see Additional file 1 : Appendix 1). They were introduced to each setting by the local coordinator and had the chance to meet the ED directors and potential participants, They also identified local issues and salient characteristics of each site, and had time to get acquainted with the study’s participants. This process allowed the investigators to check their personal perspectives and predispositions, and enhance their familiarity with the study setting. This strategy also allowed participants to become familiar with the project and the research team.

Interviewing process and techniques

In order to increase credibility of the data collected and of the subsequent results, we took a further step of calibrating the level of awareness and knowledge of the research protocol. The research team conducted training sessions, teleconferences, induction meetings and pilot interviews with the local coordinators. Each of the interviewers conducted one or two pilot interviews to refine the overall process using the interview protocol, time-management and the overall running of the interviews.

The semi-structured interview procedure also allowed focus and flexibility during the interviews. The interview protocol (Additional file 1 : Appendix 1) included several prompts that allowed the expansion of answers and the opportunity for requesting more information, if required.

Establishing investigators’ authority

In relation to credibility, Miles and Huberman [ 35 ] expanded the concept to the trustworthiness of investigators’ authority as ‘human instruments’ and recommended the research team should present the following characteristics:

Familiarity with phenomenon and research context : In our study, the research team had several years’ experience in the development and implementation of 4HR/NEAT in Australian EDs and extensive ED-based research experience and track records conducting this type of work.

Investigative skills: Investigators who were involved in data collections had three or more years’ experience in conducting qualitative data collection, specifically individual interview techniques.

Theoretical knowledge and skills in conceptualising large datasets: Investigators had post-graduate experience in qualitative data analysis and using NVivo software to manage and qualitative research skills to code and interpret large amounts of qualitative data.

Ability to take a multidisciplinary approach: The multidisciplinary background of the team in public health, nursing, emergency medicine, health promotion, social sciences, epidemiology and health services research, enabled us to explore different theoretical perspectives and using an eclectic approach to interpret the findings.

These characteristics ensured that the data collection and content were consistent across states and participating hospitals.

Collection of referential adequacy materials

In accordance with Guba’s recommendation to collect any additional relevant resources, investigators maintained a separate set of materials from on-site data collection which included documents and field notes that provided additional information in relation to the context of the study, its findings and interpretation of results. These materials were collected and used during the different levels of data analysis and kept for future reference and secure storage of confidential material [ 26 ].

Peer debriefing

We conducted several sessions of peer debriefing with some of the Project Management Committee (PMC) members. They were asked at different stages throughout the analysis to reflect and cast their views on the conceptual analysis framework, the key concepts identified during the first level of analysis and eventually the whole set of findings (see Fig. 1 ). We also have reported and discussed preliminary methods and general findings at several scientific meetings of the Australasian College for Emergency Medicine.

Dependability

Rich description of the study protocol.

This study was developed from the early stages through a systematic search of the existing literature about the four-hour rule and time-target care delivery in ED. Detailed draft of the study protocol was delivered in consultation with the PMC. After incorporating all the comments, a final draft was generated for the purpose of obtaining the required ethics approvals for each ED setting in different states and territories.

To maintain consistency, we documented all the changes and revisions to the research protocol, and kept a trackable record of when and how changes were implemented.

Establishing an audit trail

Steps were taken to keep a track record of the data collection process [ 24 ]: we have had sustained communication within the research team to ensure the interviewers were abiding by an agreed-upon protocol to recruit participants. As indicated before, we provided the investigators with a supporting material package. We also instructed the interviewers on how to securely transfer the data to the transcriber. The data-analysis team systematically reviewed the transcripts against the audio files for accuracy and clarifications provided by the transcriber.

All the steps in coding the data and identification of key concepts were agreed upon by the research team. The progress of the data analysis was monitored on a weekly basis. Any modifications of the coding system were discussed and verified by the team to ensure correct and consistent interpretation throughout the analysis.

The codebook (see Additional file 2 : Appendix 2) was revised and updated during the cycles of coding. Utilisation of the mind-mapping process described above helped to verify consistency and allowed to determine how precise the participants’ original information was preserved in the coding [ 31 ].

As required by relevant Australian legislation [ 36 ], we maintained complete records of the correspondence and minutes of meetings, as well as all qualitative data files in NVivo and Excel on the administrative organisation’s secure drive. Back-up files were kept in a secure external storage device, for future access if required.

Stepwise replication—measuring the inter-coders’ agreement

To assess the interpretative rigour of the analysis, we applied inter-coder agreement to control the coding accuracy and monitor inter-coder reliability among the research team throughout the analysis stage [ 37 ]. This step was crucially important in the study given the changes of staff that our team experienced during the analysis stage. At the initial stages of coding, we tested the inter-coder agreement using the following protocol:

Step 1 – Two data analysts and principal investigator coded six interviews, separately.

Step 2 – The team discussed the interpretation of the emerging key concepts, and resolved any coding discrepancies.

Step 3 – The initial codebook was composed and used for developing the respective conceptual framework.

Step 4 – The inter-coder agreement was calculated and found a weighted Kappa coefficient of 0.765 which indicates a very good agreement (76.5%) of the data.

With the addition of a new analyst to the team, we applied another round of inter-coder agreement assessment. We followed the same steps to ensure the inter-coder reliability along the trajectory of data analysis, except for step 3—a priori codebook was used as a benchmark to compare and contrast the codes developed by the new analyst. The calculated Kappa coefficient 0.822 indicates a very good agreement of the data (See Table  4 ).

Confirmability

Reflexivity.

The analysis was conducted by the research team who brought different perspectives to the data interpretation. To appreciate the collective interpretation of the findings, each investigator used a separate reflexive journal to record the issues about sensitive topics or any potential ethical issues that might have affected the data analysis. These were discussed in the weekly  meetings.

After completion of the data collection, reflection and feedback from all the investigators conducting the interviews were sought in both written and verbal format.

Triangulation

To assess the confirmability and credibility of the findings, the following four triangulation processes were considered: methodological, data source, investigators and theoretical triangulation.

Methodological triangulation is in the process of being implemented using the mixed methods approach with linked data from our 16 hospitals.

Data source triangulation was achieved by using several groups of ED staff working in different states/territories and performing different roles. This triangulation offered a broad source of data that contributed to gain a holistic understanding of the impact of 4HR/NEAT on EDs across Australia. We expect to use data triangulation with linked-data in future secondary analysis.

Investigators triangulation was obtained by consensus decision making though collaboration, discussion and participation of the team holding different perspectives. We also used the investigators’ field notes, memos and reflexive journals as a form of triangulation to validate the data collected. This approach enabled us to balance out the potential bias of individual investigators and enabling the research team to reach a satisfactory consensus level.

Theoretical triangulation was achieved by using and exploring different theoretical perspectives such as the carousel model and models of care approach [ 16 , 32 , 33 , 34 ]. that could be applied in the context of the study to generate hypotheses and theory driven codes [ 16 , 32 , 38 ].

Transferability

Purposive sampling to form a nominated sample.

As outlined in the methods section, we used a combination of three purposive sampling techniques to make sure that the selected participants were representative of the variety of views of ED staff across settings. This representativeness was critical for conducting comparative analysis across different groups.

Data saturation

We employed two methods to ensure data saturation was reached, namely: operational and theoretical. The operational method was used to quantify the number of new codes per interview over time. It indicates that the majority of codes were identified in the first interviews, followed by a decreasing frequency of codes identified from other interviews.

Theoretical saturation and iterative sampling were achieved through regular meetings where progress of coding and identification of variations in each of the key concepts were reported and discussed. We also used iterative sampling to reach saturation and interpret the findings. We continued this iterative process until no new codes emerged from the dataset and all the variations of an observed phenomenon were identified [ 39 ] (Fig.  2 ).

Data saturation gain per interview added based on the chronological order of data collection in the hospitals. Y axis = number of new codes, X axis = number of interviews over time

Scientific rigour in qualitative research assessing trustworthiness is not new. Qualitative researchers have used rigour criteria widely [ 40 , 41 , 42 ]. The novelty of the method described in this article rests on the systematic application of these criteria in a large-scale qualitative study in the context of emergency medicine.

According to the FDC, similar findings should be obtained if the process is repeated with the same cohort of participants in the same settings and organisational context. By employing the FDC and the proposed strategies, we could enhance the dependability of the findings. As indicated in the literature, qualitative research has many times been questioned in history for its validity and credibility [ 3 , 20 , 43 , 44 ].

Nevertheless, if the work is done properly, based on the suggested tools and techniques, any qualitative work can become a solid piece of evidence. This study suggests that emergency medicine researchers can improve their qualitative research if conducted according to the suggested criteria. The triangulation and reflexivity strategies helped us to minimise the investigators’ bias, and affirm that the findings were objective and accurately reflect the participants’ perspectives and experiences. Abiding by a consistent method of data collection (e.g., interview protocol) and conducting the analysis with a team of investigators, helped us minimise the risk of interpretation bias.

Employing several purposive sampling techniques enabled us to have a diverse range of opinions and experiences which at the same time enhanced the credibility of the findings. We expect that the outcomes of this study will show a high degree of applicability, because any resultant hypotheses may be transferable across similar settings in emergency care. The systematic quantification of data saturation at this scale of qualitative data has not been demonstrated in the emergency medicine literature before.

As indicated, the objective of this study was to contribute to the ongoing debate about rigour in qualitative research by using our mixed methods study as an example. In relation to innovative application of mixed-methods, the findings from this qualitative component can be used to explain specific findings from the quantitative component of the study. For example, different trends of 4HR/NEAT performance can be explained by variations in staff relationships across states (see key concept 1, Table 3 ). In addition, some experiences from doctors and nurses may explain variability of performance indicators across participating hospitals. The robustness of the qualitative data will allow us to generate hypotheses that in turn can be tested in future research.

Careful planning is essential in any type of research project which includes the importance of allocating sufficient resources both human and financial. It is also required to organise precise arrangements for building the research team; preparing data collection guidelines; defining and obtaining adequate participation. This may allow other researchers in emergency care to replicate the use of the FDC in the future.

This study has several limitations. Some limitations of the qualitative component include recall bias or lack of reliable information collected about interventions conducted in the past (before the implementation of the policy). As Weber and colleagues [ 45 ] point out, conducting interviews with clinicians at a single point in time may be affected by recall bias. Moreover, ED staff may have left the organisation or have progressed in their careers (from junior to senior clinical roles, i.e. junior nursing staff or junior medical officers, registrars, etc.), so obtaining information about pre/during/post-4HR/NEAT was a difficult undertaking. Although the use of criterion-based and maximum-variation sampling techniques minimised this effect, we could not guarantee that the sampling techniques could have reached out all those who might be eligible to participate.

In terms of recruitment, we could not select potential participants who were not working in that particular ED during the implementation, even if they had previous working experience in other hospital EDs. This is a limitation because people who participated in previous hospitals during the intervention could not provide valuable input to the overall project.

In addition, one would claim that the findings could have been ‘ED-biased’ due to the fact that we did not interview the staff or administrators outside the ED. Unfortunately, interviews outside the ED were beyond the resources and scope of the project.

With respect to the rigour criteria, we could not carry out a systematic member checking as we did not have the required resources for such an expensive follow-up. Nevertheless, we have taken extensive measures to ensure confirmation of the integrity of the data.

Conclusions

The FDC presented in this manuscript provides an important and systematic approach to achieve trustworthy qualitative findings. As indicated before, qualitative research credentials have been questioned. However, if the work is done properly based on the suggested tools and techniques described in this manuscript, any work can become a very notable piece of evidence. This study concludes that the FDC is effective; any investigator in emergency medicine research can improve their qualitative research if conducted accordingly.

Important indicators such as saturation levels and inter-coder reliability should be considered in all types of qualitative projects. One important aspect is that by using FDC we can demonstrate that qualitative research is not less rigorous than quantitative methods.

We also conclude that the FDC is a valid framework to be used in qualitative research in the emergency medicine context. We recommend that future research in emergency care should consider the FDC to achieve trustworthy qualitative findings. We can conclude that our method confirms the credibility (validity) and dependability (reliability) of the analysis which are a true reflection of the perspectives reported by the group of participants across different states/territories.

We can also conclude that our method confirms the objectivity of the analyses and reduces the risk for interpretation bias. We encourage adherence to practical frameworks and strategies like those presented in this manuscript.

Finally, we have highlighted the importance of allocating sufficient resources. This is essential if other researchers in emergency care would like to replicate the use of the FDC in the future.

Following papers in this series will use the empirical findings from longitudinal data linkage analyses and the results from the qualitative study to further identify factors associated with ED performance before and after the implementation of the 4HR/NEAT.

Abbreviations

Four Hour Rule/National Emergency Access Target

Australian Capital Territory

Emergency department(s)

Four-Dimensions Criteria

Health Research Ethics Committee

New South Wales

Project Management Committee

Western Australia

Rees N, Rapport F, Snooks H. Perceptions of paramedics and emergency staff about the care they provide to people who self-harm: constructivist metasynthesis of the qualitative literature. J Psychosom Res. 2015;78(6):529–35.

Article   PubMed   Google Scholar  

Jessup M, Crilly J, Boyle J, Wallis M, Lind J, Green D, Fitzgerald G. Users’ experiences of an emergency department patient admission predictive tool: a qualitative evaluation. Health Inform J. 2016;22(3):618–32. https://doi.org/10.1177/1460458215577993 .

Article   Google Scholar  

Choo EK, Garro AC, Ranney ML, Meisel ZF, Morrow Guthrie K. Qualitative research in emergency care part I: research principles and common applications. Acad Emerg Med. 2015;22(9):1096–102.

Article   PubMed   PubMed Central   Google Scholar  

Hjortdahl M, Halvorsen P, Risor MB. Rural GPs’ attitudes toward participating in emergency medicine: a qualitative study. Scand J Prim Health Care. 2016;34(4):377–84.

Samuels-Kalow ME, Rhodes KV, Henien M, Hardy E, Moore T, Wong F, Camargo CA Jr, Rizzo CT, Mollen C. Development of a patient-centered outcome measure for emergency department asthma patients. Acad Emerg Med. 2017;24(5):511–22.

Manning SN. A multiple case study of patient journeys in Wales from A&E to a hospital ward or home. Br J Community Nurs. 2016;21(10):509–17.

Ranney ML, Meisel ZF, Choo EK, Garro AC, Sasson C, Morrow Guthrie K. Interview-based qualitative research in emergency care part II: data collection, analysis and results reporting. Acad Emerg Med. 2015;22(9):1103–12.

Forero R, Hillman KM, McCarthy S, Fatovich DM, Joseph AP, Richardson DB. Access block and ED overcrowding. Emerg Med Australas. 2010;22(2):119–35.

Fatovich DM. Access block: problems and progress. Med J Aust. 2003;178(10):527–8.

PubMed   Google Scholar  

Richardson DB. Increase in patient mortality at 10 days associated with emergency department overcrowding. Med J Aust. 2006;184(5):213–6.

Sprivulis PC, Da Silva J-A, Jacobs IG, Frazer ARL, Jelinek GA. The association between hospital overcrowding and mortality among patients admitted via Western Australian emergency departments.[Erratum appears in Med J Aust. 2006 Jun 19;184(12):616]. Med J Aust. 2006;184(5):208–12.

Richardson DB, Mountain D. Myths versus facts in emergency department overcrowding and hospital access block. Med J Aust. 2009;190(7):369–74.

Geelhoed GC, de Klerk NH. Emergency department overcrowding, mortality and the 4-hour rule in Western Australia. Med J Aust. 2012;196(2):122–6.

Nugus P, Forero R. Understanding interdepartmental and organizational work in the emergency department: an ethnographic approach. Int Emerg Nurs. 2011;19(2):69–74.

Nugus P, Holdgate A, Fry M, Forero R, McCarthy S, Braithwaite J. Work pressure and patient flow management in the emergency department: findings from an ethnographic study. Acad Emerg Med. 2011;18(10):1045–52.

Nugus P, Forero R, McCarthy S, McDonnell G, Travaglia J, Hilman K, Braithwaite J. The emergency department “carousel”: an ethnographically-derived model of the dynamics of patient flow. Inte Emerg Nurs. 2014;21(1):3–9.

Jones P, Chalmers L, Wells S, Ameratunga S, Carswell P, Ashton T, Curtis E, Reid P, Stewart J, Harper A, et al. Implementing performance improvement in New Zealand emergency departments: the six hour time target policy national research project protocol. BMC Health Serv Res. 2012;12:45.

Forero R, Hillman K, McDonnell G, Fatovich D, McCarthy S, Mountain D, Sprivulis P, Celenza A, Tridgell P, Mohsin M, et al. Validation and impact of the four hour rule/NEAT in the emergency department: a large data linkage study (Grant # APP1029492). Vol. $687,000. Sydney, Perth, Brisbane, Canberra: National Health and Medical Research Council; 2013.

Google Scholar  

Forero R, Hillman K, McDonnell G, Tridgell P, Gibson N, Sprivulis P, Mohsin M, Green S, Fatovich D, McCarthy S, et al. Study Protocol to assess the implementation of the Four-Hour National Emergency Access Target (NEAT) in Australian Emergency Departments. Sydney: AIHI, UNSW; 2013.

Morse JM. Critical analysis of strategies for determining rigor in qualitative inquiry. Qual Health Res. 2015;25(9):1212–22.

Schou L, Hostrup H, Lyngso EE, Larsen S, Poulsen I. Validation of a new assessment tool for qualitative research articles. J Adv Nurs. 2012;68(9):2086–94.

Tuckett AG. Part II. Rigour in qualitative research: complexities and solutions. Nurse Res. 2005;13(1):29–42.

Lincoln YS, Guba EG. But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. N Dir Eval. 1986;1986(30):73–84.

Lincoln YS, Guba EG. Naturalistic inquiry, 1st edn. Newbury Park: Sage Publications Inc; 1985.

Guba EG, Lincoln YS. Epistemological and methodological bases of naturalistic inquiry. ECTJ. 1982;30(4):233–52.

Guba EG. Criteria for assessing the trustworthiness of naturalistic inquiries. ECTJ. 1981;29(2):75.

Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health Ment Health Serv Res. 2015;42(5):533–44.

Patton MQ. Qualitative research and evaluation methods. 3rd edn. Thousand Oaks: Sage Publishing; 2002.

McLellan E, MacQueen KM, Neidig JL. Beyond the qualitative interview: data preparation and transcription. Field Methods. 2003;15(1):63–84.

QSR International Pty Ltd: NVivo 11 for windows. 2015.

Whiting M, Sines D. Mind maps: establishing ‘trustworthiness’ in qualitative research. Nurse Res. 2012;20(1):21–7.

Forero R, Nugus P, McDonnell G, McCarthy S. Iron meets clay in sculpturing emergency medicine: a multidisciplinary sense making approach. In: Deng M, Raia F, Vaccarella M, editors. Relational concepts in medicine (eBook), vol. 2012. Oxford: Interdisciplicary Press; 2012. p. 18.

FitzGerald G, Toloo GS, Romeo M. Emergency healthcare of the future. Emerg Med Australas. 2014;26(3):291–4.

NSW Ministry of Health. In: Health NDo, editor. Emergency department models of care. North Sydney: NSW Ministry of Health; 2012. p. 65.

Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook: Sage; 1994.

Council NHaMR. In: Council NHaMR, editor. Australian code for the responsible conduct of research: Australian Government; 2007.

Kuckartz U. Qualitative text analysis: a guide to methods, practice and using software: Sage; 2014.

Ulrich W, Reynolds M. Critical systems heuristics. In: Systems approaches to managing change: a practical guide, Version of record. London: Springer; 2010. p. 243–92.

Chapter   Google Scholar  

Bowen GA. Naturalistic inquiry and the saturation concept: a research note. Qual Res. 2008;8(1):137–52.

Krefting L. Rigor in qualitative research - the assessment of trustworthiness. Am J Occup Ther. 1991;45(3):214–22.

Article   CAS   PubMed   Google Scholar  

Hamberg K, Johansson E, Lindgren G, Westman G. Scientific rigour in qualitative research - examples from a study of womens health in family-practice. Fam Pract. 1994;11(2):176–81.

Tobin GA, Begley CM. Methodological rigour within a qualitative framework. J Adv Nurs. 2004;48(4):388–96.

Kitto SC, Chesters J, Grbich C. Quality in qualitative research. Med J Aust. 2008;188(4):243–6.

Tuckett AG, Stewart DE. Collecting qualitative data: part I: journal as a method: experience, rationale and limitations. Contemp Nurse. 2003;16(1–2):104–13.

Weber EJ, Mason S, Carter A, Hew RL. Emptying the corridors of shame: organizational lessons from England’s 4-hour emergency throughput target. Ann Emerg Med. 2011;57(2):79–88.e71.

Download references

Acknowledgements

We acknowledge Brydan Lenne who was employed in the preliminary stages of the project, for ethics application preparation and ethics submissions, and her contribution in the planning stages, data collection of the qualitative analysis and preliminary coding of the conceptual framework is appreciated. Fenglian Xu, who was also employed in the initial stages of the project in the data linkage component. Jenine Beekhuyzen, CEO Adroit Research, for consultancy and advice on qualitative aspects of the manuscript and Liz Brownlee, owner/manager Bostock Transcripts services for the transcription of the interviews. Brydan Lenne, Karlene Dickens; Cecily Scutt and Tracey Hawkins who conducted the interviews across states. We also thank Anna Holdgate, Michael Golding, Michael Hession, Amith Shetty, Drew Richardson, Daniel Fatovich, David Mountain, Nick Gibson, Sam Toloo, Conrad Loten, John Burke and Vijai Joseph who acted as site contacts on each State/Territory. We also thank all the participants for their contribution in time and information provided. A full acknowledgment of all investigators and partner organisations is enclosed as an attachment (see Additional file  3 : Appendix 3).

This project was funded by the Australian National Health and Medical Research Council (NH&MRC) Partnership Grant No APP1029492 with cash contributions from the following organisations: Department of Health of Western Australia, Australasian College for Emergency Medicine, Ministry of Health of NSW and the Emergency Care Institute, NSW Agency for Clinical Innovation, and Emergency Medicine Foundation, Queensland.

Availability of data and materials

All data generated or analysed during this study are included in this published article and its supplementary information files have been included in the appendix. No individual data will be available.

Author information

Authors and affiliations.

The Simpson Centre for Health Services Research, South Western Sydney Clinical School and the Ingham Institute for Applied Research, Liverpool Hospital, UNSW, Liverpool, NSW, 1871, Australia

Roberto Forero, Shizar Nahidi & Josephine De Costa

Psychiatry Research and Teaching Unit, Liverpool Hospital, NSW Health, Sydney, Australia

Mohammed Mohsin

School of Psychiatry, Faculty of Medicine, University of New South Wales, Sydney, Australia

School - Public Health and Social Work, Queensland University of Technology (QUT), Brisbane, Qld, Australia

Gerry Fitzgerald

Australasian College for Emergency Medicine (ACEM), Melbourne, VIC, Australia

Gerry Fitzgerald & Sally McCarthy

School of Nursing and Midwifery, Edith Cowan University (ECU), Perth, WA, Australia

Nick Gibson

Emergency Care Institute (ECI), NSW Agency for Clinical Innovation (ACI), Sydney, Australia

Sally McCarthy

Clinical Support Directorate, System Policy & Planning Division, Department of Health WA, Perth, WA, Australia

Patrick Aboagye-Sarfo

You can also search for this author in PubMed   Google Scholar

Contributions

RF, GF, SMC made substantial contributions to conception, design and funding of the study. RF, SN, NG, SMC with acquisition of data. RF, SN, JDC for the analysis and interpretation of data. RF, SN, JDC, MM, GF, NG, SMC and PA were involved in drafting the manuscript and revising it critically for important intellectual content and gave final approval of the version to be published. All authors have participated sufficiently in the work to take public responsibility for appropriate portions of the content; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Corresponding author

Correspondence to Roberto Forero .

Ethics declarations

As indicated in the background, our study received ethics approval from the respective Human Research Ethics Committees of Western Australian Department of Health (DBL.201403.07), Cancer Institute NSW (HREC/14/CIPHS/30), ACT Department of Health (ETH.3.14.054) and Queensland Health (HREC/14/QGC/30) as well as governance approval from the 16 participating hospitals. All participants received information about the project; received an invitation to participate and signed a consent form and agreed to allow an audio recording to be conducted.

Consent for publication

All the data used from the interviews were de-identified for the analysis. No individual details, images or recordings, were used apart from the de-identified transcription.

Competing interests

RF is an Associate Editor of the Journal. No other authors have declared any competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:.

Appendix 1. Interview form. Text. (PDF 445 kb)

Additional file 2:

Appendix 2. Codebook NVIVO. Text code. (PDF 335 kb)

Additional file 3:

Appendix 3. Acknowledgements. Text. (PDF 104 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Forero, R., Nahidi, S., De Costa, J. et al. Application of four-dimension criteria to assess rigour of qualitative research in emergency medicine. BMC Health Serv Res 18 , 120 (2018). https://doi.org/10.1186/s12913-018-2915-2

Download citation

Received : 05 May 2017

Accepted : 06 February 2018

Published : 17 February 2018

DOI : https://doi.org/10.1186/s12913-018-2915-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Emergency department
• Four-hour rule
• Policy assessment
• Qualitative methods
• Research design

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]